This project has moved. For the latest updates, please go here.

Real time data synchronization and losses, and access to historic data

Oct 29, 2009 at 4:07 PM

Hello all. My name is Marcelo Agostini, I'm with MedFasee Project Team at Federal University of Santa Catarina, Brazil. We have worked with the synchrophasor technology since 2003, and we built a SPMS connected to the outlet voltage at several Brazilian universities (www.medfasee.ufsc.br). We developed a PDC prototype that is working since 2004, collecting data from those PMUs. Recently we downloaded, installed and have been tested TVA openPDC in our SPMS, operating with our PDC. Both PDCs are working well, with openPDC sending "resynchronized data" to our PDC. We notice some specific characteristics of the openPDC configuration and operation, and we'll appreciate your comments about that. Please feel free to break the comments in separated discussions if you consider appropriate.

1) How can we configure the "resynchronization buffer"? It's possible to configure the minimum and maximum time delays that the openPDC waits that the data of all PMUs in the system arrive? We found the "LagTime" and "LeadTime" parameters at the "OutputStream" configuration (we are using the Access database); are these parameters related to the resynchonization proccess?

2) Our SPMS is working with 60 synchrophasors per second. All system (PMUs and PDCs) are configured to operate with this data frame rate. We are capturing the real time data flow sent by an "OutputStream" configured in the openPDC, and we have notice that there is a data loss of exactly 6,67%; in fact, always the data frames 6, 21, 36 and 51 of the 60 data frames in each second aren't sent by the openPDC. The data frames that weren't sent by the OutputStream, in fact exist in the historic. In other words, those data frames arrived to the openPDC, but weren't sent by the OutputStream. Is this behavior known by you? We didn't test another frame rate yet, but we'll do this soon.

3) About the historic data, we're trying to get data of small time periods, like 1s, 2s, but in those cases the openPDC returns an empty XML. For example, the http link:

http://<openPDC-IP>:8700/historian/timeseriesdata/read/historic/24/29-out-2009%2008:00:00/29-out-2009%2008:00:01/xml

returns:

<TimeSeriesData>
<TimeSeriesDataPoints/>
</TimeSeriesData>

    If we increase the second of final time to 02, 03, ..., the returned XML remains the same empty file. When the second is made equal to 13, finally the openPDC returns the correctly data of the measurement "24", related to the period <29-out-2009 08:00:00> - <29-out-2009 08:00:13>. The minimum period of search that works well varies among 12s, 13s, until 3s or 4s; it depends of each search. Is this behavior correct? Is there a limitation of minimum time period to get data from the historic? Can we get only few data, like the follow http link (3 data points only, for example):

http://<openPDC-IP>:8700/historian/timeseriesdata/read/historic/24/29-out-2009%2008:00:00/29-out-2009%2008:00:00.5/xml

    We are pleased to contribute to the openPDC development. Thanks in advance for your comments about these points.

Best regards,
Marcelo

Coordinator
Oct 29, 2009 at 9:27 PM
Edited Oct 29, 2009 at 9:28 PM

First and foremost thank you for your most valuable feedback – we will look into each issue individually and get you an answer for each.

To answer your first question about LagTime and LeadTime, yes, these are the “resynchronization” parameters, they operate is as follows:

LagTime is a double precision number that defines the allowed past time deviation tolerance, in seconds (can be subsecond). It defines the time sensitivity to past measurement timestamps and is the number of seconds allowed before assuming a measurement timestamp is too old. This becomes the amount of delay introduced by the concentrator to allow time for data to flow into the system.

LeadTime is a double precision number that defines the allowed future time deviation tolerance, in seconds (can be subsecond). It defines the time sensitivity to future measurement timestamps and is the number of seconds allowed before assuming a measurement timestamp is too advanced. This becomes the tolerated +/- accuracy of the local clock to real-time.

Because the measurements being received by remote phasor devices are usually measured relative to GPS time, these timestamps are typically more accurate than the local system clock. As a result, we can use the latest received timestamp as the best local time measurement we have (ignoring transmission delays); but, even these times can be incorrect so we still have to apply reasonability checks to these times. To do this, we use the local system time and the LeadTime value to validate the latest measured timestamp. If the newest received measurement timestamp gets too old or creeps too far into the future (both validated + and - against defined lead time property value), we will fall back on local system time. Note that this creates a dependency on a fairly accurate local clock - the smaller the lead time deviation tolerance, the better the needed local clock accuracy. For example, a lead time deviation tolerance of a few seconds might only require keeping the local clock synchronized to an NTP time source; but, a sub-second tolerance would require that the local clock be very close to GPS time.

Note that your lag time should be defined as it relates to the rate at which data is coming into the concentrator. Make sure you allow enough time for transmission of data over the network allowing any needed time for possible network congestion.  Lead time should be defined as your confidence in the accuracy of your local clock (e.g., if you set lead time to 2, this means you trust that your local clock is within plus or minus 2 seconds of real-time.)

The other important property is UseLocalClockAsRealTime which defines a flag that determines whether or not to use the local clock time as real time. You should only use your local system clock as real time only if the time is locally GPS hardware-synchronized or if the incoming measurement values being sorted were not measured relative to a GPS-synchronized clock (e.g., non PMU device).

Thanks!
Ritchie

Coordinator
Oct 31, 2009 at 1:04 AM
Edited Oct 31, 2009 at 1:10 AM

We've been doing some extra testing with 60 samples per second and have been successful (see below) - are you building code yourself or are you using install package? Perhaps an issue has been resolved in the meantime? Is it possible to get a short sample of the source data for testing?

Thanks again for your testing.

Here are the timestamps and frame indexes for a full one second of data (60 samples per second) captured from an IEEE C37.118 formatted outgoing data stream generated using the openPDC:

1 10/30/2009 23:54:57.000 31 10/30/2009 23:54:57.500
2 10/30/2009 23:54:57.016 32 10/30/2009 23:54:57.516
3 10/30/2009 23:54:57.033 33 10/30/2009 23:54:57.533
4 10/30/2009 23:54:57.050 34 10/30/2009 23:54:57.550
5 10/30/2009 23:54:57.066 35 10/30/2009 23:54:57.566
6 10/30/2009 23:54:57.083 36 10/30/2009 23:54:57.583
7 10/30/2009 23:54:57.100 37 10/30/2009 23:54:57.600
8 10/30/2009 23:54:57.116 38 10/30/2009 23:54:57.616
9 10/30/2009 23:54:57.133 39 10/30/2009 23:54:57.633
10 10/30/2009 23:54:57.150 40 10/30/2009 23:54:57.650
11 10/30/2009 23:54:57.166 41 10/30/2009 23:54:57.666
12 10/30/2009 23:54:57.183 42 10/30/2009 23:54:57.683
13 10/30/2009 23:54:57.200 43 10/30/2009 23:54:57.700
14 10/30/2009 23:54:57.216 44 10/30/2009 23:54:57.716
15 10/30/2009 23:54:57.233 45 10/30/2009 23:54:57.733
16 10/30/2009 23:54:57.250 46 10/30/2009 23:54:57.750
17 10/30/2009 23:54:57.266 47 10/30/2009 23:54:57.766
18 10/30/2009 23:54:57.283 48 10/30/2009 23:54:57.783
19 10/30/2009 23:54:57.300 49 10/30/2009 23:54:57.800
20 10/30/2009 23:54:57.316 50 10/30/2009 23:54:57.816
21 10/30/2009 23:54:57.333 51 10/30/2009 23:54:57.833
22 10/30/2009 23:54:57.350 52 10/30/2009 23:54:57.850
23 10/30/2009 23:54:57.366 53 10/30/2009 23:54:57.866
24 10/30/2009 23:54:57.383 54 10/30/2009 23:54:57.883
25 10/30/2009 23:54:57.400 55 10/30/2009 23:54:57.900
26 10/30/2009 23:54:57.416 56 10/30/2009 23:54:57.916
27 10/30/2009 23:54:57.433 57 10/30/2009 23:54:57.933
28 10/30/2009 23:54:57.450 58 10/30/2009 23:54:57.950
29 10/30/2009 23:54:57.466 59 10/30/2009 23:54:57.966
30 10/30/2009 23:54:57.483 60 10/30/2009 23:54:57.983
Coordinator
Nov 2, 2009 at 1:10 PM

I also thought of one other thing to check related to your 60 samples per second - you can run the following application to determine your system's maximum timer:

    http://www.lucashale.com/timerresolution/

This will determine if the system is operating with a good timer resolution. Since you are broadcasting 60 samples per second, you are very close to the edge of the default resolution of 15 milliseconds and may need some higher accuracy. If you use this application and set the timer to its maximum resolution and the issues goes away let me know and we will make a call to set the timer resolution from within the openPDC.

Nov 3, 2009 at 4:48 PM

Richie, thanks for your comments. About the LagTime and LeadTime, it's ok. We understood and configured these parameters according to our phasor system rate and delay, and it's working fine. By the way, we also use the latest received timestamp of the remote phasor devices as the PDC "real time" in our PDC implementation, to providing the correlation ("resynchronization") of the data frames sent by each remote device.

About the installation, we tried both: building code and install package, downloaded from "http://openpdc.codeplex.com". The losses are the same that I already described. I tried to use the "Set Timer Resolution" from "http://www.lucashale.com/timerresolution/", as you suggested, but the losses remain. The openPDC behavior remained the same.

Here are the timestamps and frame indexes for a full one second at 60, 30, 15 and 10 samples per second. Both the PMU and the openPDC "OutputStream" were set to the same rate, in each test. Notice that the losses have similar characteristics in all situations. I used the same format of your post to make easier the comparison. The date is in the format used in Brazil (day/month/year).

Rate: 60 samples per second - 4 missing frames per second

01  03/11/2009 10:26:21.000        31  03/11/2009 10:26:21.500
02  03/11/2009 10:26:21.017        32  03/11/2009 10:26:21.517
03  03/11/2009 10:26:21.033        33  03/11/2009 10:26:21.533
04  03/11/2009 10:26:21.050        34  03/11/2009 10:26:21.550
05  03/11/2009 10:26:21.067        35  03/11/2009 10:26:21.567
06   -- MISSING --   (21.083)         36  -- MISSING --   (21.583)
07  03/11/2009 10:26:21.100        37  03/11/2009 10:26:21.600
08  03/11/2009 10:26:21.117        38  03/11/2009 10:26:21.617
09  03/11/2009 10:26:21.133        39  03/11/2009 10:26:21.633
10  03/11/2009 10:26:21.150        40  03/11/2009 10:26:21.650
11  03/11/2009 10:26:21.167        41  03/11/2009 10:26:21.667
12  03/11/2009 10:26:21.183        42  03/11/2009 10:26:21.683
13  03/11/2009 10:26:21.200        43  03/11/2009 10:26:21.700
14  03/11/2009 10:26:21.217        44  03/11/2009 10:26:21.717
15  03/11/2009 10:26:21.233        45  03/11/2009 10:26:21.733
16  03/11/2009 10:26:21.250        46  03/11/2009 10:26:21.750
17  03/11/2009 10:26:21.267        47  03/11/2009 10:26:21.767
18  03/11/2009 10:26:21.283        48  03/11/2009 10:26:21.783
19  03/11/2009 10:26:21.300        49  03/11/2009 10:26:21.800
20  03/11/2009 10:26:21.317        50  03/11/2009 10:26:21.817
21   -- MISSING --   (21.333)         51  -- MISSING --   (21.833)
22  03/11/2009 10:26:21.350        52  03/11/2009 10:26:21.850
23  03/11/2009 10:26:21.367        53  03/11/2009 10:26:21.867
24  03/11/2009 10:26:21.383        54  03/11/2009 10:26:21.883
25  03/11/2009 10:26:21.400        55  03/11/2009 10:26:21.900
26  03/11/2009 10:26:21.417        56  03/11/2009 10:26:21.917
27  03/11/2009 10:26:21.433        57  03/11/2009 10:26:21.933
28  03/11/2009 10:26:21.450        58  03/11/2009 10:26:21.950
29  03/11/2009 10:26:21.467        59  03/11/2009 10:26:21.967
30  03/11/2009 10:26:21.483        60  03/11/2009 10:26:21.983

Rate: 30 samples per second - 2 missing frames per second

01  03/11/2009 10:42:32.000        16  03/11/2009 10:42:32.500
02  03/11/2009 10:42:32.033        17  03/11/2009 10:42:32.533
03  03/11/2009 10:42:32.067        18  03/11/2009 10:42:32.567
04  03/11/2009 10:42:32.100        19  03/11/2009 10:42:32.600
05  03/11/2009 10:42:32.133        20  03/11/2009 10:42:32.633
06  03/11/2009 10:42:32.167        21  03/11/2009 10:42:32.667
07  03/11/2009 10:42:32.200        22  03/11/2009 10:42:32.700
08  03/11/2009 10:42:32.233        23  03/11/2009 10:42:32.733
09  03/11/2009 10:42:32.267        24  03/11/2009 10:42:32.767
10  03/11/2009 10:42:32.300        25  03/11/2009 10:42:32.800
11   -- MISSING --   (32.333)         26  -- MISSING --   (32.833)
12  03/11/2009 10:42:32.367        27  03/11/2009 10:42:32.867
13  03/11/2009 10:42:32.400        28  03/11/2009 10:42:32.900
14  03/11/2009 10:42:32.433        29  03/11/2009 10:42:32.933
15  03/11/2009 10:42:32.467        30  03/11/2009 10:42:32.967
                        
Rate: 15 samples per second - 1 missing frame per second

01  03/11/2009 11:05:13.000        09  03/11/2009 11:05:13.533
02  03/11/2009 11:05:13.067        10  03/11/2009 11:05:13.600
03  03/11/2009 11:05:13.133        11  03/11/2009 11:05:13.667
04  03/11/2009 11:05:13.200        12  03/11/2009 11:05:13.733
05  03/11/2009 11:05:13.267        13  03/11/2009 11:05:13.800
06   -- MISSING --   (13.333)         14  03/11/2009 11:05:13.867
07  03/11/2009 11:05:13.400        15  03/11/2009 11:05:13.933
08  03/11/2009 11:05:13.467

Rate: 10 samples per second - NO LOSSES!!!

01  03/11/2009 11:29:47.000        06  03/11/2009 11:29:47.500
02  03/11/2009 11:29:47.100        07  03/11/2009 11:29:47.600
03  03/11/2009 11:29:47.200        08  03/11/2009 11:29:47.700
04  03/11/2009 11:29:47.300        09  03/11/2009 11:29:47.800
05  03/11/2009 11:29:47.400        10  03/11/2009 11:29:47.900

Notice there is a pattern among the case tests above. One of 15 data frames wasn't sent by the openPDC. The fractional timestamps "0.333" and "0.833" were lost every time they exist. When I used a rate less than 15 samples per second, 10 for example, the losses disappear. The same happened when I used 12.

We used different PMUs in the tests, and the results were the same. However, all PMUs are of the same manufacturer. I'll do more tests using a PMU data frame simulator, to trying to understand better the problem. I'll post here any news.

 

Coordinator
Nov 6, 2009 at 4:29 PM
Edited Nov 18, 2009 at 3:07 AM

We have confirmed this to be an issue after connecting to device - we have added a work item to track this issue (http://openpdc.codeplex.com/WorkItem/View.aspx?WorkItemId=4759), we will get this corrected ASAP.

Thanks for finding this.

UPDATE: This issue has been corrected, work item has been closed.

Coordinator
Dec 3, 2009 at 6:41 PM
This discussion has been copied to a work item. Click here to go to the work item and continue the discussion.