This project has moved and is read-only. For the latest updates, please go here.

Retrieving historical data from remote PDC

Oct 30, 2013 at 7:05 PM
Hello,

I'd like to retrieve some historical data from a remote openPDC over the network.

Applying the code presented here: http://gsf.codeplex.com/SourceControl/latest#Main/Source/Applications/DataSubscriberTest/Program.cs)
I used the SynchronizedSubscribe method (using port 6165) and received real-time data packets just like when using GEP Subscription Tester.

In order to use the code linked above to retrieve historical data, do I need to configure something in the remote openPDC or just adapt the code (i.e. setting startTime and endTime for the SubsciptionInfo and a different port for subscriber.ConnectionString)?

Thank's in advance.

Rodolfo
Oct 30, 2013 at 9:04 PM
Sure - same port, same connection. You just simply set the start time and/or end time parameters. This can be a specific date/time or relative time, e.g.:
            remotelySynchronizedInfo.StartTime = "*-10M";
            remotelySynchronizedInfo.StopTime = "*";
or
            remotelySynchronizedInfo.StartTime = "10/10/2013 04:45:00.000";
            remotelySynchronizedInfo.StopTime = "10/10/2013 05:20:00.000";
Here are some relative time expression examples:
  • * = current time (UTC)
  • *-20s = 20 seconds ago
  • *-10m = 10 minutes ago
  • *-1h = 1 hour ago
  • *-1d = 1 day ago
Another parameter you can control is:
            remotelySynchronizedInfo.ProcessingInterval = 120; // Initial speed
Note that the processing interval parameter can by adjusted dynamically to speed up or slow down replay speed:
            subscriber.ProcessingInterval = 60;
Dynamic adjustments are made to the subscriber instance not the subscription info.

Setting value to zero means to replay as fast as possible, setting value to -1 means you default playback speed otherwise value is minimum millisecond delay between samples. Note that the data will only flow back as fast as your historian can deliver the data.

Once the historical replay has completed system returns to real-time - this is indicated by raising the "ProcessingComplete" event.

Hope that helps!

Thanks,
Ritchie
Marked as answer by ritchiecarroll on 11/1/2013 at 11:40 AM
Oct 31, 2013 at 3:03 PM
Also - don't forget, the DataSubcriber API is available in Java, C++, Unity 3D platform as well as .NET...

Thanks,
Ritchie
Oct 31, 2013 at 5:02 PM
Hello,

First of all, thank you Ritchie for your quick response.

Well, I tried those configuration sets that you described to adjust StartTime, StopTime and ProcessingInterval parameters, without success.
Basically, my subscription info is defined as follows:
            remotelySynchronizedInfo.LagTime = 4.0D;
            remotelySynchronizedInfo.LeadTime = 4.0D;
            remotelySynchronizedInfo.FilterExpression = "FILTER ActiveMeasurements WHERE SignalType='FREQ'";
            remotelySynchronizedInfo.StartTime = "10-26-2013 23:50:00.000";
            remotelySynchronizedInfo.StopTime = "10-26-2013 23:50:59.999";  
The connection is being normally stablished (using the same 6165 port) but I'm not receiving any data packet. The response codes observed when checking
OnReceivedServerResponse event are in this order:

1º UpdateSignalIndexCache
2º Succeeded
3º (and so on) NoOP

While when requesting real-time data I get:

1º UpdateSignalIndexCache
2º Succeeded
3º DataStartTime
4º (and so on) DataPacket

...and then the event NewMeasurements attached to the subscriber is raised.

Any clues of what am I doing wrong?
Thank's for your help.
Rodolfo
Oct 31, 2013 at 5:51 PM
Let's get a few obvious things out of the way...

When you are using the openPDC Manager on the Graph Measurements screen so you see a play-back option? Is so, does it function?

If not - what kind of historian do you have configured?

Thanks,
Ritchie
Oct 31, 2013 at 6:39 PM
No, I can't see a play-back option on Graph Measurements screen.
It was configured a local historian.

Thanks.
Rodolfo
Nov 1, 2013 at 3:30 PM
Look at screen below - it shows the historical playback option. What kind of historian are you using? Also, what is your openPDC version?

Image
Nov 1, 2013 at 4:00 PM
Hi, well I meant a local in-process historian (".d" files). We are using the 1.5.143.0 version (both server and manager).
I can't see that historical playback region on the Graph Measurements screen here.

Thanks.
Rodolfo
Nov 1, 2013 at 5:52 PM
Edited Nov 1, 2013 at 5:55 PM
I think once we get that corrected your API calls will work.

OK, a few things we can check and/or try:
  1. A custom input adapter called "PPAREADER" exists (could be called something else if historian acronym is not PPA). Make sure:
    A. Input adapter is enabled
    B. archiveLocation parameter has correct path
    C. instanceName and sourceIDs match historian acronym
  2. Make sure historian archive locations are setup correctly in openPDC.exe.config (can run XML Configuration Editor for this)
    A. <historianacronym>ArchiveFile \ FileName has correct path
    B. <historianacronym>IntercomFile \ FileName has correct path
    C. <historianacronym>MetadataFile \ FileName has correct path
    D. <historianacronym>StateFile \ FileName has correct path
  3. Check StatusLog.txt and ErrorLog.txt for any errors during startup related to custom local input adapter (e.g., PPAREADER) or local historian output adapters (e.g., PPA)
  4. Make sure you can trend data that is being read from the historian using the "Historian Trending Tool". If this fails, try these troubleshooting steps:
    A. Stop openPDC service
    B. Delete the following files in the historian archive location (this defaults to "C:\Program Files\openPDC\Archive"):
    • scratch.dat
    • ppa_dbase.dat
    • ppp_startup.dat
C. Start openPDC service and retry trending tool
D. If the trending tool still doesn't work, one of the data files (ends in .D) may be corrupted - things to try:
  • Keep moving .D files to another folder and keep "re-running" (i.e., start / stop / restart) historian trending tool until it works
  • If you have moved all the files except for the active files (e.g., ppa_archive.d) - stop the openPDC, move this file to another folder, then restart the openPDC and try the trending tool again.
Marked as answer by RodolfoLeandro on 11/1/2013 at 11:04 AM
Nov 1, 2013 at 6:01 PM
Alright, I'm going to check that.

We just installed an openPDC version 2.0.83.0 in a different machine and we're able to retrieve some historical data now.
We defined the same kind of historian and just allowed synchronized subscriptions for external and internal data publishers.

One last thing, can I configure subscriber.ConnectionString to accept an IPv4 server address?

Thanks for your help!
Rodolfo
Nov 1, 2013 at 6:52 PM
Simply suffix the connection string with:
; interface=0.0.0.0
This says use "default network interface"; optionally you can set this to actual IP of the network interface over which you would like your data to flow.

The format of the interface definition defines the IP flavor, e.g., "interface=::0" would mean force IPv6.

Note that you can't use the format of the server address to determine which IP stack to use since this can use a DNS name.

Thanks!
Ritchie
Nov 1, 2013 at 7:04 PM
Edited Nov 5, 2013 at 8:47 PM
Right. Got it.

Well, your first tip solved our issue here, PPAREADER wasn't enabled. We just enabled that and everything is working fine now.

Thank you very much Ritchie.
Rodolfo
Nov 6, 2013 at 9:39 PM
Edited Nov 7, 2013 at 11:36 AM
Well, actually almost everything...

Now, checking the openPDC console when subscribing for some historical data we get this:
[PHASOR!SERVICES] Client subscribed as compact synchronized with a temporal constraint.

[PHASOR!SERVICES] [PPAREADER#<::ffff:XXX.XXX.XX.XX>@2013-11-05 12:00:00] Attempting connection...

[PHASOR!SERVICES] [PPAREADER#<::ffff:XXX.XXX.XX.XX>@2013-11-05 12:00:00] Connection established.

[PHASOR!SERVICES] [PPAREADER#<::ffff:XXX.XXX.XX.XX>@2013-11-05 12:00:00] Building list of historic archive files...

[PHASOR!SERVICES] [PPAREADER#<::ffff:XXX.XXX.XX.XX>@2013-11-05 12:00:00] No measurement keys have been requested for reading, historian reader is idle.

[PHASOR!SERVICES] [PPAREADER#<::ffff:XXX.XXX.XX.XX>@2013-11-05 12:00:00] Processing completed.
We are using the same subscription info parameters as posted before.
Do we need to define any extra parameters or maybe our filter expression is the problem?

Thank you.
Rodolfo
Nov 7, 2013 at 7:10 PM
Looks like your filter expression is not returning any points. Use the GEP Subscription Tester with your filter expression and see what gets returned...

Thanks!
Ritchie
Nov 20, 2013 at 8:12 PM
Hi.

Well, as we didn't identify what we were doing wrong, we decided to install openPDC v2.0. So, now we're receiving the requested historical data normally.

However, we noticed some problems with measurements' timestamps. When requesting less than six measurements in the filter expression we got the correct requested timestamps. But, when requesting six or more measurements, all the measurements' timestamps were equal to 01/01/0001 00:00:00.

We got the same behavior using the following two definitions for the filter expression:
remotelySynchronizedInfo.FilterExpression = "PPA:63;PPA:64;PPA:65;PPA:66;PPA:67;PPA:68;PPA:69";
remotelySynchronizedInfo.FilterExpression = "FILTER ActiveMeasurements WHERE Device='PMU1' AND SignalType IN ('FREQ', 'VPHA', 'VPHM')";
We tested all the measurements listed individually, always getting the correct timestamps. The problem just appeared when listing six or more measurements.
Is there any problem with our filter expression?

Thanks!
Rodolfo
Nov 20, 2013 at 8:21 PM
Hi Rodolfo,

It looks like we currently only support unsynchronized and locally synchronized subscriptions with temporal sessions. Can you switch to a locally synchronized subscription to see if that helps fix your problem?

Thanks,
Stephen
Nov 21, 2013 at 12:43 PM
Hi Stephen,

I tried an unsynchronized subscription, cause I'm dealing with a remote openPDC. Now I'm receiving the correct timestamps.
Thank you very much Stephen.

Rodolfo