This project has moved. For the latest updates, please go here.

openPDC Data Access Options:


  1. Real-time data acquisition
  2. Near real-time data acquisition
  3. Historical data acquisition

 1) The openPDC supports real-time subscription based data acquisition via the Gateway Exchange Protocol. You can see the API in action on the Graph Measurements screen inside the openPDC Manager - every time you check a new point in the list it "subscribes" via this real-time API. Code usage is fairly simple - there is an API that can be called to "subscribe" to data in multiple formats (example below). API's exist for .NET, Java, C++ and Unity 3D:

 

Assembly: GSF.TimeSeries.dll

Class: GSF.TimeSeries.Transport.DataSubscriber

Relevant Methods:

 

        /// <summary>

        /// Subscribes (or re-subscribes) to a data publisher for a synchronized set of data points.

        /// </summary>

        /// <param name="compactFormat">Boolean value that determines if the compact measurement format should be used. Set to <c>false</c> for full fidelity measurement serialization; otherwise set to <c>false</c> for bandwidth conservation.</param>

        /// <param name="framesPerSecond">The desired number of data frames per second.</param>

        /// <param name="lagTime">Allowed past time deviation tolerance, in seconds (can be subsecond).</param>

        /// <param name="leadTime">Allowed future time deviation tolerance, in seconds (can be subsecond).</param>

        /// <param name="filterExpression">Filtering expression that defines the measurements that are being subscribed.</param>

        /// <param name="useLocalClockAsRealTime">Boolean value that determines whether or not to use the local clock time as real-time.</param>

        /// <param name="ignoreBadTimestamps">Boolean value that determines if bad timestamps (as determined by measurement's timestamp quality) should be ignored when sorting measurements.</param>

        /// <param name="allowSortsByArrival"> Gets or sets flag that determines whether or not to allow incoming measurements with bad timestamps to be sorted by arrival time.</param>

        /// <param name="timeResolution">Gets or sets the maximum time resolution, in ticks, to use when sorting measurements by timestamps into their proper destination frame.</param>

        /// <param name="allowPreemptivePublishing">Gets or sets flag that allows system to preemptively publish frames assuming all expected measurements have arrived.</param>

        /// <param name="downsamplingMethod">Gets the total number of downsampled measurements processed by the concentrator.</param>

        /// <returns><c>true</c> if subscribe was successful; otherwise <c>false</c>.</returns>

        public virtual bool SynchronizedSubscribe(bool compactFormat, int framesPerSecond, double lagTime, double leadTime, string filterExpression, bool useLocalClockAsRealTime = false, bool ignoreBadTimestamps = false, bool allowSortsByArrival = true, long timeResolution = Ticks.PerMillisecond, bool allowPreemptivePublishing = true, DownsamplingMethod downsamplingMethod = DownsamplingMethod.LastReceived)

 

        /// <summary>

        /// Subscribes (or re-subscribes) to a data publisher for an unsynchronized set of data points.

        /// </summary>

        /// <param name="compactFormat">Boolean value that determines if the compact measurement format should be used. Set to <c>false</c> for full fidelity measurement serialization; otherwise set to <c>false</c> for bandwidth conservation.</param>

        /// <param name="throttled">Boolean value that determines if data should be throttled at a set transmission interval or sent on change.</param>

        /// <param name="filterExpression">Filtering expression that defines the measurements that are being subscribed.</param>

        /// <param name="lagTime">When <paramref name="throttled"/> is <c>true</c>, defines the data transmission speed in seconds (can be subsecond).</param>

        /// <param name="leadTime">When <paramref name="throttled"/> is <c>true</c>, defines the allowed time deviation tolerance to real-time in seconds (can be subsecond).</param>

        /// <param name="useLocalClockAsRealTime">When <paramref name="throttled"/> is <c>true</c>, defines boolean value that determines whether or not to use the local clock time as real-time. Set to <c>false</c> to use latest received measurement timestamp as real-time.</param>

        /// <returns><c>true</c> if subscribe was successful; otherwise <c>false</c>.</returns>

        public virtual bool UnsynchronizedSubscribe(bool compactFormat, bool throttled, string filterExpression, double lagTime = 10.0D, double leadTime = 5.0D, bool useLocalClockAsRealTime = false)

 

Here is an operational example:

 

                http://timeseriesframework.codeplex.com/SourceControl/changeset/view/61589#1011313

 

Note that all this code exists in the Time Series Framework - this is important since phasor gateways, historians, etc. in the future can all use this same code.

 

These two API calls provide the following possible real-time data subscriptions:

 

1)      A synchronized (i.e., concentrated by time) set of subscribed data points

2)      An on-change unsynchronized set of subscribed data points

3)      A throttled (e.g., downsampled to every few seconds)  unsynchronized set of subscribed data points

 

To pick up data you simply attach to the NewMeasurements event and receive a collection of measurements:

 

        static DataSubscriber subscriber = new DataSubscriber();

       

        subscriber.NewMeasurements += subscriber_NewMeasurements;

       

        // Initialize subscriber

        subscriber.ConnectionString = "server=localhost:6165";

        subscriber.Initialize();

 

        // Start subscriber connection cycle

        subscriber.Start();

 

        static void subscriber_NewMeasurements(object sender, EventArgs<ICollection<IMeasurement>> e)

        {

            dataCount += e.Argument.Count;

 

            if (dataCount % (5 * 60) == 0)

                Console.WriteLine(string.Format("{0} total measurements received so far: {1}", dataCount, e.Argument.ToDelimitedString(", ")));

        }

 

To better understand using this API, open the TimeSeriesFramework solution and evaluate the DataSubscriberTest application. You can run this application in debug mode (multiple instances actually) as well as starting an instance of the associated DataPublisherTest app to see it in action.

 

2) With the development of this new real-time API, the web service based data acquisition for near-real time data that has always existed will now be optional - that is, if a user "chooses" to install a local historian (this is being added as a configuration setup screen step) then the near real-time data web service will then be available - otherwise it will not be available for data. It should be noted that the statistics historian is a separate local historian and will always be available. This data access option uses the web service API:

 

http://openpdc.codeplex.com/wikipage?title=Getting Started (Developers)&referringTitle=Documentation#time_series_web_service

            

3) When accessing the statistics historian or the optional local historian for historical data you can always use the web service aforementioned in item 2 (details in the web-link above). Note however that there is another, more direct, way to access the data. For faster historical data acquisition you can use the historian API directly. For an example of how to use this you can look at the statistics reader code recently added to the phasor protocol's library:

 

http://openpdc.codeplex.com/SourceControl/changeset/view/61586#1084098

 

This API reads data directly from the historical .D files over a given time span as fast as possible. Additionally, you can look at the source code for the Historian Playback Utility for another example of how to use the data read / extraction API directly.

 

Last edited Jun 10, 2014 at 3:08 PM by ritchiecarroll, version 3

Comments

No comments yet.