openPDC

Source code checked in, #42564

ritchiecarroll — Fri, 19 Mar 2010 04:01:33 GMT

Synchrophasor: Version change for build Synchrophasor_v1.1.18.42559.

New Post: bugs in openPDC

cheka — Fri, 19 Mar 2010 00:53:32 GMT

first, I am sorry to reply to you so late. My laptop is broken, making me unable to receive e-mails for a couple days.

and thank you very much for your praise. We should thank you and the whole openPDC team to make openPDC such a wonderful achievement, as well as make it open source, to help a lot more people.

But for "PhasorDataConcentratorBase", I still think that there is race condition. I have noticed that the derived class doesn't use the dictionary of the frame, so I agree with you that there is no race condition on that dictionary, so there is no need to lock and unlock the dictionary. But the frame itself is also shared by multiple threads:

1. if there are multiple input adapters receiving data from different devices, each input adaper will run on its own thread, and try to put its newly-received measurements into that frame, so there is race condition among multiple "writers".

2. there is another thread to call "PublishFrame". "PublishFrame" will read the content of a frame and publish it out, so there is race condition among the only one "reader" and multiple "writers".

so, the race condition doesn't occur on that dictionary, but on the other collections within the frame, such as the "PhasorValueCollection", "DigitalValueCollection", "AnalogValueCollection". these collections are all shared among multiple threads, so synchronization is still needed, in my opinion.

I am trying to extend openPDC. In our system, the data for openPDC to process is not PMU measurements, but substation level SE result. And the data will be transmitted by GridStat middleware. I hope my work can make contributions to openPDC later.

Thanks.

New Post: bugs in openPDC

cheka — Fri, 19 Mar 2010 00:29:58 GMT

Hello Pinal:

first I am sorry to reply to you so late. there is some problems with my laptop, which make me unable to access the internet for a couple of days.

Thank you very much for the information you provided. Based on those information, I will re-write some of my codes, to post multiple asynchronous IO operation at the same time, to achieve a higher concurrency.

Source code checked in, #42559

ritchiecarroll — Thu, 18 Mar 2010 19:15:35 GMT

Synchrophasor: Updated PMU Connection Tester to accept IPv6, DNS and UDP Multicast addresses - forced IPv4 is now an optional application setting.

Source code checked in, #42549

mthakkar — Thu, 18 Mar 2010 15:05:33 GMT

Synchrophasor: openPDCManager - Added Device Measurements screen with real-time measurement values which refreshes every 30 seconds.

Source code checked in, #42546

ritchiecarroll — Thu, 18 Mar 2010 04:02:00 GMT

Synchrophasor: Version change for build Synchrophasor_v1.1.17.42526.

Source code checked in, #42545

ritchiecarroll — Thu, 18 Mar 2010 04:01:34 GMT

Historian: Version change for build Historian_v1.1.7.42533.

Source code checked in, #42533

pinalpatel — Wed, 17 Mar 2010 18:08:06 GMT

Historian: Added File and Serial output options to Historian Playback Utility.

Source code checked in, #42530

pinalpatel — Wed, 17 Mar 2010 17:12:42 GMT

Framework: Version change for build Framework_v1.1.11.42529.

Source code checked in, #42529

pinalpatel — Wed, 17 Mar 2010 16:50:47 GMT

Framework: Version change for build Framework_v1.1.10.42357.

Source code checked in, #42526

ryanzuo — Wed, 17 Mar 2010 13:35:25 GMT

Synchrophasor:Automatically copy EventDetection.dll to output directory

Source code checked in, #42490

jpatterson — Tue, 16 Mar 2010 18:50:27 GMT

updated archive parsing and manipulation code

New Post: bugs in openPDC

ritchiecarroll — Tue, 16 Mar 2010 18:41:27 GMT

First, we are very honored to have one of Anjan’s students reviewing the code. We have a tremendous respect for Dr. Bose, Dr. Bakken as well as Dr. Venkatasubramanian (and WSU as a whole) all of whom have had an instrumental influence on NASPI and synchrophasor technologies as a whole. We also would like to thank you for taking time to evaluate the system - in the end people like yourself will be the one’s adding the most value to the system by finding and helping to correct issues.

1. openPDC has some problem in supporting IPV6.

We have successfully deployed the openPDC in both IPv6 and IPv4 environments - Pinal has already weighed in on this issue (above). If his suggested configuration settings do not resolve the issue for you please let us know. Thanks!

2. there is race condition in "PhasorDataConcentratorBase"

Very observant! I am the author so I should explain my behavioral “deviation” :) The base class is designed around the concept of a “frame” of measurements (implemented as an abstract interface, “IFrame”, containing a “dictionary” of measurement values). This frame becomes the object that holds all time-aligned measurements. As such this dictionary is used for thread synchronization so that the base class can safely stop sorting measurements into the frame before offering it to the consumer for publication. This allows the consumer to use the frame without worry about locking and further modification to the frame by other threads.

The area of code you noticed in “PhasorDataConcentratorBase” is a derived class designed to construct a frame of synchrophasor data in a standard protocol from these constituent measurements. The important thing to note is that although the synchrophasor frame implements the “IFrame” interface as is required by “ConcentratorBase” - it doesn’t actually use this dictionary of measurements. It instead handles assignment of measurements to its frame as a “special case” since the measurements are destined for “special locations “within the frame - not simply maintained as a keyed-list. As a result, I chose not to lock the unused dictionary as an optimization - especially given that processing is already limited to 1 / fps (typically 1/30th or 1/60th of a second). By not using the dictionary the race condition will be avoided even if its not locked in the “AssignMeasurementToFrame” override. It’s rather impressive that you actually were able to dig deep enough to find this however! :)

3. there is some bug when generating CRC number for IEEE 1344 protocol.

The phasor protocol library is written to be completely “bi-directional” - that is, not only can it “parse” data from the synchrophasor protocols but it can also “generate” data in this format as well. In your above example you are using the buffer as provided for “parsing” to validate the CRC - you are then also using the “generation” properties (i.e., this.BinaryImage and this.BinaryLength) to create an image and then validate the same CRC. However, simply because the protocols were designed to be bi-directional, not all of them have been thoroughly tested. Typically we only use either IEEE C37.118 or BPA PDCstream in “generate” mode - looks like you found an issue here! The problem is that the generation method in the “ConfigurationFrame” of IEEE 1344 was coded using CRC16 and the parsing is using CRC-CCITT (which is done correctly) - this has been corrected as of change number 42489. Thanks for finding this!

4. Asynchronous Sending Problems

Hoping that Pinal’s answer to this query (above) has resolved this issue. To summarize, only thing happening asynchronously here is egress from “PublishFrame” method.

5. Extension Problem

Reusability of the synchrophasor parsing classes is open and accomplished through deriving from Stream which means data can from any source. However, like you mention the class “MultiProtcolFrameParser” is a closed by using the protocol enumeration. My only defense is that this particular class exists only to simplify consumption of phasor protocols by combining typical network layer and transports into one class using enumerations - as well as to provide a good example on how to use these libraries. Using this class is not required to use the phasor protocols, but it does mean the consumer needs manage their own transport layer as well as creating the protocol specific parsing classes to do otherwise.

That being said I am excited that you are considering integrating GridStat as a communications layer into the openPDC. Since GridStat is not really a phasor protocol in itself (i.e., it can “carry” phasor data but is not a standard phasor protocol) - I would not recommend extending the phasor protocol library to accommodate the GridStat system. Instead, I recommend writing one or more “extensions” to the openPDC. Extending the openPDC is handled by directly extending “InputAdapterBase” (to map incoming data to measurements), “ActionAdapterBase” (to time-align and process desired measurements) or “OutputAdapterBase” (to queue measurements for archival) - see http://openpdc.codeplex.com/wikipage?title=Custom%20Adapter&referringTitle=Documentation.

You could simply writing a “GridStatInputAdapter” to allow the openPDC to receive time-series style data from GridStat as well as a “GridStateOutpuAdapter” to allow the openPDC to publish time-series data to GridStat. I think this would accomplish the “integration” mission and both adapters could live in a single assembly (e.g., GridStatAdapters.dll). The openPDC Synchrophasor source code includes a “MySQLAdapters” project which can be used as a template of how to write an InputAdapter and an OutputAdapter that would operated in a similar fashion.

Thanks!

J. Ritchie Carroll

Source code checked in, #42489

ritchiecarroll — Tue, 16 Mar 2010 18:10:20 GMT

Synchrophasor: Corrected IEEE 1344 configuration frame generation code to use CRC-CCITT instead of CRC16.

Source code checked in, #42488

mthakkar — Tue, 16 Mar 2010 18:09:06 GMT

Synchrophasor: Data - Modified MeasurementDetail view to retrieve NodeID from Historian table instead of Device table.

New Post: bugs in openPDC

pinalpatel — Tue, 16 Mar 2010 17:29:02 GMT

Hi Chuanlin,

I haven't seen anything officially published by Microsoft, but here's a couple of link about asynchronous I/O in .NET and completion ports:

- Wintellect Slidedeck - These guys train Microsoft new hires, so very credible.
- Dr. Dobb's Article - Internal .NET implementation has changed somewhat, but the overall idea stays the same.

Another thing you can do for yourself is to disassemble mscorlib using .NET Reflector and see how Socket class is implemented; you will see that all asynchronous socket operations make a call to Socket.BindToCompletionPort() internal method.

- Pinal

New Post: Manager problems

mthakkar — Tue, 16 Mar 2010 16:28:42 GMT

Matt,

One more suggestion for the error message "Failed to Retrieve Node". Please modify web.config file for openPDCManagerServices to set includeExceptionDetailInFaults to true. Please see below. This will return exact detail of the error message. Please let me know if this helps or not.

Thank you,
Mehul Thakkar

New Post: Manager problems

mthakkar — Tue, 16 Mar 2010 14:01:08 GMT

Matt,

Did you run SampleDataSet.sql after setting up mySQL database? If not, then would you please try running this script and see if openPDCManager works. Please let me know.

Thanks,
Mehul Thakkar

New Post: bugs in openPDC

cheka — Tue, 16 Mar 2010 03:18:23 GMT

Hi Pinalpatel:

I am very glad to receive your reply.

I will try your method to solve the IPV6 problem. But I still think there is some other problems. The strangest thing is that openPDCConsole can connect with openPDC successfully at first, but a few days later, it cannot. I am sure that I didn't modify any source code or configuration during that period of time, so I guess there is conflict between openPDC and some windows updates.

Another thing I am very interested in is the "IO Completion Port" problem. I have used IOCP before. When I switched to C#, I also try to find how to use IOCP in c#. But all the information I get from the web told me that the only way to use IOCP in .NET is to call the managed codes written in c++, and I find no evidence to relae the asynchronous socket API in c# with IOCP, even in MSDN.

Would you please tell me where you find the information that the asynchronous socket API in c# were internally implemented with IOCP, and I want to know more technical details. If it is true, I am afraid some parts of my program need to be re-written, to achieve a higher concurrency.

Thank you.

Source code checked in, #42468

ritchiecarroll — Mon, 15 Mar 2010 18:28:51 GMT

Synchrophasor: Added auto-reset of device statistics when reconnected - also added adpater commands to do the same manually on-demand.