This project has moved. For the latest updates, please go here.

scalability problem

Developer
Jun 14, 2010 at 12:18 AM

Hello:

I am the research associate of EECS, WSU. Our research group is working on a software system based on openPDC. In this system, we generate PMU measurements based on the simulation result of Transient Simulation Tool. Since we are dealing with a simulated power system, so we can install PMU device wherever we want, so there will be a large number of PMU devices when we are simulating a large system. Then here comes the problem, how many measurement devices can openPDC handles simultaneously?

From the the SQL script "openPDC.sql" and source codes in "PhasorMeasurementMapper.cs", I can see that "openPDC.exe" will use an individual "PhasorMeasurementMapper" for each PMU device.  This "PhasorMeasurementMapper" will run on its own thread, and if it is a remote device, "PhasorMeasurementMapper" will use its own socket to connect and receive data. That is to say, if we want to receive measurements simultaneously from N device, we must need N threads and N sockets allocated. So I guess that is the reason why the author limits the maximum device number to 100 when createing "RuntimeDevice" view in database.

100 devices maybe not enough for our simulation purpose. In my opinion, there will be two methods which can walk around this limitation:
(1)  Design a new input adapter, which just use a single thread and a single socket, but can receive data from multiple devices. (Actually, I have already made a simple one. )
(2)  Add some concentrators in that simulated power system. For example, add a concentrator for each substation, and let openPDC only receive data from concentrators rather than single devices, which will highly reduces the threads and sockets required.

Above is just my opinion, and I will appreciate any advice or comment from the author of openPDC.
Thank you very much.

                                               Chuanlin Zhao

Coordinator
Jun 14, 2010 at 4:11 PM

Hi Chuanlin,

The RuntimeDevice view actually does not limit the maximum number of devices. Perhaps you are referring to the "TOP (100) PERCENT" added to the SELECT statement in the SQL Server script? That statement is only used by SQL Server and does not limit the number of records in the view.

Since the openPDC can make no guarantees about whether it can connect to each of the devices defined in the configuration, connections are made in separate threads so that devices that cannot be reached do not block other device connections. Each device needs a socket in order for the openPDC to issue commands those devices. If you do experience issues resulting from the initiation of too many connections, may we suggest using a tiered architecture with multiple PDCs? You can forward the data to another instance of the openPDC through an OutputStream or you could try using the RemoteOutputAdapter in the HistorianAdapters project. You can find our thoughts on using the RemoteOutputAdapter for scalability purposes at this link:

http://openpdc.codeplex.com/Project/Download/FileDownload.aspx?DownloadId=116120

Stephen Wills

Developer
Jun 14, 2010 at 5:47 PM

Thank you very much for your reply. so you prefer the second method I proposed, which is, maybe add a concentrator for each substation. I will try it.

the next question is just from my curiosity. I just started to use .NET 4.0 in my codes recently.   .net 4.0 has a lot of improvement in the domain of "Concurrent Programming", such as Event-based Asynchronous Pattern, or, Asynchronous Workflow in F#. And I can see that concurrency is very important to openPDC, because each adapter runs on its own thread.  Concurrent strategy will determine that how many PMU devices can be dealt with simultaneously and how many functions can be performed at the same time.

so I am just curious about whether openPDC has some intention or plan to apply these new concurrent features of .NET 4.0 into its codes?

thank you very much.

Coordinator
Jun 14, 2010 at 5:50 PM

There are no direct limitations in code - the Access database does not support triggers however, so we preloaded ID's for the runtime table - but you can add more. We recommend only using Access for development purposes however - MySQL, SQL Server or other database should be used for production or large scale implementations.

Keep in mind that because the system is using non-blocking asynchronous completion ports for all communications there will be no limits applied to a number of connections and because transfers are made using the thread pool the system will continue to process all data up to yor hardware limits.

Typically there is one mapper applied per input stream connection - which can contiain many PMU's

Thanks! Ritchie

Coordinator
Jun 14, 2010 at 5:52 PM
Edited Jun 14, 2010 at 9:54 PM

FYI - .NET 4.0 based openPDC is coming this summer - thanks!