This project has moved and is read-only. For the latest updates, please go here.

Moving .d files

Mar 12, 2015 at 6:42 PM
Edited Mar 12, 2015 at 6:46 PM
Hi there everyone.
I work at the brazilian power grid operator, and we've been using openPDC for about a year and a half now. We've begun the synchrophasors project with 22 units, we have 36 now, and we'll grow to about 750 in a few years.
For obvious storage issues - .d files corresponding to one single day take up to 30Gb - we're creating a policy of moving old .d files to storage in the cloud. Our intention was to recover these files when we wanted to treat the data that they contain. However, when we do that, when we move the files back from the storage in the cloud to a local computer, we're not able to read the data in it - we use the service provided by openPDC in order to recover the data in json/xml format.
We've made attempts to move the most recent .dat files, plus the ppa_archive.d file along to the local folder, but it still doesn't work.
Has anybody faced this situation? How did you solve it?
Thanks a lot, and best regards,
Romulo Menezes
Mar 13, 2015 at 6:47 PM
I would expect that this would work.

You might want to validate .D integrity, i.e., take a CRC of file before post to cloud, the push to cloud, then pull from cloud, then check the CRC again.

Mar 13, 2015 at 9:06 PM
Hi Romulo,

The openPDC uses the same module to write to the archive as it uses to read from the archive file when it produces output for the historian web services. That module tracks the files that are in the archive directory and stores a list of historic archive files including the time range of the data in those files so that it knows what files to look in to retrieve data for a query. However, the file watcher that tracks files as they enter and leave the directory is turned off by default, meaning that if you simply drop a new file in that directory, the archive module may never know it.

A quick, manual test to see if this is the problem would be to drop a historic file into the archive directory, then initialize the PPA historian adapter (PPA is the default name for that adapter, but it might be different in your system), and then query the web service for data from that file. If the test is successful, I would recommend turning on the file tracking functionality in the archive so that it will be able to keep track of files you drop into that directory. The setting to track files in that directory is in the openPDC.exe.config file in the "ppaArchiveFile" section (again, dependent on the name of your archive), and it is called "MonitorNewArchiveFiles". Stop the openPDC service, set the value of that setting to "True", then start the service again.

Note that we originally turned off all the file watchers in the system because of a memory leak in .NET 4.0 related to the FileSystemWatcher. If you are using openPDC v1.5, I would recommend upgrading to the latest version which uses .NET 4.5 so that you won't introduce a memory leak.

Apr 7, 2015 at 9:51 PM
Hi Stephen. Thanks for you answer.
I apologize if wthat is coming is too newbie a question, but what do you mean by "initialize the PPA historian adapter"? Is it just start the historian playback utility, or is it something less simple than that?

What I did so far was turning on that flag you mentioned (MonitorNewArchiveFile, in the ppaArchiveFile section of the openPDC.exe.config) and after that I droped some historic files in the openPDC\Archive folder. When I tried to read the data in the historic file, both with the Historian Playback Utility and by calling the web service directly, it didn't work =/

There's one thing I didn't mention, and I was wondering if that may influence or even cause the problem: we use two Amazon EC2 instances (two virtual machines) with openPDC, a "production" station, the one that's online, receiving data from the phasors, and another one in which we process the data. (One reason for that is when we call the web service to read the data, the flow of the data that is being received in real time is disturbed.) So the processing station does not receive real time data, and at the end of each day we copy the .d files from the production station to an Amazon storage (S3), and from there to the processing station. The problem that I've been having happens in the __processing__ station, when I try to read the data in the .d files in its Archive folder. Like I said, it doesn't work either with the Historian Playback Utility or by callilng the web service. Mind you that the processing station is a "hard copy" of the production one, only with a different IP address - and obviously the data have evolved differently since the copy was made. We simple made a snapshot of the station and turned it to another station. We did not reinstall the openPDC. Do you think this can influence or cause the problem?

Thanks again,