This project has moved and is read-only. For the latest updates, please go here.

Handling Historical Data

May 31, 2015 at 2:38 PM
Me again...

My Raspberry Pi PDC has been up and running for a while, so I've started pulling some data off of it for analysis. I have a few issues and wanted to check on them.
  1. Handling .d files - I'd ideally like to move the .d file to a Windows based machine and export data from there, but I haven't had any success doing that. I read the other thread and tried a couple of things, but it didn't help. I really didn't want to be running the whole OpenPDC package on my Windows box... all I want to do is analyze data from the files.
  2. Data corruption when exporting to CSV on the Pi. I'm handling 4 hour chunks of data, looking at only frequency right now. In one 4 hour chunk, I'm getting a number of corrupted lines (by a null character), and a few pieces where data is repeated (perhaps 40 or so measurements repeated). The export should contain 432001 lines, but it's typically more than that. After a couple hours of data manipulation, massaging, and re-export of certain time chunks, I can get a correct file. But it's a lot of work. :(
  3. I'd like to get data all in a single row, not multiple rows. PMU, Timestamp, Frequency, Voltage, Phase Angle all in columns - rather than F/V/Phase Angle each in their own rows. Any way to do that?

Jun 3, 2015 at 8:53 PM
Hi Andrew,

CSV is probably not your best choice under any platform - this is what the historian was designed for, however.

Are having trouble transferring .D files to Windows machine then extracting data, or are you trying to directly access files from via a shared folder on the PI?

There are a few things you need to do once you get the files locally on Windows machine for analysis, for example:
  1. Copy the .DAT file as well as the .D files that you need - note that when the openPDC is running, copying the active "ppa_archive.d" may cause file corruption, it is suggested to not copy this file while the openPDC is running
  2. When you copy a set of historical .D files without the active file, you need to make one of the historical files the "active" file - you can simply do this by taking the newest of the .D files and renaming it to "ppa_archive.d"
  3. Point your tool, e.g., the Historian Trending Tool, to the folder that contains your .D files (including your "active" .D file) and the .DAT file (this is the metadata for the points) and you should see the names of the points that have been archived.
  4. Now you can select points and time-range to visually trend and/or export data to a file, e.g., COMTRADE or CSV.