This project has moved. For the latest updates, please go here.

Suggestions for Improving the Historian Playback Utility

Developer
Nov 24, 2010 at 10:44 PM

Dear openPDC Coordinators,

Along with my MSc student, we extracted 15 days of data from the historian from a PMU streaming data to our openPDC in Stockholm.

The process was the following: i) select the device signal, ii) select the date, iii) assign a file name, iv)

We latter took the .csv files and built a MySQL databse. This database was used to automatically import data into Matlab structures using the MySQL-Matlab connector: http://mmf.utoronto.ca/resrchres/mysql/

We go through this process because we want the measurements to be sorted, in the past I have found out that this might not be the case because of the way the historian database is built (correct me if I am wrong). I have found out that is extremely time consuming to sort the measurements in other analysis programs such as Matlab and Octave.

From these experiences, I would like to make the following suggestions:

1) A queuing feature, so many jobs can be listed and we can leave the server to do the work

2) A Time Stamp conversion feature. Time stamps can be converted into a double while doing the extraction. This can be based on UNIX stamps or nicely done with the approach from the MathWorks: http://www.mathworks.com/help/techdoc/ref/datenum.html

Reason: it is much more easier to deal with numerical time stamps than string time stamps in post-processing the data, also it is quite time consuming for many applications to do this process efficiently.Moreover, perhaps is possible to add this intrinsically to the Historian Database so its not necessary to do it while performing the output to plain-text.

3) User defined time stamp format for the output data.

4) User defined output format for the plain-text data.

5) Sometimes, depending on the analysis, it is not necessary to have all 30-50 samples from each second. It would be a very nice feature if a fixed number of samples could be extracted. For example, only the first sample every second, or 5, or 10 samples for each second. The first option we did in Matlab after importing the data, and its also possible in MySQL by querying only for a certain time-stamp format, i.e. HH:MM:00.000, however, I'm sure you guys could do something smarter.

Well, that's all for now. These are only suggestions that would be useful for a data analyst, nevertheless, I realize that they might not be of important priority for the project as there are more pressing things to deal with.

Best,

Luigi

Developer
Nov 24, 2010 at 10:45 PM

I apologize, the missing step iv) above was to set up the output-data string such as: {0:I};{1:T};{2:V}

I forgot to mention that documentation regarding on how to set up this string could be beneficial.

Best,

Luigi

Coordinator
Nov 26, 2010 at 3:28 PM
This discussion has been copied to a work item. Click here to go to the work item and continue the discussion.