This project has moved and is read-only. For the latest updates, please go here.

Reading Archive Data from .d files manually

Aug 12, 2011 at 8:23 PM
Edited Aug 12, 2011 at 8:41 PM

Hi,

I would appreciate your help on a few questions I've got by working with Archive Files!

    I' trying to create a tool for students to analyze historical files on their own without having access to OpenPDC interface or files basically providing archive ".d" files for their own analysis.

            1. Could i utilize TVA.Historian.ArchiveFiles library to open manually archive "*.d" files with C#?

                     1.1 It looks like there is a possibility of utilizing ArchiveFile class by setting up file name and call Open but Open require one of the configuration files...

                          1.1.1 So is there any configuration files to copy along with ".d"?

           2. Is there any other products designed for Archive Files except Hadoop & Historian Playback?

           3. I've created archive reader from binary format documentation, but: 

                    3.1 Where is signal ID  !?

                    3.2 What is a meaning of <short flags 2 bytes> on DataPoint block, quality of signal and value?

Thanks,

DK.

P.s. for my understanding such a tools might be needed to scan, by various security patters, inside of an archive files produced by built-in historian. AdoAdapter needed for preliminary analysis of incoming data, basically, to design robust security algorithms before processing produced archive files. Any notes/advices in regards of this topic will be very helpful...

Aug 12, 2011 at 9:10 PM

A good place to start is to open the Historian project and look at the code for the "Historian Playback Utility" - this application does basically what you are referring to (i.e., opening the archive files directly and querying data):

http://openpdc.codeplex.com/SourceControl/changeset/view/69632#720925

Another example can be found in the StatisticsReader class currently in the phasor protocols library:

http://openpdc.codeplex.com/SourceControl/changeset/view/69632#1084098

For your reference, the format of the .D is documented here:

http://www.codeplex.com/Project/Download/FileDownload.aspx?ProjectName=openpdc&DownloadId=109343

Hope that helps...

Thanks!
Ritchie

Aug 15, 2011 at 3:16 PM
Edited Aug 15, 2011 at 4:20 PM

  Thanks  Richie,

       Reference to Historian Playback was very helpful. 

       What about format.d files i already created this utility but i can't find SignalID on description given and meaning of Short Flags on DataPoint Block... ? Is  there any convention for historian file names, if yes do you have documentation on it also?

 

Regards,

Denis.

Aug 15, 2011 at 8:49 PM
Edited Aug 15, 2011 at 8:50 PM

SignalID is in the metadata database - the best place to look is the ActiveMeasurement view - you can correlate the data in the archive and the metadata using the measurement "key" - which is defined as the "ID" column in the measurement metadata. This is defined as "Source:PointID" where source is the acronym of the destination historian and point ID is the historian point index, you could lookup a signal ID using a SQL Expression like "SELECT SignalID FROM ActiveMeasurement WHERE ID='PPA:15'".

The "short flags" on the data point in a data block you reference are the flags for the data point which happen to be a short-integer, that is 16-bits, defined as follows:

  • Bits 0 - 4 are used to describe 30 possible quality states for the data point
  • Bits 5 - 15 are used to store the "milliseconds" related to this data point - remember the "data block" is many points associated with a given time in "seconds" - so each archived data point stores its associated milliseonds.

Predefined quality states:

    /// <summary>
    /// Indicates the quality of time-series data.
    /// </summary>
    public enum Quality
    {
        /// <summary>
        /// Unknown
        /// </summary>
        Unknown,
        /// <summary>
        /// DeletedFromProcessing
        /// </summary>
        DeletedFromProcessing,
        /// <summary>
        /// CouldNotCalculate
        /// </summary>
        CouldNotCalculate,
        /// <summary>
        /// FrontEndHardwareError
        /// </summary>
        FrontEndHardwareError,
        /// <summary>
        /// SensorReadError
        /// </summary>
        SensorReadError,
        /// <summary>
        /// OpenThermocouple
        /// </summary>
        OpenThermocouple,
        /// <summary>
        /// InputCountsOutOfSensorRange
        /// </summary>
        InputCountsOutOfSensorRange,
        /// <summary>
        /// UnreasonableHigh
        /// </summary>
        UnreasonableHigh,
        /// <summary>
        /// UnreasonableLow
        /// </summary>
        UnreasonableLow,
        /// <summary>
        /// Old
        /// </summary>
        Old,
        /// <summary>
        /// SuspectValueAboveHiHiLimit
        /// </summary>
        SuspectValueAboveHiHiLimit,
        /// <summary>
        /// SuspectValueBelowLoLoLimit
        /// </summary>
        SuspectValueBelowLoLoLimit,
        /// <summary>
        /// SuspectValueAboveHiLimit
        /// </summary>
        SuspectValueAboveHiLimit,
        /// <summary>
        /// SuspectValueBelowLoLimit
        /// </summary>
        SuspectValueBelowLoLimit,
        /// <summary>
        /// SuspectData
        /// </summary>
        SuspectData,
        /// <summary>
        /// DigitalSuspectAlarm
        /// </summary>
        DigitalSuspectAlarm,
        /// <summary>
        /// InsertedValueAboveHiHiLimit
        /// </summary>
        InsertedValueAboveHiHiLimit,
        /// <summary>
        /// InsertedValueBelowLoLoLimit
        /// </summary>
        InsertedValueBelowLoLoLimit,
        /// <summary>
        /// InsertedValueAboveHiLimit
        /// </summary>
        InsertedValueAboveHiLimit,
        /// <summary>
        /// InsertedValueBelowLoLimit
        /// </summary>
        InsertedValueBelowLoLimit,
        /// <summary>
        /// InsertedValue
        /// </summary>
        InsertedValue,
        /// <summary>
        /// DigitalInsertedStatusInAlarm
        /// </summary>
        DigitalInsertedStatusInAlarm,
        /// <summary>
        /// LogicalAlarm
        /// </summary>
        LogicalAlarm,
        /// <summary>
        /// ValueAboveHiHiAlarm
        /// </summary>
        ValueAboveHiHiAlarm,
        /// <summary>
        /// ValueBelowLoLoAlarm
        /// </summary>
        ValueBelowLoLoAlarm,
        /// <summary>
        /// ValueAboveHiAlarm
        /// </summary>
        ValueAboveHiAlarm,
        /// <summary>
        /// ValueBelowLoAlarm
        /// </summary>
        ValueBelowLoAlarm,
        /// <summary>
        /// DeletedFromAlarmChecks
        /// </summary>
        DeletedFromAlarmChecks,
        /// <summary>
        /// InhibitedByCutoutPoint
        /// </summary>
        InhibitedByCutoutPoint,
        /// <summary>
        /// Good
        /// </summary>
        Good
    }

Covention for the historian file names is like this:

FilePath.GetDirectoryName(m_fileName) + 
(FilePath.GetFileNameWithoutExtension(m_fileName) + "_" + 
m_fat.FileStartTime.ToString("yyyy-MM-dd HH:mm:ss.fff") + "_to_" + 
m_fat.FileEndTime.ToString("yyyy-MM-dd HH:mm:ss.fff") + 
FileExtension).Replace(':', '!');

Thanks!
Ritchie

Aug 15, 2011 at 10:54 PM

Thank you Richie, 

   I've facing some interesting behavior with TVA.Historian library.   

  - Data can only be directly read from files that are Active

Below is the code for Open() function inside ArchiveFile class and it basically means i can't call ReadData function because archive is not active.!?

if (m_fileType != ArchiveFileType.Active)  return ;

 And i can't set it to active since ti's private value and ArchiveFile class accept only .d & .stanby .

 

Is there any way for me to utilize ReadData function from ArchiveClass for non Active files?

Regards,

DK.

Aug 16, 2011 at 12:04 AM

Just copy an existing blank or partial "xxx_archive.d" file to the folder - that will be consider the "active" one...

Thanks!
Ritchie

Aug 16, 2011 at 3:16 PM
Edited Aug 16, 2011 at 4:14 PM

Yes that did for "*_archive.d" file only, but what about the other ones with time stamps on it they have "*_archive_*.d" format? Is there any redirection mechanism for these files like archive.d always active and other files will be uploaded automatically, does it mean i have always read archive.d file to get information from other files in archive folder?

Aug 16, 2011 at 6:42 PM

The code automatically scans an entire folder for all files as a collection. When you specify a start time and a stop time in the API, it will automatically load the the data from the appropriate file. You don't open a specific file - the API does this for you...

Make sense?

Ritchie

Aug 16, 2011 at 8:32 PM
Edited Aug 16, 2011 at 8:35 PM

Everything is working now, just needed to confirm, I saw API scan mechanism, as you can see from my previous post, but wasn't sure for logical purpose.

Regards,

DK

Aug 24, 2011 at 5:54 PM

Hi,

    Is there any way to access "#archive.d" file with File Share option so i can read it from different processes without looking?  Every 60 sec message, from OpenPDC, pops up: "cannot open *archive.d for write access".

Regards,

DK.

Aug 28, 2011 at 3:03 PM

Something like this should work:

    FileStream m_fileStream = new FileStream(m_fileName, FileMode.Open, FileAccess.ReadWrite, FileShare.ReadWrite);

There are interop files that engaged though when "rolling-over" to a new file, etc.

Normally, it's best to use the API since it will automatically "cooperate" with the other processes with file-sharing and access...

Ritchie

Aug 29, 2011 at 8:17 PM

Thank you Ritchie!