HI,

I have installed OPenPDC on window machine and installled Hadoop on another machine runing SUSE OS. I want to upload the Historian Data (c:\OpenPDC\archive) into HDFS. How i can do it?

Need help.

Many thanks

Lot's of fun in this exercise - suggest you start this journey here:

• Processing openPDC Data with Hadoop
• Other useful links

Thanks!
Ritchie

Many thanks Ritchie,

But i want directly store the data into HDFS. As i have mentioned that OpenPDC software runing on one machine and hadoop runing on another machine.

Do you know when configuring the OpenPDC, there is option remote server for historian data. How to configure so that the historian data store on remote computer.

Will highly appricitae your response.

Many thanks again

Ah - yes. There are two components that work together to make this happen:

1. The Java based HDFS bridge code - this "accepts" files onto Hadoop using a modified version of FTP that will validate the checksum of the uploaded file:
   • http://openpdc.codeplex.com/SourceControl/changeset/view/78875#727431
2. The C#.NET based local openHistorian Hadoop replication provider which will auotmatically push .D files to the HDFS bridge and retry if checksum doesn't match:
   • http://openpdc.codeplex.com/SourceControl/changeset/view/78875#753947

The Hadoop replication provider can be enabled in the openPDC.exe.config file as it is not enabled by default. You can use the "XML Configuration Editor" to modify the config file and adjust settings for the Hadoop provider and local historian.

Relevant settings:

• ArchiveLocation - specifies the local paths for .D files to push to Hadoop, multiple paths can be separated by a semi-colon
• ReplicaLocation - specifes the FTP style URI of the Hadoop HDFS bridge

Basically the local historian archives data to a local .D as normal - once this local file is "full" and gets timestamped (i.e., rolled over and closed), it will automatically get pushed to Hadoop (i.e., replicated to Hadoop over the HDFS bridge).

You can make the local historian files very small (again settings in the config file) so that they roll over very often and you can set the maximum number of files you want to maintain locally for posterity,

It is recommended you maintain at least some cache of the files locally so you can validate the data is working properly using the provided historian tools, e.g., the Historian Viewer and Historian Playback Utility.

Once the files are on Hadoop you will need to use Map/Reduce jobs to query, analyze and extract data. There is source code provided to do map/reducing of the .D files, and there have been several tools developed by other parties such as the following one to "extract" .D archived data once stored on Hadoop, e.g.:

https://pdihistorian.svn.sourceforge.net/svnroot/pdihistorian/PDI_Historian/

Hope that helps!

Ritchie

You do have to use both components in order to move files to Hadoop.

The HDFS bridge code defines a special "FTP" service for Hadoop that will validate the checksum of historian files that were uploaded using the openPDC Hadoop replication provider.

You will need to run the HDFS bridge to start listening for historian files to be transfered from the openPDC. This is a java application that will need to be started on Hadoop, we typically run this application on the head node. I'm not going to be the best person to provide detailed instructions on how to get things going on Hadoop however.

Note that it will not transfer a file until the local openPDC fills up one historian file (i.e., files will not transfer until the .D file "rolls-over"). This could take a while with the default settings and not many reporting PMU devices.

The default archive size is 100MB, if you want to test the transfer quicker make sure the default openPDC archive file size is very small (say 5 megabytes or so). This setting can be found in the openPDC.exe.config file under settings/ppaArchiveFile/FileSize.

Thanks!
Ritchie

many thanks

1. The ReplicaLocation setting specifes the URI of the Hadoop HDFS bridge (e.g., ftp://134.83.35.24:25) - the HadoopReplicator running on the openPDC will connect to the HDFS Bridge running on Hadoop using this URL. Again you will need to run the HDFS bridge Java based application on the Hadoop system - this then will "listen" for files from the openPDC running on another computer.
2. Your settings look good except for ReplicaLocation (see item 1)
3. You have options here. You can certainly change your local historian to save data to MySQL instead of openHistorian then use other tools to migrate and query MySQL data to Hadoop. Be forwarned however, the openHistorian is designed to handle archving of high-speed PMU data - you can only get so much speed when archiving through MySQL (see this post and search for MySQL in discussions list for more details on archiving to MySQL)

Thanks!
Ritchie

I have installed the hdfs-over-ftp mentioned (http://openpdc.codeplex.com/wikipage?title=Using Hadoop (Developers)&referringTitle=Documentation#copy_data_into_hadoop) . The hdfs-over-ftp is running as process on hadoop machine. On OpenPDC site, i have enabled HadoopReplicationProvider, set the archive location (c:\program files\openpdc\statistics\stat_archive.d  and replication location (ftp:134.83.35.24:9000). But still files are not uploaded directly to hadoop.

Any body have any suggestion?

Many thanks

Hi,

Any one know about the following statment (HadoopReplication)

string[] credentials = replicaUri.UserInfo.Split(':');

Here mean the username and password?

Many thanks