This project has moved and is read-only. For the latest updates, please go here.

Any Hadoop Implementations? Production or Experimental?

Dec 13, 2011 at 12:52 AM

Is anyone using Hadoop with openPDC?

Are you using it in Production or just Experimentation?

If you are, can you share some of your experiences, procedures, or references that might help kick start other researchers that may be interested in using Hadoop?

Arnold

Dec 18, 2011 at 11:58 PM

Arnold,

TVA uses Hadoop in production with both HDFS for cheap scalable storage and MapReduce for processing large amounts of time series data. Some materials you may find helpful:

 

Original Whitepapers:

http://www.cloudera.com/resource/hadoop-platform-smartgrid-tva-josh-patterson

http://www.cloudera.com/blog/2009/06/smart-grid-hadoop-tennessee-valley-authority-tva/


OSCON-data presentation (good TVA story here):

http://www.slideshare.net/jpatanooga/oscon-data-2011-lumberyard


High Level articles/coverage of project:

http://www.slideshare.net/cloudera/hadoop-as-the-platform-for-the-smartgrid-at-tva

http://www.tva.gov/news/releases/octdec09/data_collection_software.htm

http://jpatterson.floe.tv/index.php/2009/10/29/the-smartgrid-goes-open-source/

http://gigaom.com/cleantech/the-google-android-of-the-smart-grid-openpdc/

http://news.cnet.com/8301-13846_3-10393259-62.html

http://gigaom.com/cleantech/how-to-use-open-source-hadoop-for-the-smart-grid/

http://openpdc.codeplex.com/



Engineering Literature:

http://openpdc.codeplex.com/

https://openpdc.svn.codeplex.com/svn/Hadoop/Current%20Version/

https://openpdc.svn.codeplex.com/svn/Hadoop/Current%20Version/docs/openPDC%20Datamining%20Tools%20Guide.pdf

 

General time series processing with Hadoop (along with another source code example):

http://www.cloudera.com/blog/2011/03/simple-moving-average-secondary-sort-and-mapreduce-part-1/
http://www.cloudera.com/blog/2011/03/simple-moving-average-secondary-sort-and-mapreduce-part-2/
http://www.cloudera.com/blog/2011/04/simple-moving-average-secondary-sort-and-mapreduce-part-3/

 

Hope that helps,

 

Josh

Dec 20, 2011 at 1:26 AM
Edited Dec 20, 2011 at 2:08 AM

That was by far the most excellent answer I have ever received!

Thanks!

Arnold

PS:  I added your references to the Hadoop section of the FAQ

http://openpdc.codeplex.com/wikipage?title=FAQ#how_are_openpdc_and_hadoop_used