This project has moved. For the latest updates, please go here.

Enable Data Compression?

May 18, 2011 at 4:57 PM

I'm trying to enable data compression for my historian so I set the CompressData value to true in the config. I also noticed that this value was overridden to false in the documentation so I recompiled the HistorianAdapters.dll with the override set to true but there is still no effect on data size. Is there any way to enable this feature?

 

Thanks,

Dave

Coordinator
May 18, 2011 at 5:31 PM

Compression comes by setting a compression ratio per point - you will need to modify the HistorianMetaData view to change the compression ratio per signal type.

May 23, 2011 at 3:11 PM

I looked at that view and it appears that the compression is achieved by down-sampling. Is it possible to use lossless compression?

 

Dave

Coordinator
May 23, 2011 at 6:12 PM

Right - basically slope based point compression - losses compression is in the feature list for the next version...

Thanks!
Ritchie

May 24, 2011 at 9:01 PM

I'm not sure whether or not this will be helpful, but I decided to implement a solution that compresses the data via LZMA compression (using this library: http://www.eggheadcafe.com/tutorials/aspnet/064b41e4-60bc-4d35-9136-368603bcc27a/7zip-lzma-inmemory-com.aspx) after each historian file rollover. It then decompresses only the needed data files when retrieved via the web services. It compresses our 100MB files down to about 8.2MB saving a large amount of space. Here's what I did:

HistorianAdapter.dll -> LocalOutputAdapter.cs:

//added in the init() function at line 314
            m_archive.RolloverComplete += new EventHandler(m_archive_RolloverComplete);

void m_archive_RolloverComplete(object sender, EventArgs e)
        {
            if (m_archive.CompressData == true)
            {
                var fileArray = m_archive.FileName.Split('\\');
                string fName = fileArray[fileArray.Length - 1];
                string fPath = Regex.Replace(m_archive.FileName, fName, "");
                string[] archiveFiles = Directory.GetFiles(fPath, ".", SearchOption.TopDirectoryOnly);
                string[] dataFiles = archiveFiles.Where(f => f.EndsWith(".d") && f != m_archive.FileName).ToArray();

                foreach (var file in dataFiles)
                {
                    try
                    {
                        OnStatusMessage("Compressing (" + dataFiles.Count() + "): " + file);
                        //If there is no current zip file with this name then try to zip the file
                        //otherwise, delete it.
                        if (archiveFiles.Where(f => f == Regex.Replace(file, "\\.d$", ".z")).Count() == 0)
                        {
                            FileStream source = new FileStream(file, FileMode.Open, FileAccess.Read, FileShare.Read);
                            FileStream dest = new FileStream(Regex.Replace(file, "\\.d$", ".z"), FileMode.OpenOrCreate, FileAccess.Write);
                            byte[] buffer = new byte[source.Length];
                            source.Read(buffer, 0, buffer.Length);
                            source.Close();
                            var compressedBuffer = SevenZipHelper.Compress(buffer);
                            dest.Write(compressedBuffer, 0, compressedBuffer.Length);
                            dest.Close();
                            OnStatusMessage(file + " compressed");
                        }
                        else
                        {
                            OnStatusMessage("Deleting: " + file);
                            File.Delete(file);
                        }
                    }
                    catch (Exception ex)
                    {
                        OnStatusMessage("Exception in rollover: " + ex.Message);
                    }
                }
            }
        }

This will compress the previous rollover file and delete any other .d files that are in the directory. The idea is that any .d files that were decompressed via the dataservice will remain decompressed in the directory until the next rollover for any further data service calls.

TVA.Historian.dll -> ArchiveFile.cs

//I changed this function to run the DecompressNeededFiles function before the ReadData function at line 1728        
public IEnumerable ReadData(int historianID, string startTime, string endTime)
        {
            DecompressNeededFiles(startTime, endTime);
            return ReadData(historianID, TimeTag.Parse(startTime), TimeTag.Parse(endTime));
        }

private void DecompressNeededFiles(string startTime, string endTime)
        {
            try
            {
                var requestStart = DateTime.Parse(startTime);
                var requestEnd = DateTime.Parse(endTime);
                var fName = this.FileName.Split('\\').Last();
                var fPath = Regex.Replace(FileName, fName + "$", "");
                var allFiles = Directory.GetFiles(fPath, ".", SearchOption.TopDirectoryOnly);
                string dtFormat = "yyyy-MM-dd HH!mm!ss.fff";
                bool decompressionPerformed = false;
                //get a list of compressed files in the request's date range by parsing the file name for the corresponding "from" and "to"
		//date strings.
                var neededFiles = allFiles.Where(f =>
                    {
                        if (!f.EndsWith(".z")) return false;
                        var fileBase = Regex.Replace(fName, "\\.d$", "");
                        var fileComponents = Regex.Replace(f.Split('\\').Last(), "^" + fileBase + "_", "").Split('_');
                        var fileStart = DateTime.ParseExact(fileComponents.First(), dtFormat, System.Globalization.CultureInfo.InvariantCulture);
                        var fileEnd = DateTime.ParseExact(Regex.Replace(fileComponents.Last(),"\\.z$",""), dtFormat, System.Globalization.CultureInfo.InvariantCulture);
                        if ((fileStart <= requestStart && requestStart <= fileEnd) ||
                            (requestStart <= fileStart && fileEnd <= requestEnd) ||
                            (fileStart <= requestEnd && fileEnd >= requestEnd))
                            return true;
                        return false;
                    }).ToList();
                //decompress any files that need it.
                foreach (string file in neededFiles)
                {
                    var dFile = Regex.Replace(file, "\\.z$", ".d");
                    //do nothing if file is already decompressed
		    if(File.Exists(dFile)) return;
                    FileStream source = new FileStream(file, FileMode.Open, FileAccess.Read, FileShare.Read);
                    FileStream dest = new FileStream(Regex.Replace(file, "\\.z$", ".d"), FileMode.OpenOrCreate, FileAccess.Write);
                    byte[] buffer = new byte[source.Length];
                    source.Read(buffer, 0, buffer.Length);
                    source.Close();
                    var decBuffer = SevenZip.Compression.LZMA.SevenZipHelper.Decompress(buffer);
                    dest.Write(decBuffer, 0, decBuffer.Length);
                    dest.Close();
                    decompressionPerformed = true;
                }
                //if any files were decompressed then it is nescessary to re-read the history files.
                if (decompressionPerformed)
                {
                    m_historicArchiveFiles = null;
                    BuildHistoricFileList();
                }
            }
            catch (Exception ex)
            {
            }
        }

Coordinator
May 24, 2011 at 9:15 PM

Wow! Thanks so much for the code implementation proposal. Good to know that lossless compression implementation will make a big difference!

Ritchie