This project has moved. For the latest updates, please go here.

Help debugging a data loss issue

Sep 22, 2011 at 3:38 PM

Hello, I'm having trouble diagnosing a data loss issue with openPDC and am hoping someone has some insight on what might be going on. OpenPDC will archive just fine for several days and sometimes for as much as several weeks with no data loss, however, eventually some data is no longer archived. Data will be missing for a second, then return for half a second then be missing again. Here is an example of the data that I am seeing: . The straight lines show where the data is missing (the visualization program I am using will just connect the valid data points instead of set them to zero where they are missing). 

There doesn't seem to be any coinciding errors in the error log and resource availability on the server looks good. Restarting the OpenPDC service will fix the problem for a time. Any ideas on what might be going on? I'm using version 1.5 but it happens with version 1.4 as well. I don't think this is a network issue as I can record data just fine with stream reader or rtdms.




Sep 23, 2011 at 1:46 PM

Thanks Dave - would like to try to replicate here to see if we can get the same result.

Can you give an idea of your environment so we can attempt to emulate (e.g., connecting to PMU, PDCS, output streams, etc.)?


PS - Very nice graphing tool there! Is this reading history files using the API or using the web service?


Sep 23, 2011 at 2:51 PM

Hi Ritchie, I'm running this on two machines with the same behavior. Machine 1 is a physical Windows 2003 server with quad core intel Xeon 3GHz processors and 3.5GB of RAM. Machine 2 is a virtual windows 2008 R2 with two 2.66GHz CPU instances and 4GB of RAM. OpenPDC is getting data from one PDC (SEL-3306) and has no output streams. There are only the two default historians (statistics and phasor data). Let me know if you need anything else. Thanks! The visualization tool I created is using the web service to retrieve data.

Sep 23, 2011 at 2:55 PM

Also, I forgot to mention, both machines are running 64-bit operating systems with the 64-bit version of OpenPDC.

Sep 23, 2011 at 4:45 PM

I am following this issue and interested in how it is resolved.  I think there is a chance it might be a Virtualization Host performance or OS/hardware compatibility issue (in the case of your WS2003 physical server).  Can you provide the following details please?

1.  Host Server's Operating System and version.  (e.g. clean install WS 2008 R2 with Hyper-V, all current updates)

2.  Host Server's architecture (chipset and any virtualization specific enhancements), CPU(s) identification (e.g. Intel Xeon L5520 @2.27GHz (2 processors))

3.  Virtualization Software, version, and platform (Hyper-V R2 or VMware ESXi).  Is the hardware platform "certified campatible" for the virtualization software?

Can you perform the following rough observation please?   In the virtual machine, observe the Windows Clock (in round wall clock style).  When things are working correctly, the seconds will tick by steadily.  If things are not working, how do the seconds behave?  Are you able to observe abnormal timing in the virtual machine's system clock display?  If you can roughly observe abberations in the system clock's seconds display, then chances are this issue is a platform hardware/OS performance issue and not a software application issue.  If everything looks fine in the seconds display, system timing may still be an issue, but it will just take finer test software to determine this.


PS:  I experienced a similar issue with a SuperMicro server Intel Xeon L5520 @2.27GHz (2 processors)) running Windows Server 2008 Hyper-V.  The issue was visually observable in the virtual machines' clocks.  The issue was resolved by a combination of a BIOS update on the SuperMicro server AND Microsoft's eventual release of Hyper-V R2.  Your issue looks similar to what I experienced, except for the physical Windows 2003 machine.  I didn't try WS2003 on the physical Xeon L5520 - however, the VMs I was trying to get to work on the Xeon L5520 Hyper-V host were mostly Windows 2003 and 2003 R2 Server (32 bit).  I was not experiencing any problems at the time running the same Windows versions on AMD Phenom processor servers - which led me to open up support tickets with both Microsoft and Intel.  The problem drove me nuts for almost 4 months!