This project has moved. For the latest updates, please go here.

Problem with publishing frames in ActionAdapter

Apr 13, 2016 at 6:32 PM
Hello,

In action adapters, depending on specified LagTime, calculated values has NaN values for certain frames (instances).
My first objective is to receive the data in real-time to make decisions (If I miss some data due to any delays, I would replace them using a prediction algorithm). So I tried to replace the NaN values with a predicted value (let's say a fixed value for sake of simplicity). This way I can avoid NaNs in calculated output.

So I think that if the values are not received yet, the Action Adapter would not even publish the frame. In that case, what can I do to instantly replace the non-received frame measurements with predefined values?

I would really appreciate any help.

Thank You!

Reza

P.S.

Here is a code I used to solve this issue. But I still get NaNs for some frames. I think It's because that, if the values are not received within the LagTime, PublishFrame would not publish the measurements.

/// Extract the measurement values from the IFrame object
        foreach (IMeasurement measurement in frame.Measurements.Values)
        {
            if (!double.IsNaN(measurement.Value))    // If there are measurements in this frame 
            {
                listOfMeasurements.Add(measurement.Value);
                inputMeasurementKeys.Add(measurement.Key.ToString());
            }
            else   // If there are no measurements in this frame 
            {
                listOfMeasurements.Add(0.0D); // Test for replacing non-received measurements
                inputMeasurementKeys.Add(measurement.Key.ToString());
            }
        }
Coordinator
Apr 14, 2016 at 1:29 PM
Hello Reza,

I think the easiest solution would be to set up an input adapter that simply generates a measurement for every frame you would like to process. This input adapter would force the action adapter to generate a frame for each of the measurements generated by your input adapter so there wouldn't be any gaps in publication. Your action adapter would need to include that measurement in its inputs, but your analysis routine can ignore it. The MovingValueInputAdapter is probably the simplest example of an adapter that is capable of doing this, but you may want to design an even simpler adapter that preemptively produces dummy measurements so you can guarantee it will reach the action adapter before the lag time expires.

https://github.com/GridProtectionAlliance/gsf/blob/master/Source/Libraries/Adapters/TestingAdapters/MovingValueInputAdapter.cs

Thanks,
Stephen
Apr 29, 2016 at 3:36 PM
Hello Stephen,

Thanks for the suggestion.

I developed a predictor by which I am able to determine the next frame's measurements. I would like to assign the predicted values in each frame to the output of the next timestamp. This way I can apply your idea of giving this predicted measurement (which is always available before the lag time passes) to my action adapter and ignore it if the actual measurement is received or if the actual one is not received within the lagtime use the predicted one instead.

Normally we use : output.Add(Measurement.Clone(outputMeasurements, predictedValue, frame.Timestamp));
I should change the current code so that depending of FramePerSecond the current calculated value can be assigned to the "next" frame.Timestamp.:

Considering the structure of OpenPDC Manager, Is it possible ?

Thanks,

Reza
Coordinator
Apr 29, 2016 at 7:11 PM
That's a good idea. I hadn't thought of doing that. I believe the code below will do what you're asking for.
public class MyConcentratedAdapter : ActionAdapterBase
{
    private Ticks[] m_subsecondDistribution;

    public override void Initialize()
    {
        // Call base.Initialize() first to initialize the FramesPerSecond property
        base.Initialize();

        // Force preemptive publishing to false
        // IMPORTANT: This strategy will not work with preemptive publishing because every
        //            call to PublishFrame automatically fills the next frame with measurements
        AllowPreemptivePublishing = false;

        // Force the downsampling method to filtered
        DownsamplingMethod = DownsamplingMethod.Filtered;

        // Get the subsecond distribution for the configured frame rate
        m_subsecondDistribution = Ticks.SubsecondDistribution(FramesPerSecond);
    }

    public override void SortMeasurements(IEnumerable<IMeasurement> measurements)
    {
        base.SortMeasurements(measurements.Select(measurement =>
        {
            // Clone the measurement and set the filter to the priority filter
            IMeasurement clone = Measurement.Clone(measurement);
            clone.MeasurementValueFilter = PriorityFilter;
            return clone;
        }));
    }

    protected override void PublishFrame(IFrame frame, int index)
    {
        // Determine the index of the next frame after this one
        int nextFrameIndex = (index + 1) % m_subsecondDistribution.Length;

        // Calculate the timestamp of the next frame
        Ticks baselinedTimestamp = frame.Timestamp.BaselinedTimestamp(BaselineTimeInterval.Second);
        Ticks nextTimestamp = baselinedTimestamp + m_subsecondDistribution[nextFrameIndex];

        if (nextFrameIndex < index)
            nextTimestamp += Ticks.PerSecond;

        // TODO: Process your measurements here

        // Sort the measurements into the next frame
        base.SortMeasurements(frame.Measurements.Values.Select(measurement =>
        {
            // Get the predicted value for the next measurement
            // TODO: Apply your prediction logic here
            double predictedValue = measurement.Value;

            // Create the predicted measurement
            IMeasurement predictedMeasurement = Measurement.Clone(measurement, predictedValue, nextTimestamp);
            predictedMeasurement.MeasurementValueFilter = PredictedFilter;
            return predictedMeasurement;
        }));
    }

    // Define the filter for incoming measurements which are not predicted.
    private double PriorityFilter(IEnumerable<IMeasurement> measurements)
    {
        return Filter(measurements);
    }

    // Define the filter for predicted measurements.
    private double PredictedFilter(IEnumerable<IMeasurement> measurements)
    {
        return Filter(measurements);
    }

    [MethodImpl(MethodImplOptions.AggressiveInlining)]
    private double Filter(IEnumerable<IMeasurement> measurements)
    {
        // Get the most recent measurement which is not predicted
        IMeasurement measurement = measurements.LastOrDefault(m => m.MeasurementValueFilter != PredictedFilter);

        // If all we have is predicted measurements, just get the last measurement
        measurement = measurement ?? measurements.Last();

        return measurement.Value;
    }

    public override bool SupportsTemporalProcessing
    {
        get
        {
            return false;
        }
    }
}
May 2, 2016 at 4:02 PM

Thanks a lot for the code!

I just took a quick look at it and it seems very interesting. For now, I used your other idea of having a dummy input that forces the the action adapter to generate frames for each time stamp.
I'm using "TestingAdapters.FrameBasedRandomValueInputAdapter" and it works very well until I chose small Lag Times for my Action Adapter . I should mention that for my predictor to be more effective and interfere faster, I need to decrease the lag time as much as possible and at least equal to one cycle.
When I chose the lag time smaller than 0.2 seconds, the dummy input itself won't be received by action adapter within the lag time (however, most of time the other measurements are received).
What I can conclude is that, this input adapter is relatively slow. I'm going to study the input adapter and its structure for that matter.

Any advice and suggestions will be greatly appreciated.

Thanks,

Reza

Coordinator
May 2, 2016 at 4:32 PM
Edited May 2, 2016 at 4:33 PM
I don't think that the input adapter is particularly slow, but it relies on asynchronous calls to ThreadPool.RegisterWaitForSingleObject() to keep up with timing. I'm not familiar with the implementation details of ThreadPool.RegisterWaitForSingleObject(), however I do know that timers on the Windows operating system have a minimum precision of 1 millisecond, and they can slip into a mode where they have a minimum precision of 15 milliseconds when the system believes it is under stress. Even so, the only guarantee that timer APIs typically provide is that they will not trigger before the wait time has expired, which means they could potentially trigger tens or hundreds of milliseconds after the wait time has expired. The ScheduledTask class used by FrameBasedRandomValueInputAdapter tries very hard to align its timer with the next publication time to the nearest millisecond, but I'm not sure how reliable it truly is.

Also note that there is an undocumented connection string parameter on the FrameBasedRandomValueInputAdapter called "latency" which defaults to 125 milliseconds. This parameter is designed to inject an artificial latency for testing purposes, which could partially explain why you can't get your lag time down below 200 milliseconds.

Thanks,
Stephen
May 2, 2016 at 5:12 PM
Thanks for the detailed explanation.

First I'm going to try to change DefaultLatency to a smaller value. Since I have not yet worked with input adapters, I should study FrameBasedRandomValueInputAdapter.cs to see if decreasing the DefaultLatency is possible and has any effect on the performance.

Best regards,

Reza