Analysis¶

The Video Analyzer SDK enables analysis of video frames as they are displayed to users. This guide covers how to set up frame analysis, process video data, and handle the results. You can get frames from a camera, a video file, or any other source that provides video frames, as long as you can provide them as 8-bit YUV 4:2:0 frames (ImageFormat.YUV_420_888).

Getting Started¶

Once your application is registered, you can start analyzing video frames in real-time.

Step 1: Initialize the VideoAnalyzerManager¶

Create a VideoAnalyzerManager instance to handle frame analysis:

import com.aveq.videoanalyzer.VideoAnalyzerManager;

// Initialize with the frame rate of the video (null for 30 frames per second)
VideoAnalyzerManager manager = new VideoAnalyzerManager(context, 30.0);

The frame rate is used for the timestamps of frames that are passed without one, and for the temporal metrics. Frames are analyzed at the size they are passed in. To analyze camera frames at a smaller size, set the resolution of the CameraX ImageAnalysis use case.

Step 2: Set Up Frame Metrics Callback¶

Register a callback to receive metrics as each frame is processed:

manager.registerFrameMetricsCallback(new FrameMetricsCallback() {
    @Override
    public void onFrameMetrics(SingleFrameMetrics metrics) {
        // This callback is called for each analyzed frame
        // Update your UI or process metrics on the main thread
        runOnUiThread(() -> {
            // do whatever you want with the metrics
        });
    }
});

Step 3: Process Video Frames¶

The SDK analyzes 8-bit YUV 4:2:0 frames, passed as the Y, U and V planes of an android.media.Image in ImageFormat.YUV_420_888. This is the format of CameraX ImageAnalysis (ImageProxy.getPlanes()) and of most MediaCodec decoders. The SDK copies each frame, so you can close the image right after the call.

// In ImageAnalysis.Analyzer.analyze(ImageProxy image)
ImageProxy.PlaneProxy[] planes = image.getPlanes();
manager.analyzeFrame(image.getWidth(), image.getHeight(),
        planes[0].getBuffer(), planes[0].getRowStride(),
        planes[1].getBuffer(), planes[1].getRowStride(),
        planes[2].getBuffer(), planes[2].getRowStride(),
        planes[1].getPixelStride(),  // 1: separate planes (I420), 2: interleaved (NV12, NV21)
        true,                        // full range (camera); false for limited range (most decoded video)
        frameIndex);
image.close();

The parameters:

• The plane buffers must be direct ByteBuffers. Each plane starts at the position of its buffer.
• The row strides are the distances between the starts of two rows, in bytes. The U and V planes have half the width and height of the Y plane (rounded up).
• The chroma pixel stride is the distance between two samples of a row of the U and V planes (1 or 2 on Android). The SDK splits interleaved chroma into separate planes.
• The full range flag is true for full range (0-255, as camera frames usually are) and false for limited range (16-235, as decoded video usually is).
• An overload takes the presentation timestamp in seconds as last parameter. Without it, the timestamp is the frame index divided by the frame rate.

Invalid buffers (not direct, or too small for the given size and strides) throw an IllegalArgumentException.

For an android.media.Image (for example from ImageReader or MediaCodec), you can pass the image directly:

manager.analyzeImage(image, false, frameIndex);  // false: limited range
image.close();

The metrics of each frame are passed to the callback registered in step 2. You can also read them with getFrameMetrics().

Next, learn about the specific metrics that the SDK provides.