Automating Quality Checks¶

The toolkit runs without user interaction and writes machine-readable output, so it fits into encoding pipelines, watch folders and CI jobs. This page shows a quality gate that rejects encodes below a threshold, a batch job in Docker, and a CSV export for spreadsheets.

Quality gate in a script¶

The following script compares an encode with its source and fails if the mean VMAF is below 90 or the 5^th percentile below 80. It uses summary.json, where we have one entry per input.

#!/usr/bin/env bash
set -euo pipefail

source_file="$1"
encode_file="$2"
out="$(mktemp -d)"

surfmeter-media compare -q --no-psnr --no-ssim -o "$out" "$source_file" "$encode_file"

status=$(jq -r '.run_info.status' "$out/summary.json")
if [ "$status" != "completed" ]; then
  echo "Comparison did not complete: $status" >&2
  jq -r '.run_info.errors[]?.message' "$out/summary.json" >&2
  exit 2
fi

read -r mean p5 < <(jq -r '
  .inputs[] | select(.id == "1") | .summaries[]
  | select(.record_type == "frame_fullref" and .metric == "vmaf" and (.frame_type | not))
  | "\(.mean) \(.p5)"' "$out/summary.json")

echo "VMAF mean=$mean p5=$p5"
awk -v m="$mean" -v p="$p5" 'BEGIN { exit !(m >= 90 && p >= 80) }' || {
  echo "Encode rejected" >&2
  exit 1
}

Without a reference, use analyze and check the P.1204.3 score instead:

surfmeter-media analyze -q --layers bitstream,p1204 -o "$out" "$encode_file"
jq -r '.inputs[0].model_scores[] | select(.model == "p1204.3") | .score' "$out/summary.json"

Batch job in Docker¶

For a CI job or a container that starts fresh for every run, use an ephemeral registration. It registers at the start and returns the license at the end, so it does not use up license slots. Pass the key as a secret environment variable:

docker run --rm --user "$(id -u):$(id -g)" \
  -v /mnt/encodes:/data \
  -e SURFMETER_MEDIA_KEY \
  registry.aveq.info/surfmeter/surfmeter-media:latest \
  analyze --ephemeral -q -o results/$(date +%Y%m%d) incoming/*.mp4

Note that the shell on the host expands incoming/*.mp4 only if the files exist at the same relative path on the host. To expand the pattern inside the container, run it through a shell:

docker run --rm --user "$(id -u):$(id -g)" -v /mnt/encodes:/data -e SURFMETER_MEDIA_KEY \
  --entrypoint sh registry.aveq.info/surfmeter/surfmeter-media:latest \
  -c 'surfmeter-media analyze --ephemeral -q -o results incoming/*.mp4'

Using a permanent registration

On a permanent machine, such as an encoding server with a watch folder, register once instead of using --ephemeral (see Registration).

Exporting to CSV¶

For analysis in a spreadsheet or in tools such as pandas or R, add csv to the output formats:

surfmeter-media analyze --format ndjson,json,csv -o results recordings/*.ts

The toolkit writes one CSV file per record type, for example:

stream_info.csv: one row per stream of each input
frame_bitstream.csv: one row per frame with QP, frame type, frame size and motion statistics
frame_pixels.csv: one row per frame with the pixel metrics
model_score.csv: the P.1204.3 scores per second, per GOP and overall
frame_fullref.csv: one row per frame with VMAF, PSNR and SSIM (compare only)
summary.csv: statistics of every metric per input

Every row has an input column with the input ID. The file names that belong to the IDs are listed in summary.json under inputs.

Processing the NDJSON stream¶

Without -o, the records go to standard output, which you can process directly, for example in Python:

import json
import subprocess

proc = subprocess.Popen(
    ["surfmeter-media", "analyze", "-q", "--layers", "bitstream", "input.ts"],
    stdout=subprocess.PIPE,
    text=True,
)
for line in proc.stdout:
    record = json.loads(line)
    if record["type"] == "frame_bitstream" and record.get("decode_error"):
        print(f"Decoder error in frame {record['n']} at {record['pts']:.3f} s")
proc.wait()

surfmeter-media schema prints the JSON Schema of all records. You can use it to validate the output, or to generate types for your programming language. See Output Format for details.