Skip to content

Comparing Encodes Against a Reference

Use compare when you have the source (reference) of one or more encodes, for example to tune a bitrate ladder, to evaluate encoder settings, or to accept an encode against its mezzanine file.

compare takes the reference as the first argument, followed by one or more encodes (the distorted inputs):

surfmeter-media compare -o results/ladder source.mkv \
  ladder/1080p_6000k.mp4 ladder/1080p_4500k.mp4 ladder/720p_3000k.mp4 ladder/540p_1500k.mp4

By default, compare:

  • aligns the frames of each encode with the reference by presentation time (PTS), see Aligning Inputs
  • scales each encode to the resolution of the reference
  • deinterlaces interlaced inputs
  • computes VMAF, PSNR and SSIM of each encode against the reference, frame by frame
  • runs the bitstream analysis and P.1204.3 on all inputs, so that you can see the no-reference scores next to the full-reference ones

Reading the results

When the run is done, compare prints a table with one row per encode on standard error, for example:

VMAF model: vmaf_v1.0.16_3d0h, libvmaf 3.2.1
Input                Frames   Offset   VMAF    min     p5  hmean  PSNR-Y   SSIM dropped repeated
1080p_6000k.mp4         300   +0.000  95.12  88.40  91.02  95.08   43.21 0.9871       0        0
1080p_4500k.mp4         300   +0.000  93.47  85.13  88.76  93.41   42.05 0.9839       0        0
720p_3000k.mp4          300   +0.000  88.90  79.52  82.11  88.79   39.87 0.9752       0        0
540p_1500k.mp4          300   +0.000  80.33  69.07  72.40  80.10   37.12 0.9601       0        0

The columns are:

  • Frames: number of frames of the encode that were matched to a reference frame
  • Offset: time offset between the encode and the reference in seconds (see Aligning Inputs)
  • VMAF, min, p5, hmean: mean, minimum, 5th percentile and harmonic mean of the per-frame VMAF scores
  • PSNR-Y: PSNR of the luma plane, computed from the mean MSE of all frames, as FFmpeg reports it
  • SSIM: mean SSIM over all planes
  • dropped, repeated: reference frames without a matching encode frame, and encode frames that were mapped to the same reference frame as the previous one

The minimum and low percentiles show short drops in quality that the mean hides. A large difference between the mean and the 5th percentile usually points to difficult scenes or encoder rate-control problems.

The same values are available in summary.json, together with the P.1204.3 scores and the bitstream statistics of each input:

jq '.inputs[] | {label,
  vmaf: (.summaries[] | select(.metric == "vmaf") | {mean, p5, min}),
  p1204: [.model_scores[].score]}' results/ladder/summary.json

The comparisons of each encode list the difference of every summary value against the reference, for example how much higher the average QP is than in the reference.

Exploring frame by frame

Open the terminal view to see where the quality drops:

surfmeter-media compare --tui -o results/ladder source.mkv ladder/*.mp4

The VMAF, PSNR and SSIM tracks are shown next to QP, frame sizes and the P.1204.3 score. Frames with low VMAF appear as events. Press v to switch between the comparison view (all inputs on top of each other) and the analysis view of a single input.

To find the worst frames of an encode in the records:

jq -s -c '[.[] | select(.type == "frame_fullref" and .input == "1")]
  | sort_by(.vmaf.score) | .[:10][] | {n, pts, vmaf: .vmaf.score, reference_frame: .reference}' \
  results/ladder/records.ndjson

Input "0" is the reference, and the encodes are numbered from "1" in the order you gave them.

Choosing the metrics

All three full-reference metrics are on by default. Turn off the ones you do not need to save time:

surfmeter-media compare --no-psnr --no-ssim -o results source.mkv encode.mp4

--layers sets the no-reference analysis of all inputs. The default is bitstream,p1204. --pixels adds the pixel metrics:

# Add the no-reference pixel metrics
surfmeter-media compare --pixels -o results source.mkv encode.mp4

# Only full-reference metrics, no other analysis
surfmeter-media compare --layers none -o results source.mkv encode.mp4

Choosing the VMAF model

By default, the toolkit uses the latest VMAF v1 model and picks one by the resolution and frame rate of the reference:

  • up to 1080 lines, up to 30 fps: 3d0h (HD TV at 3 times the picture height)
  • up to 1080 lines, above 30 fps: hfr_3d0h
  • above 1080 lines, up to 30 fps: 1d5h_2160 (UHD TV at 1.5 times the picture height)
  • above 1080 lines, above 30 fps: hfr_1d5h_2160

VMAF is computed at 10-bit precision, also for 8-bit sources. The model used is printed with the results and stored in each frame_fullref record.

To use a different model, pass its name or the path of a JSON model file with --vmaf-model. For example, to compare with older results based on VMAF 0.6.1:

surfmeter-media compare --vmaf-model vmaf_v0.6.1 -o results source.mkv encode.mp4

Scaling

Encodes with a lower resolution are scaled up to the resolution of the reference, as recommended for VMAF.

To compare at a fixed display resolution instead, for example when the reference is UHD but your viewers watch on HD screens:

surfmeter-media compare --scale 1920x1080 -o results source_2160p.mkv ladder/*.mp4

The resolution used is stored in the alignment record of each encode (scaled_to).

Speeding up long comparisons

  • --duration <seconds> compares only the first seconds of the encodes.
  • --subsample <n> computes VMAF on every n-th frame only.
  • --jobs <n> sets how many encodes are compared in parallel, and --threads <n> how many threads each comparison uses. By default, the toolkit divides the CPU cores among the jobs.
surfmeter-media compare --duration 120 --subsample 2 -o results source.mkv ladder/*.mp4