Skip to main content

Metrics

This page describes the metrics the toolkit computes, grouped by analysis layer. The names are the field names in the records and the column names in the CSV files.

Bitstream statistics​

The bitstream statistics come from videoparser-ng. They are written as one frame_bitstream record per frame, in presentation order, for MPEG-1/2 video, H.264, HEVC, VP9 and AV1.

Please refer to the public repository for more information.

Motion vector statistics

The toolkit uses the "legacy" build of videoparser-ng, which P.1204.3 requires. For H.264, HEVC and VP9, the motion vector statistics are normalized by the temporal distance to the reference frame. They therefore differ from those of a standard videoparser-ng build. All other values are identical.

Pixel metrics​

The pixel metrics come from AVEQ's video analyzer, the same analyzer as in the Video Analyzer SDK. They are no-reference metrics: they are computed from the decoded pictures alone, without a reference. They are written as one frame_pixels record per analyzed frame.

The metrics are grouped into features, which you select with --pixel-features. Features marked with "default" are computed when you do not set the option. Metrics of features that were not computed are left out of the records.

FeatureDefaultMetricsRange and meaning
blockinessyesblockiness0 (none) to 1 (severe). Multi-scale detection of block edges. As a rough guide: 0.2 minimal, 0.4 noticeable, 0.6 significant.
blockiness_ntia, blockiness_vp9Blockiness by the NTIA and VP9 methods, unscaled
blurrinessyesblurriness0 (sharp) to 1 (blurry), scaled to 1080p
blurriness_rawBlur magnitude before the mapping to 0–1
brightnessyesbrightnessAverage gray value, 0 to 1
blackscreenyesblack_screenWhether the frame is completely black
spatialcomplexityyesspatial_informationSpatial information (SI) after ITU-T P.910 (legacy definition)
noiseyesnoiseNoise estimate on the luma plane
temporalchangeyestemporal_informationTemporal information (TI) after ITU-T P.910 (legacy definition)
freezingWhether the frame repeats the previous one
scene_change, scene_scoreWhether a scene change starts at this frame, and its score
sad_normalizedDifference to the previous frame, 0 to 1
jerkinessyesjerkinessJerkiness from repeated frames in a group, 0 to 1
chromanochromaColorfulness, 0 (gray) to 1
contrastnocontrast, contrast_michelsonContrast from the 1st and 99th percentile, and Michelson contrast, 0 to 1
radialprofilenosum_of_high_frequenciesSum of high frequencies of the radial frequency profile
autoenhancementnowhite_level, black_levelIrregular white and black levels, 0 to 1, higher is worse
finedetailnofine_detailFine detail, 0 to 1, higher is worse

--pixel-features all computes all features, which takes about twice the CPU time of the default set. Each frame_pixels record also contains the width and height of the analyzed picture.

P.1204.3 quality score​

ITU-T P.1204.3 is a standardized bitstream-based model that estimates the quality a viewer perceives, as a MOS from 1 (bad) to 5 (excellent). It uses the bitstream statistics (QP, motion, frame sizes and types). See ITU-T P.1204 for background on the model.

The toolkit writes model_score records with model set to p1204.3 in three scopes:

  • per_second: one score per second of media time
  • per_gop: one score per group of pictures
  • overall: one score for the whole stream

Each record contains the start and end of the scored time range, the device (pc or mobile) and the display resolution the model was set up for (--device and --display), and in details the intermediate values of the model.

When the model cannot produce a score, the record has no score, but a reason with a code:

  • unsupported_codec: the codec is not covered by P.1204.3 (for example MPEG-2 or AV1)
  • insufficient_data: the stream is too short
  • model_error: the model failed

Full-reference metrics​

compare computes the full-reference metrics with FFmpeg's libvmaf, psnr and ssim filters. The values are identical to what you get from FFmpeg with the same settings. They are written as one frame_fullref record per frame of each encode.

VMAF (vmaf):

  • score: VMAF score of the frame, usually 0 to 100
  • model, model_version: the model used, for example vmaf_v1.0.16_3d0h
  • features: the elementary features of the model under stable names (for example adm3, motion3, cambi)

PSNR (psnr):

  • y, u, v: PSNR of the luma and the two chroma planes in dB
  • avg: PSNR over all planes in dB
  • mse_y, mse_u, mse_v, mse_avg: the mean squared errors

When a plane is identical to the reference, its MSE is 0 and the PSNR is infinite. The PSNR value is then left out of the record.

SSIM (ssim):

  • y, u, v: SSIM of each plane
  • all: SSIM over all planes, and all_db the same in dB

Each frame_fullref record also has a reference field with the reference frame it was compared with, and a mapping field that says how the frame was matched:

  • exact: the timestamps match after applying the offset
  • nearest: the nearest reference frame in time was used (different frame rates or timestamp jitter)
  • repeated: the same reference frame as for the previous frame

Audio loudness​

The audio layer measures every audio stream according to ITU-R BS.1770 and EBU R 128, with the loudness range of EBU Tech 3342. The measurement has been checked against the EBU Tech 3341 and 3342 test signals.

Units

Loudness is given in LUFS (the same as LKFS), loudness differences in LU, the true peak in dBTP, and the sample peak and RMS level in dBFS.

The measured values are as follows:

  • momentary: loudness over the last 400 ms
  • short_term: loudness over the last 3 seconds
  • integrated: gated loudness from the start of the stream, as defined in BS.1770. For the whole stream, this is the value that is checked against a target.
  • lra: loudness range from the start of the stream, in LU, after EBU Tech 3342
  • true_peak: highest peak of the signal, including peaks between the samples (measured with oversampling, as in BS.1770)
  • sample_peak: highest sample value
  • rms: RMS level of one channel over one second

The channels are weighted as in BS.1770, and LFE is excluded from the loudness score. The surround channels get +1.5 dB. The audio is measured as decoded, without dynamic range control, without downmix and without resampling. A mono stream counts once, as BS.1770 says; played on two loudspeakers, it sounds 3 dB louder.

The toolkit also finds these events:

  • silence: a channel stays below -60 dBFS for 2 seconds or longer
  • clipping: 3 or more samples in a row at full scale. Clips less than 1 second apart are one event.
  • true_peak_over: the true peak is above the maximum true peak of a target. The message names the targets.
  • dialnorm_change: the dialnorm of an AC-3 or E-AC-3 stream changes

Loudness targets​

At the end of each stream, the toolkit checks the integrated loudness and the true peak against each target (see Basic Usage for the list). Both values are rounded to 0.1 before the check. The results per target are:

  • deviation: the integrated loudness minus the target level, in LU
  • loudness_ok: the deviation is within the tolerance
  • true_peak_ok: the true peak is at or below the maximum true peak
  • dialnorm_ok: the dialnorm matches the loudness (only for targets that check dialnorm)
  • ok: all checks passed

Dialnorm​

AC-3 and E-AC-3 streams can carry their dialog level as dialnorm (-1 to -31 dB). Our toolkit reads it from every frame, together with the bitstream mode (bsmod, for example complete main or commentary), the channel mode (acmod, for example 3/2), the LFE flag, the compression value (compr), bit rate and sample rate.

For the summary, the toolkit takes the dialnorm that was used for the longest time. dialnorm_difference is the integrated loudness minus this dialnorm. ATSC A/85 expects a value close to 0. For atsc-a85, dialnorm_ok is true when the difference is within ±2 dB. See Checking Loudness Compliance for examples.

Limitations
  • For E-AC-3 streams with several programs (independent substreams), only program 0 is measured, with a warning.
  • Lost and damaged audio packets count as silence and give a warning with the number of decoding errors.
  • The targets are checked on the integrated loudness of the whole stream. There is no event yet for parts of the stream that are too loud or too quiet.

Summary statistics​

For every metric, the toolkit writes summary records with descriptive statistics over the whole stream:

  • count: number of values
  • mean, std: mean and population standard deviation
  • min, max
  • p1, p5, p50, p95: 1st, 5th, 50th (median) and 95th percentile
  • harmonic_mean: for VMAF only

record_type and metric name the field the statistics are computed from, for example frame_bitstream and qp_avg. For bitstream metrics, there are additional summaries per frame type, with frame_type set to I, P or B. For full-reference metrics, reference gives the ID of the reference input.

In compare, comparison records give the difference of each summary between an encode and the reference (delta = encode minus reference).