Skip to content

AVPlayer Quality Test

The AVPlayerQualityTest plays a video stream with AVFoundation's AVPlayer and scores the playback with the ITU-T Rec. P.1203 model. It is the iOS counterpart of the Android ExoPlayer Quality Test, and reports its results under the subject avplayer.

The test plays for a configurable duration, then stops playback, builds a P.1203 input report from what the player did, and computes the quality scores on the device. Have a look at AVPlayerTestView.swift in the demo app for a complete example.

Starting the Test

In the example below the test plays an HLS stream for 30 seconds. The maximum test duration is a timeout for the whole test in case of network problems, and the analysis duration is a timeout for computing the statistics once playback is over.

import SurfmeterQualitySDK

let test = try AVPlayerQualityTestBuilder(manifest: URL(string: "https://example.com/stream.m3u8")!)
    .setMaxPlaybackDuration(durationMs: 30_000)
    .setMaxTestDuration(durationMs: 45_000)
    .setMaxAnalysisDuration(durationMs: 10_000)
    .setProgressInterval(intervalMs: 1000)
    .setResultListener(listener: self)
    .build()

try test.start()

Both build() and start() throw, so wrap them in a do/catch and handle the error. build() rejects invalid combinations, for example a maximum test duration shorter than the maximum playback duration. start() throws when the test has already been started.

To stop a running test early, call stop(). If playback was already running, the test finishes normally and you get a result for what was played so far. If it had not started yet, you get an error instead.

try test.stop()

Builder Options

AVPlayerQualityTestBuilder is created with the manifest URL and supports the following methods, all of which return the builder so they can be chained:

  • setMaxPlaybackDuration(durationMs:) – maximum playback duration in milliseconds, 30000 by default
  • setMaxTestDuration(durationMs:) – maximum test duration in milliseconds, 45000 by default. Must be at least the playback duration
  • setMaxAnalysisDuration(durationMs:) – timeout for computing the statistics, in milliseconds, 10000 by default
  • setProgressInterval(intervalMs:) – how often onPlaybackProgress is called, in milliseconds, 1000 by default
  • setResultListener(listener:) – the QualityTestResultListener that receives results, errors and state changes
  • setCalculationSettings(calculationSettings:) – P.1203 calculation settings, see below
  • attachToView(view:) – attach the player to a UIView so the video is visible during the test
  • setReportedDisplaySize(width:height:) – the display size reported to P.1203, in device pixels
  • setMetadata(_:) – arbitrary key-value pairs that are stored alongside the measurement

There is also a setExternalPlayer(player:) method, for handing the test an AVPlayer you created yourself. It is not functional yet, see below. It cannot be combined with attachToView; build() throws if both are set.

Tip

You can get test URLs from the hls.js demo. Note that AVPlayer supports HLS and progressive MP4, but not DASH, so the DASH streams that work with the Android SDK will not play here.

Reported Display Size

P.1203 scores partly on how far the video has to be scaled to fit the display, so the model needs to know which display the measurement stands for.

This is deliberately independent of attachToView. A test running in a small preview, or with no view at all, is usually meant to represent what a full-screen viewing would look like, so the SDK does not measure the player's actual size. Left unset, the report assumes a 1920x1080 display.

try builder.setReportedDisplaySize(width: 1920, height: 1080)

Calculation Settings

You can select the P.1203 models and switch on the amendments through setCalculationSettings:

try builder.setCalculationSettings(calculationSettings: [
    "pv": "P1203PvExtended",
    "pq": "P1203PqExtended",
    "amendment1Audiovisual": true
])

Valid values for pv are P1203PvExtended, P1203PvHveiExtended, P1203PvRetrained and P12043BitstreamMode0. Valid values for pq are P1203PqExtended and P1203PqM. The keys amendment1Audiovisual, amendment1Stalling, amendment1App2 and longMode take a boolean. Anything else makes the call throw.

Using an External Player

Not supported yet

setExternalPlayer(player:) exists on the builder, as the counterpart of the Android SDK's option of the same name, but the iOS test does not yet observe a player it did not create. A test built with an external player never starts collecting data and never delivers a result. Please do not use it for now, and let us know if you need it.

Receiving Test Results

To receive the outcomes of the test, your class must conform to the QualityTestResultListener protocol. This protocol includes methods for handling successful results, errors, state changes, and playback progress.

import SurfmeterQualitySDK

class TestRunner: QualityTestResultListener {
    func onTestResult(_ result: [String: AnyHashable]) {
        // the finished measurement report
    }

    func onTestError(_ error: String) {
        // the test could not be completed
    }

    func onTestStateChanged(_ state: QualityTestState) {
        // the test moved to a new state
    }

    func onPlaybackProgress(_ progress: Int) {
        // playback progress in percent
    }
}

Exactly one of onTestResult or onTestError is called per test run. The SDK holds a strong reference to the listener until that final callback has been delivered, then releases it, so a listener that also owns the test does not leak.

Test Success

When the test completes, onTestResult is called with the full measurement report:

func onTestResult(_ result: [String: AnyHashable]) {
    guard let stats = result["statistic_values"] as? [String: AnyHashable] else { return }
    print("Overall MOS: \(stats["p1203OverallMos"] ?? "n/a")")
}

The report has the same shape as the one the Android SDK produces, so both platforms can be read the same way. The quality scores are under statistic_values, and include p1203OverallMos, p1203AverageVideoQuality, p1203AverageAudioQuality and p1203StallingQuality, alongside playback statistics such as initialLoadingDelay, averageVideoBitrate and averageStallingTime. The full P.1203 computation that produced them is under statistics_input, and the raw input report and per-segment performance data are under client_reports.

The values correspond to the measurement data, so please refer to that page for a description of each one.

To serialize the report, for example to store it or to display it:

if let data = try? JSONSerialization.data(withJSONObject: result, options: .prettyPrinted),
   let json = String(data: data, encoding: .utf8) {
    print(json)
}

Test Errors

If the test cannot be completed, onTestError is called with a message instead:

func onTestError(_ error: String) {
    print("Test error: \(error)")
}

This happens when the test is stopped before playback started, or when computing the statistics times out.

Playback Progress

While the video is playing, onPlaybackProgress reports the position as a percentage of the expected playback duration, at the interval you set through setProgressInterval. It is called once more with 100 when playback finishes.

func onPlaybackProgress(_ progress: Int) {
    print("Progress: \(progress)%")
}

Test State Changes

You can follow the lifecycle of the test through onTestStateChanged:

func onTestStateChanged(_ state: QualityTestState) {
    print("Test state: \(state)")
}

The video test passes through the following states, in this order:

  • STATE_READY — the test has been built but not started
  • STATE_STARTING — start() was called, setup is in progress
  • STATE_PLAYING — playback is running
  • STATE_ANALYZING — playback is over, statistics are being computed
  • STATE_END — the test is finished, the result callback follows

If the test is aborted, it moves to STATE_ABORTED instead.

Showing the Video

By default the test plays without any visible output, which is enough for a measurement. To show the video, hand the test a view to attach the player layer to:

builder.attachToView(view: myPlayerView)

The player layer is sized to the view's bounds when the test is built, so make sure the view has its final size by then.

Background Execution

The test needs the app to be in the foreground. AVPlayer stops delivering playback events when the app is suspended, which leaves gaps in the P.1203 input and makes the resulting score meaningless. Running quality tests from a background task is not supported.