Statistic Values¶
Overview¶
This page lists the statistic values that can be found in the statistic_values field of our measurement data, and explains how they are calculated. The statistic_values field is a dictionary of statistic values, where the keys are the names of the statistic values, and the values are the actual values. The names are the headings below, only that they use camelCase.
Note that statistic values are, by definition, calculated fields that we extract from other information gathered while a measurement is made. Sometimes these are very simple mappings (e.g., multiplying a milliseconds value by 1,000 to get seconds), sometimes they are more complex (e.g., calculating the average of a set of values). In some cases, calculations can get very complex (e.g., when calculating the MOS score for a video stream).
We also provide Client Reports as a way to get more detailed background information. We do not perform any calculations on these reports, but instead provide the raw data that we collected during the measurement. Therefore, these are not listed here, and you can find a reference in the Client Reports page.
Finally, be aware that for some measurement types we do not calculate explicit statistic values, as the results are implicitly all contained within the Client Report. Here, the Client Reports are “forwarded” as statistic values, and you can find their reference in the Client Reports page.
Video Measurement Statistics¶
This section lists the statistc values available for video measurements.
Video statistics are essentially KPI and KQI values. They are based on a measurement of a video stream with all the important events and metadata captured by Surfmeter. The specific implementation of the event tracking and metadata collection depend on the service measured; the description of the KPIs/KQIs in the following however are generic and service-independent.
Quality-Related KPIs/KQIs¶
Quality-related KPIs/KQIs are based on the application of a QoE model on the measured video session.
P.1203 Overall MOS¶
Unit: MOS (Valid Range: 1–5)
The overall MOS, taking into account audiovisual quality, quality switches, and initial loading/stalling.
This corresponds to the output O.46 of the P.1203.3 integration module.
Unless otherwise specified, the MOS is calculated using Mode 0 of the ITU-T Rec. P.1203.1 model. By default, P.1203.3 Amendment 1 is enabled. This introduces an adjustment of the audiovisual quality of ITU-T P.1203.3 for the case of very low audio quality and long stalling events.
For more information, see the explanation of the video QoE model.
Note
Depending on the customer configuration, the MOS may be calculated using a different model, or using a different version of the model.
P.1203 Overall Audiovisual Quality¶
Unit: MOS (Valid Range: 1–5)
The overall audiovisual quality, not including any stalling.
This corresponds to the output O.35 of the P.1203.3 integration module.
P.1203 Average Audio Quality¶
Unit: MOS (Valid Range: 1–5)
Average per-second audio MOS.
The average is calculated over the O.21 values produced by the P.1203.2 audio quality module.
In case of a stream not using any audio, this value will be fixed to 5.
P.1203 Average Video Quality¶
Unit: MOS (Valid Range: 1–5)
Average per-second video MOS.
The average is calculated over the O.22 values produced by the P.1203.1 video quality module.
P.1203 Stalling Quality¶
Unit: MOS (Valid Range: 1–5)
The overall quality impact of the stalling itself. A low number indicates a bad experience; a high number indicates a good experience.
This corresponds to the O.23 indicator produced by the P.1203.3 integration module.
P.1203 Max Theoretical MOS¶
Unit: MOS (Valid Range: 1–5)
The MOS that could have been achieved if there was no initial loading delay, no stalling, and continuous playback of the highest played out video and audio representation. "Highest" in this case is the representation with the highest bitrate and resolution observed during the playback.
For example, if a video was played with the following chunks, where a chunk is a contiguous region of playback with the same resolution and (nominal) bitrate:
- 0–20 s: 1080p, 3500 kBit/s
- 20–40 s: 720p, 1500 kBit/s
- 40–60 s: 1080p, 3500 kBit/s
The maximum theoretical MOS is the result of a MOS calculation assuming that the video was played out with a single chunk from 0–60s, with 1080p, 3500 kBit/s.
The maximum theoretical MOS helps the user understand the quality in relative terms. For instance, a video that is only available in 720p resolution will never yield a high MOS, even if it is always played at 720p, due to the comparably low resolution.
Note
In the case of bad network conditions, a video playback may never reach the highest possible resolution that is available on the video server or indicated in the video manifest. In this case, the maximum theoretical MOS is only calculated with respect to the highest observed representation during playout.
P.1203 Max MOS Ratio¶
Unit: %
The ratio between actual MOS and the maximum theoretical MOS.
Bitrate-/Framerate-Related KPIs¶
Average Audio Bitrate¶
Unit: Kbit/s
Mean of the audio bitrate, weighted by playback duration of the respective representations using that bitrate.
For the calculation procedure, see the section on average video bitrate.
Average Video Bitrate¶
Unit: Kbit/s
Mean of the video bitrate, weighted by playback duration of the respective representations using that bitrate.
The calculation is realized as follows, where the bitrates and durations are lists of measured bitrates and the durations for which they have been measured:
bitrates = [...]
durations = [...]
sum = 0
for bitrate, duration in (bitrates, durations):
sum = sum + bitrate * duration
sum_of_weights = 0
for weight in durations:
sum_of_weights = sum_of_weights + weight
average_bitrate = sum / sum_of_weights
Average Total Bitrate¶
Unit: Kbit/s
Mean of the video and audio bitrate, weighted by playback duration of the respective representations.
For the calculation procedure, see the section on average video bitrate.
Average Video Framerate¶
Unit: fps
Mean of the video framerate, weighted by playback duration of the respective representations using that framerate.
Resolution-Related KPIs¶
Initial Resolution¶
Unit: Pixels
The height (in pixels) of the video when playback was first started.
The initial resolution is an indicator of the player's ability to load a video with high quality. Users may prefer videos to start with high quality.
Longest Played Chunk¶
Unit: Pixels
The representation that was played out the longest during the measurement. In other words, this is the height of the chunk with the longest duration, where a chunk is a contiguous region of playback with the same resolution and (nominal) bitrate.
Largest Played Video Size¶
Unit: Pixels
The highest resolution (in terms of height in pixels) that was played out during the measurement.
Performance-Related KPIs¶
Note that most of the following KPIs are directly mapped from the PerformanceClientReport.
Average Buffer Length¶
Unit: Seconds
The average length of the video buffer over the course of the entire measurement.
Max. Buffer Length¶
Unit: Seconds
The highest length of the video buffer over the course of the entire measurement.
Min. Buffer Length¶
Unit: Seconds
The lowest length of the video buffer over the course of the entire measurement.
Average Latency¶
Unit: Seconds
The average latency of the video player over the course of the entire measurement.
Note that this is currently only available for low-latency CMAF streams played with the DASH.js player. Other players may support reporting latency. If this is the case, the value will be available in Surfmeter too.
Latency is measured as "distribution latency", that is the latency between the video segment being available at the CDN vs the time it was received at the client. This is shown in the below figure.
Contribution latency cannot be measured from the client side.
Max. Latency¶
Unit: Seconds
The maximum latency of the video player over the course of the entire measurement.
Min. Latency¶
Unit: Seconds
The minimum latency of the video player over the course of the entire measurement.
Dropped Frames¶
Unit: N/A
The number of frames dropped during playback.
This is not an issue with the streaming performance itself, but the performance of the device decoding the video. A high number of dropped frames indicates issues with the playout that suggest that a higher-powered device is needed, or that the screen and/or video resolution should be reduced.
Video Response Time¶
Unit: Seconds
The time between the first request for a video segment and the first response for that segment.
When only using Surfmeter Lab (without Automator), this statistic may not always be available. If it is, it uses the difference between the requestStart and responseStart timing, per the W3C Resource Timing 2 specification. Due to CORS restrictions, this field is not always present.
When using Surfmeter Automator, this statistic is always available when also enabling --logNetworkRequests. It is calculated from the sendEnd to the receiveHeadersStart field of the Chrome DevTools Protocol Timing Specification. If that field is not available, the receiveHeadersEnd field is used instead.
Quality-Switch-Related KPIs¶
Number of Quality Switches¶
Unit: N/A
Number of total quality switches, that is switches between chunks of different resolutions and bitrates.
A chunk is a contiguous region of playback with the same resolution and (nominal) bitrate.
Quality Switch Up Count¶
Unit: N/A
Number of quality switches to a higher resolution.
Quality Switches Down Count¶
Unit: N/A
Number of quality switches to a lower resolution.
Timing-Related KPIs¶
Total Play Time¶
Unit: Seconds
The total time spent playing video in media time. This excludes any stalling event. Thus, it is equal to the sum of all chunk lengths.
Video Completion Ratio¶
Unit: N/A
The fraction of last point in the video that was played (in media time) vs. total duration of the video.
This is not available for live video measurements.
Stalling-Related KPIs¶
Initial Loading Delay¶
Unit: Seconds
The initial startup time of the stream, measured from a specific point in time (T0) until the first successful playback of the video.
The recording of T0 depends on the specific implementation of the video website. Generally, unless otherwise specified, T0 is equal to the wall clock time of the moment the video element is created the first time and is configured ready to load a playout. In cases where a video starts directly together with the service's content page, T0 is equal to the wall clock time of the page's domContentLoadedEventEnd event (see also Section 3.2).
Some video services may use additional video loading strategies that involve showing a spinning indicator to the user ("buffering") before a video player object is instantiated. In such cases, T0 is set to the time at which such an indicator appears, as this will better reflect the subjective experience of the end user.
In cases where the video element is pre-loaded but hidden until the user interacts with the web page, T0 references the wall clock time of either the next shown spinning loader or – if such an indicator is not displayed – the moment the video becomes visible before the actual playout.
Number of stalling events¶
Unit: N/A
The total number of stalling events. This excludes initial loading. Thus, a stream with only initial loading has 0 stalling events.
Average Stalling Time¶
Unit: Seconds
The average time spent stalling for each event. This excludes initial loading.
Total Stalling Time¶
Unit: Seconds
The total time spent stalling. This excludes initial loading.
Stalling Ratio¶
Unit: N/A
The fraction between total stalling time and total play time.
Other KPIs¶
Content Server IP Address¶
Unit: N/A
This is the IP address of the content server that was used to stream the video, based on the frequency of the IP address in the received responses matching video/audio MIME types or URL signatures.
For this feature to be available, the --logNetworkRequests option must be enabled in Surfmeter Automator.
Content Server Hostname¶
Unit: N/A
This is the hostname of the content server that was used to stream the video, based on the frequency of the hostname in the received responses matching video/audio MIME types or URL signatures.
For this feature to be available, the --logNetworkRequests option must be enabled in Surfmeter Automator.
Note
The hostname is not necessarily the same as the domain name of the video service. For example, a video service may use a CDN to deliver video content, in which case the hostname will be the CDN hostname, not the video service's domain name.
Web Measurement Statistics¶
This section lists the KPIs that are available for web measurements.
Web measurement KPIs are realized with either the W3C APIs, as implemented in the respective browser versions used, or Google Lighthouse. Note that these are two different types of measurements, and that the KPIs are not directly comparable, unless they are listed in both sections.
W3C-based KPIs¶
The following APIs are used:
Underlying to the following KPIs is the following processing model (Source: W3C).
Specifically, for basic web load performance, we are interested in the timing from the start of the domain lookup until the end of the load event.
DNS Load Time¶
Unit: Seconds
The time from the start of domain lookup (domainLookupStart) until end of domain lookup (domainLookupEnd).
These fields are exposed in the PerformanceResourceTiming interface.
Specifically, the DNS timing of the main page that is requested will be calculated. Timings for other resources on the measured website are not included.
Note
DNS lookups may be cached by the operating system or the customer router, so that the DNS lookup time could become 0 (or close to 0). This is not an issue with the website, but with the operating system or router. Ensure that the DNS cache is cleared before running the measurement (which Surfmeter Automator attempts to do, for example). For avoiding router caching, use a different DNS server than the one provided by your ISP, or use a new set of host names to test. On the other hand, if you want to avoid DNS impacting your web tests at all, warm up the cache by performing DNS requests against the host names you want to test before running the test.
Document Ready Load Time¶
Unit: Seconds
The time from connect start (connectStart) until end of the DOM content loaded event (domContentLoadedEventEnd).
These fields are exposed in the PerformanceNavigationTiming interface.
Page Load Time¶
Unit: Seconds
The time from connect start (connectStart) until end of the load event (loadEventEnd).
These fields are exposed in the PerformanceNavigationTiming interface.
Server Response Time¶
Unit: Seconds
The time from the start of the load (startTime) until the response begins (responseStart).
These fields are exposed in the PerformanceNavigationTiming interface.
Also called Time to First Byte (TTFB), this metric measures the time it takes for the server to respond to a request. The lower the value, the better.
First Contentful Paint¶
Unit: seconds
The First Contentful Paint (FCP) marks the first time when the browser renders any text, image (including background images), non-white canvas or SVG.
This is exposed in the PerformancePaintTiming interface.
Largest Contentful Paint¶
Unit: seconds
The Largest Contentful Paint (LCP) marks the point in the page load timeline when the page's main content has likely loaded. This must be contrasted with First Contentful Paint which only captures the beginning of the loading experience.
This is exposed in the PerformancePaintTiming interface.
Google Lighthouse-based KPIs¶
These are directly exposed by Google Lighthouse in the audits section of the JSON report produced by the lighthouse command line tool.
For a detailed description of some of the below metrics, see the Google Lighthouse documentation.
The overall scoring system from Lighthouse is explained here, with an interactive scoring calculator being available.
Server Response Time¶
Unit: seconds
The time from the start of the load (startTime) until the response begins (responseStart).
These fields are exposed in the PerformanceNavigationTiming interface.
Also called Time to First Byte (TTFB), this metric measures the time it takes for the server to respond to a request. The lower the value, the better.
Total Byte Weight¶
Unit: bytes
Summarizes the total byte size of the page and all its network requests.
DOM Size¶
Unit: elements
The size of the DOM in elements. Higher values may indicate that the page is heavy on DOM manipulation.
Interactive¶
Unit: seconds
Also called Time to Interactive (TTI), this metric measures the time it takes for the page to become fully interactive. The lower the value, the better.
First Contentful Paint¶
Unit: seconds
The First Contentful Paint (FCP) marks the first time when the browser renders any text, image (including background images), non-white canvas or SVG.
This is exposed in the PerformancePaintTiming interface.
Largest Contentful Paint¶
Unit: seconds
The Largest Contentful Paint (LCP) marks the point in the page load timeline when the page's main content has likely loaded. This must be contrasted with First Contentful Paint which only captures the beginning of the loading experience.
This is exposed in the PerformancePaintTiming interface.
Speed Index¶
Unit: seconds
The Speed Index (SI) is a metric that summarizes the visual progress of a page load. It is calculated based on the speed of visual progress and the visual completeness of the page.
Cumulative Layout Shift¶
Unit: unitless
The Cumulative Layout Shift (CLS) metric measures the instability of content on a web page during the loading process. It quantifies how much the layout of a page shifts around.
Total Blocking Time¶
Unit: seconds
The Total Blocking Time (TBT) metric measures the total amount of time between First Contentful Paint and Time to Interactive (TTI) where the main thread was blocked for long enough to prevent input responsiveness.
First Meaningful Paint¶
Unit: seconds
The First Meaningful Paint (FMP) measures when the primary content of a page is visible to the user.
Warning
Per Google, “First Meaningful Paint (FMP) is deprecated in Lighthouse 6.0. In practice FMP has been overly sensitive to small differences in the page load, leading to inconsistent (bimodal) results. Additionally, the metric's definition relies on browser-specific implementation details, which means it cannot be standardized nor implemented in all web browsers. Moving forward, consider using Largest Contentful Paint instead.”
Speedtest Measurement Statistics¶
This section lists the KPIs that are available for speedtest measurements.
The interpretation of speedtest measurement KPIs depend on the type of speedtest being run. For example, speedtests run via the browser will exhibit limitations for very high levels of bandwidths, whereas speedtests that were run natively do not have such limitations.
Speedtests do not have KQIs due to the complex relationship between observed bandwidth and actual customer experience.
Note
Speedtest measurements have no statistic_values property, but instead provide the raw data that we collected during the measurement as direct attributes of the measurement. You can find a reference in the Measurement Data page.
Download¶
Unit: Mbit/s
The download speed as indicated by the speedtest provider.
Details on the methodology can be obtained from the respective provider, for instance:
Upload¶
Unit: Mbit/s
The upload speed as indicated by the speedtest provider.
Latency¶
Unit: ms
The latency as indicated by the speedtest provider.
The latency is to be interpreted with respect to the remote speedtest server chosen. The remote server may be available as an extra attribute of the measurement depending on whether the speedtest provider shows this information.
Network Measurement Statistics¶
Network measurements do not calculate any statistic values, but instead provide the raw data that we collected during the measurement. You can find a reference in the Client Reports page.
Conferencing Measurement Statistics¶
This section lists the KPIs that are available for conferencing measurements. Note that some KPIs overlap with the video measurement statistics, so we do not list them here again. The following KPIs are therefore specific to conferencing measurements.
Inbound Audio¶
These statistics relate to the inbound RTP audio stream.
Packets Received¶
Unit: N/A
The number of audio packets received.
Packets Lost¶
Unit: N/A
The number of audio packets lost during transmission.
Minimum Jitter¶
Unit: Seconds
The minimum observed jitter in the audio stream.
Maximum Jitter¶
Unit: Seconds
The maximum observed jitter in the audio stream.
Average Jitter¶
Unit: Seconds
The average jitter calculated over the duration of the audio stream.
Total Interruption Duration¶
Unit: Seconds
The total duration of interruptions in the audio stream.
Inbound Video¶
Packets Received¶
Unit: N/A
The number of video packets received.
Packets Lost¶
Unit: N/A
The number of video packets lost during transmission.
Minimum Jitter¶
Unit: Seconds
The minimum observed jitter in the video stream.
Maximum Jitter¶
Unit: Seconds
The maximum observed jitter in the video stream.
Average Jitter¶
Unit: Seconds
The average jitter calculated over the duration of the video stream.
Total Pauses Duration¶
Unit: Seconds
The total duration of pauses in the video stream, where a pause is defined as exceeding 5 seconds without a rendered frame.
See also the WebRTC specification for pauseCount and totalPausesDuration.
Total Freezes Duration¶
Unit: Seconds
The total duration of freezes in the video stream. It is a freeze if frame duration, which is time interval between two consecutively rendered frames, is equal or exceeds Max(3 * avg_frame_duration_ms, avg_frame_duration_ms + 150), where avg_frame_duration_ms is linear average of durations of last 30 rendered frames.
See also the WebRTC specification for freezeCount and totalFreezesDuration.
Frames Dropped¶
Unit: N/A
The number of frames dropped in the video stream. The total number of frames dropped prior to decode or dropped because the frame missed its display deadline for this receiver's track.
See also the WebRTC specification for framesDropped and framesDecoded.
Outbound Audio¶
Retransmitted Packets Sent¶
Unit: N/A
The number of audio packets sent as retransmissions.
Total Packet Send Delay¶
Unit: Seconds
The total delay for all packets sent in the audio stream.
NACK Count¶
Unit: N/A
The number of Negative Acknowledgment (NACK) messages sent for the audio stream.
Outbound Video¶
Retransmitted Packets Sent¶
Unit: N/A
The number of video packets sent as retransmissions.
Total Packet Send Delay¶
Unit: Seconds
The total delay for all packets sent in the video stream.
NACK Count¶
Unit: N/A
The number of Negative Acknowledgment (NACK) messages sent for the video stream.
FIR Count¶
Unit: N/A
The total count of Full Intra Requests (FIR) sent for the video stream.
PLI Count¶
Unit: N/A
The total count of Picture Loss Indications (PLI) sent for the video stream.
Quality Limitation Durations¶
These statistics relate to the quality limitations that may affect the video stream.
Bandwidth¶
Unit: Seconds
The total duration where bandwidth limitations affected the video quality.
CPU¶
Unit: Seconds
The total duration where CPU limitations affected the video quality.
None¶
Unit: Seconds
The total duration where there were no quality limitations.
Other¶
Unit: Seconds
The total duration where other factors limited video quality.
Local Candidate¶
Candidate Type¶
Unit: N/A
The type of the local candidate. The following types are possible: host, srflx, prflx, relay.
IP¶
Unit: N/A
The IP address of the local candidate.
Network Adapter Type¶
Unit: N/A
The type of network adapter used (e.g., "ethernet").
Network Type¶
Unit: N/A
The type of network (e.g., "ethernet").
Port¶
Unit: N/A
The port number used by the local candidate.
Protocol¶
Unit: N/A
The protocol used (e.g., "udp", "tcp", "tls").
VPN¶
Unit: N/A
Indicates whether a VPN is being used (true/false).
Remote Candidate¶
Candidate Type¶
Unit: N/A
The type of the remote candidate. The following types are possible: host, srflx, prflx, relay.
IP¶
Unit: N/A
The IP address of the remote candidate.
Port¶
Unit: N/A
The port number used by the remote candidate.
Protocol¶
Unit: N/A
The protocol used (e.g., "udp", "tcp", "tls").
AV Sync¶
We calculate the AV sync as the difference between the audio and video timestamps. Specifically, we sample the jitter buffer delay for the audio and video buffers, respectively, every second, and calculate the difference between the two.
ITU-R BT.1359-1 gives us some indication as to which delays are noticeable or not. The lip sync detection thresholds are +45 ms to -125 ms of audio leading video, and these experiments were done with talking head (news readers). The acceptability threshold is +90 and -185 ms, respectively.
Other studies like this one show that the latency sensitivity depends on the content type, so it may be different for non-talking-head videos.
Minimum AV Sync¶
Unit: Milliseconds
The minimum observed audiovisual synchronization delay.
Maximum AV Sync¶
Unit: Milliseconds
The maximum observed audiovisual synchronization delay.
Average AV Sync¶
Unit: Milliseconds
The average audiovisual synchronization delay calculated over the duration of the stream.