Network Requests and HAR Files
Automator can record the network requests made by the browser for any study you run. This captures the timing, size, remote IP, and other details of each request, which helps when diagnosing issues with the browser or the network.
Logging Network Requests
By default, Automator logs all network requests of browser studies and uses them for further analysis. This enables additional statistics like the video response time and content server hostname/IP address.
If you need to disable this, for example due to performance concerns on low-end devices, use --noLogNetworkRequests. To disable it for all your studies, add the option to the schedule config:
{
"version": 2,
"globalScheduleSettings": {
"options": {
"noLogNetworkRequests": true
}
}
}
The log is deleted after the run, unless you specify --keepNetworkRequests. It is then stored in ~/.surfmeter/network-requests/, as surfmeter-requests-<id>.ldjson.gz. The format is gzipped line-delimited JSON. To view it, run gunzip -c <path>.
Creating HAR Files
To record the HTTP traffic of a browser study in a HAR file, use --enableHarFile:
surfmeter study start --studyId STUDY_YOUTUBE --enableHarFile
This writes one file per browser study to ~/.surfmeter/har-files/, or to the directory set with --harFileDir, named <time>_<study ID>_<id>.har. It contains the requests without their response bodies. You can open it in the network panel of Chrome's DevTools.
Sending Network Requests
You can also send all network requests to the server with the --sendNetworkRequests option.
The data is available in the Surfmeter Export API under the network_requests resource. It is not indexed in the Analytics Dashboard.
This causes a lot of network traffic and should be used with care. If the connection is lost, the request will fail, since there is no retry.
The data is deleted after at most 30 days. We recommend using the API to get the data on a daily basis if you need more long-term storage.
With --sendNetworkPerformanceReports, Automator sends a network performance report with each browser web measurement. With --sendByteCountReport, it sends a report with the bytes transferred per second and statistics on the throughput.
Getting Network Requests via the API
Retrieve network requests through the network_requests resource of the Export API.
The API defaults to a page size of 20. To get more requests, increase the per_page parameter, or use pagination.
Data Format
Automator records network requests through the Chrome DevTools Protocol. The fields follow these events:
Each API request record includes its response. Field names use snake_case; Chrome uses camelCase.
The network_response.data.finished_or_failed_at_timestamp field records when the request finished or failed, based on the loadingFinished or loadingFailed event. Use it to calculate the total request duration, as shown below.
Example
This example shows a request and its response:
{
"id": 411,
"measurement_id": 123,
"client_time": "2025-08-19T08:18:00.115Z",
"client_time_skewed": "2025-08-19T08:18:00.115Z",
"created_at": "2025-08-19T08:18:26.956Z",
"updated_at": "2025-08-19T08:18:26.956Z",
"data": { // (1)
"type": "Script",
"request": {
"url": "https://example.com/some-script.js",
"method": "GET",
"headers": {
"referer": "https://example.com/some-page.html",
"sec_ch_ua": "\"Not)A;Brand\";v=\"8\", \"Chromium\";v=\"138\", \"Google Chrome\";v=\"138\"",
"user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
"sec_ch_ua_mobile": "?0",
"sec_ch_ua_platform": "\"Linux\""
}
},
"frame_id": "570AF5E850C2F859374DDF9BB8CB68E5",
"study_id": "STUDY_YOUTUBE", // (2)
"loader_id": "277B02E037CBA0628F1D53F0B76FDAF1",
"timestamp": 6378353.008456,
"wall_time": 1755591480.115987,
"request_id": "1132753.12",
"document_url": "https://example.com/some-page.html"
},
"network_response": {
"id": 411,
"network_request_id": 4104074,
"client_time": "2025-08-19T08:18:00.173Z",
"client_time_skewed": "2025-08-19T08:18:00.173Z",
"created_at": "2025-08-19T08:18:26.959Z",
"updated_at": "2025-08-19T08:18:26.959Z",
"data": {
"type": "Script",
"frame_id": "570AF5E850C2F859374DDF9BB8CB68E5",
"response": {
"url": "https://example.com/some-script.js",
"status": 200,
"timing": {
"dns_end": -1,
"ssl_end": -1,
"push_end": 0,
"send_end": 1.617,
"dns_start": -1,
"proxy_end": -1,
"ssl_start": -1,
"push_start": 0,
"send_start": 1.565,
"connect_end": -1,
"proxy_start": -1,
"request_time": 6378353.035412,
"worker_ready": -1,
"worker_start": -1,
"connect_start": -1,
"worker_fetch_start": -1,
"receive_headers_end": 30.537,
"receive_headers_start": 30.501,
"worker_respond_with_settled": -1
},
"charset": "",
"headers": {
"date": "Tue, 19 Aug 2025 08:18:00 GMT",
"etag": "\"40a-63269338b8380\"",
"server": "Apache",
"content_type": "text/javascript",
"accept_ranges": "bytes",
"last_modified": "Thu, 10 Apr 2025 09:27:58 GMT",
"content_length": "1034"
},
"protocol": "h2",
"mime_type": "text/javascript",
"remote_port": 443,
"status_text": "",
"connection_id": 7951,
"response_time": 1755591480173.442,
"from_disk_cache": false,
"connection_reused": true,
"remote_ip_address": "1.2.3.4",
"encoded_data_length": 39,
"from_prefetch_cache": false,
"from_service_worker": false
},
"study_id": "STUDY_YOUTUBE",
"loader_id": "277B02E037CBA0628F1D53F0B76FDAF1",
"timestamp": 6378353.091208,
"request_id": "1132753.12",
"finished_or_failed_at_timestamp": 6378353.070411 // (3)
}
}
}
- This is the request data per the DevTools Protocol
requestWillBeSentevent. - Use the study ID to filter requests when running multiple studies in parallel.
- The timestamp records when the request finished or failed.
Calculating Total Response Times
This example calculates the total request duration, including the response body download, using API data. See also the timing calculations in chrome-har-capturer.
def calculate_total_request_time(network_request_data):
"""Return the request duration in milliseconds, or None if unavailable."""
request_data = network_request_data["data"]
response_data = network_request_data["network_response"]["data"]
timing_info = response_data["response"]["timing"]
request_start_time = timing_info["request_time"]
request_finished_time = response_data.get("finished_or_failed_at_timestamp")
if not request_finished_time or request_finished_time <= 0:
return None
def to_milliseconds(timestamp):
"""Infer seconds or milliseconds from the number of integer digits."""
digit_count = len(str(int(timestamp)))
if digit_count >= 13:
# Already in milliseconds (e.g., 1755591481408.136)
return timestamp
else:
# In seconds, need to convert (e.g., 1755591481.408136)
return timestamp * 1000
start_time_ms = to_milliseconds(request_start_time)
finish_time_ms = to_milliseconds(request_finished_time)
total_duration_ms = finish_time_ms - start_time_ms
return total_duration_ms
# Example request:
video_request = {
"data": {
"timestamp": 6378354.292458, # Browser internal timestamp
"request": {"url": "https://example.com/video_segment.ts"}
},
"network_response": {
"data": {
"response": {
"timing": {
"request_time": 6378354.294126, # Request started here
"send_start": 0.343,
"send_end": 0.481,
# ... other fields
"receive_headers_end": 5.5
}
},
"finished_or_failed_at_timestamp": 6378354.301374 # Request completed here
}
}
}
total_time = calculate_total_request_time(video_request)
print(f"Total request time: {total_time:.2f} ms") # Output: 7.25 ms