Logging

Surfmeter Automator uses pino for structured JSON logging. Each measurement run produces log output that includes startup information, browser events, measurement progress, and results. Understanding how this output is captured depends on whether you run Automator natively or inside Docker.

Log Levels

You can control the verbosity of log output with the LOG_LEVEL environment variable. The available levels, from least to most verbose, are:

• error -- only errors
• warning -- errors and warnings
• info -- the default; normal operational messages
• debug -- additional diagnostic output
• trace -- very detailed tracing
• extreme -- maximum verbosity

Set the variable when running Automator directly:

LOG_LEVEL=debug ./surfmeter-automator-headless startStudy --studyId STUDY_YOUTUBE

Or add it to a schedule entry's env block in your automatorConfig.json:

{
    "id": "STUDY_YOUTUBE",
    "studyId": "STUDY_YOUTUBE",
    "cronSchedule": "*/15 * * * *",
    "env": {
        "LOG_LEVEL": "debug"
    }
}

You can also set it globally for all schedules via globalScheduleSettings.env:

"globalScheduleSettings": {
    "env": {
        "LOG_LEVEL": "debug"
    }
}

Logging in Native Deployments

When Automator runs natively (outside Docker), scheduled measurements are executed by cron. By default, cron delivers command output via its mail transfer agent (MTA). On most systems this MTA is not configured, which means log output is silently discarded.

To persist the logs, set cronLogFile in your automatorConfig.json. You can set it per schedule entry or globally:

"globalScheduleSettings": {
    "cronLogFile": "$HOME/logs/surfmeter-automator.log"
}

The $HOME and $TMP variables are expanded automatically. Make sure the target directory exists before loading the schedule:

mkdir -p ~/logs

When you run a study manually (not via cron), output goes to your terminal's stdout as usual.

You must manage log rotation yourself

When writing to a file via cronLogFile, logs accumulate indefinitely. On a busy schedule this file can grow to several gigabytes. Set up logrotate to prevent disk exhaustion – see the section at the bottom of this page.

Logging in Docker Deployments

Docker containers write their stdout and stderr to the Docker logging driver, which you can inspect with:

docker compose logs -f surfmeter

The Surfmeter Docker entrypoint produces its own structured JSON log lines for container lifecycle events (startup, registration, schedule loading, etc.).

Cron output forwarding

Cron jobs inside a container cannot write to container stdout directly, because cron runs as a separate process without access to PID 1's file descriptors. To solve this, the Docker entrypoint sets up a named pipe (FIFO) at the path defined by the SURFMETER_CRON_LOG environment variable (default: /tmp/surfmeter-cron.log). When a schedule entry does not specify a cronLogFile, Automator automatically redirects the cron job's output to this FIFO. The entrypoint reads from the pipe in the background and forwards lines to container stdout:

• JSON lines (pino output from Automator) are forwarded as-is
• Non-JSON lines are wrapped in the entrypoint's own JSON log format

This means that with the default Docker setup, all measurement logs appear in docker compose logs without any extra configuration.

Overriding with cronLogFile

If you set a cronLogFile in your automatorConfig.json, cron output is written to that file instead of the FIFO. This bypasses Docker's logging entirely for those schedule entries.

File-based logging in Docker requires manual rotation

If you explicitly set cronLogFile to a file path inside the container (i.e. one that you have not mapped to a volume or bind mount), logs will not appear in docker compose logs and will not be managed by Docker's log rotation. You must handle rotation yourself – see Log Rotation below. Notably, the container itself can become very large if the log file grows without bound, which can lead to performance issues. And if you restart the container, you will lose all logs.

Disabling the FIFO

To disable the FIFO forwarding entirely, set the environment variable to an empty string in your docker-compose.yml:

environment:
  SURFMETER_CRON_LOG: ""

Be aware that without the FIFO and without a cronLogFile, cron output inside Docker is lost.

Summary

Setup	Default behavior	Where logs appear
Native, manual run	stdout	Terminal
Native, cron, no cronLogFile	Sent to cron MTA	Usually lost (MTA not configured)
Native, cron, with cronLogFile	Appended to file	The specified file
Docker, no cronLogFile	FIFO forwarding to container stdout	docker compose logs
Docker, with cronLogFile	Appended to file	The specified file (not in docker compose logs)

Log Rotation

Whenever Automator writes logs to a file -- either because you set cronLogFile in a native deployment or because you opted for file-based logging inside Docker -- you are responsible for rotating those files.

On Linux, the standard tool for this is logrotate. Create a configuration file, for example /etc/logrotate.d/surfmeter:

/home/surfmeter/logs/surfmeter-automator.log {
    daily
    rotate 14
    compress
    missingok
    notifempty
    copytruncate
}

This rotates the log daily, keeps 14 compressed copies, and uses copytruncate so that the running process does not need to be signaled. Adjust the path and retention to your needs.

Docker's built-in log rotation

When using the default FIFO-based forwarding (i.e., not setting cronLogFile), Docker manages log rotation through its logging driver. The default json-file driver supports the max-size and max-file options, which you can configure globally in /etc/docker/daemon.json:

{
    "log-driver": "json-file",
    "log-opts": {
        "max-size": "50m",
        "max-file": "5"
    }
}

Or per container in docker-compose.yml:

services:
  surfmeter:
    logging:
      driver: json-file
      options:
        max-size: "50m"
        max-file: "5"

This is another advantage of letting logs flow through Docker's standard mechanism rather than redirecting to a file.