insights/.agents/skills/writing-evals/references/running-evals.md

Running evals locally and remotely

Use this reference to prepare a machine for the standalone eval harness and choose between local Docker sandboxes and remote Modal sandboxes. Both providers keep the eval database, Django server, LLM gateway, MCP server, Temporal, personinsights, case setup, and scoring on the machine running insightscli evals. The provider changes only where the coding-agent sandbox runs. Prefer remote Modal for multi-case sandboxed evals when its prerequisites are available. Remote sandboxes run in parallel without consuming local Docker memory, which usually shortens the run substantially. Use local Docker for a single-case smoke test, offline work, or when Modal credentials or a Funnel-capable tailnet are unavailable.

Shared setup

Run evals from the repository root in a flox shell. The personinsights build needs flox's Rust toolchain, pkg-config, and OpenSSL. When a flox shell is not already active, wrap the whole command instead of starting an interactive activation:

flox activate -- bash -c "insightscli evals --list"

Use insightscli evals for normal runs. It loads the repository's .env, .env.local, .env.development, and .env.services files, with existing shell variables taking precedence. Keep personal credentials in the gitignored .env.local or in the shell. The direct python -m products.insights_ai.eval_harness.harness entry point loads only .env, so export any other credentials before using it.

The required variables depend on the selected suite and runtime:

Selection	Required variables
Every suite	BRAINTRUST_API_KEY, LLM_GATEWAY_ANTHROPIC_API_KEY
Any sandboxed suite	SANDBOX_JWT_PRIVATE_KEY in addition to the variables above
Codex runtime	LLM_GATEWAY_OPENAI_API_KEY in addition to the variables above

The local-development sandbox JWT key ships in .env.example and is normally copied into .env as part of repository setup. Use that key only for local development, never for production. Preflight reports every missing variable before it starts the database or any services.

List discovered suite IDs before the first real run:

insightscli evals --list

Product-owned suites have IDs in the form <product>/<module>::<function>. Start with one suite and one case so setup errors are cheap to diagnose:

insightscli evals '<product>/eval_example::eval_my_thing' --eval '<case-name>'

One-shot-only selections do not use a sandbox provider. Do not pass --provider, --max-sandboxes, or other sandbox-only flags for them; preflight rejects those flags rather than ignoring them.

The harness also limits the team-cloning and case-seeding phase separately from sandbox concurrency. That setup limit is one on an ordinary local machine and automatically increases to four when either CODER or CI is set. The higher limit lets managed development and CI environments prepare cases in parallel while keeping local Datastore memory use conservative.

Local setup with Docker

Local sandboxed evals use Docker by default. Install Docker, start the daemon, and confirm the CLI can reach it:

docker info

No manual sandbox-image build is required. Before a run, the harness checks whether insights-sandbox-base is current and rebuilds it when the Dockerfile or published agent version changed. The first personinsights build and any required image build can take several minutes; later runs reuse their outputs.

Run one case with one live sandbox first:

insightscli evals '<suite-selector>' \
    --eval '<case-name>' \
    --provider docker \
    --max-sandboxes 1

--provider docker is optional because Docker is the default. The default local cap is four concurrent sandboxes, and each sandbox is configured for up to 16 GB of memory. Keep --max-sandboxes low unless the host has enough memory for the requested concurrency.

Normal teardown removes each case's container and sweeps leftovers from the current run. For a failed case that needs container inspection, rerun it with --keep-sandbox-containers; remove the retained container after debugging. Use --rebuild-sandbox-image only when a forced image rebuild is part of the diagnosis.

Remote setup with Modal (preferred)

Remote sandboxed evals use the dedicated insights-sandbox-evals Modal app. Install the modal and tailscale CLIs and make sure both are on PATH inside the flox environment.

Authenticate Modal once:

modal token new

This normally writes ~/.modal.toml. For non-interactive environments, set both MODAL_TOKEN_ID and MODAL_TOKEN_SECRET instead.

Connect this machine to your tailnet once:

tailscale up

Funnel additionally needs the funnel node attribute granted to the tailnet and this node in the admin console, and HTTPS certificates enabled for the tailnet. The harness then exposes the local Django API, LLM gateway, and MCP server over Tailscale Funnel for the duration of the run, one service per Funnel public port (443, 8443, 10000). Do not manually set SANDBOX_API_URL, SANDBOX_LLM_GATEWAY_URL, or SANDBOX_MCP_URL for an eval run.

Funnel serves only those three public ports, so the harness refuses to start when your tailnet node already serves one of them rather than clobbering it. Free the port first — tailscale funnel --https=443 off, and likewise for 8443 and 10000.

Start with one remote sandbox to verify credentials, Funnel, and the remote image build:

insightscli evals '<suite-selector>' \
    --eval '<case-name>' \
    --provider modal \
    --max-sandboxes 1

The first Modal run builds the eval image remotely. Later runs reuse that image until its Dockerfile, build context, or bundled local skills change. Keep the local process and network connection alive because the remote sandbox calls back into the services running on the local machine.

Modal concurrency is unbounded by default, so always choose an intentional --max-sandboxes value before running a product or domain with many cases:

insightscli evals '<product-or-domain>' --provider modal --max-sandboxes 4

The harness turns its Funnel mappings off during teardown. It also terminates each case's task-tagged sandbox and sweeps leftovers belonging to the current run. Funnel config lives in tailscaled and outlives the harness process, so a run killed with SIGKILL can leave the three ports public — check with tailscale serve status and turn them off by hand. The Funnel URLs are reachable by anyone on the internet who learns them while the run is active, so do not share them or leave a run unattended after an interruption.

Diagnosing setup failures

Read the preflight error first; it names the missing executable, credential, or environment variable and gives the expected fix. Frequent causes are:

• Running outside flox, which makes the personinsights Rust build fail on its toolchain or OpenSSL dependencies.
• A stopped Docker daemon in local mode.
• Missing ~/.modal.toml or incomplete MODAL_TOKEN_ID / MODAL_TOKEN_SECRET credentials in remote mode.
• A stopped tailscaled, a node that is not logged in, or a tailnet without the funnel node attribute or HTTPS certificates, which stops Funnel from serving the three remote-reachable services.
• A Funnel public port (443, 8443, 10000) already serving something else, which the harness refuses to overwrite.
• Passing provider flags to a one-shot-only selection.
• Omitting LLM_GATEWAY_OPENAI_API_KEY when selecting --agent-runtime codex.

Every real invocation mirrors stdout and stderr to products/insights_ai/eval_harness/logs/harness/<timestamp>_<id>.log. The final terminal line is that transcript's absolute path, and logs/harness/latest.log points to the newest run. After a case starts, its agent log directory contains <case>.jsonl, <case>.artifacts.json, and <case>.summary.txt for deeper debugging.