Module 07: running evaluations

This module runs the full offline evaluation pipeline: the predictor calls the live workflow for every dataset item, the scoring service computes metrics, and the gate decides whether the run passes.

Run every command in this module from your scaffold pod directory, the one where you ran uv sync in Module 01 §1.2.

7.1 Run the evaluation

Make sure you're still on VPN. Both shared services already run for you in the cloud (see Module 01 §1.1).

uv run python evaluation/offline_eval/main.py run \
  --run-name workshop_local \
  --no-fail-on-gate

--no-fail-on-gate keeps the process exit code 0 even when the gate fails so you can inspect results without the run aborting. Remove the flag once you are happy with the metric thresholds.

How scoring actually runs

Scoring a batch is a scoring job, not one blocking call: main.py run submits the batch to the scoring service and polls it automatically until it finishes, so the command can never time out regardless of how long a batch takes. Each judge call within the batch runs concurrently (bounded by the scoring service's own configured limit), rather than one metric at a time, so a batch across several metrics finishes faster than the sum of every individual judge call.

While it waits, main.py run's own terminal prints a progress line on every poll, naming the job's status and how many judge calls have finished:

Scoring job 3f2a1c9e...: running (7/12 evaluations)

A reasoning-tier LLM-judge call typically takes 9 to 30 seconds, so seeing the same job sit at running for a while between progress lines is normal for a larger batch or a slower judge model. It isn't evidence of a stuck process. If you want to confirm independently, the scoring service's own terminal also logs when it starts and finishes a submitted job.

7.2 Inspect results

Results are written to evaluation/offline_eval/outputs/. Each run creates a timestamped folder containing:

File	Contents
predictions.jsonl	Raw workflow outputs per item, as PredictionRecords
scores.jsonl	Per-item, per-metric MetricResults from the scoring service
aggregate_metrics.json	Per-metric averages, min/max, and pass rates
gating_result.json	Gate result (pass/fail, and which metrics failed if any)
run_summary.json	Combined overview: item counts, aggregate metrics, gate result, and the Langfuse experiment (dataset-run) URLs this run published to

The run also publishes an experiment (a dataset run) to Langfuse where you can compare scores across runs and see item-level metric details.

7.3 Read the gate result

{
  "passed": false,
  "mode": "all_metrics",
  "minimum_item_count": 1,
  "total_items": 10,
  "failing_metrics": [
    {
      "scope": "reply_generation",
      "metric_name": "reply_correctness",
      "actual": 3.32,
      "threshold": 3.8,
      "reason": "Aggregate score 3.32 for 'reply_correctness' did not meet threshold 3.8."
    }
  ]
}

failing_metrics names exactly which metric(s) on which scope(s) missed their threshold and by how much, enough to decide whether the failure is a real regression, an unlucky LLM-judge sample (Module 06), or a rubric that needs tuning, without re-deriving it from scores.jsonl by hand.

Continue to Module 08.