Offline evaluation

An offline evaluation run exercises a system under test against a fixed golden dataset, scores the predictions via the scoring service, and publishes the results to Langfuse. The central entry point is execute_offline_evaluation().

See 01-getting-started.md for the quick-start walkthrough. See 04-run-config.md for RunManifest, MetricConfig, GateConfig, and the evaluator catalogue.

PredictionRunner Protocol

PredictionRunner (semantic_evaluation.prediction) is the formal Protocol that pod-side predictor adapters implement. A run file names the class in predictor_adapter, as module.path:ClassName.

class PredictionRunner(Protocol):
    name: str  # identifies the predictor in run artifacts and Langfuse trace metadata

    async def predict_items(
        self, items: list[EvaluationItem]
    ) -> list[PredictionRecord]: ...

The framework calls predict_items once per repeat, one after another, repeat 1 first, each time with the items of every selected scope. A run without repeats has one repeat. A run resumed from saved predictions never calls it; see A run that stopped after predicting. The implementation controls its own internal concurrency (fan-out, semaphores, batching). It returns one PredictionRecord per input item; missing records cause the runner to skip scoring for those items.

predict_items may also take a keyword repeat: int, the repeat the call is for, counted from 1. The runner passes it when the signature has a parameter named repeat or accepts **kwargs, and omits it otherwise:

    async def predict_items(
        self, items: list[EvaluationItem], *, repeat: int = 1
    ) -> list[PredictionRecord]: ...

A predictor may also define one optional method, used only when the run declares trace metrics:

    async def fetch_trace(self, trace_id: str) -> Any: ...

It returns the trace the measurements are read from. A pod with local trace capture returns await fetch_local_trace_via_http(f"{base_url}/debug/trace/{trace_id}") (see semantic_tracing's local capture docs), which is complete the moment the turn ends. Without the method the runner reads the trace from Langfuse: it flushes its own Langfuse client, then reads the trace every 5 seconds until three reads in a row agree, for up to 120 seconds, because Langfuse ingests spans and computes cost a while after the turn ends, out of order, and an early read measures part of the turn. That is at least ten seconds per trace, eight traces at a time; local capture is both faster and certain to be complete.

A predictor whose runtime emits no Langfuse trace sets measurements on each record instead, and the run file sets prediction_traces: create_when_missing; see Predictions from a runtime without a Langfuse trace.

PredictionRecord

Field	Type	Notes
`item_id`	`str`	Must match the source `EvaluationItem.item_id`
`scope`	`str`	Scope the item belongs to
`dataset_name`	`str`	Source dataset name
`predictor_name`	`str`	From `PredictionRunner.name`
`input`	`dict[str, Any]`	Copy of `EvaluationItem.input`
`expected_output`	`dict[str, Any]`	Copy of `EvaluationItem.expected_output`
`prediction`	`dict[str, Any]`	The system's output (field names must match what evaluators expect)
`trace_id`	`str \\| None`	Langfuse trace ID, which links the prediction to an observable trace. May be omitted when the run sets `prediction_traces: create_when_missing`; otherwise the runner refuses a record without it
`observation_id`	`str \\| None`	Optional specific observation within the trace
`metadata`	`dict[str, Any]`	Anything the predictor wants on the trace; written onto it as `prediction_metadata` when the run is published
`measurements`	`PredictionMeasurements \\| None`	What the predictor measured itself: `wall_time_s`, `input_tokens`, `output_tokens`, `cached_tokens`, `reasoning_tokens`, `cost_usd`, `tool_calls`, `tool_calls_by_name`, and `notes`. `None` for a field means not measured. When set, trace metrics are taken from it and no trace is read
`source_transcript`	`TranscriptReference \\| None`	Where the platform that ran the agent keeps its own transcript: `platform`, `conversation_id` and an optional `url`. A reference only; it is written to the prediction trace's metadata as `offline_eval.source_transcript`
`repeat`	`int`	Set by the runner: which repeat of the item this is, from 1. A predictor leaves it alone
`error`	`str \\| None`	Why the agent produced no answer to read: no conversation was opened, no transcript arrived in time, or the transcript cannot be read. An errored record is saved and published but not scored; see Errored predictions. An answer that was read but is wrong or invalid is not an error

`execute_offline_evaluation()`

from semantic_evaluation.orchestration.runner import execute_offline_evaluation

result = await execute_offline_evaluation(
    manifest=manifest,                       # from load_resolved_run_manifest()
    manifest_path=Path("evaluation/runs/my_run.yaml"),
    scope_override=None,           # comma-separated scope names; overrides manifest.scopes
    run_label=None,                # overrides manifest.run_label for this invocation
    scoring_service_base_url="http://localhost:8001",
    scoring_service_timeout_seconds=120.0,
    outputs_dir=Path("evaluation/offline_eval/outputs"),
    # optional:
    run_config=None,               # merged over manifest.run_config for this invocation
    repeats=None,                  # replaces manifest.repeats for this invocation
    predictor=None,                # built from manifest.predictor_adapter when None
    scoring_service_api_key=None,
    resume_from=None,              # a stopped run's directory; see "A run that stopped after predicting"
)

Parameters

Parameter	Type	Description
`manifest`	`RunManifest`	The resolved run definition
`manifest_path`	`Path`	The run file's path, recorded in the run's artifacts
`predictor`	`PredictionRunner \\| None`	Default `None`: the runner builds the class `manifest.predictor_adapter` names, with the run's `run_config` and `repeats` where its constructor takes them, and asks it to describe its candidate; see Configurations reach the agent through the predictor. A predictor that needs other arguments is built by the caller, from `resolve_predictor_class(manifest)`, and passed here; it must be an instance of that class, or the run stops with a `TypeError`
`scoring_service_base_url`	`str`	Base URL of the scoring service
`scope_override`	`str \\| None`	Comma-separated scope names to run instead of all manifest scopes
`run_label`	`str \\| None`	Experiment label for this invocation. Resolved as `run_label or manifest.run_label`; the CLI flag wins over the YAML default. Written into the Langfuse run name as `{manifest.name}_{run_label}_{timestamp}`.
`scoring_service_timeout_seconds`	`float`	HTTP timeout for each individual call to the scoring service — the job submission and every status poll — not the whole batch (default `120.0`). See 05-scoring.md — ScoringServiceClient.
`scoring_service_api_key`	`str \\| None`	Your pod's own caller key for the scoring service, sent as `X-API-Key` with every call. When `None`, the runner sends `SCORING_SERVICE_API_KEY` from the environment; a key passed here wins, and with neither no key is sent. The deployed service identifies a known caller on the scoring routes and still accepts a call without a key during a warning phase, so set one (default `None`).
`outputs_dir`	`Path`	Where to write per-run artifacts
`run_config`	`Mapping[str, str \\| int \\| float \\| bool] \\| None`	The configuration under test for this invocation, merged over `manifest.run_config` (its keys win). See Comparing configurations
`repeats`	`int \\| None`	How many times each item is predicted, replacing `manifest.repeats` (default `None`: the manifest's value, itself `1` by default)
`resume_from`	`Path \\| None`	Default `None`. The run directory of a run that stopped after predicting. The runner then calls no `predict_items`, scores and publishes the saved predictions, and writes into that directory. See A run that stopped after predicting

Pipeline

execute_offline_evaluation() runs the full pipeline in order:

graph TD
    P["Build or check the predictor\nthe manifest names,\nand describe its candidate"]
    A["Load items from Langfuse\n(per scope)"]
    B["Call predict_items()\n(once per repeat)"]
    T["Read trace metrics\n(only when the manifest declares them)"]
    C["Submit and poll one scoring job\nfor every repeat's scored predictions"]
    F["Write run artifacts\nto outputs_dir"]
    D["Create Langfuse process trace"]
    E["Publish experiments\nto Langfuse"]
    G["Return OfflineEvaluationExecutionResult"]

    P --> A --> B --> T --> C --> F --> D --> E --> G

Step C is a scoring job: the runner submits the batch and polls until it completes, logging each poll's progress (completed_evaluations/total_evaluations) so a long-running batch never sits silent. This wait is bounded only by ScoringServiceClient's own max_wait_seconds (unset by default — the runner waits as long as the job takes), never by scoring_service_timeout_seconds, which only bounds each individual submit/poll call.

The client retries transient failures with a doubling backoff of 2, 4, 8, 16 and 32 seconds, six tries in all. A status poll (a GET) is retried on any transport error and on HTTP 502, 503 and 504. The job submission is retried only when it provably never reached the app: a connection error, a connect timeout or HTTP 503. A submission that may have been received is never repeated, so a job is never created twice. The tries and the first delay are the client's retry_attempts and retry_initial_backoff_seconds arguments.

Artifacts are written before anything is published, so a Langfuse failure never loses the run.

Errored predictions

A prediction that carries an error is saved with the others and is not scored. The runner logs how many there are, then leaves them out of the trace metrics, the scoring request (an item whose every repeat errored is not sent at all), the per-group aggregates and the agreement. The scoring service never sees them, so it cannot apply a tolerance; the runner does. The gate fails for a scope whose share of errored predictions is above the run file's max_error_fraction (default 0: any error fails it), with a failing metric named errored_predictions that gives the share and the tolerance. The share is counted over every repeat's predictions of the scope.

What publication writes to Langfuse

Langfuse calls a dataset run an experiment, in its UI and its API; a run published here is one experiment, and runs published earlier appear as experiments under their original ids. Publishing requires Langfuse tracing to be enabled, because an experiment item is a span; with tracing off the client raises RuntimeError.

For each evaluation item the runner adds one span, offline_eval_item, to the prediction trace the agent already wrote. The span sits in Langfuse's experiment environment, sdk-experiment, so the trace shows a second root that the SDK's trace readers ignore. The agent's own observations are never modified. The span carries:

the experiment attributes that make it the item in the experiment view;
the dataset item's category labels (tags), as trace tags;
the whole dataset item metadata, as the experiment item's metadata (see What each dataset item metadata field means);
the run's metadata, as trace metadata (see run_config).

Per-item scores, metrics and trace metrics alike, attach to that span, which is where Langfuse's experiment view reads them. Run scores attach to the experiment: the aggregates, their spread and gate_passed, and, when the run file asks for them, the per-group aggregates and the agreement scores. The score names:

Score name	Value
`offline_eval.{metric}`	Per item, on the item span: the item's normalized score (a trace metric's raw measurement). The comment is `[execution name \\| dataset version] justification`, with `dataset unversioned` when the item has no version and the judge's justification when there is one
`{scope}.{metric}`	The metric's mean (a binary metric's pass rate) over the whole run
`{scope}.{metric}.median`, `.p90`, `.stdev`	Its spread
`{scope}.{metric}.by.{attr}.{value}`	The same mean over the items whose metadata attribute `attr` is `value`, with `.median`, `.p90` and `.stdev` under the same switches as the whole-run spread
`{scope}.agreement.{field}.unanimous_share`, `.mean_modal_share`	Agreement of the repeats on `field`, over the scope; with `.by.{attr}.{value}` appended, over one group
`{scope}.agreement.{field}.run_shifts`	How many (unit, repeat) pairs shifted
`gate_passed`	`1.0` or `0.0`

The offline_eval. prefix keeps a run's per-item scores apart from human-review scores on the same trace. Run scores sit on the experiment, so they carry no prefix.

What each dataset item metadata field means for a prediction trace

Dataset item metadata field	Where it appears
`tags`	Trace tags on every prediction trace the item produces; category labels only
`changed_last_in_dataset_version`	Named in each per-item score's comment, and in the experiment item metadata
`trace_id`, `observation_id`	The source trace the item was taken from, not the prediction trace; experiment item metadata only
`product_id`, `created_by`, `created_date`, `workflow_version`, `selected_agent`, `raw_metadata`	Experiment item metadata only, never trace tags or trace metadata. `raw_metadata` holds the promotion keys such as `ground_truth_status`, `seed_source`, `review_outcome`, `human_review_scores` and `reviewer_comments`

A prediction with an error is published like any other, but its trace carries the error as offline_eval.error in its metadata and the item span has no scores, since the prediction was not scored. An errored prediction without a trace_id gets a created trace under either prediction_traces setting, because it has no agent trace to attach to.

Langfuse's own latency and cost columns in the experiment view describe the item span, not the agent's work. Read the offline_eval.wall_time_s and offline_eval.cost_usd scores instead.

Run artifacts

Each run gets its own folder, {outputs_dir}/{manifest.name}/{timestamp}_{run_label} (default when the run has no label), holding six files, and two more when the run file asks for per-group aggregates or agreement. The runner writes the folder in two steps. effective_config.json and predictions.jsonl are written as soon as predicting ends, before trace metrics are read and before scoring, so an agent run that took an hour is never lost to a failure after it. The other files follow once scoring is done:

File	Holds
`effective_config.json`	The run manifest as the run used it, with the effective `run_config`, `repeats` and `run_label`, the scopes, and the run file's path
`predictions.jsonl`	One `PredictionRecord` per line, every repeat's, errored ones included
`scores.jsonl`	One `MetricResult` per line: every metric on every prediction, trace metrics included
`aggregate_metrics.json`	One `AggregateMetricResult` per metric and scope: the scored metrics, then the trace metrics (those have `raw_scale` set)
`grouped_aggregates.json`	One `AggregateMetricResult` per metric, scope and group, with `group_by` and `group_value` set; written when `aggregate_by` is set
`agreement.json`	The `AgreementResult`: per item, per group and per run-shift unit; written when `agreement` is set and the run has repeats
`gating_result.json`	The `GateResult`, with each failing metric and why; includes a scope whose errored predictions exceed `max_error_fraction`
`run_summary.json`	Item and result counts, `prediction_count` and `errored_prediction_count`, `grouped_aggregate_count`, the `agreement` summaries, the artifact paths, and once published, `langfuse_publication`: the execution name, the label and every experiment (dataset run) with its URL; or the error when publication failed

The timestamp in the folder name and the one in the Langfuse execution name are taken separately and can differ by a second; the execution name in run_summary.json is the one Langfuse knows. eval-report reads these files.

A run that stopped after predicting

When trace metrics, scoring or a later step fails (a scoring service that is down, a DNS error, a ConnectError), the runner logs an error that names the run directory and re-raises the failure. The directory already holds effective_config.json and predictions.jsonl, so the predictions need not be produced again. Run again with resume_from set to that directory, or with the pod CLI:

python -m evaluation.offline_eval.main run --run-file evaluation/runs/regression.yaml \
  --resume-from evaluation/offline_eval/outputs/my_run/20261105T101500_default

The runner loads the items again, calls no predict_items, scores the saved predictions, publishes them, and writes the remaining artifacts into the same directory.

A resumed run must be the run that saved the predictions. The runner compares the saved effective_config.json with this invocation and refuses on any difference in the run file's name, predictor_adapter, run_config, repeats, the scopes or the run label, and lists the differing fields. When the runner builds the predictor, it describes the candidate again first, so a candidate that changed since the saved run, with another content_sha256 for example, is refused as a difference in run_config. Pass the same run file, --config, --repeats, --scopes and --run-label as the first time. It also refuses saved predictions that do not fit the loaded items: a record for an item that was not loaded, a repeat outside 1..repeats, or the same item and repeat twice. A directory without both saved files is refused as well.

OfflineEvaluationExecutionResult

Field	Type	Description
`gate_passed`	`bool`	Whether the quality gate passed
`summary`	`dict[str, Any]`	Aggregated run results and artifact paths
`publication_succeeded`	`bool`	Whether Langfuse publication succeeded
`publication_error`	`str \\| None`	Error message if publication failed

LLM judge score variability

LLM-based evaluators are non-deterministic. Two runs against identical items using the same evaluator configuration can return slightly different normalized_score values. Use aggregate means and pass-rates for trend comparisons rather than individual item scores, and set gate thresholds with enough margin to absorb normal variation. See 05-scoring.md — Normalized 1-to-5 scale for the scoring service's normalization behaviour.

Comparing configurations

A run can compare one agent across configurations (model, reasoning effort, feature switches) on time and cost as well as on its scores. Three manifest fields serve that, all optional:

name: drainage_local
predictor_adapter: evaluation.predictor:DrainagePredictor
scopes: [e2e]
run_config:
  model: gpt-5
  reasoning_effort: high
  computation_service: true
  python_sandbox: false
repeats: 3
trace_metrics: [wall_time_s, input_tokens, output_tokens, reasoning_tokens, cost_usd, tool_calls]

A pod usually keeps one manifest and passes the configuration per invocation, through execute_offline_evaluation(run_config=..., repeats=...) from its CLI.

`run_config`

Flat key: value pairs with string, number or boolean values. It is written:

into every experiment's metadata, as run_config, beside repeat and repeats;
onto every prediction trace, through its offline_eval_item span, as one top-level metadata key per entry: run_config.model, run_config.reasoning_effort, and so on. Langfuse filters traces by top-level metadata keys, so "every trace with run_config.reasoning_effort = high" is one filter;
into the run's effective_config.json and the summary the runner returns.

The same span carries offline_eval.execution_name, offline_eval.dataset_run_name, offline_eval.repeat and offline_eval.repeats, the prediction's own PredictionRecord.metadata as prediction_metadata, and, when the prediction names one, offline_eval.prediction_observation_id. The agent emitted its traces in its own process and its observations stay as written, so a trace is filterable by these keys through the span, not through the agent's own observations.

Keys that describe the candidate

These key names are recommended, not enforced. They describe the candidate being evaluated and are the same on every runtime, so runs on different runtimes can be compared and filtered by the same run_config.<key>.

Key	Holds	Written by
`platform`	Runtime family: `open_webui`, `copilot_studio`, `self_hosted`, `stub`	The predictor's `describe_candidate()`, merged into `run_config` before the run; the platform predictors write every key in this table except the reserved ones
`candidate`	The runtime's name or id of the instance evaluated	Same
`model`	Model or deployment the candidate runs on	Same
`delivery`	How the input reaches it: `attachment`, `text`, ...	Same
`content_sha256`	Hash of the candidate's content snapshot	Same
`instructions_version`	Version of the candidate's instructions	Reserved: only a release service that keeps a release record writes it; the SDK never fills it
`knowledge_version`	Version of the candidate's knowledge	Reserved, as above
`release_id`	Id of the release the candidate belongs to	Reserved, as above

Runtime-specific keys (for example bot_id, workflow, upload_mode, file_context) sit free-form beside these. candidate replaces per-runtime names such as assistant or agent rather than sitting beside them. Code other than such a release service leaves the three reserved names unset in run_config. Production traces carry the same three keys in their trace metadata when the agent sets its release with semantic_tracing's Release (see Releases).

Configurations reach the agent through the predictor

run_config labels a run; it does not change the agent. The runner builds the predictor, offering the run's effective run_config and repeats as keyword arguments, each only when the constructor takes it (a parameter of that name, or **kwargs). A predictor that wants the configuration declares it:

class SupportPredictor:
    name = "support"

    def __init__(self, run_config: Mapping, repeats: int = 1) -> None:
        self.model = run_config["model"]

The predictor then hands each value to the agent (a model deployment, a reasoning effort, a feature switch). A key the predictor does not use still labels the run, and two runs that differ only in such a key run the same agent.

A built predictor may also define describe_candidate(), which returns a mapping of run-config values. The runner calls it once, before it loads items, and merges the result over the run's run_config (see Keys that describe the candidate). The merged configuration labels the artifacts, the dataset runs and the traces, and is what a resumed run is compared against. A describe_candidate() that raises stops the run before anything is predicted, which is how a predictor refuses a candidate it must not run on. The platform predictors use it to check and snapshot the candidate.

A project whose predictor needs other arguments (a service URL, a client) builds it itself and passes it in. The runner does not ask a predictor it was given to describe its candidate, so the project merges the description first:

from semantic_evaluation.orchestration.runner import (
    execute_offline_evaluation,
    resolve_predictor_class,
    resolve_run_config,
)

run_config = resolve_run_config(manifest.run_config, {"reasoning_effort": "high"})
predictor = resolve_predictor_class(manifest)(client=client, run_config=run_config)
result = await execute_offline_evaluation(
    manifest=manifest, ..., run_config=run_config, predictor=predictor
)

`repeats`

An agent's time and cost vary from run to run, so a comparison needs several executions per item and a spread rather than one value. With repeats: N the runner calls predict_items N times, one after another, repeat 1 first (passing repeat to a predictor that declares it), and submits all N × items predictions as one scoring job, so the scoring service aggregates and gates over every execution together.

In Langfuse each repeat is an experiment of its own, named {execution_name}_r1 … _rN, so every experiment holds exactly one execution of each item, which is the shape Langfuse's experiment comparison expects. The run-level aggregates are over all repeats, because together they describe one configuration, and by default every repeat's run carries them; the per-item scores on each experiment's item spans are that repeat's own. With repeats: 1 the run name is the execution name, as before. To write the aggregates once rather than N times, see Fewer scores per run.

repeats above 1 needs a scoring service from semantic_evaluation 1.4.0 or later, which copies PredictionRecord.repeat onto each result. Against an older one the run stops after scoring, before it writes artifacts or publishes, because results could not be matched to their predictions.

Aggregates per group of items

A whole-run mean can hide where an agent fails. aggregate_by names item metadata attributes (keys of the dataset item's metadata), and the runner then aggregates every metric a second time per value of each attribute, over the items that carry it:

aggregate_by: [expected_verdict]

Each is an AggregateMetricResult with group_by (the attribute) and group_value (its value), with the same statistics as the whole-run aggregate, over every repeat of the group's items. Trace metrics are grouped as well. An item without the attribute is left out of that attribute's groups, and a value is read as text.

Take a screening agent with a binary flagged metric that passes when the agent flags the item, and aggregate_by: [expected_verdict]. Its pass rate over items expected YES is the share of risks it caught; the miss rate is the complement. Its pass rate over items expected NO is the false-flag rate. One whole-run mean would blend the two.

Grouped aggregates describe; they never enter the gate, so the gate decision is the same with or without aggregate_by. They appear in the artifacts, the Langfuse run scores and eval-report. With aggregates_on: last_repeat they are written on the last repeat's run only, like the whole-run aggregates.

Agreement across repeats

An agent that gives a different answer to the same item on each repeat is not reliable, even when its mean score looks fine. agreement compares the repeats on one field of the prediction:

repeats: 5
agreement:
  field: verdict          # a dotted path into the prediction, e.g. decision.label
  run_shift:
    per: document_id      # an item metadata attribute naming the unit
    min_share: 0.25       # default

It needs repeats of 2 or more; with one repeat the runner logs a warning and computes nothing. Values compare by their JSON form, so a dict or list compares by content. A prediction that lacks the field counts as None in the item's agreement.

Term	Meaning
unanimous	Every repeat gave the same value for the item
modal share	The share of an item's repeats that gave its most frequent value
unanimous share	The share of items that are unanimous
mean modal share	The mean of the items' modal shares

The runner reports, per scope, the unanimous share and the mean modal share, and the same two per group when aggregate_by is set.

run_shift finds a different failure: one repeat that moved a whole unit. The unit is the items sharing the per attribute, for example every item cut from one source document. A repeat of a unit is a whole-run shift when its values differ from the strict majority of the other repeats on at least min_share of the unit's items. A tie among the other repeats is no majority, so that item does not count as differing, and a missing field never counts as a value here. Scattered disagreement lowers the modal share; a whole-run shift is the same disagreement concentrated in one repeat of one unit, which usually means one run read that document differently.

With exactly two repeats, a unit that differs marks both repeats, because each is the one other repeat's majority; the check cannot say which run moved. The runner warns, and run_shift needs three or more repeats to point at one run.

Agreement is written to agreement.json, to the Langfuse run scores ({scope}.agreement.{field}.unanimous_share and .mean_modal_share, with .by.{attr}.{value} per group, and {scope}.agreement.{field}.run_shifts, the number of shifted repeats of units) and to eval-report. It never enters the gate.

Trace metrics

Measurements taken per prediction, beside the scored metrics. For each record the runner uses the first source that applies:

the record's measurements, used as given, with no trace read;
the record's trace_id, whose trace is read as described under PredictionRunner Protocol;
neither, which leaves every trace metric unmeasured for that record.

The table names where each value is read from a trace; a predictor supplying measurements provides the same quantities itself. In the score's justification, "read from the trace." and "supplied by the predictor." tell the two apart, and the metric's details.measured_from is trace, predictor or null.

Name	Unit	Read from
`wall_time_s`	seconds	Langfuse: the trace's `latency`, first observation start to last end, ignoring the `sdk-experiment` span. Local: last span end minus first span start
`input_tokens`	tokens	Every observation's usage, cached input included
`output_tokens`	tokens	Every observation's usage, reasoning included
`cached_tokens`	tokens	The cached part of the input
`reasoning_tokens`	tokens	The reasoning part of the output
`cost_usd`	usd	Langfuse: the sum of the observations' costs, which Langfuse computes from its model prices. Local: each span's `cost_details`, where the pod sets it
`tool_calls`	count	Tool observations (`TOOL` type, or a span named `execute_tool …`)
`tool_calls_by_name`	count	One metric per tool, `tool_calls.<tool>`; an item that did not call a tool counts 0 for it

Each becomes a MetricResult per prediction, with the value in raw_score, the unit in raw_scale, and normalized_score, threshold and passed left None. They are measurements, not scores, so they never fail the gate. A value that cannot be read (no trace and no measurements, a trace that cannot be fetched, a field the predictor left None, a local trace from a pod older than semantic_tracing 0.7.0 for wall time) is recorded with no value and the reason as its justification, rather than dropped. A trace metric may not share a name with a scored metric in the manifest.

A record whose measurements.tool_calls_by_name is None has no value for every tool_calls.<tool>, because the tools it called are unknown.

Read the trace from Langfuse when you want cost: Langfuse computes it from its model prices. A local trace is complete sooner and gives tokens and wall time, but it holds spans before FilteringExporter runs, so usage on a span named in suppress_usage_on_spans is counted here even though Langfuse never receives it.

Aggregates and their spread

Every aggregate, scored or measured, reports median_score, p90_score and stdev_score beside the mean, minimum and maximum, over the whole run and, with aggregate_by, per group. In Langfuse each experiment gets, per metric, {scope}.{metric} (the mean, as before) and {scope}.{metric}.median, .p90 and .stdev. A trace metric's aggregate has raw_scale set and its statistics are in that unit. eval-dataset promote-baseline skips the spread scores (grouped ones included) and the trace-metric aggregates: they are not a metric's baseline. It keeps the scored metrics' grouped means and the agreement scores, so a later run can be compared against them.

Fewer scores per run

Langfuse bills every score as a unit. A manifest that sets nothing writes every score above, as 1.4.0 did. With repeats most of them are copies, so the run manifest's score_publication block leaves out the ones that repeat another or say nothing:

repeats: 3
score_publication:
  aggregates_on: last_repeat        # default every_repeat
  spread_statistics: [median, stdev] # default: all of median, p90, stdev
  informative_spread_only: true     # default false
  trace_metric_item_scores: false   # default true

Switch	What it leaves out
`aggregates_on: last_repeat`	The aggregates and `gate_passed` on every repeat's run but the last. Every copy holds the same values
`spread_statistics`	The statistics not named. The mean is always written
`informative_spread_only: true`	Spread for a binary metric (every result has `raw_scale: binary`), whose pass rate already says everything, and for an aggregate over one value
`trace_metric_item_scores: false`	Each trace metric's score on every item span; its aggregate stays. Langfuse shows a trace's time, tokens and cost itself, but the experiment view's per-item latency and cost columns describe the item span, so keep these scores when you compare items there

None of them changes the gate decision, the local artifacts, or what the aggregates are computed from. With aggregates_on: last_repeat, point promote-baseline at the last repeat's run, {execution_name}_rN: the others carry no aggregates. For a configuration of 3 repeats over 2 items with one binary, one 1-to-5 and one trace metric, the four switches above cut the run scores from 39 to 8 and the item scores from 18 to 12.

Predictions from a runtime without a Langfuse trace

Some agents run on platforms that semantic_tracing cannot instrument, such as Open WebUI or Copilot Studio. Each platform keeps its own transcript of a run (Open WebUI through its chat API, Copilot Studio in the Dataverse conversationtranscripts table). For those two platforms, Platform predictors is a ready-made predictor: the project writes only a conversation hook. For any other runtime, the predictor reads the answer, the tool calls, the timing and the spend from the runtime's transcript and fills the record directly:

class OpenWebUiPredictor:
    name = "open_webui"

    async def predict_items(self, items: list[EvaluationItem]) -> list[PredictionRecord]:
        records = []
        for item in items:
            chat = await self.platform.ask(item.input["message"])  # hypothetical client
            records.append(
                PredictionRecord(
                    item_id=item.item_id,
                    scope=item.scope,
                    dataset_name=item.dataset_name,
                    predictor_name=self.name,
                    input=item.input,
                    expected_output=item.expected_output,
                    prediction={"response": chat.answer, "tool_calls": chat.tool_calls},
                    measurements=PredictionMeasurements(
                        wall_time_s=chat.duration_s,
                        input_tokens=chat.usage.input,
                        output_tokens=chat.usage.output,
                        cost_usd=chat.usage.cost,
                        tool_calls=len(chat.tool_calls),
                    ),
                    source_transcript=TranscriptReference(
                        platform="open_webui", conversation_id=chat.id
                    ),
                )
            )
        return records

The run file asks the runner to write the traces the platform does not provide:

name: assistant_open_webui
predictor_adapter: evaluation.predictor:OpenWebUiPredictor
scopes: [e2e]
prediction_traces: create_when_missing   # default: required
trace_metrics: [wall_time_s, input_tokens, output_tokens, cost_usd, tool_calls]

With the default required, a record without trace_id stops the run with Prediction for evaluation item '…' is missing trace_id. With create_when_missing, publication writes one thin trace per record that has none, so the experiment item and the per-item scores have a trace to attach to. A record that has a trace_id keeps it.

Field of the created trace	Holds
`id`	Derived from the dataset run name and the item id: one per repeat, and the same on re-publication of the same run name
`name`	`offline_eval_prediction/<predictor name>`, the trace's only span, which is the experiment item (default environment)
`input`	`item_id`, `scope` and `dataset_name` only
`output`	The record's `prediction`
`metadata`	The usual `offline_eval.*` keys, `run_config.<key>`, `prediction_metadata`, and `offline_eval.prediction_trace: created_by_runner`
`tags`	The dataset item's category labels

A record's source_transcript is written to offline_eval.source_transcript in its trace's metadata, whether the trace is the agent's own or created by the runner. run_summary.json reports langfuse_publication.created_trace_count, and the experiment's metadata carries prediction_traces.

What reaches Langfuse. Whatever the predictor puts into prediction, metadata and source_transcript. The transcript itself never goes there, only the reference. A predictor over a confidential corpus keeps document text out of all three and carries verdicts, integrity facts and counts instead.

Two limits follow from the trace being written after the fact:

A created trace's timestamp and latency are those of publication, not of the prediction. The prediction's wall time is the wall_time_s measurement.
A predictor that fans one platform conversation out into several records (one contract review producing one record per review criterion) decides how to attribute that conversation's measurements to the records. The runner counts whatever each record carries.

Reporting runs: `eval-report`

eval-report turns runs' artifacts into Markdown, so a terminal, a chat and a CI job summary show the same thing. It needs no network.

eval-report evaluation/outputs/<run name>/<timestamp>_<label>   # one run
eval-report evaluation/outputs/<run name> --latest 2            # the last two runs of it
eval-report <run folder> <run folder> --output report.md        # also write a file

A folder that holds run_summary.json is one run; any other folder stands for the runs inside it, oldest first, and --latest N keeps the last N. For each run the report gives its label and run_config, items, repeats and predictions, how many of those predictions errored (errored: n of N, shown only when some did), the gate with each failing metric, a table of every scored metric (mean, median, p90, standard deviation, pass rate, threshold, result), a table of the trace metrics (mean, median, p90, standard deviation and total, in their unit), a table per aggregate_by attribute ("By attr", one row per metric and value), an agreement table (unanimous share and mean modal share per scope and group), a run-shift table listing each repeat of each unit with how many items differ, and a Langfuse link per dataset run. With two or more runs it starts with a comparison: a column per run, each metric's mean ± standard deviation, the same per group, the agreement shares, an Errored row when any run has errored predictions, and the gate.

A run whose cost_usd is zero or missing is flagged: Langfuse prices a generation only when a model definition matches its model name, so an unpriced model reads as free in a cost comparison.

From Python, load_run_report(run_dir) and render_markdown(reports) in semantic_evaluation.orchestration.reporting.run_report do the same.

Local tuning workflow

The local tuning loop lets engineers iterate on prompts and configuration by comparing candidate runs side-by-side against an established baseline in Langfuse — without committing to CI or involving the full deployment pipeline.

Step 1: establish a baseline

Mark a run config as a baseline by setting baseline: true in the manifest YAML (or by passing --run-label baseline to the pod's CLI):

# evaluation/runs/my_run.yaml
name: pathfinder-local-all-scopes
baseline: true
# run_label defaults to "baseline" when baseline is true

Run the offline evaluation:

python -m evaluation.run --manifest evaluation/runs/my_run.yaml

The execution name is printed to stdout when the run completes and is also visible in Langfuse under the experiment name pathfinder-local-all-scopes_baseline_<timestamp>. Copy this name — you will need it in step 5.

Snapshot the baseline scores into the manifest so they serve as a reference for future comparisons:

eval-dataset promote-baseline \
  --run-name pathfinder-local-all-scopes_baseline_20260702T143200Z \
  --manifest evaluation/evaluation_manifest.yaml

Step 2: experiment

Change prompts, retrieval parameters, or any other configuration that affects the scores you care about.

Step 3: run again with a descriptive label

Give the experimental run a label that describes what changed:

python -m evaluation.run \
  --manifest evaluation/runs/my_run.yaml \
  --run-label v2_routing_fix

This produces a Langfuse run named pathfinder-local-all-scopes_v2_routing_fix_<timestamp>.

Step 4: compare in Langfuse

Both the baseline run and the experimental run share the same experiment_name (pathfinder-local-all-scopes), so Langfuse groups them for side-by-side comparison on the experiment page. Locally, eval-report <outputs_dir>/pathfinder-local-all-scopes --latest 2 prints the two runs side by side (Reporting runs).

Step 5: promote when the experiment beats the baseline

If the experimental run improves on the baseline, promote it so the manifest reflects the new reference point:

eval-dataset promote-baseline \
  --run-name pathfinder-local-all-scopes_v2_routing_fix_20260702T153000Z \
  --manifest evaluation/evaluation_manifest.yaml

To also update the gated evaluator thresholds to match the new scores, add --update-gates:

eval-dataset promote-baseline \
  --run-name pathfinder-local-all-scopes_v2_routing_fix_20260702T153000Z \
  --manifest evaluation/evaluation_manifest.yaml \
  --update-gates \
  --yes

The baseline holds the run's whole-run aggregates, the scored metrics' grouped means ({scope}.{metric}.by.{attr}.{value}) and the agreement scores; the spread scores are left out. --update-gates only reads the whole-run metric scores and ignores the grouped and agreement ones. Only evaluators that already have a threshold field in the manifest are touched. Non-gated metrics are never promoted to gated by this command. When a metric appears in multiple scopes, the most conservative (minimum) value across scopes is used as the updated threshold.

Step 6: commit

Review and commit the manifest changes alongside the prompt or configuration changes that produced the improvement:

git diff evaluation/evaluation_manifest.yaml
git add evaluation/evaluation_manifest.yaml evaluation/runs/my_run.yaml
git commit -m "eval: promote v2_routing_fix baseline, update thresholds"

Committing the manifest with the code change that caused the improvement creates an auditable record of what each baseline represents. Reviewers can see the score delta in evaluation_manifest.yaml and trace it back to the prompt or config diff in the same PR.

Scopes and `resolve_scopes()`

from semantic_evaluation.orchestration.runner import resolve_scopes

# Validates the requested scopes against what the manifest declares
active_scopes = resolve_scopes(
    manifest_scopes=manifest.scopes,
    scope_override="e2e,routing",   # or None to use all manifest scopes
)

resolve_scopes raises RuntimeError if any requested scope is not in the manifest, or if the resulting scope list is empty.