Run configuration

A project configures its offline evaluation in two kinds of YAML file, both under evaluation/:

evaluation/evaluation_manifest.yaml, one per project: every metric with the scopes it scores (the metric catalogue), the dataset of each scope, how datasets are synced, and the baselines eval-dataset promote-baseline writes.
evaluation/runs/<run>.yaml, one per run: the regression gate, each experiment. A run file names its scopes and whatever it changes, and inherits everything else from the project manifest.

evaluation/
  evaluation_manifest.yaml          the project manifest
  evaluation_manifest.schema.json   its JSON Schema, for the editor
  runs/
    regression.yaml                 one run file per run
    run_file.schema.json            their JSON Schema
  datasets/
    archive/                        the snapshot of each released dataset, committed
    incoming/                       working copies, gitignored

Both kinds of file are read as YAML 1.2: only true and false (in any of true, True, TRUE, false, False, FALSE) are booleans. An unquoted yes, no, on or off, in any casing, loads as a string, and the loader logs a warning naming the file and the line. Write true or false for a boolean, and quote a string that could be read as one, for example an enumeration value such as "YES" or "NO".

The runner runs a RunManifest: a run file resolved against the project manifest, with every scope's dataset and metrics spelled out.

See 01-getting-started.md for the quick-start walkthrough. See 04-offline-evaluation.md for execute_offline_evaluation(), PredictionRunner, and the local tuning workflow.

The project manifest: `evaluation_manifest.yaml`

EvaluationManifestFile (semantic_evaluation.contracts) is the shape of the file. Every block is optional, and an unknown key is refused, so a misspelled block fails when the file is loaded rather than being ignored.

Field	Type	Description
`evaluators`	`dict[str, CatalogueMetricConfig]`	The metric catalogue, keyed by metric name
`dataset_sync`	`DatasetSyncConfig`	The dataset of each scope, and where `push-all`, `pull-all` and `archive` read and write
`baselines`	`dict[str, dict[str, float]]`	Per run name, the aggregate scores `eval-dataset promote-baseline` promoted from one of its dataset runs

# yaml-language-server: $schema=evaluation_manifest.schema.json
evaluators:
  groundedness:
    framework: azure_ai_evaluation
    evaluator: GroundednessEvaluator
    threshold: 4.0
    scopes: [grounding, playbook_adherence]
  tool_call_match:
    framework: builtin_deterministic
    evaluator: ToolCallMatch
    threshold: 5.0
    evaluator_params:
      tool_name: search_documents
      arg_name: query
    scopes: [routing]
dataset_sync:
  scope_to_dataset:
    grounding: product_001_grounding_v1.0
    playbook_adherence: product_001_playbook_v1.0
    routing: product_001_routing_v1.0
  scoped_dir: datasets/incoming
  archive_dir: datasets/archive
baselines: {}

load_evaluation_manifest_file(path) returns the whole file; load_evaluation_manifest(path) returns the part a run resolves against (the catalogue and scope_to_dataset), as an EvaluationManifest.

CatalogueMetricConfig

CatalogueMetricConfig extends MetricConfig with one field:

Field	Type	Description
`scopes`	`list[str]`	Every scope that uses this metric

The catalogue is keyed by metric name, not by scope, so a metric several scopes use (a groundedness check shared by grounding and playbook_adherence, for example) is declared once with both scopes listed.

DatasetSyncConfig

Field	Type	Default	Description
`scope_to_dataset`	`dict[scope, str]`	`{}`	The Langfuse dataset of each scope. A run inherits it; `push-all` and `pull-all` need it
`scoped_dir`	`str \\| None`	`None`	One working copy per scope, `{scoped_dir}/{scope}_dev.json` (Pattern B)
`merged_path`	`str \\| None`	`None`	One working copy holding every scope (Pattern A), with `canonical_name`
`canonical_name`	`str`	`""`	The `dataset_name` written into the merged working copy
`archive_dir`	`str`	`datasets/archive`	Where `eval-dataset archive` files the snapshot of each release
`create_on_missing`	`bool`	`false`	Let `push-all` create a Langfuse dataset that does not exist yet
`input_schema`, `expected_output_schema`	`str \\| None`	`None`	`module.path:ClassName` of a Pydantic model, or a JSON Schema file

Relative paths resolve against the manifest's own directory, not the directory a command runs from. scoped_dir: datasets/incoming in evaluation/evaluation_manifest.yaml is evaluation/datasets/incoming whether eval-dataset runs from the project root or from evaluation/. 03-scope-sync.md covers the two working-copy patterns.

Run files: `evaluation/runs/<run>.yaml`

RunFile (semantic_evaluation.contracts) is the shape of a run file. Like the project manifest, it refuses an unknown key.

Field	Type	Required	Description
`name`	`str`	yes	Run name, used in artifact filenames and Langfuse labels
`description`	`str \\| None`	no	Free-form description
`predictor_adapter`	`str`	yes	The predictor class, as `module.path:ClassName`; see The predictor a run names
`scopes`	`list[str]`	yes	The scopes this run evaluates
`datasets`	`dict[scope, DatasetSourceConfig]`	no	Replaces a scope's dataset from `scope_to_dataset`, for example when two runs evaluate one scope on different datasets
`evaluators`	`dict[scope, overrides]`	no	Narrows or adjusts a scope's metrics; see How a run file resolves
`gate`	`GateConfig`	no (defaults to `GateConfig()`)	Run-level quality gate
`max_error_fraction`	`float` (0 to 1)	no (default `0`)	The largest share of a scope's predictions that may carry an `error` before the gate fails. Errored predictions are never scored; the runner applies this limit, not the scoring service; see Errored predictions
`run_label`	`str \\| None`	no	Default experiment label in the Langfuse execution name, `{name}_{run_label}_{timestamp}`; a pod's CLI overrides it per invocation with `--run-label`
`baseline`	`bool`	no (default `false`)	When `true`, sets `run_type = "baseline"` in Langfuse and adds `is_baseline: true` to run metadata; defaults `run_label` to `"baseline"` when none is set
`run_config`	`dict[str, str \\| int \\| float \\| bool]`	no (default `{}`)	The configuration under test, written onto the dataset runs and onto every prediction trace as `run_config.<key>`; merged with the `run_config` argument of `execute_offline_evaluation()`
`repeats`	`int` (at least 1)	no (default `1`)	How many times each item is predicted; each repeat is its own Langfuse dataset run
`aggregate_by`	`list[str]`	no (default `[]`)	Item metadata attributes (keys of the dataset item's metadata) to aggregate every metric by, as well as over the whole run; see Aggregates per group of items
`agreement`	`{field, run_shift}`	no	How the repeats agree on a dotted `field` of the prediction, for example `verdict` or `decision.label`; needs `repeats` of 2 or more. `run_shift` (`per`, `min_share`, default `0.25`) also checks whether one repeat moved a whole unit's items; see Agreement across repeats
`trace_metrics`	`list[str]`	no (default `[]`)	Measurements taken per prediction, from its trace or from the predictor's `measurements`: `wall_time_s`, `input_tokens`, `output_tokens`, `cached_tokens`, `reasoning_tokens`, `cost_usd`, `tool_calls`, `tool_calls_by_name`
`prediction_traces`	`"required" \\| "create_when_missing"`	no (default `required`)	Whether a prediction without a `trace_id` stops the run, or the runner writes a thin trace for it on publication. For agents on platforms that cannot be traced; the predictor then supplies `measurements`
`score_publication`	`ScorePublicationConfig`	no (defaults write every score)	Which scores are written to Langfuse: `aggregates_on` (`every_repeat` or `last_repeat`), `spread_statistics` (any of `median`, `p90`, `stdev`), `informative_spread_only`, `trace_metric_item_scores`

max_error_fraction, run_config, repeats, aggregate_by, agreement, trace_metrics and score_publication are described in 04-offline-evaluation.md: Comparing configurations; prediction_traces in 04-offline-evaluation.md: Predictions from a runtime without a Langfuse trace.

# evaluation/runs/regression.yaml
# yaml-language-server: $schema=run_file.schema.json
name: product_001_regression
predictor_adapter: evaluation.predictor:MyPredictor
scopes: [grounding, routing]
gate:
  mode: all_metrics
  minimum_item_count: 10

Langfuse run naming

`run_label`	`baseline`	Langfuse run name
`null`	`false`	`{name}_{timestamp}`
`"experiment"`	`false`	`{name}_experiment_{timestamp}`
`null`	`true`	`{name}_baseline_{timestamp}`
`"v2"`	`true`	`{name}_v2_{timestamp}`

With repeats above 1, each repeat's dataset run appends _r1, _r2, … to the name above, and the execution name without the suffix is in each run's metadata.

How a run file resolves

For each scope in scopes:

Dataset: the run's datasets.{scope} if given, used as it is; otherwise scope_to_dataset[scope] with the default fetch_items_page_size (50). A scope with neither raises RunManifestResolutionError.
Metrics: every catalogue metric whose scopes list contains the scope, then the run's evaluators.{scope} applied by the rules below. A scope that resolves to no metric raises RunManifestResolutionError.

Under `evaluators.{scope}`	Effect
`only: [name, ...]`	Starts the scope from exactly these catalogue metrics, dropping the other candidates. Naming a metric not tagged for this scope is an error.
`{name}: {field: value, ...}`, `name` in the catalogue	Merges the given fields onto the catalogue's definition and adds the metric to the scope, even if `only:` left it out. `evaluator_params` merges key by key, one level deep; every other field is replaced. The metric must be tagged with this scope in the catalogue.
`{name}: {...}`, `name` not in the catalogue	A complete standalone `MetricConfig`; every required field must be present. Always included. Use it to try an evaluator before adding it to the catalogue.

Because a named override always resolves against the whole catalogue, read only: as "start from just these metrics" rather than "never include anything else".

A run narrowing a scope, tightening one metric's threshold, and trying a metric the catalogue does not have yet:

# evaluation/runs/fast_grounding_check.yaml
# yaml-language-server: $schema=run_file.schema.json
name: product_001_fast_grounding_check
predictor_adapter: evaluation.predictor:MyPredictor
scopes: [grounding]
evaluators:
  grounding:
    only: [groundedness]
    groundedness:
      threshold: 4.5        # tighten just this run's threshold
    citation_precision:     # not in the catalogue yet
      framework: builtin_deterministic
      evaluator: CitationPrecision
      threshold: 5.0

This resolves to two metrics for grounding: groundedness from the catalogue with its threshold at 4.5, and citation_precision as a standalone definition.

RunManifest: the resolved run

RunManifest (semantic_evaluation.contracts) is a run fully resolved: the run file's fields (including max_error_fraction, aggregate_by and agreement), with datasets naming every scope's dataset and evaluators.{scope}.metrics spelling out every metric as a MetricConfig. It is what execute_offline_evaluation() runs and what a run's artifacts record.

from pathlib import Path
from semantic_evaluation.orchestration.config import load_resolved_run_manifest

manifest, run_path = load_resolved_run_manifest(
    run_manifest_path=Path("evaluation/runs/regression.yaml"),
    evaluation_manifest_path=Path("evaluation/evaluation_manifest.yaml"),
)

resolve_run_manifest(run_source, pod) does the same for a run file already loaded as a dict and a project manifest already loaded with load_evaluation_manifest(). To see what a run will execute:

eval-dataset resolve-run \
  --run-file evaluation/runs/regression.yaml \
  --manifest evaluation/evaluation_manifest.yaml

It prints the RunManifest as YAML:

name: product_001_regression
predictor_adapter: evaluation.predictor:MyPredictor
scopes: [grounding, routing]
datasets:
  grounding:
    dataset_name: product_001_grounding_v1.0
    fetch_items_page_size: 50
  # ...
evaluators:
  grounding:
    metrics:
      groundedness:
        framework: azure_ai_evaluation
        evaluator: GroundednessEvaluator
        threshold: 4.0
        # ...

The predictor a run names

predictor_adapter names the project's PredictionRunner class as module.path:ClassName, importable from where the run is started, for example evaluation.offline_eval.predictors:SupportWorkflowPredictor. A bare label such as support_workflow is refused when the file is loaded.

execute_offline_evaluation() imports that class. Called without a predictor, it builds one, passing the run's run_config and repeats as keyword arguments to a constructor that takes them, and none to one that takes neither. It then calls the predictor's describe_candidate(), if it has one, and merges the result over run_config; see Configurations reach the agent through the predictor. A project whose predictor needs other arguments (a service URL, a client) builds it from resolve_predictor_class(manifest) and passes it; the runner then checks it is an instance of the class the run names, and refuses a run given any other predictor.

semantic_evaluation.platforms:PlatformPredictor is a predictor the SDK ships for agents on Open WebUI and Copilot Studio; see Platform predictors.

Editor validation

Both files start with a yaml-language-server line naming a JSON Schema in the same directory, which the YAML extension in VS Code uses to check the file as you type:

# yaml-language-server: $schema=evaluation_manifest.schema.json

# yaml-language-server: $schema=run_file.schema.json

eval-dataset schemas writes both schemas from the installed SDK, so they always match the version that will run the files: evaluation/evaluation_manifest.schema.json and evaluation/runs/run_file.schema.json (--dir names another evaluation directory). Commit them, and run it again after upgrading semantic-evaluation. The schemas this repository commits under sdk/semantic_evaluation/schemas/ are the same files, plus run_manifest.schema.json for the resolved form.

DatasetSourceConfig

Controls how dataset items are loaded from Langfuse for one scope.

Field	Type	Default	Description
`dataset_name`	`str`	required	Name of the Langfuse dataset to load items from
`fetch_items_page_size`	`int`	`50`	Items per page when fetching from Langfuse
`input_schema`	`str \\| None`	`None`	Dotted `module.path:ClassName` reference to a Pydantic model. Resolved by `LangfuseDatasetProvider.load_items()` to drive schema-aware unwrap of a single-key-wrapped `input`; see Normalizing single-key wrapped input

MetricConfig

Declares one evaluator and its gating parameters: a catalogue entry (with scopes), or one metric of a resolved run's ScopeEvaluatorConfig.

Field	Type	Default	Description
`framework`	`str`	required	Source library: `"builtin_deterministic"`, `"azure_ai_evaluation"`, `"deepeval"`
`evaluator`	`str`	required	Concrete evaluator name within the framework
`enabled`	`bool`	`True`	Set `false` to skip the metric without removing it from the catalogue
`threshold`	`float \\| None`	`None`	Pass threshold on the normalized 1-to-5 scale
`direction`	`"higher_is_better" \\| "lower_is_better"`	`"higher_is_better"`	Score comparison direction
`model`	`str \\| None`	`None`	Optional evaluator-side model label override
`evaluator_params`	`dict[str, Any]`	`{}`	Framework-specific constructor arguments forwarded to the evaluator

The metrics field on ScopeEvaluatorConfig is a dict[str, MetricConfig] keyed by metric name. That name appears in MetricResult.metric_name and in Langfuse score labels.

See 05-scoring.md for MetricResult, AggregateMetricResult, and the scoring service's normalized scale.

GateConfig

Run-level gating policy applied after all scores are collected.

Field	Type	Default	Description
`mode`	`"all_metrics"`	`"all_metrics"`	Only `all_metrics` is supported: gate passes if every enabled metric passes its threshold
`minimum_item_count`	`int`	`1`	Gate fails if fewer items were evaluated than this value

A metric passes the gate when its mean normalized score over all items, and all repeats, reaches its threshold. A gate means something only on a released dataset ({name}_v{M.m}), because a dev dataset can change between two runs and a score change then says nothing about the agent; 03-datasets-lifecycle.md explains the two states. A run on a dev dataset checks that the pieces work, and is not a regression gate. How to set a threshold for a deterministic check and for an LLM judge is in the scoring service's evaluator catalogue.