Module 04: run manifest

This module covers the two YAML files that configure an evaluation: the project manifest, which declares every metric and dataset once, and the run file, which names one run's scopes and inherits the rest. Together they resolve to a RunManifest: which datasets, evaluators, thresholds, and gate rules define one evaluation run. Everything in Modules 05 to 07 executes against a run you understand by the end of this one.

Every file path in this module is relative to your scaffold pod directory, the one where you ran uv sync in Module 01 §1.2.

4.1 Inspect the two files

evaluation/
  evaluation_manifest.yaml          the project manifest: metrics and datasets
  evaluation_manifest.schema.json   its JSON Schema, for the editor
  runs/
    workshop_local.yaml             the run file this workshop runs
    run_file.schema.json            its JSON Schema

Open evaluation/runs/workshop_local.yaml first:

# yaml-language-server: $schema=run_file.schema.json
name: workshop_local
description: Workshop offline-evaluation run over both support-assistant scopes.
predictor_adapter: evaluation.offline_eval.predictors:SupportWorkflowPredictor
scopes:
- intent_classification
- reply_generation
gate:
  mode: all_metrics
  minimum_item_count: 1

Field	Meaning
`name`	Identifies this run: becomes part of the Langfuse dataset-run name and the output folder path
`predictor_adapter`	The predictor class the run uses, as `module.path:ClassName` (Module 05)
`scopes`	Which evaluation scopes this run covers
`gate`	Run-level pass/fail policy (4.3)

Everything else comes from evaluation/evaluation_manifest.yaml:

Block	Meaning
`evaluators`	Every metric this pod uses, each listing the `scopes` it scores (4.2)
`dataset_sync.scope_to_dataset`	Which Langfuse dataset backs each scope
`dataset_sync.scoped_dir`, `archive_dir`	Where `eval-dataset` keeps working copies and release snapshots (Module 08)
`baselines`	Scores promoted from a reference run; empty until you promote one

A run file can also replace a scope's dataset, narrow a scope to some of its metrics, or change one metric's threshold for that run only, without touching the project manifest. To see exactly what a run will execute, resolve it:

uv run eval-dataset resolve-run \
  --run-file evaluation/runs/workshop_local.yaml \
  --manifest evaluation/evaluation_manifest.yaml

It prints the resolved RunManifest, with each scope's dataset and metrics spelled out.

The first line of each file names its JSON Schema, so VS Code (with the YAML extension) checks the file as you type: an unknown key, a missing field, or a predictor_adapter that is not module.path:ClassName is underlined before anything runs.

4.2 Anatomy of one metric

Each entry under evaluators in the project manifest is one metric, keyed by its name:

Field	Required	Meaning
`framework`	Yes	Which scoring-service framework handles this metric: `deepeval`, `azure_ai_evaluation`, `builtin_deterministic`, or `custom_rubric`
`evaluator`	Yes	The evaluator class name within that framework, e.g. `GEval`, `StructuredFieldMatch`, `inline`
`scopes`	Yes	Every scope this metric scores
`enabled`	No (default `true`)	Set `false` to keep a metric defined but skip running it
`threshold`	No	Pass/fail cutoff on the normalized 1-to-5 scale, regardless of the framework's native scale
`direction`	No (default `higher_is_better`)	`higher_is_better` or `lower_is_better`
`evaluator_params`	No	Framework-specific constructor arguments, forwarded verbatim to the evaluator class. The scoring service's catalogue declares which parameter names each evaluator accepts; an unknown key is rejected, not silently ignored

Two worked examples from the current manifest: one LLM-judge metric, one deterministic metric:

evaluators:
  intent_correctness:
    framework: deepeval
    evaluator: GEval
    threshold: 3.8
    evaluator_params:
      name: IntentCorrectness
      criteria: >-
        Score 1.0 if the predicted intent exactly matches the expected intent. ...
      evaluation_params:
      - ACTUAL_OUTPUT
      - EXPECTED_OUTPUT
      input_key: email_body
      actual_output_key: intent
      expected_output_key: intent
    scopes:
    - intent_classification
  account_status_match:
    framework: builtin_deterministic
    evaluator: StructuredFieldMatch
    threshold: 5.0
    evaluator_params:
      field: account_status
      expected_field: expected_account_status
    scopes:
    - reply_generation

StructuredFieldMatch is a generic deterministic evaluator: it compares prediction.prediction[field] against item.expected_output[expected_field] for exact equality (expected_field defaults to field when both sides use the same key). Because a match/mismatch only ever produces normalized score 5.0/1.0, threshold: 5.0 means only an exact match passes. A 3.0 threshold would accept nothing, since there is no partial-credit value this evaluator can produce.

The two pods' manifests aren't identical, and that's expected

langchain-pod's manifest adds a third reply_generation metric, answer_relevancy (DeepEval's AnswerRelevancyMetric), that agent-framework-pod's does not define. The project manifest is pod-local configuration, not a fixed shared shape; a pod adds, removes, or tunes metrics independently of any other pod's manifest, as long as each metric resolves against the scoring service's evaluator catalogue.

4.3 The gate

gate:
  mode: all_metrics
  minimum_item_count: 1

mode: all_metrics means every enabled metric across every scope must meet its own threshold for the run to pass. One failing metric fails the whole run. minimum_item_count guards against a false-positive pass on an accidentally-empty dataset (e.g. a typo'd dataset name that resolves to zero items).

4.4 Hands-on: extend the manifest

Pick any existing catalogue evaluator you have not yet used in this manifest (check scoring_app/docs/01-evaluator-catalogue.md for the full list) and add it as a new metric under evaluators in evaluation/evaluation_manifest.yaml, with the scope it should score in scopes. A builtin_deterministic evaluator is the fastest to verify, since its outcome is a plain equality check rather than an LLM judgment. Run eval-dataset resolve-run (4.1) and check the metric appears under its scope, then re-run the CLI (Module 07) and confirm your new metric appears in aggregate_metrics.json.

Continue to Module 05.