Glossary

Terms used throughout this workshop. Repository-wide terms already defined in the repository glossary are linked from here rather than redefined.

Dataset scope

One named evaluation "track" within a pod, for example intent_classification or reply_generation. Each scope maps to exactly one Langfuse dataset per lifecycle state (dev/release) and one entry in a run manifest's scopes/datasets/evaluators maps. A scope is not the same as a dataset name: the scope is the stable logical identifier; the dataset name it resolves to can change (e.g. across a product rename) without the scope itself changing.

Dev dataset

A mutable Langfuse dataset, named {product_id}_{scope}_dev, that grows over time as seed items, human-reviewed items, and manual edits are pushed to it. Contrast with a release dataset. See Module 08.

Release dataset

An immutable snapshot of a dev dataset's state at one point in time, created by promote(). Used to pin exactly what a CI gate or a reported evaluation run measured against, so a later change to the dev dataset can't silently change what an already-published result means. See Module 08.

Local capture

An in-process cache of the current trace's observations, populated by setup_tracing(..., capture_local_traces=True) and read synchronously via get_local_trace(trace_id), no network round trip to Langfuse required. Used whenever a prediction needs a value that a workflow's own return value doesn't carry, such as a tool call's raw result. See Module 02 §2.3 and Module 05.

Observation

One recorded unit of work in a trace (a span, in OpenTelemetry terms) with a name, as_type (chain/agent/generation/tool/event/span/trace), input, output, and observation_id. A trace is a tree of observations rooted at one top-level trace-type observation. span is the generic type used when none of the more specific ones applies; Langfuse's API also reports the LangChain scaffold's @step observations as SPAN where the Agent Framework scaffold's come back as AGENT, so you meet it on either path.

EvaluationItem

The shared contract for one golden-dataset item, regardless of where it came from (seed file, human review, or a promoted release): item_id, scope, dataset_name, input, expected_output, metadata. See Module 08 §8.1.

PredictionRecord

The shared contract for one workflow response, shaped for scoring: item_id, scope, prediction (a dict of whatever fields the metrics need to read), trace_id, observation_id, plus the original input/expected_output for convenience. Produced by a PredictionRunner's predict_items(). See Module 05.

Run manifest

What defines one evaluation run (RunManifest): its name, predictor_adapter, scopes, the dataset and metrics of each scope, and its gate. A pod writes it as two YAML files: a run file under evaluation/runs/ naming the run's scopes and gate, which inherits each scope's dataset and metrics from the project manifest, evaluation/evaluation_manifest.yaml. See Module 04.

Gate

The run-level pass/fail policy attached to a run manifest. This workshop uses mode: "all_metrics": every enabled metric across every scope must meet its own threshold for the run to pass. See Module 07 §7.3.

product_id

A short, stable identifier for which pod a dataset or evaluation run belongs to, not a description of what the pod's application does. This workshop's fictional codename is northwind; the demo application itself is described in prose as the "support assistant," a distinction the {product_id}_{scope}_{state} naming convention depends on. See Module 01.

{product_id}_{scope}_dev naming convention

The dataset-naming pattern this workshop follows for custom-field datasets (no canonical schema type in play; see Module 08 §8.5): {product_id}_{scope}_dev for the mutable working dataset, e.g. northwind_reply_generation_dev. A dataset using one of the four canonical schema types additionally includes the schema type in its name ({product_id}_{schema_type}_{scope}_{state}), not used by this workshop's own datasets, which are custom-field.

Tool-call / local-capture exercise pattern

The pedagogical pattern this workshop uses to make local capture concrete: a workflow step (lookup_account_status) calls a tool whose result is never returned to the caller, only recorded as its own traced observation. Recovering that result for scoring requires reading the trace via get_local_trace() + find_observations(), not just reading the workflow's response, demonstrating that a prediction's available data isn't limited to a function's return value. See Module 05.