Glossary

This glossary defines terms that are local to the SDK package. For repository terms such as pod, shared service, review sync, or offline evaluation, use the repository glossary.

Shared Contract

A shared contract is a data shape or API surface used by more than one package in this workspace. These contracts let the pod and the shared services exchange requests, results, dataset items, and review bindings without importing each other's runtime code.

Shared Client

A shared client is a helper in sdk/semantic_evaluation/ that calls a shared HTTP service by using the shared request and response contracts.

Transport-Neutral Helper

A transport-neutral helper works on normalized Python objects and stays independent of FastAPI wiring, CLI wiring, or pod-specific runtime code.

Step

A traced unit of work implemented as a decorated function. The @step decorator records the function's input, output, and metadata as a Langfuse observation. Nested @step calls form parent–child relationships automatically via OTel context propagation.

Span

A traced unit of work implemented as a context-manager block (rather than a full function). Use span() when the code boundary doesn't map to a single callable — for example, a loop body or a conditional branch within a larger function.

Event

A fire-and-forget trace marker with no duration. Events record point-in-time occurrences (cache hits, fallback triggers, policy decisions) without starting a child span.

Observation

Langfuse's generic term for any node in a trace tree — a step, span, event, or generation. Each observation has a unique ID retrievable via get_observation_id().

TracerProvider

The OpenTelemetry object that owns tracer instances and span processors. setup_tracing() makes Langfuse's internal TracerProvider the global one so all frameworks emit spans into the same trace.

API major

The major version of a shared service's HTTP API, carried in the URL path (/v1). A breaking change adds the next major, and each semantic_evaluation major calls exactly one major per service. See service API versions.

Evaluator Catalogue Response

The Pydantic model returned by ScoringServiceClient.list_evaluators(). It wraps a list of EvaluatorCatalogueEntry objects, each carrying framework, evaluator, scope, raw_scale, and supported_params. This is the SDK-side mirror of the scoring service's internal catalogue.

Evaluator Parameters

Per-metric keyword arguments that a manifest author declares under MetricConfig.evaluator_params. The scoring service forwards these into the evaluator framework's constructor after checking them against the catalogue's supported_params list. Unknown keys cause a hard failure.

PredictionRunner

The formal Protocol in semantic_evaluation.prediction that pod-side predictor adapters implement. It declares one async batch method, predict_items(items: list[EvaluationItem]) -> list[PredictionRecord]. The generic offline-evaluation runner calls it once per repeat with the items of every requested scope; pods choose their own internal concurrency. A run file names the class in predictor_adapter as module.path:ClassName, and the runner builds it, or checks that the predictor it is given is an instance of it.

Candidate

The copy of the agent an evaluation run talks to, never the live one: an assistant or agent whose id carries a prefix the platform configuration names. Its description (platform, candidate, model, delivery, content_sha256) is merged into the run's run_config before the run. See Keys that describe the candidate.

Platform predictor

PlatformPredictor in semantic_evaluation.platforms: a PredictionRunner for an agent on Open WebUI or Copilot Studio. It holds real conversations with a candidate through a Platform, reads the platform's own transcript of each, and builds the PredictionRecords with the help of the project's conversation hook. See Platform predictors.

Conversation hook

The project's half of a platform predictor: a class with conversations(items), next_message(conversation, reply, turn), predictions(conversation, outcome) and describe(run_config). It decides what to ask, how to reply and what an answer predicts, and knows nothing about the platform. See The three parts.

Errored prediction

A PredictionRecord whose error is set because the agent produced no answer to read: no conversation was opened, no transcript arrived in time, or the transcript cannot be read. A wrong or invalid answer is not an error. It is not scored, and the gate fails for a scope whose share of them is above the run file's max_error_fraction. See Errored predictions.

Prediction trace

The Langfuse trace a prediction is linked to in a dataset run and scored on. It is the agent's own trace when the record carries a trace_id; under prediction_traces: create_when_missing the runner writes a thin one for each record without. See Predictions from a runtime without a Langfuse trace.

Dataset run

One published execution of an offline evaluation over a dataset: its item results and its run scores (aggregates, spread, gate_passed). Langfuse shows a dataset run as an experiment, in its UI and API; runs created earlier appear as experiments under their original ids. The score API still names the id dataset_run_id, and so does DatasetRunPublication. Each item is an offline_eval_item span added to the prediction trace. See What publication writes to Langfuse.

Trace metrics

Measurements reported beside the scored metrics (time, tokens, cost, tool calls) and never part of the gate. Declared in a run file as trace_metrics; read from the prediction's trace, or supplied by the predictor in PredictionRecord.measurements. See Trace metrics.

Grouped aggregate

An aggregate of one metric over the items of one group: those whose dataset item metadata attribute (named in a run file's aggregate_by) has one value. It is an AggregateMetricResult with group_by and group_value set, published as {scope}.{metric}.by.{attr}.{value}. It describes and never enters the gate. See Aggregates per group of items.

The share of items on which every repeat gave the same value for the agreement field named in a run file's agreement; a missing field counts as a value of its own (None). See Agreement across repeats.

For one item, the share of its repeats that gave its most frequent value. The mean modal share is the mean of these over the items. An item is unanimous when its modal share is 1 and every repeat has a prediction. See Agreement across repeats.

Whole-run shift

One repeat of a unit (the items sharing the per attribute of agreement.run_shift, for example one source document) whose values differ from the strict majority of the other repeats on at least min_share of the unit's items. A tie among the other repeats is no majority, so it does not count as differing, and a missing field never counts as a value. See Agreement across repeats.

run_label

A free-text experiment identifier that becomes part of the Langfuse run name ({manifest.name}_{run_label}_{timestamp}). Declared as run_label in the YAML run config and overridable per invocation via the pod's --run-label CLI flag. The effective label is resolved as run_label or manifest.run_label — the CLI wins over the YAML default. See also baseline run.

baseline run

An evaluation run marked as a reference point for local-tuning comparisons. Activated by setting baseline: true in the run config YAML. When set:

run_type is set to "baseline" in Langfuse run metadata.
is_baseline: true is added to the run metadata dict.
If no run_label is given, it defaults to "baseline".

Baseline runs appear in Langfuse alongside experiment runs so teams can track regressions against a known good configuration. See also run_label.

Run file

One YAML file under evaluation/runs/ describing one run (the regression gate, an experiment), validated as the RunFile model. It names its scopes, its predictor_adapter and whatever it changes, and inherits each scope's dataset and metrics from evaluation_manifest.yaml. Resolved, it becomes a RunManifest, the form execute_offline_evaluation() runs and eval-dataset resolve-run prints. See 04-run-config.md: Run files.

Manifest schemas

The JSON Schemas of the two files a project writes, generated from their models: evaluation_manifest.schema.json (EvaluationManifestFile) and run_file.schema.json (RunFile). eval-dataset schemas writes them from the installed SDK into evaluation/ and evaluation/runs/, next to the files whose first line names them for in-editor validation in VS Code:

# yaml-language-server: $schema=run_file.schema.json

This repository also commits them, with run_manifest.schema.json for the resolved form, under sdk/semantic_evaluation/schemas/.

Canonical hash

A SHA-256 digest computed by semantic_evaluation.datasets.canonical_hash(item) over the content-relevant fields of an EvaluationItem: input, expected_output, and the metadata Langfuse stores for the item (raw_metadata with the typed fields such as product_id applied over it), with tags removed. Tags are category labels, not content, so two items that differ only in their tags have the same canonical hash and dedupe keeps one of them. Dataset diff compares the tags separately, so a re-tag still counts as a change to push.

Dataset sync

The local-to-Langfuse round-trip implemented by semantic_evaluation.datasets. Core operations: pull downloads a dataset, push upserts new and changed items (never deletes), diff compares the two sides, validate checks structure, dedupe removes content duplicates, promote transitions a dev dataset to an immutable release snapshot, and archive files that release's snapshot in the project's archive directory. Multi-scope flows use SyncConfig, pull_all, and push_all. The eval-dataset CLI wraps the same functions. Langfuse is the operational source of truth for all golden dataset items; archive snapshots (one per released version) are the only local files committed permanently to the pod repository.

See 03-datasets-lifecycle.md for the full workflow.

Coverage map

evaluation/coverage.md in a project: what its offline evaluation covers. One row per behaviour the agent must keep, with the dataset items that prove it and the metrics that check it, the datasets and their releases, and a table naming, for each kind of change to the agent, what to update. It is a project file, not something the SDK reads, and it is updated in the same change that changes the agent's behaviour. See 03-datasets-lifecycle.md: The coverage map.

Dataset diff

A three-way bucketing produced by semantic_evaluation.datasets.diff(local, remote) and returned as a DatasetDiff object with fields added, removed, changed, unchanged. Items are bucketed by item_id; equality within a shared id means the same canonical hash and the same tags; a difference in either puts the item in changed. The changed list contains the local version, since that is what a push would upsert.

PushResult

The return value of semantic_evaluation.datasets.push(). Carries lists of item IDs: created (new items written), updated (existing items overwritten), unchanged (items whose canonical hash and tags already matched the remote), backfilled (a subset of unchanged that had no changed_last_in_dataset_version and got one stamped anyway, so an item added straight to Langfuse still joins the version scheme without needing a real content edit), and would_delete (remote items absent locally, reported for visibility only — never deleted automatically). In dry-run mode the same information is exposed as would_create, would_update, would_backfill, and would_delete, with no writes performed. Every upsert push() performs, in any of these buckets, re-supplies the item's source_trace_id/ source_observation_id so a pull-edit-push cycle never silently drops the item's link back to its originating trace.

SyncConfig

The central configuration object for multi-scope dataset operations. Declares which scopes exist and which Langfuse dataset each maps to (scope_to_dataset), how local files are laid out (Pattern A — one merged file — or Pattern B — one file per scope), and optionally the Pydantic class (or JSON Schema dict) used as input_schema / expected_output_schema when creating or updating Langfuse datasets.

create_on_missing (default False): when True, a missing Langfuse dataset is auto-created on the first push instead of raising a 404. Set it only for the initial bootstrap; after the datasets exist, leave it False so a misspelled name fails loudly rather than silently creating a stray dataset.

See 03-scope-sync.md.

Wrapped input unwrap

The correction normalize_dataset_item() applies when a dataset item's input has the shape produced by Langfuse's own "+ Add to dataset" action on a semantic_tracing-traced observation: a single top-level key, named after the traced function's own argument, wrapping the real payload (e.g. {"request": {...}}). Detected schema-aware, via a registered input_schema, or, absent a schema, by a bare structural heuristic.

See 02-schema-types.md — Normalizing single-key wrapped input.

added_via_langfuse_ui

A derived, read-only property on EvaluationItemMetadata: True when an item's ground truth most likely came directly from Langfuse's own "+ Add to dataset" action on a trace, rather than this SDK's own seed or promote tooling. Computed entirely from existing fields, with no new tagging convention: a populated trace_id (only ever set by an actual traced observation) combined with the absence of ground_truth_status in raw_metadata (unconditionally set by every one of this SDK's own publish paths).

See 03-datasets-lifecycle.md — Curation workflow.

changed_last_in_dataset_version

A per-item metadata label stored in raw_metadata.changed_last_in_dataset_version. Stamped on each item that is created, updated, or backfilled during a push — a backfill covers an otherwise-unchanged item that had no version label yet (see PushResult), so an item added straight to Langfuse still joins the version scheme without needing a real content edit. Once an item carries a version label, it retains that label across later pushes as long as its content doesn't change, regardless of how many subsequent pushes declared higher version labels.

derive_version(items) returns the maximum changed_last_in_dataset_version across a list. pull --version vX.Y filters on this field.

Contrast with dev_version, which reflects the declared push label on the dataset object regardless of whether any items changed.

dev_version

A metadata field on the Langfuse dataset object itself (metadata.dev_version). Stamped by push() on every push, even when all items were already up to date and nothing was written.

pull_all() reads this field after pulling items and uses it as the local file envelope version, so the file's version field always matches the last declared push label — not the max per-item changed_last_in_dataset_version, which can lag when a push finds all items unchanged.

Contrast with changed_last_in_dataset_version, which only advances per item when that item's content actually changes.

evaluation_manifest.yaml

The project manifest: one per project, at evaluation/evaluation_manifest.yaml, validated as the EvaluationManifestFile model, which refuses unknown keys. Contains three top-level blocks:

dataset_sync: the scope-to-dataset mapping every run inherits, the working-copy and archive directories, and optional input_schema / expected_output_schema, consumed by eval-dataset pull-all, push-all and archive. Relative paths resolve against the manifest's directory.
evaluators: the metric catalogue, one CatalogueMetricConfig entry per metric name, each tagged with the scopes that use it. Loaded into the EvaluationManifest model by load_evaluation_manifest() and consumed by resolve_run_manifest() to fill in each run file.
baselines: per-run-name score snapshots written by eval-dataset promote-baseline.

EvaluationManifest

The Pydantic model of what a run resolves against: a project's evaluators catalogue and scope_to_dataset map, returned by load_evaluation_manifest(path) from evaluation_manifest.yaml. Fields: evaluators: dict[str, CatalogueMetricConfig] and scope_to_dataset: dict[scope, str]. The whole file is EvaluationManifestFile, from load_evaluation_manifest_file(path). Consumed by resolve_run_manifest() to fill in whatever a run file leaves unspecified.

See 04-run-config.md: The project manifest.

CatalogueMetricConfig

A MetricConfig plus one extra field, scopes: list[str], naming every scope that uses this metric. This is the entry shape of an EvaluationManifest's evaluators dict — keyed by metric name rather than by scope, so a metric shared across scopes (a groundedness check used in both grounding and playbook_adherence, for example) is declared once instead of once per scope.

See 04-run-config.md — CatalogueMetricConfig.

baselines block

The baselines: section of evaluation_manifest.yaml. Stores named snapshots of aggregate evaluation scores, keyed by the full Langfuse run name (e.g. pathfinder-local-all-scopes_baseline_20260702T143200Z). Written and updated by eval-dataset promote-baseline. Pods commit the manifest after promoting so there is an auditable record linking each baseline snapshot to the code change that produced it. Run configs that set baseline: true produce runs whose names can be promoted into this block.

SchemaChangeLevel

An enum returned inside a SchemaChangeReport that classifies how much the field structure of a dataset changed between two versions. Values:

Level	Meaning
`NONE`	Identical field sets — no structural change
`MINOR`	New field present in some but not all items — the field is absent from at least one item, making it optional across the dataset.
`MAJOR`	Field removed, or new field present in all items (effectively required)

Used by promote() to enforce appropriate version bumps.

SchemaChangeReport

A frozen dataclass returned by schema_change_report(new_items, old_items). Fields: level (SchemaChangeLevel), removed_fields (list of "input.x" or "expected_output.x" key strings), added_required_fields, and added_optional_fields. Fields are prefixed with "input." or "expected_output." to distinguish the two sub-schemas.

RAGAgentInput

Pydantic model for the RAG-Agent schema type's input fields. Inherits messages (required), context, and document_ids from RAGInput and adds available_tools: list[str] | None. Use when a system both retrieves context and invokes tools. Export path: semantic_evaluation.schemas.RAGAgentInput.

RAGAgentExpectedOutput

Pydantic model for the RAG-Agent schema type's expected-output fields. Inherits response (required), citations, and retrieval_ground_truth from RAGExpectedOutput and adds tool_calls: list[ToolCall] | None and handoff_to: str | None. Export path: semantic_evaluation.schemas.RAGAgentExpectedOutput.

VersionBumpError

An exception raised by promote() when the proposed version number is inconsistent with the detected SchemaChangeLevel. The exception carries three attributes: change_level, proposed_version, and previous_version. Pass force=True to promote() to override schema-rule blocks; same-or-lower version is always blocked regardless of force.

See 03-datasets-lifecycle.md — Version-bump enforcement.

PromoteResult

A frozen dataclass returned by a successful promote() call. Fields: dev_name (source), release_name (the newly created dataset name), version (the M.m string), items_promoted (count), schema_change (SchemaChangeLevel), dry_run (bool), and push_result (PushResult or None in dry-run mode).

Trace reader

The read-side helpers in semantic_tracing.trace_reader: fetch_trace, find_observations, as_dict, coerce_text. They pull structured information back out of a finished Langfuse trace so pod predictors can compose pod-specific extractors on top of one shared polling + coercion implementation. The inverse direction of the write-side decorators (@step, span, event) provided by the semantic-tracing package.

QueueToDatasetBinding

The declarative recipe stored by the review-sync service for one (annotation queue, golden dataset, scope) triple. A binding carries a ReviewPolicy, a DatasetTarget, an input_projection and expected_output_projection (each a Projection of ProjectionField entries that source values via ValueSelector), a MetadataMapping, and a QueueCleanupConfig. Pods register one binding per scope and the service replays it on every sync run.

See 06-review-sync.md for full field reference and examples.

ReviewPolicy

Binding-local promotion policy for normalized human-review scores. A reviewed record is only promoted when every score listed in required_score_names is present and within the [normalized_score_min, normalized_score_max] range. When any score falls at or below low_score_correction_threshold, the reviewer must supply an explicit corrected output for the record to be promotable.

See 06-review-sync.md — ReviewPolicy.

Projection

The shape contract for one side of a QueueToDatasetBinding — either the dataset record's input or its expected_output. Each Projection is an ordered list of ProjectionField entries; each ProjectionField declares a dataset_field (the key on the resulting record) and one or more ValueSelector entries (the source values, evaluated as a first-non-null-wins fallback chain). Splitting projection from selection keeps the dataset's schema visible in one place even when the underlying queue payload evolves.

ValueSelector

One source of a value pulled out of a reviewed queue item. Provides exactly one of queue_item_path (dotted lookup into the payload) or literal (constant), plus an optional coerce (ValueCoercion) that casts the raw value before it lands in the dataset. Multiple selectors on the same ProjectionField form a fallback chain that survives queue-payload renames or partial migrations.

ValueCoercion

An enum that controls type coercion applied to a value read from a Langfuse annotation-queue payload before it is written into a dataset field. Langfuse stores queue payloads loosely (a reviewer may type a number as a string, a boolean as "true", etc.); the coercion runs at the boundary so the dataset record has strict types. Values: identity (no change), string, boolean, integer, number.

ReviewSyncServiceClient

The HTTP client in semantic_evaluation.orchestration.review_sync that pods use to register QueueToDatasetBinding objects and trigger sync jobs against the review_sync_app shared service. Key methods: upsert_binding, validate_binding, run_sync, get_job. All methods are async.

See 06-review-sync.md — Managing bindings.

default_dataset_name(product_id, scope, version, *, schema_type=None)

Helper that returns a dataset name following the naming convention in 03-datasets-lifecycle.md:

With schema_type: "{product_id}_{schema_type}_{scope}_v{version}"
Without schema_type: "{product_id}_{scope}_v{version}"

Use schema_type when the dataset belongs to a recognized schema type ("rag", "agent", "rag_agent", etc.). Omit it for QA datasets where the schema type adds no extra fields. 03-datasets-lifecycle.md is the source of truth for the naming convention and the list of schema types.

Exported from semantic_evaluation.orchestration.review_sync.