Repository Glossary

This glossary defines terms that are used across more than one package in this workspace. Package-specific terms should be defined in the nearest local docs/glossary.md and link back here when they reuse these definitions.

Pod

A pod is one cross-functional group that builds one product. The pod therefore has code to be semantically evaluated, the product-specific implementations that owns workflow code, predictor adapters, manifests, and local service entrypoints. In this workspace, reference_pod_implementations/agent-framework-pod/ and reference_pod_implementations/langchain-pod/ are the two concrete pods — equivalent implementations of the same support-assistant product on different agent frameworks, not two different products.

A pod is not the shared scoring service, the shared review-sync service, or the shared SDK package.

Shared Service

A shared service is an HTTP service used by one or more pods instead of being implemented inside a pod. In this repository, review_sync_app/ and scoring_app/ are the shared services.

The term describes who consumes the service. It does not imply one deployment model. 'Shared' means shared within the ML and Data Engineering chapter for pods using the semantic evaluation system.

API Major

The major version of a shared service's HTTP API, carried in the URL path (/v1). A breaking change to the API adds the next major beside the current one. Each semantic_evaluation major calls exactly one API major per service.

Review Sync

Review sync is the step that reads completed human reviews from a Langfuse annotation queue, normalizes the reviewed data, maps it into a target schema, and writes approved records into a Langfuse golden dataset.

It does not include reviewer assignment, queue authoring, or queue cleanup in the Langfuse UI.

Reviewer Comments

Reviewer comments are the ordered free-text notes extracted from score-level comment fields on completed annotation and correction scores in Langfuse.

Object-level comments (comments on the trace or observation itself in the Langfuse UI) are not included. Those are stored in raw artifacts for debugging but do not flow into promoted dataset metadata.

Review sync copies reviewer comments unchanged into the promoted golden-dataset record metadata so later curators can see why a human reviewer approved or corrected the output.

Reviewer comments are curated dataset metadata. They are not a substitute for required review scores and are not a binding-configurable selector surface.

Human Review Scores

Human review scores are the normalized score objects preserved in promoted golden-dataset record metadata. Each score object contains the numeric value, the optional score-level comment, the Langfuse score ID, and the data type.

Review sync copies these scores unchanged from the Langfuse annotation into promoted dataset metadata so later consumers can see which scores the reviewer assigned and why. The policy evaluator uses the numeric value from these objects to decide whether promotion is allowed.

The review policy evaluator accepts both the normalized dict format (with value, comment, score_id, and data_type keys) and bare numeric values when checking score thresholds.

Annotation Queue

An annotation queue is the reviewer inbox in Langfuse. Reviewers add traces or observations to the queue and complete their assessments there.

An annotation queue is review input. It is not the long-term curated dataset used for trusted evaluation runs.

Golden Dataset

A golden dataset is a curated Langfuse dataset used as trusted evaluation input. In this repository, golden datasets are the output of human review and the input to offline evaluation.

A seeded dataset or raw annotation queue is not automatically a golden dataset.

Offline Evaluation

Offline evaluation is a non-live evaluation run that loads a trusted dataset, generates predictions, sends those predictions to the scoring service, writes local artifacts, and can publish its results back to Langfuse as an experiment (a dataset run).

It is separate from live tracing and separate from queue-based human review.

Dataset Scope

A dataset scope is the label a pod uses to distinguish evaluation slices such as routing or end_to_end. The shared SDK carries scope names, but it does not define their business meaning.

The meaning of each scope stays with the pod that owns it.

Eval Case

An eval case is the structured evaluation example derived from one traced workflow run. Before entering a dataset, it is the local object that the trace-to-review flow builds and optionally writes to JSONL or publishes into Langfuse. Once published or promoted into a dataset, it becomes a dataset item.

The code-level contract is EvalCaseRecord in semantic_evaluation.

Trace-To-Review Flow

The trace-to-review flow is the path from a traced workflow execution to queued eval cases ready for human review. It runs the workflow, derives scope-specific eval cases, and publishes them as seeded Langfuse dataset items or submits them to annotation queues.

In this workspace, reference_pod_implementations/{agent-framework-pod,langchain-pod}/evaluation/trace_to_review/ implements the pod-side entrypoint.

Review-To-Dataset Sync

Review-to-dataset sync is the path from completed human reviews to promoted golden-dataset records. It reads completed queue items, enforces the review policy, maps reviewed data into the target schema, and writes approved records into the golden dataset.

The shared service implementation lives in review_sync_app/. The pod-side CLI trigger lives in reference_pod_implementations/{agent-framework-pod,langchain-pod}/evaluation/review_to_dataset/.

Evaluator Framework

A label that identifies the source library or origin of an evaluator implementation. Current values:

Framework	Source
builtin	Exact-match routing evaluators implemented inside scoring_app/
builtin_deterministic	Deterministic pattern-matching evaluators for tool calls, tool call ordering, tool outputs, URL allow-lists, message content patterns, router-mode assertions, and policy ID validation
deepeval	Evaluators backed by the DeepEval library
azure_ai_evaluation	Evaluators backed by the Azure AI Evaluation SDK
custom_rubric	Pod-defined LLM-judge evaluators in two forms: inline (full rubric definition in the manifest) and {name} (rubric committed to scoring_service/custom_evaluators/accepted/ by the eval core team and referenced by name from any pod manifest)
decision_judge	Microsoft Decision-1 evaluator for one fixed-option judgement about free text, configured inline in the manifest

Manifests must spell the framework name exactly; unknown framework names cause the scoring service to reject the request.

The framework name is paired with an evaluator name in the manifest. The (framework, evaluator) pair is the dispatch key the scoring service uses to build one evaluator instance per metric.

Evaluator Catalogue

The single source of truth for which (framework, evaluator) pairs the scoring service supports. Each catalogue entry declares the framework, evaluator class name, scope, raw scale, supported parameter names, and factory. The catalogue is exposed to callers via GET /v1/evaluators.

The catalogue lives in scoring_app/scoring_service/scoring/catalogue.py. A manifest that names a pair the catalogue does not know causes a hard failure with a message that lists every supported pair.

Normalized Score

The 1-to-5 floating-point value every evaluator emits as normalized_score on each MetricResult. It is the only scale used for thresholds, gate decisions, and aggregate scores in this repository. Manifest thresholds, the gate configuration, and the values published to Langfuse all live on this scale.

The normalized score is not the framework-native value. Each evaluator maps its native output onto 1-to-5 once and records the result.

Raw Score

The framework-native numeric score an evaluator received before normalization. It is preserved on each MetricResult as raw_score together with raw_scale (for example "0-1" or "binary") so reviewers can see what the underlying framework actually returned. A categorical evaluator has no native numeric score, so it sets raw_score to null and keeps its selected label and probabilities in details.

The raw score is informational only. Threshold comparisons, gate evaluation, and aggregates use the normalized score, not the raw score.

Scoring Job

A scoring job is one batch of predictions submitted to the scoring service for evaluation. Submitting a ScoreRequest (POST /v1/score-jobs) does not score it inline; it returns a job ID immediately, and the batch runs as a background task, scoring items concurrently up to a configured limit. Callers poll GET /v1/score-jobs/{job_id} for status and, once the job reaches a terminal state, the result.

A job moves through four states: pending (accepted, not yet started), running (evaluating; completed_evaluations/total_evaluations report progress), completed (a ScoreResponse is available), and failed (an error message is available). semantic_evaluation's ScoringServiceClient exposes submit_score()/get_score_job() for direct control, and a convenience score() that submits and polls to completion, so most callers never handle job IDs themselves.

This shape exists because scoring a batch against a reasoning-tier LLM judge can legitimately take longer than any single HTTP connection should stay open. Decoupling the client's wait from server-side processing time, rather than raising a client-side timeout, is what actually fixes that.

RACI

RACI is the Responsible / Accountable / Consulted / Informed framework used to assign ownership for each activity in the Product Owner adoption operating model, across both the data-first and trace-to-data feedback-capture modes. See docs/adoption/po_adoption_operating_model.md for the actual RACI table.

RACI is a general project-management convention, not something specific to this codebase — defined here because not every reader will already know the acronym.

Data-First

Data-first is the practice of agreeing evaluation criteria — and ideally a concrete example — alongside building a new product, proof of concept, or feature, instead of only reacting to it once it already exists. See docs/adoption/po_adoption_operating_model.md for how this fits the wider Product Owner adoption model.

Data-first does not prescribe who drafts the criteria or the example: a Product Owner and engineer may co-create both together, split the work, or draft and review in either order.

Data-first is the counterpart to trace-to-data: data-first applies before or while something new is being built, trace-to-data applies once it is already running.

Trace-To-Data

Trace-to-data is converting a production trace directly into a dataset entry — through a human-review step when domain judgment is needed, without one when it isn't — instead of authoring dataset examples from scratch. See docs/adoption/po_adoption_operating_model.md.

Trace-to-data is not the same thing as an annotation queue: the annotation queue is where the human-review step happens when one is needed. Trace-to-data is the broader practice of using production traces as the source material for growing a golden dataset in the first place, whether or not that particular trace needs human review.

Trace-to-data is the counterpart to data-first: data-first applies before or while something new is being built, trace-to-data applies once it is already running.