Scoring — overview

scoring_app/ is the shared scoring service for the workspace. It is a single FastAPI application (scoring_service/) that scores already-generated predictions: it runs evaluator metrics, aggregates the results, evaluates a pass/fail gate, and exposes an evaluator-discovery endpoint so manifest authors can find out what metrics exist before they write a run manifest.

Definition: release-facing scoring Release-facing scoring is the scoring used to support keep, change, or promote decisions for workflow changes. In this repository, it means calling this shared service over HTTP, not running ad hoc local scoring inside a pod. See release-facing scoring.

The problem this service solves

Every pod that evaluates a workflow needs the same handful of things: a place to run DeepEval and Azure AI Evaluation metrics, a fixed-option Microsoft Decision-1 evaluator for free-text judgements, a way to run deterministic checks without paying for an LLM call, a way to define a pod-specific LLM-judge rubric, and a gate decision at the end. Building that once per pod means duplicated evaluator wiring, duplicated provider credentials scattered across pod codebases, and no shared place to add a new metric.

scoring_app/ centralizes all of it behind one HTTP boundary. Pods send already-generated evaluation items and predictions; the service decides how to score them and returns metrics, aggregates, and a gate result. Evaluator backend credentials (Azure OpenAI or Foundry Decision endpoints and keys) live only inside this service's connection profiles — callers never see or send them.

Where evaluator definitions live

The service is pod-agnostic: it never stores which metrics a pod uses, at what threshold, or with what parameters. Every metric definition — MetricConfig's (framework, evaluator) pair, threshold, evaluator_params, and so on — travels inline inside the evaluators field of each ScoreRequest, built pod-side from whatever run manifest or catalogue that pod maintains (see Run configuration: RunManifest for how a pod builds one). This is true for every framework, not only custom_rubric — a deepeval/GEval metric's criteria and threshold are just as request-local as a custom_rubric/inline rubric.

What this service registers instead is the evaluator catalogue: a fixed set of generic (framework, evaluator) types — for example builtin_deterministic/ToolCallMatch — each with a factory and a supported_params allow-list. The catalogue says which evaluator types exist and what parameters they accept. It says nothing about which pod uses them, under what metric name, or at what threshold.

The one thing that is both server-side and shared across pods is a named custom_rubric/{name} entry: a rubric committed to custom_evaluators/accepted/ after review becomes a catalogue entry any pod can reference by name instead of repeating the full rubric definition inline on every request. See 03-custom-rubric-and-registration.md.

Current responsibilities

Area	Owned here
HTTP scoring runtime	FastAPI app, request logging, health route, scoring-job submission and status via POST /v1/score-jobs/GET /v1/score-jobs/{job_id}
Evaluator catalogue and discovery	Catalogue-driven (framework, evaluator) dispatch, GET /v1/evaluators
Evaluator dispatch and execution	One evaluator instance per logical metric, built through framework-specific factories; built-in routing and deterministic checks, DeepEval, Azure AI Evaluation, custom-rubric, and Decision-1 judge metrics, run concurrently up to OFFLINE_SCORING_MAX_CONCURRENT_JUDGE_CALLS per job
Score normalization	Linear mapping of every framework-native score onto the 1-to-5 scale
Evaluator parameter validation	evaluator_params keys are checked against the catalogue's supported_params before construction
Aggregation and gating	Mean/pass-rate aggregation per (scope, metric) pair and gate evaluation on the normalized scale
Connection selection	Service-owned named scoring connection profiles loaded from a local JSON file, resolved via Azure Key Vault when configured
Pod-driven rubric registration	Reviewing and accepting named custom_rubric evaluators submitted by pods

Current non-responsibilities

Area	Owned elsewhere
Shared request/response contracts (ScoreRequest, MetricConfig, MetricResult, ...)	sdk/semantic_evaluation/
The HTTP client that calls this service	sdk/semantic_evaluation/ (ScoringServiceClient)
Predictor execution — running the workflow under test	reference_pod_implementations/{agent-framework-pod,langchain-pod}/ (or any other pod)
Dataset loading	Pod-side runners through semantic_evaluation
Review-sync execution	review_sync_app/
Langfuse dataset-run publication	The shared offline-evaluation runner in sdk/semantic_evaluation/

The service does not run the system under test. It only evaluates the items and predictions a caller sends to it.

Architecture

graph TD
  subgraph Caller["Pod side caller"]
    A["Submit ScoreRequest"]
    P["Poll for job status"]
  end
  subgraph Service["scoring_app scoring_service"]
    B["FastAPI app"]
    J["Job store shared state mount"]
    C["Connection profile registry"]
    D["Evaluator catalogue"]
    E["Evaluator executor concurrent judge calls"]
    F["Aggregation and gate"]
  end
  subgraph Shared["sdk semantic_evaluation"]
    G["Shared scoring contracts"]
  end
  A --> G
  G --> B
  B -->|"pending record"| J
  B --> D
  D --> E
  C --> E
  E --> F
  F -->|"completed record"| J
  P --> B
  B -->|"reads record"| J
  style Caller fill:#e9ebec,stroke:#273943,stroke-width:2px
  style Service fill:#e5f4fd,stroke:#0098eb,stroke-width:2px
  style Shared fill:#e7eeec,stroke:#125a40,stroke-width:2px

Submitting a ScoreRequest (items, predictions, per-scope metric configuration, and gate configuration) creates a pending scoring job and returns immediately. In the background, the app looks up each requested (framework, evaluator) pair in the catalogue, which builds one evaluator instance per metric — resolving a connection profile first if the metric needs one — and runs the resulting judge calls concurrently. Every evaluator returns results on the normalized 1-to-5 scale; those results are aggregated per (scope, metric) pair and fed into the gate, which decides pass/fail for the whole run. The finished record is written to the job store, which the caller polls for status and, once complete, the result.

Module map

Path	Purpose
scoring_service/app.py	FastAPI app: lifespan, request-logging middleware (logs api_version and user_agent, adds retirement headers), /health, and the v1 router mounted under /v1 and unversioned
scoring_service/api/v1.py	Every route of API v1 and the API-key dependency
scoring_service/api/versioning.py	SUPPORTED_MAJORS, api_version_for_path(), and RETIREMENT_SCHEDULE with the Deprecation/Sunset/Link headers it produces
scoring_service/openapi_snapshot.py	Writes openapi/<major>.json; the root test gate fails when the committed file is stale, and the PR pipeline diffs it for breaking changes
scoring_service/main.py	Local entrypoint (uv run python scoring_service/main.py)
scoring_service/service_config.py	Environment-backed settings, .env loading, profile-registry construction
scoring_service/scoring/catalogue.py	The evaluator catalogue — every supported (framework, evaluator) pair and its factory
scoring_service/scoring/profiles.py	Azure OpenAI and Foundry Decision profile models plus ScoringProfileRegistry (registration, backend assignment, JSON persistence)
scoring_service/scoring/evaluator_requests.py	Pod-driven rubric submission model and the pending-submission file registry
scoring_service/scoring/evaluators/	One module per evaluator family: DeepEval, Azure AI Evaluation, custom rubric, Decision-1, routing exact-match, and the nine builtin_deterministic pattern matchers
scoring_service/scoring/gating/	Aggregation (aggregate_metric_results) and gate evaluation (build_gate_result)
scoring_service/scoring/executor/	prepare_score_request() (fast validation and evaluator construction) and run_prepared_score_request() (concurrent judge-call execution, aggregation, and gating) — score_request() composes both for any caller that wants the all-in-one behavior
scoring_service/scoring/jobs.py	Scoring-job persistence and background execution: creates and updates job records under OFFLINE_SCORING_STATE_DIR/jobs/, and runs the background task POST /v1/score-jobs dispatches
scoring_service/custom_evaluators/accepted/	Committed accepted rubric JSON files, loaded at startup as custom_rubric/{name} catalogue entries
scoring_service/tests/	Service test suite — see 05-testing.md

Design rules

• Every MetricConfig a pod uses travels inline in its ScoreRequest, built pod-side; the catalogue only fixes generic evaluator types, never a pod's specific metrics. See Where evaluator definitions live.
• Provider credentials never travel inside a scoring request. They live only in registered connection profiles on the service side.
• The catalogue is the single source of truth for evaluator dispatch. Adding a metric that belongs to an already-supported framework needs one catalogue entry — no new adapter code.
• Every evaluator normalizes its output onto the same 1-to-5 scale before aggregation or gating ever sees a value. Thresholds, gate decisions, and aggregates always operate on that normalized scale, never the framework-native one. See normalized score.
• Rubric definitions committed to custom_evaluators/accepted/ become named custom_rubric/{name} catalogue entries automatically at startup — no code change is needed to add one.
• Deterministic checks (builtin, builtin_deterministic) never call an LLM and never require a connection profile, so routing-only evaluation always works with an empty profile registry.

Reading order

File	What it covers
getting-started.md	Install, run locally, first scoring-job submission, first connection-profile registration
00-scoring-overview.md ← you are here	What this service is, why it exists, boundary, architecture, module map
01-evaluator-catalogue.md	The catalogue concept, every framework family, every catalogue entry, score normalization, aggregation and gating
02-connection-profiles.md	The connection-profile model, registration, backend keys, resolution and failure behavior
03-custom-rubric-and-registration.md	Inline vs. named custom rubrics, the pod-driven registration lifecycle
03-decision-evaluator.md	Fixed-option Microsoft Decision-1 questions, score mappings, and result semantics
04-api-reference.md	The full HTTP API surface
05-testing.md	Test-suite layers and the dependency-upgrade workflow
glossary.md	Definitions for terms local to this service