Scoring

The scoring service evaluates predictions against golden dataset items and returns per-item metric results, aggregates, and a gate verdict. The SDK provides the request/response contracts and an HTTP client.

Normalized 1-to-5 scale

All scoring-service evaluators map their framework-native output onto a common 1-to-5 normalized scale before returning results:

Scale anchor	Value
Minimum (worst)	1.0
Maximum (best)	5.0

threshold in MetricConfig is always expressed on this scale, regardless of the evaluator framework's native range. Example mappings from native to normalized:

`raw_scale`	Native range	Normalized result
`"1-5"`	1.0 – 5.0	Passed through unchanged
`"0-1"`	0.0 – 1.0	Linearly scaled: 0.0 → 1.0, 1.0 → 5.0
`"binary"`	0 or 1	0 → 1.0, 1 → 5.0

MetricResult.normalized_score is comparable across evaluator frameworks. MetricResult.raw_score and MetricResult.raw_scale carry the framework-native value and scale tag for debugging and traceability.

MetricConfig

Declares one evaluator within a ScopeEvaluatorConfig.metrics dict. See 04-run-config.md — MetricConfig for the full field reference.

MetricResult

Per-item metric result returned by the scoring service. One MetricResult is produced per (item_id, repeat, metric_name): once per item, or once per repeat of it when the run sets repeats.

A run's trace metrics are MetricResults too, computed by the runner rather than the scoring service. They are measurements, not scores: normalized_score, threshold and passed are None, and the value is raw_score in the unit raw_scale names (seconds, tokens, usd, count).

Field	Type	Description
`item_id`	`str`	Links the result back to its source `EvaluationItem`
`scope`	`str`	Scope the item belongs to
`metric_name`	`str`	The metric key from `ScopeEvaluatorConfig.metrics`
`repeat`	`int \\| None`	Which repeat of the item the scored prediction was, from 1; the scoring service copies it from `PredictionRecord.repeat`
`normalized_score`	`float \\| None`	Score on the 1-to-5 scale; `None` if scoring failed
`justification`	`str \\| None`	Reviewer-facing explanation from the evaluator
`raw_score`	`float \\| None`	Framework-native score before normalization
`raw_scale`	`str \\| None`	Tag describing the native scale (e.g. `"binary"`, `"0-1"`, `"1-5"`)
`threshold`	`float \\| None`	Pass threshold (copied from `MetricConfig`, on 1-to-5 scale)
`direction`	`"higher_is_better" \\| "lower_is_better"`	Inherited from `MetricConfig`
`passed`	`bool \\| None`	`True` if `normalized_score` meets `threshold`; `None` if no threshold set
`details`	`dict[str, Any]`	Evaluator-specific supplementary data

AggregateMetricResult

Aggregated result across all items for one (scope, metric_name) pair, counting every repeat of every item. The scoring service computes these from the per-item MetricResult list with aggregate_metric_results, which lives in the SDK (semantic_evaluation.scoring.aggregation) and applies the gate on top. The runner computes the ones for trace metrics, with the same statistics, and sets raw_scale on them. It also computes the per-group aggregates a run file's aggregate_by asks for, with the same function, over the results of the items in one group; those carry group_by and group_value and never reach the gate.

Field	Type	Description
`scope`	`str`	Scope
`metric_name`	`str`	Metric key
`threshold`	`float \\| None`	Pass threshold (from `MetricConfig`)
`direction`	`MetricDirection`	From `MetricConfig`
`item_count`	`int`	Results aggregated: items, times repeats
`scored_count`	`int`	Results that returned a non-`None` `normalized_score`
`mean_score`	`float \\| None`	Mean of `normalized_score` across scored results
`min_score`	`float \\| None`	Minimum `normalized_score`
`max_score`	`float \\| None`	Maximum `normalized_score`
`median_score`	`float \\| None`	Median `normalized_score`
`p90_score`	`float \\| None`	90th percentile, interpolated between the closest ranks, so within the observed values
`stdev_score`	`float \\| None`	Sample standard deviation; `None` for fewer than two results
`pass_rate`	`float \\| None`	Fraction of scored results where `passed=True`
`passed`	`bool`	`True` if `mean_score` meets `threshold`; used for the gate decision. Always `True` for a trace metric
`raw_scale`	`str \\| None`	Set only on a trace metric's aggregate: its statistics are then in this unit, not on the normalized scale
`details`	`dict[str, Any]`	Supplementary data
`group_by`	`str \\| None`	The item metadata attribute this aggregate is grouped by; `None` on a whole-run aggregate
`group_value`	`str \\| None`	The attribute's value for the group; `None` on a whole-run aggregate

Agreement contracts

The runner builds these from a run file's agreement block with compute_agreement (semantic_evaluation.scoring.agreement); they are not part of the scoring service's request or response. See 04-offline-evaluation.md: Agreement across repeats for what a run file asks for and how the terms are defined.

Contract	Holds
`ItemAgreement`	One item: `values` (the field in each repeat, `None` where missing), `modal_value` (`None` when values tie for most frequent), `modal_share` and `unanimous`
`AgreementSummary`	One scope, or one group of it (`group_by`, `group_value`): `item_count`, `unanimous_count`, `unanimous_share`, `mean_modal_share`
`RunShift`	One repeat of one unit (`per`, `group_value`, `repeat`): `item_count` items with a value, `differing_count` of them off the strict majority of the other repeats, and `shifted`
`AgreementResult`	The `field`, the `repeats`, and the `items`, `summaries` and `run_shifts` lists

GateResult

The gate verdict returned with every ScoreResponse. See also 04-run-config.md — GateConfig.

Field	Type	Description
`passed`	`bool`	Overall gate pass/fail
`mode`	`str`	Gate mode used (currently always `"all_metrics"`)
`minimum_item_count`	`int`	Minimum item count required to pass
`total_items`	`int`	Total items across all scopes that were evaluated
`failing_metrics`	`list[GateFailure]`	Details of each failing metric
`evaluated_at`	`str`	ISO 8601 UTC timestamp

Each GateFailure carries scope, metric_name, actual (mean score), threshold, and reason.

ScoringServiceClient

HTTP client for the centralized scoring service. The client is stateless — create an instance per request or reuse across requests. Scoring is a scoring job: submitting a batch returns a job ID immediately, and the batch is scored concurrently in the background rather than inline in the request. This decouples the wait for a batch's judge calls from any single HTTP connection's lifetime — a reasoning-tier LLM judge scoring a large batch can legitimately take longer than any client should hold a connection open for.

from semantic_evaluation.scoring import ScoringServiceClient

client = ScoringServiceClient(
    base_url="http://localhost:8001",
    timeout_seconds=120.0,       # HTTP timeout for each individual submit/poll call
    poll_interval_seconds=3.0,   # how often score() polls job status
    max_wait_seconds=None,       # overall wait bound; None waits indefinitely
    api_key=None,                # X-API-Key header value; omit or None to send no header
    retry_attempts=6,            # tries per HTTP call, the first included
    retry_initial_backoff_seconds=2.0,  # delay before the first retry; doubles each time
)

Each HTTP call retries transient failures, waiting 2, 4, 8, 16 and 32 seconds between tries by default, so a DNS failure or a restarting service does not end a run. Reads (GET) retry on any transport error and on HTTP 502, 503 and 504. The job submission retries only when the request cannot have reached the service: a connection error, a connect timeout or HTTP 503. A read timeout on submission is raised at once, because the job may already exist. Each retry logs a warning.

base_url is the service host only. The client adds the API major (/v1) to every route, so a product never configures a version; see service API versions.

api_key is your pod's own caller key for the scoring service, which the client sends as X-API-Key on every request. The deployed service is in a warning phase for POST /v1/score-jobs and GET /v1/score-jobs/{job_id}: a call with a known key is identified in the service logs, and a call without one is still accepted. A key becomes required from an enforcement date announced in the scoring service's changelog, so pass yours now. The connection-profile, evaluator-backend and evaluator-request routes, which this client does not call, need a known key already.

`score(request: ScoreRequest, *, on_progress=None) -> ScoreResponse`

Submit a scoring request and poll until the job completes, returning the parsed result. This is the method most callers use — it is built from submit_score()/get_score_job() below, so it never blocks on one long HTTP call regardless of batch size.

from semantic_evaluation.contracts import ScoreRequest

response = await client.score(
    ScoreRequest(
        items=items,
        predictions=predictions,
        evaluators=manifest.evaluators,
        gate=manifest.gate,
        scopes=manifest.scopes,
    )
)

# per-item results
for r in response.metric_results:
    print(r.metric_name, r.normalized_score, r.passed)

# aggregate + gate
for agg in response.aggregate_results:
    print(agg.metric_name, agg.mean_score, agg.passed)

print("gate passed:", response.gate_result.passed)

Pass on_progress to observe the job while it runs — called once per poll with the current ScoreJobResult:

def log_progress(job):
    print(f"{job.status.value}: {job.completed_evaluations}/{job.total_evaluations}")

response = await client.score(request, on_progress=log_progress)

Raises RuntimeError if the job ends in failed status, or if any HTTP call returns an error status. Raises TimeoutError if max_wait_seconds is set and exceeded while the job is still pending/running.

ScoreRequest fields:

Field	Type	Description
`items`	`list[EvaluationItem]`	Items to score
`predictions`	`list[PredictionRecord]`	Corresponding predictions
`evaluators`	`dict[scope, ScopeEvaluatorConfig]`	Metric configuration per scope
`gate`	`GateConfig`	Gating policy
`scopes`	`list[str] \\| None`	Subset of scopes to score; `None` means all

ScoreResponse fields:

Field	Type	Description
`metric_results`	`list[MetricResult]`	Per-item metric results
`aggregate_results`	`list[AggregateMetricResult]`	Per-scope, per-metric aggregates
`gate_result`	`GateResult`	Overall gate verdict
`item_count_by_scope`	`dict[str, int]`	Item count per scope

`submit_score(request: ScoreRequest) -> ScoreJobSubmitted`

Submit a scoring request as a background job and return immediately, without waiting for it to finish. Use this directly (with get_score_job()) for manual control over polling; most callers use score() instead.

submitted = await client.submit_score(request)
print(submitted.job_id, submitted.status.value)  # "<uuid>", "pending"

`get_score_job(job_id: str) -> ScoreJobResult`

Return one job's current status, and its result once it has completed.

job = await client.get_score_job(submitted.job_id)
if job.status.value == "completed":
    response = job.result

ScoreJobResult fields:

Field	Type	Description
`job_id`	`str`	The job's identifier
`status`	`ScoreJobStatus`	`pending`, `running`, `completed`, or `failed`
`submitted_at`	`str`	ISO 8601 UTC timestamp
`started_at`	`str \\| None`	Set once the job leaves `pending`
`finished_at`	`str \\| None`	Set once the job reaches `completed`/`failed`
`completed_evaluations`	`int \\| None`	Judge calls finished so far
`total_evaluations`	`int \\| None`	Total judge calls this job will run
`result`	`ScoreResponse \\| None`	Set once `status` is `completed`
`error`	`str \\| None`	Set once `status` is `failed`

`list_evaluators() -> EvaluatorCatalogueResponse`

Fetch the catalogue of supported (framework, evaluator) pairs from the service.

catalogue = await client.list_evaluators()
for entry in catalogue.entries:
    print(entry.framework, entry.evaluator, entry.supported_params)

EvaluatorCatalogueEntry fields:

Field	Type	Description
`framework`	`str`	Framework identifier
`evaluator`	`str`	Evaluator class name
`scope`	`str`	Default scope this evaluator targets
`raw_scale`	`str`	Native scoring scale (e.g. `"1-5"`, `"binary"`, `"0-1"`)
`supported_params`	`list[str]`	Allowed keys in `MetricConfig.evaluator_params`

Passing an unknown evaluator_params key causes the scoring service to reject the request before evaluator construction.