API reference

The base URL is the host (http://127.0.0.1:8001 by default); every API route sits under the API major /v1. Every response carries an X-Request-ID header for log correlation. Callers identify themselves with their own caller key in an X-API-Key: <value> header. Who may call which route:

Route	Who
`GET /health`, `GET /v1/evaluators`	Open
`POST /v1/connection-profiles`, `DELETE /v1/connection-profiles/{name}`, `PUT /v1/evaluator-backends/{backend}/connection-profile`, `GET /v1/evaluators/requests`	The platform-admin caller only
`GET /v1/connection-profiles`, `GET /v1/connection-profiles/{name}`, `GET /v1/evaluator-backends`, `POST /v1/evaluators/requests`	Any known caller
`POST /v1/score-jobs`, `GET /v1/score-jobs/{job_id}`	Warning phase: a known caller is identified, and a call without a known key is still accepted and logged. A key becomes required from an enforcement date announced in the changelog

A request without a key, or with a key no caller has, gets HTTP 401 Invalid or missing API key.; a pod key on an admin route gets HTTP 403 This route needs the platform-admin caller. When CALLER_KEY_VAULT_URL is unset (local development and tests) every route is open. See Authentication.

Versions

The API has one supported major, v1. Every API route sits under it (/v1/...); the base URL is the host alone. /health, /docs, /redoc and /openapi.json are unversioned. The API major policy (supported majors, retirement window, what counts as a breaking change) is written once in Service API versions.

The same paths without the /v1 prefix (for example /evaluators) remain as deprecated aliases: the same operations, deprecated on 2026-11-06 and removed no earlier than 2027-02-04. Every response on an alias carries three headers:

Deprecation: @1793923200
Sunset: Thu, 04 Feb 2027 00:00:00 GMT
Link: <https://blue-sand-0fe957b03.7.azurestaticapps.net/docs/scoring-service/04-api-reference/>; rel="deprecation"

Deprecation is the deprecation instant as a Unix timestamp (RFC 9745), Sunset is the earliest removal date (RFC 8594), and Link points at this API reference. Responses under /v1 carry none of them.

/docs shows both mounts, with the aliases marked deprecated. The request log line carries api_version (v1, unversioned, or none for paths such as /health) and user_agent, so traffic still on the aliases is visible before they are removed.

Base URLs and interactive docs

Environment	Base URL
Local	`http://127.0.0.1:8001`
Non-prod deployed	`https://scoring-service.bravecliff-5469885d.swedencentral.azurecontainerapps.io`

The deployed service sits behind the VPN. Both environments serve the generated OpenAPI documentation, which is always in step with the running build in a way this page cannot promise to be:

Path	What it is
`/docs`	Swagger UI, with a request form for every route
`/redoc`	The same schema as a reference page
`/openapi.json`	The raw schema, for generating a client

Method	Path	Auth	Purpose
`GET`	`/health`	no	Liveness check
`GET`	`/v1/evaluators`	no	Return the full evaluator catalogue
`POST`	`/v1/score-jobs`	no	Submit items and predictions as a scoring job; returns a job ID immediately
`GET`	`/v1/score-jobs/{job_id}`	no	Return one scoring job's status and, once complete, its result
`POST`	`/v1/connection-profiles`	yes	Register or replace a connection profile
`GET`	`/v1/connection-profiles`	yes	List every connection profile, without their credentials
`GET`	`/v1/connection-profiles/{name}`	yes	Fetch one connection profile, without its credential
`DELETE`	`/v1/connection-profiles/{name}`	yes	Remove a connection profile
`GET`	`/v1/evaluator-backends`	yes	Read which connection profile each evaluator backend uses
`PUT`	`/v1/evaluator-backends/{backend}/connection-profile`	yes	Point an evaluator backend at a connection profile
`POST`	`/v1/evaluators/requests`	yes	Submit a named custom rubric for review
`GET`	`/v1/evaluators/requests`	yes	List pending custom-rubric submissions

`GET /health`

Returns {"status": "ok"}. Used for local and cloud liveness checks.

`GET /v1/evaluators`

Returns an EvaluatorCatalogueResponse: a list of entries, each with framework, evaluator, scope, raw_scale, and supported_params. See 01-evaluator-catalogue.md for the full contents today and what each field means.

`POST /v1/score-jobs`

Submits a batch of predictions for scoring as a scoring job and returns immediately — it does not wait for scoring to finish. Accepts a ScoreRequest:

Field	Type	Notes
`items`	`list[EvaluationItem]`	Trusted evaluation items (`item_id`, `scope`, `dataset_name`, `input`, `expected_output`, `metadata`)
`predictions`	`list[PredictionRecord]`	One prediction per item (`item_id`, `scope`, `dataset_name`, `predictor_name`, `input`, `expected_output`, `prediction`, optional `trace_id`/`observation_id`)
`evaluators`	`dict[scope, ScopeEvaluatorConfig]`	Per-scope `{metric_name: MetricConfig}` map; `MetricConfig` carries `framework`, `evaluator`, `enabled`, `threshold`, `direction`, `model`, `evaluator_params`
`gate`	`GateConfig`	`mode` (currently always `"all_metrics"`) and `minimum_item_count`; defaults to `mode="all_metrics"`, `minimum_item_count=1`
`scopes`	`list[str] \\| null`	Explicit scope list; defaults to every distinct scope present in `items`

A prediction whose item_id has no matching item, or a metric naming an unknown (framework, evaluator) pair, or a metric whose backend has no connection profile assigned all return HTTP 400 with a detail message before any job is created — this validation runs synchronously, so a structurally bad request never occupies a job slot. See 01-evaluator-catalogue.md and 02-connection-profiles.md for the exact failure conditions.

On success, returns HTTP 202 with a ScoreJobSubmitted:

Field	Type	Notes
`job_id`	`str`	Pass to `GET /v1/score-jobs/{job_id}` to check status and fetch the result
`status`	`str`	Always `"pending"` at submission time

The actual scoring then runs as a background task: judge calls across the batch run concurrently, bounded by OFFLINE_SCORING_MAX_CONCURRENT_JUDGE_CALLS (see 06-deployment.md), rather than one at a time. See getting-started.md for a complete routing-only submit-and-poll walkthrough.

When a prediction carries prediction["_trace_fetch_failed"]: true (set by a pod's predictor when it could not retrieve the underlying trace), every MetricResult for that prediction gets a suffix appended to its justification noting that the score was computed from empty prediction data.

`GET /v1/score-jobs/{job_id}`

Returns a ScoreJobResult for one job, or HTTP 404 if job_id is unknown:

Field	Type	Notes
`job_id`	`str`	Echoes the requested ID
`status`	`str`	`"pending"`, `"running"`, `"completed"`, or `"failed"`
`submitted_at`	`str`	ISO 8601 UTC timestamp
`started_at`	`str \\| null`	Set once the job leaves `pending`
`finished_at`	`str \\| null`	Set once the job reaches `completed`/`failed`
`completed_evaluations`	`int \\| null`	Judge calls finished so far
`total_evaluations`	`int \\| null`	Total judge calls this job will run
`result`	`ScoreResponse \\| null`	Set once `status` is `"completed"`
`error`	`str \\| null`	Set once `status` is `"failed"`

A completed job's result is a ScoreResponse:

Field	Type	Notes
`metric_results`	`list[MetricResult]`	One entry per `(item, metric)` pair that ran
`aggregate_results`	`list[AggregateMetricResult]`	One entry per `(scope, metric_name)` pair
`gate_result`	`GateResult`	`passed`, `mode`, `minimum_item_count`, `total_items`, `failing_metrics`
`item_count_by_scope`	`dict[str, int]`	Item counts per requested scope

Job records are persisted under OFFLINE_SCORING_STATE_DIR, so a poll can land on either Container Apps replica and still see the same job.

`POST /v1/connection-profiles`

Registers or replaces an azure_openai or foundry_decision connection profile (registering a name that already exists overwrites it). Returns the stored profile without its credential. See 02-connection-profiles.md.

`GET /v1/connection-profiles`

Returns every registered profile as a list.

`GET /v1/connection-profiles/{name}`

Returns one profile, or HTTP 404 if name is not registered.

`DELETE /v1/connection-profiles/{name}`

Removes a profile and any backend assignment that referenced it. Returns {"deleted": "<name>"}, or HTTP 404 if name was never registered.

`GET /v1/evaluator-backends`

Returns the backend to profile-name map, e.g. {"deepeval_llm_judge": "azure-default"}. A backend with no profile assigned is absent from the map, which is also what makes a scoring request using it fail with HTTP 400.

`PUT /v1/evaluator-backends/{backend}/connection-profile`

{backend} is an evaluator backend key such as deepeval_llm_judge or decision_llm_judge. Body: {"connection_profile_name": "<registered profile name>"}. Returns {"backend": "<backend>", "connection_profile_name": "<name>"}. Returns HTTP 400 if backend is not a known backend key or the profile is not registered, and HTTP 422 if the body carries any other field.

One profile can serve several backends; a backend has at most one profile, so repeating the call replaces the previous assignment rather than accumulating.

`POST /v1/evaluators/requests`

Submits an EvaluatorRegistrationRequest (name, criteria, scale, input_fields, output_fields, expected_output_fields, model, submitted_by) for eval-core-team review. Writes a pending submission file and returns the resulting AcceptedEvaluatorRecord (HTTP 201).

Returns HTTP 400 if name is "inline" (reserved).
Returns HTTP 409 if name is already accepted (committed to custom_evaluators/accepted/).
Re-submitting a name that is still pending overwrites the pending file.

See 03-custom-rubric-and-registration.md.

`GET /v1/evaluators/requests`

Returns every pending submission as a list of AcceptedEvaluatorRecord objects, for the eval core team to review before accepting.