Skip to content

Changelog

Changes to the scoring service that a caller can notice: routes, request and response fields, evaluators, and when an API major starts and ends its retirement. Each entry is dated by the deploy that shipped it and names the API major it belongs to. Changes inside the service that no caller sees are not listed. The list starts when the API took its first major, v1.

Callers using semantic-evaluation reach the service through its client, which picks the API major itself; the SDK's own changelog says which SDK release calls which major. How majors are added and retired is described under Versions.

Unreleased: API v1

Added

  • Each caller has its own caller key, sent as X-API-Key as before. The service identifies the caller from it and logs caller=<id> on every request. A pod asks the platform team for its key and sets it where it set the shared key.
  • FieldEquals accepts expected_one_of_field, a dotted path into the item's expected output that holds a list of accepted values; the metric passes when the prediction's field equals any of them. Items without that value are skipped, and a value that is not a non-empty list fails the item with the reason.
  • decision_judge and custom_rubric results carry details.latency_ms, the duration of the model call. custom_rubric results also carry details.usage with input_tokens, output_tokens and, when reported, reasoning_tokens.

Changed

  • Registering or deleting a connection profile, assigning an evaluator backend and listing pending evaluator submissions need the platform-admin caller; a pod key gets HTTP 403. Reading profiles and backend assignments and submitting an evaluator work with any caller key.
  • POST /v1/score-jobs and GET /v1/score-jobs/{job_id} identify a caller that sends a key. A call without a known key is still accepted and logged. These routes will require a caller key from an enforcement date announced here; send your pod's key now.

Fixed

  • A connection profile whose api_key is a Key Vault secret URI resolves on the deployed service. The image now installs azure-keyvault-secrets; before, every scoring request using such a profile was rejected with HTTP 400.

2026-11-06: API v1

Deprecated

  • The unversioned routes (/evaluators, /evaluator-backends, /connection-profiles, /score-jobs) are deprecated aliases of v1. Every response on them carries Deprecation, Sunset (Thu, 04 Feb 2027) and Link headers. They are removed no earlier than 4 February 2027, and only once the request logs show no more calls on them. semantic-evaluation 2.5.0 and later call /v1; scripts that call the service directly move to /v1 now.

2026-11-05: API v1

Added

  • Every route is served under /v1 (for example GET /v1/evaluators, POST /v1/score-jobs). Additive changes stay in v1; a breaking change goes into a new major, /v2, which ships together with a new semantic-evaluation major.
  • The unversioned routes stay available as aliases of v1, with the same operations and responses.
  • /health, /docs, /redoc and /openapi.json are not versioned. /docs lists both mounts.