Skip to content

Scoring app glossary

This glossary defines terms that are local to scoring_app/ — the shared scoring service and its supporting docs. For terms used across more than one package in the workspace (for example shared service, offline evaluation, evaluator framework, evaluator catalogue, normalized score, or raw score), see the repository glossary instead of looking for a second definition here.

Release-Facing Scoring

Release-facing scoring is the scoring used to support keep, change, or promote decisions for workflow changes.

In this repository, it means calling the shared scoring service in scoring_app/ over HTTP. It does not mean one-off local scoring experiments run inside a pod.

Scoring Connection Profile

A scoring connection profile is a named, service-owned evaluator backend configuration. It tells the scoring service which Azure OpenAI or Foundry Decision endpoint, deployment, and credential to use for a backend such as deepeval_llm_judge, azure_ai_evaluation_llm_judge, or custom_rubric_llm_judge, or decision_llm_judge.

Profiles are registered through the HTTP API and persisted to a local JSON state file. Callers refer to a backend indirectly through the metric's (framework, evaluator) pair — they never send provider credentials inline with a scoring request. See 02-connection-profiles.md for the full model.

Backend Assignment

A backend assignment maps one evaluator backend key (deepeval_llm_judge, azure_ai_evaluation_llm_judge, custom_rubric_llm_judge, or decision_llm_judge) to one registered scoring connection profile name. An evaluator backend can have at most one assignment at a time; assigning a new profile to a backend replaces the previous assignment.

Evaluator Parameters

Evaluator parameters (evaluator_params) are the per-metric keyword arguments a manifest author declares under MetricConfig.evaluator_params. They flow through the matching evaluator wrapper and, for deepeval and azure_ai_evaluation metrics, into the underlying framework's constructor or call signature. Each evaluator catalogue entry declares which parameter names it accepts as supported_params; passing any other key is a hard failure before evaluation runs. See 01-evaluator-catalogue.md.

Discovery Endpoint

The discovery endpoint (GET /v1/evaluators) returns the full evaluator catalogue as a list of entries with framework, evaluator, scope, raw_scale, and supported_params. Manifest authors and the SDK's list_evaluators() helper use this endpoint to learn which metrics are available and what parameters they accept, without reading the scoring service's source.

Accepted Evaluator

An accepted evaluator is a named custom rubric that has been reviewed by the eval core team and committed as a JSON file to scoring_service/custom_evaluators/accepted/. The service loads every file in that directory at startup and exposes it as one custom_rubric/{name} entry in the catalogue.

An accepted evaluator differs from an inline custom rubric: its rubric definition lives in committed source, not in the caller's manifest. Only the model field may be overridden per request through evaluator_params. Pods submit candidates via POST /v1/evaluators/requests; accepted entries appear in GET /v1/evaluators after the next service deployment. See 03-custom-rubric-and-registration.md.

API Major

The major version of a shared service's HTTP API, carried in the URL path (/v1). A breaking change adds the next major beside the current one, and each semantic_evaluation major calls exactly one major per service. See Service API versions.

Caller Key

A caller key is the API key that identifies one caller (a pod, the workshop, the platform team) to the two shared services. Every caller has its own key per service, stored as a Key Vault secret in semantic-eval-dev-kv named scoring-caller-<caller-id> or review-sync-caller-<caller-id>. The secret's role tag is admin (the platform-admin caller) or pod. The caller sends the key as the X-API-Key header; pods set it as SCORING_SERVICE_API_KEY and REVIEW_SYNC_API_KEY. The service reads the secrets with its managed identity and reloads them every 5 minutes. See 06-deployment.md for issuing, rotating and revoking, and the API reference for which routes need one.