Connection profiles

LLM-judge evaluator connections are service-owned. Callers send metric configuration in a ScoreRequest, never provider credentials — the service resolves those from a named, registered scoring connection profile.

The profile model

scoring_service/scoring/profiles.py supports these profile kinds:

| Kind | Fields | Notes | |---|---| | azure_openai | name, endpoint, api_key, deployment, api_version | Azure OpenAI resource endpoint and API version | | foundry_decision | name, endpoint, api_key, deployment | Microsoft Decision-1's Foundry resource endpoint, ending in /providers/microsoft, or its /v1/systemone endpoint; it has no Azure OpenAI API version | | foundry_openai_v1 | name, endpoint, api_key, deployment | Foundry's OpenAI-compatible v1 base endpoint, ending in /openai/v1; /responses and /chat/completions are also accepted and normalized to the base endpoint |

api_key accepts a secret URI of the form https://{vault}.vault.azure.net/secrets/{name}. The service resolves that URI to a real key at evaluation time using azure-identity's DefaultAzureCredential (managed identity in Azure, developer credentials locally). No read route returns api_key, whether it was registered as a literal or a Key Vault URI.

Decision profiles use the same API-key or Key Vault URI behavior as Azure OpenAI profiles. They do not support Entra ID authentication in this release.

Registering a profile

Profiles are registered via POST /v1/connection-profiles and persisted to a JSON state file (connection_profiles.json inside OFFLINE_SCORING_STATE_DIR, scoring_service/state/ by default) so they survive service restarts. On the deployed service only the platform-admin caller may register a profile (see Authentication); the commands below run against a local service, where every route is open:

curl -s -X POST http://127.0.0.1:8001/v1/connection-profiles \
  -H "Content-Type: application/json" \
  -d '{
    "name": "azure-default",
    "kind": "azure_openai",
    "endpoint": "https://your-resource.openai.azure.com/",
    "api_key": "replace-me",
    "deployment": "gpt-4.1",
    "api_version": "2024-12-01-preview"
  }'

Registering a profile alone does not connect it to any metric — a backend assignment does that.

Registering a Decision profile

curl -s -X POST http://127.0.0.1:8001/v1/connection-profiles \
  -H "Content-Type: application/json" \
  -d '{
    "name": "decision-default",
    "kind": "foundry_decision",
    "endpoint": "https://your-resource.services.ai.azure.com/providers/microsoft",
    "api_key": "replace-me",
    "deployment": "Decision-1-DZ"
  }'

The evaluator accepts either endpoint form in the table. It appends /v1/systemone to the resource base and leaves a full System One endpoint unchanged.

Registering a Foundry OpenAI v1 profile

Use this profile for custom_rubric when the judge deployment is exposed by Foundry's OpenAI-compatible v1 API rather than the Azure OpenAI versioned API:

curl -s -X POST http://127.0.0.1:8001/v1/connection-profiles \
  -H "Content-Type: application/json" \
  -d '{
    "name": "foundry-judge-default",
    "kind": "foundry_openai_v1",
    "endpoint": "https://your-resource.services.ai.azure.com/openai/v1",
    "api_key": "replace-me",
    "deployment": "gpt-5.1-DZ"
  }'

The OpenAI SDK appends its operation path to the configured base endpoint. The service also accepts the /responses or /chat/completions URL copied from a provider sample and normalizes it back to the base endpoint before calling the SDK.

Backend assignment

Definition: backend assignment A backend assignment maps one evaluator backend key to one registered connection profile name. An evaluator backend can have at most one assignment; assigning a new profile replaces the previous one. See backend assignment.

Local serviceNon-prod deployed service

curl -s -X PUT http://127.0.0.1:8001/v1/evaluator-backends/deepeval_llm_judge/connection-profile \
  -H "Content-Type: application/json" \
  -d '{"connection_profile_name": "azure-default"}'

Needs the platform-admin caller's key.

curl -s -X PUT https://scoring-service.bravecliff-5469885d.swedencentral.azurecontainerapps.io/v1/evaluator-backends/deepeval_llm_judge/connection-profile \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $SCORING_SERVICE_API_KEY" \
  -d '{"connection_profile_name": "azure-default"}'

The path names the evaluator backend and the body names the profile, in the same direction the service resolves them. connection_profile_name must already be registered, or the request fails with HTTP 400. Assigning again replaces the previous assignment, so the call is safe to repeat.

Read the current assignments back with GET /v1/evaluator-backends, which returns a backend to profile-name map. A backend with no profile is simply absent from it.

Backend keys

Backend key	Used by
`deepeval_llm_judge`	Every `deepeval` catalogue entry
`azure_ai_evaluation_llm_judge`	Every `azure_ai_evaluation` catalogue entry, including the statistical `F1ScoreEvaluator`
`custom_rubric_llm_judge`	`custom_rubric/inline` and every accepted `custom_rubric/{name}` evaluator
`decision_llm_judge`	`decision_judge/inline`

builtin and builtin_deterministic evaluators never resolve a backend — they do not appear in this table because they never need a connection profile.

Resolution behavior

graph TD
  A["POST /v1/score-jobs received"]
  B["build_evaluators checks all metrics before scoring"]
  C{"metric framework?"}
  D["builtin or builtin_deterministic"]
  E["deepeval, azure_ai_evaluation, custom_rubric, or decision_judge"]
  F{"backend key has an assignment?"}
  G["HTTP 400 - named backend has no profile assigned"]
  H["fetch registered profile from registry"]
  I{"api_key is a Key Vault URI?"}
  J["use api_key as a literal string"]
  K["resolve secret via DefaultAzureCredential at evaluation time"]
  L["build evaluator instance"]
  M["no connection profile needed"]

  A --> B
  B --> C
  C --> D
  C --> E
  D --> M
  M --> L
  E --> F
  F -- "no" --> G
  F -- "yes" --> H
  H --> I
  I -- "no" --> J
  I -- "yes" --> K
  J --> L
  K --> L

  style A fill:#e5f4fd,stroke:#0098eb,stroke-width:2px
  style G fill:#fce8e8,stroke:#cc4444,stroke-width:2px
  style L fill:#e7eeec,stroke:#125a40,stroke-width:2px

Scenario	Behavior
Routing-only request (`builtin`)	No profile required
`builtin_deterministic` metric	No profile required
`deepeval`, `azure_ai_evaluation`, `custom_rubric`, or `decision_judge` metric with a missing backend assignment	Reject the whole `POST /v1/score-jobs` submission with HTTP 400 before any evaluator runs and before any job is created, naming the missing backend
`azure_ai_evaluation/F1ScoreEvaluator` (statistical, no LLM call)	Still requires `azure_ai_evaluation_llm_judge` to be assigned — the catalogue factory resolves the backend for every `azure_ai_evaluation` entry unconditionally, even though `F1ScoreEvaluator`'s own constructor ignores the resulting credentials
Invalid or malformed state file at startup	Service fails to start
Unknown backend name inside the state file at startup	Service fails to start

The rejection happens for the whole request, not just the offending metric: build_evaluators() constructs every evaluator for every requested scope before any evaluation runs, so a missing backend assignment for one metric fails the batch before scoring begins.

Managing profiles

Route	Purpose
`GET /v1/connection-profiles`	List every registered profile, without their `api_key`
`GET /v1/connection-profiles/{name}`	Fetch one profile by name, without its `api_key` (404 if unknown)
`GET /v1/evaluator-backends`	Read back which profile each evaluator backend uses
`DELETE /v1/connection-profiles/{name}`	Remove a profile; also clears any backend assignment that referenced it

BASE=http://127.0.0.1:8001          # or the deployed FQDN
AUTH="X-API-Key: $SCORING_SERVICE_API_KEY"   # your own caller key; unused locally

curl -s -H "$AUTH" "$BASE/v1/connection-profiles"
curl -s -H "$AUTH" "$BASE/v1/connection-profiles/azure-default"
curl -s -H "$AUTH" "$BASE/v1/evaluator-backends"
curl -s -X DELETE -H "$AUTH" "$BASE/v1/connection-profiles/azure-default"

No route ever returns an api_key. The read routes answer with the profile's name, kind, endpoint, deployment, and API version only. There is no way to retrieve a stored credential, which is deliberate: it is what makes storing a Key Vault URI worth doing.

Deleting a profile does not fail if the deleted profile was assigned to a backend — the assignment is silently removed, and the next request that needs that backend fails with the standard "no scoring connection profile is configured" error until a new profile is assigned.

Authentication

Every caller has its own caller key, sent as an X-API-Key: <value> header. The deployed service splits the profile and evaluator routes by what they do:

Route	Who
`POST /v1/connection-profiles`, `DELETE /v1/connection-profiles/{name}`, `PUT /v1/evaluator-backends/{backend}/connection-profile`, `GET /v1/evaluators/requests`	The platform-admin caller only (HTTP 403 for a pod key)
`GET /v1/connection-profiles`, `GET /v1/connection-profiles/{name}`, `GET /v1/evaluator-backends`, `POST /v1/evaluators/requests`	Any known caller

Registering or deleting a profile and assigning a backend on the deployed service is therefore the platform team's job; a pod reads them. What a profile points at is internal configuration, so a caller without a key reads nothing.

GET /health, GET /v1/evaluators never require the header. The scoring endpoints (POST /v1/score-jobs, GET /v1/score-jobs/{job_id}) are in a warning phase: a known caller is identified in the logs, and a call without a known key is still accepted. They require a key from an enforcement date announced in the changelog.

When CALLER_KEY_VAULT_URL is unset every endpoint is open and no caller is identified, which is the local-development and test default. The service logs a warning at startup when that happens, so an unprotected deployment is visible in its own logs rather than indistinguishable from a healthy one. With it set, the service logs Loaded N callers: <ids> at startup. Issuing and revoking keys is described in Deployment.