Getting started — scoring_app

Choose the flow that matches your entry point. All three flows share the same HTTP API; only the host and auth requirements differ.

Local — routing onlyLocal — LLM-judge metricsNon-prod deployed service

Install and run

From scoring_app/:

uv sync
uv run python scoring_service/main.py

This starts the FastAPI service on 127.0.0.1:8001 by default. Confirm it is up:

curl -s http://127.0.0.1:8001/health
# {"status": "ok"}

No connection profile is required to start the service or to run routing-only scoring.

Environment variables

The service reads scoring_service/.env.scoring_service and then scoring_service/.env.scoring_service.local (both optional; the first value found for a given variable wins, so the .local file is for overrides that should not be committed).

Variable	Default	Purpose
OFFLINE_SCORING_HOST	127.0.0.1	Bind host
OFFLINE_SCORING_PORT	8001	Bind port
OFFLINE_SCORING_RELOAD	false	Enable uvicorn auto-reload for local development
OFFLINE_SCORING_DEFAULT_MODEL	OFFLINE_EVAL_DEEPEVAL_MODEL, or gpt-4.1	Default evaluator model label used when a metric and its connection profile do not name one
CALLER_KEY_VAULT_URL	(unset)	Key Vault whose scoring-caller-<caller-id> secrets are the caller keys, for example https://semantic-eval-dev-kv.vault.azure.net/. Unset leaves every route open and identifies no caller, with a startup warning
OFFLINE_SCORING_STATE_DIR	scoring_service/state/	Directory holding connection_profiles.json, pending custom-rubric submissions, and scoring-job records
OFFLINE_SCORING_MAX_CONCURRENT_JUDGE_CALLS	4	How many judge calls one scoring job runs concurrently

Evaluator provider secrets (API keys) are never read from environment variables — they are registered through the HTTP API. See 02-connection-profiles.md.

First scoring request

Routing-only scoring needs no connection profile. This example submits one item as a scoring job and polls it to completion, using the builtin/ExactMatch evaluator, which compares expected_output["handoff_to"] against prediction["handoff_to"]:

curl -s -X POST http://127.0.0.1:8001/v1/score-jobs \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {
        "item_id": "item-1",
        "scope": "routing",
        "dataset_name": "getting-started-demo",
        "input": {"query": "Reset my password"},
        "expected_output": {"handoff_to": "account_support"},
        "metadata": {}
      }
    ],
    "predictions": [
      {
        "item_id": "item-1",
        "scope": "routing",
        "dataset_name": "getting-started-demo",
        "predictor_name": "demo-predictor",
        "input": {"query": "Reset my password"},
        "expected_output": {"handoff_to": "account_support"},
        "prediction": {"handoff_to": "account_support"}
      }
    ],
    "evaluators": {
      "routing": {
        "metrics": {
          "handoff_correct": {
            "framework": "builtin",
            "evaluator": "ExactMatch",
            "threshold": 3.0
          }
        }
      }
    }
  }'
# {"job_id": "…", "status": "pending"}

Poll the job with the returned job_id:

curl -s http://127.0.0.1:8001/v1/score-jobs/<job_id>

A deterministic, routing-only job like this one finishes almost immediately, so the very next poll typically already shows "status": "completed" with a result carrying metric_results (one per item per metric), aggregate_results (one per scope/metric pair), and gate_result (the pass/fail decision for the whole run). See 04-api-reference.md for the full request and response shape.

In real usage, pods do not build this payload or poll by hand — the SDK's ScoringServiceClient.score() and the offline-evaluation runner in sdk/semantic_evaluation/ construct the request from a run manifest and predictor output, then submit and poll for you.

Install and run

From scoring_app/:

uv sync
uv run python scoring_service/main.py

Registering a connection profile

End-to-end LLM-judge metrics (deepeval, azure_ai_evaluation, or custom_rubric) need a registered connection profile and a backend assignment before the first request that uses them:

curl -s -X POST http://127.0.0.1:8001/v1/connection-profiles \
  -H "Content-Type: application/json" \
  -d '{
    "name": "azure-default",
    "kind": "azure_openai",
    "endpoint": "https://your-resource.openai.azure.com/",
    "api_key": "replace-me",
    "deployment": "gpt-4.1",
    "api_version": "2024-12-01-preview"
  }'

curl -s -X PUT http://127.0.0.1:8001/v1/evaluator-backends/deepeval_llm_judge/connection-profile \
  -H "Content-Type: application/json" \
  -d '{"connection_profile_name": "azure-default"}'

api_key may be a literal string (as above, for local development) or an Azure Key Vault secret URI; the service resolves the URI at evaluation time via managed identity. A local service has no CALLER_KEY_VAULT_URL, so neither call needs a key.

See 02-connection-profiles.md for the full model, including what happens when a backend has no assignment, and the backend key names for azure_ai_evaluation and custom_rubric.

Access requirements

scoring-service runs in semantic-eval-nonprod-env (Sweden Central). A VPN connection is required to reach it from a developer machine — confirmed reachable at the hostname below once connected (the .internal.-prefixed variant of this hostname does not resolve correctly from outside the VNet and should not be used).

FQDN:

scoring-service.bravecliff-5469885d.swedencentral.azurecontainerapps.io

The first request after the service has been idle can take 10-15 seconds (cold start) before it responds — this is normal, not a failure.

Authentication

Every caller has its own caller key, issued by the platform team, and sends it as the X-API-Key header; pods set it as SCORING_SERVICE_API_KEY. The profile, evaluator-backend and evaluator-request routes need a known key, and the write routes (registering or deleting a profile, assigning a backend, listing pending evaluator submissions) need the platform-admin caller's. GET /health and GET /v1/evaluators need none. The scoring endpoints (POST /v1/score-jobs, GET /v1/score-jobs/{job_id}) identify a known caller and still accept a call without a key during a warning phase, so send your key from the start. The full route table is in the API reference.

Health check

curl -s "https://scoring-service.bravecliff-5469885d.swedencentral.azurecontainerapps.io/health"
# {"status": "ok"}

Connection profiles on the deployed service

The deployed service persists connection profiles to the shared Azure Files volume (/mnt/state/connection_profiles.json), so profiles survive restarts and redeploys. The platform team registers them with the platform-admin caller's key, the same way as locally, using the deployed FQDN and including the API key header. A pod only reads them (GET /v1/connection-profiles, GET /v1/evaluator-backends):

curl -s -X POST \
  "https://scoring-service.bravecliff-5469885d.swedencentral.azurecontainerapps.io/v1/connection-profiles" \
  -H "Content-Type: application/json" \
  -H "X-API-Key: <platform-admin caller key>" \
  -d '{
    "name": "azure-default",
    "kind": "azure_openai",
    "endpoint": "https://your-resource.openai.azure.com/",
    "api_key": "https://your-kv.vault.azure.net/secrets/your-secret",
    "deployment": "gpt-4.1",
    "api_version": "2024-12-01-preview"
  }'

In production, api_key should be a Key Vault secret URI, not a literal string — the service resolves it at evaluation time via managed identity.

Scoring via the SDK

The SDK's ScoringServiceClient (from sdk/semantic_evaluation/) accepts a base_url argument pointing to either the local or the deployed service. Pods running in the peered VNet can call the deployed service directly in their offline-evaluation runs without any code change — only base_url and the caller key differ between environments.

Where to go next

File	What it covers
changelog.md	What a caller notices change, per deploy and API major
00-scoring-overview.md	Architecture, boundary, module map
01-evaluator-catalogue.md	Every supported (framework, evaluator) pair and score normalization
02-connection-profiles.md	The full connection-profile and backend-assignment model
03-custom-rubric-and-registration.md	Pod-defined rubrics, inline vs. named, the registration lifecycle
03-decision-evaluator.md	Fixed-option Microsoft Decision-1 questions, score mappings, and result semantics
04-api-reference.md	Every HTTP route
05-testing.md	Test-suite layers and the dependency-upgrade workflow
06-deployment.md	Container Apps infrastructure, CI/CD pipeline, secret management