Deployment

scoring-service runs as an Azure Container App in Semantic-Evaluation-Dev-RG, in the semantic-eval-nonprod-env Workload Profiles environment in Sweden Central. This page covers the infrastructure layout, CI/CD pipeline, secret management, and routine operational tasks.

Infrastructure overview

graph TD
  subgraph VNet["AZUSC-SemanticEvaluationSystem nonprod vnet"]
    subgraph Subnet["Container-app-subnet 10.9.75.128/27"]
      subgraph Env["semantic-eval-nonprod-env Workload Profiles"]
        SS["scoring-service"]
        RS["review-sync-service"]
      end
    end
  end
  subgraph Azure["Azure platform services"]
    KV["semantic-eval-dev-kv Key Vault"]
    AF["semevalnpstorage Azure Files share service-state"]
    AR["artifactory.ramboll.com container registry"]
  end
  SS -->|"mounts /mnt/state"| AF
  RS -->|"mounts /mnt/state"| AF
  SS -->|"reads caller keys and judge keys via managed identity"| KV
  RS -->|"reads caller keys and judge keys via managed identity"| KV
  AR -->|"image pulled at deploy"| SS
  AR -->|"image pulled at deploy"| RS
  style VNet fill:#e9ebec,stroke:#273943,stroke-width:2px
  style Subnet fill:#dde6ec,stroke:#273943,stroke-width:2px
  style Env fill:#e5f4fd,stroke:#0098eb,stroke-width:2px
  style Azure fill:#f5f0e8,stroke:#cc8844,stroke-width:2px

The environment runs in a Workload Profiles configuration because the assigned Container-app-subnet is a /27 (32 addresses), which is below the /23 minimum required for a Consumption-only environment but meets the /27 Workload Profiles minimum.

Both services share one Azure Files volume (service-state) mounted at /mnt/state. The state files each service writes are disjoint — scoring-service writes connection_profiles.json, custom_rubric_requests/, and jobs/ (one JSON record per scoring job); review-sync-service writes bindings.json, langfuse_profiles.json, jobs/, checkpoints/, and cleanup_audits/. Persisting job records to the shared mount, rather than to memory or container-local disk, is what lets a GET /v1/score-jobs/{job_id} poll land on either replica and still see the job a POST /v1/score-jobs created on the other one.

Scoring-service Container App properties

Property	Value
Container App name	`scoring-service`
Environment	`semantic-eval-nonprod-env`
Resource group	`Semantic-Evaluation-Dev-RG`
Location	Sweden Central
FQDN	`scoring-service.bravecliff-5469885d.swedencentral.azurecontainerapps.io` (VPN required; the `.internal.`-prefixed variant does not resolve correctly from outside the VNet)
Port	`8001`
Image	`artifactory.ramboll.com/semantic-evaluation/scoring-service`
Min replicas	`1`
Max replicas	`2`
State volume	Azure Files share `service-state`, mounted at `/mnt/state`
Managed identity	System-assigned — principal `18ca8816-e9fd-4e1c-9e71-61c396d0cd3e`
Caller keys	`scoring-caller-<caller-id>` secrets in `semantic-eval-dev-kv`, read by the service itself (see Caller keys)

The FQDN is assigned by Azure when the environment is created and will change if the environment is recreated. To retrieve the current value:

az containerapp show \
  --name scoring-service \
  --resource-group Semantic-Evaluation-Dev-RG \
  --query "properties.configuration.ingress.fqdn" -o tsv

Environment variables

The Container App injects these environment variables into the container at startup:

Variable	Value in Container App
`OFFLINE_SCORING_HOST`	`0.0.0.0`
`OFFLINE_SCORING_PORT`	`8001`
`OFFLINE_SCORING_RELOAD`	`false`
`OFFLINE_SCORING_DEFAULT_MODEL`	`gpt-4.1`
`OFFLINE_SCORING_STATE_DIR`	`/mnt/state`
`OFFLINE_SCORING_MAX_CONCURRENT_JUDGE_CALLS`	`4` (default; not currently overridden in this environment)
`CALLER_KEY_VAULT_URL`	`https://semantic-eval-dev-kv.vault.azure.net/`

CALLER_KEY_VAULT_URL names the vault the service reads its caller keys from. The Container App carries no API-key secret reference: the service lists the scoring-caller- secrets itself with its managed identity. At startup it logs Loaded N callers: <ids>; with the variable unset it logs a warning and leaves every route open.

CI/CD pipeline

The pipeline at .devops/backend-services-azure-pipeline.yml runs on every main merge that touches scoring_app/, review_sync_app/, sdk/ or service_auth/:

build_and_push_images job — builds both service images from the repo root (so the Dockerfiles can COPY sdk/ alongside the app folder) and pushes two tags to Artifactory: $(Build.BuildId) (unique, traceable per run) and latest.
deploy_services job — runs only on main merges (not PRs), using the semantic-eval-dev Azure DevOps service connection. Updates each Container App's active image to the $(Build.BuildId) tag just pushed, then re-pins targetPort via az containerapp ingress update to ensure it survives the image update.

To trigger a forced redeploy without a code change, rerun the pipeline manually in Azure DevOps — the $(Build.BuildId) tag of that run will be deployed.

Secret management

Keys live in semantic-eval-dev-kv as Key Vault secrets. The service reads them itself with its managed identity, through the private endpoint.

KV secret	Used by
`scoring-caller-<caller-id>`	`scoring-service`: one caller key per caller
`review-sync-caller-<caller-id>`	`review-sync-service`: one caller key per caller
`decision-judge-api-key`	`scoring-service`, through the `decision-dev-foundry` connection profile
`generative-judge-api-key`	Kept for a generative judge profile on `semantic-evaluation-dev-foundry`; no backend of the deployed service uses it

Both managed identities hold Key Vault Secrets User RBAC on semantic-eval-dev-kv. Key Vault has public network access disabled and a private endpoint in the VNet; VPN is required to access the data plane from a developer machine.

Judge keys in connection profiles

The judge keys are read differently from the caller keys. A connection profile stores the secret's URI (https://semantic-eval-dev-kv.vault.azure.net/secrets/<name>), and the service code resolves it with its managed identity when a scoring request builds the evaluator, through the private endpoint. This needs azure-keyvault-secrets, which the service gets from its semantic-evaluation[azure] dependency. See 02-connection-profiles.md.

The deployed service's judge backends are assigned like this:

Backend	Profile	Model deployment	Key
`decision_llm_judge`	`decision-dev-foundry` (`foundry_decision`)	`Decision-1-DZ` on `semantic-evaluation-dev-foundry`	Key Vault URI of `decision-judge-api-key`
`deepeval_llm_judge`, `custom_rubric_llm_judge`, `azure_ai_evaluation_llm_judge`	`azure-default` (`azure_openai`)	`gpt-5.1-1`	as registered

Store a judge key once with az keyvault secret set --vault-name semantic-eval-dev-kv --name <name> --value "$KEY" -o none, then register the profile with the URI.

Caller keys

Every caller (a pod, the workshop, the platform team) has its own key for each service, so a key can be rotated or revoked without touching anyone else. A caller key is a Key Vault secret:

Secret field	Meaning
Name	`scoring-caller-<caller-id>` (the review-sync service uses `review-sync-caller-<caller-id>`)
Value	The caller's key
Tag `role`	`admin` (the platform-admin caller) or `pod`; no tag means `pod`

Disabled and expired secrets are ignored. The service reloads the secrets every 5 minutes, so issuing, rotating or revoking a caller needs no deploy and takes effect within that interval. These callers exist:

Caller id	Role	Purpose
`platform-admin`	`admin`	The platform team: connection profiles, backend assignment, evaluator submissions
`workshop`	`pod`	Workshop attendees; its keys are rotated after each workshop
`checkmate`	`pod`	Checkmate, the legal assistant (`legal_poc` in the `Checkmate` repository)
`hr-mvp`	`pod`	The HR assistant MVP (the `Project Green` repository)
`shared`	`pod`	Holds the former shared key value so existing callers keep working during the warning phase; retired at enforcement

Which routes need which caller is in the API reference. Every request log line carries caller=<id> (unknown for an unidentified call on a warning-phase route, - when none applies).

Issuing a caller

The platform team issues keys, from a machine on VPN:

az keyvault secret set \
  --vault-name semantic-eval-dev-kv \
  --name scoring-caller-<caller-id> \
  --value "$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')" \
  --tags role=pod -o none

Repeat with --name review-sync-caller-<caller-id> for the review-sync service. Read the key to hand it over:

az keyvault secret show --vault-name semantic-eval-dev-kv \
  --name scoring-caller-<caller-id> --query value -o tsv

Hand the value over out-of-band (1:1, never a shared channel) and never commit it. The caller sets it as SCORING_SERVICE_API_KEY in the pod's environment; the SDK client sends it as X-API-Key. Someone without Key Vault Secrets User on the vault asks the platform team; access and credentials names the contact.

Rotating a caller key

Run az keyvault secret set again with the same --name, a new --value and the same --tags role=<role>, and give the caller the new value. Tags belong to a secret version, so a new version without --tags turns an admin caller into a pod caller. The old value stops working at the next reload, at most 5 minutes later.

Revoking a caller

az keyvault secret set-attributes --vault-name semantic-eval-dev-kv \
  --name <secret> --enabled false

Deleting the secret has the same effect.

Operational checks

Check running status

az containerapp show \
  --name scoring-service \
  --resource-group Semantic-Evaluation-Dev-RG \
  --query "{status:properties.runningStatus, revision:properties.latestRevisionName}"

Tail recent logs

az containerapp logs show \
  --name scoring-service \
  --resource-group Semantic-Evaluation-Dev-RG \
  --tail 50

Verify the active image tag

az containerapp revision show \
  --name scoring-service \
  --resource-group Semantic-Evaluation-Dev-RG \
  --revision "$(az containerapp show \
    --name scoring-service \
    --resource-group Semantic-Evaluation-Dev-RG \
    --query properties.latestRevisionName -o tsv)" \
  --query "properties.template.containers[0].image" -o tsv

Test the health endpoint (from inside the VNet)

curl -s "https://scoring-service.bravecliff-5469885d.swedencentral.azurecontainerapps.io/health"
# {"status": "ok"}

Reprovisioning from scratch

If the Container Apps environment needs to be recreated, the provisioning steps from Wave 7 apply. Prerequisites: semevalnpstorage storage account and service-state Azure Files share must exist; semantic-eval-dev-kv secrets must exist; the Container Apps extension and the semantic-eval-dev service connection must be active in Azure DevOps.

See docs/history/2026-07-22-wave-7-container-apps-deployment.md and docs/history/2026-07-22-wave-8-managed-identity-keyvault.md (at repository root) for the original step-by-step provisioning commands.