Module 01: Setup

This module installs dependencies, configures credentials, connects to the shared services, and publishes the seed golden dataset into your Langfuse project.

1.1 The shared services

The workshop uses two shared services, run centrally for everyone as Azure Container Apps. You never install or start them yourself:

Service	Role
`scoring_app`	Runs the offline-evaluation evaluators
`review_sync_app`	Bridges Langfuse annotation queues to datasets

Turn on your VPN now. It's required to reach both services, and again in section 1.2 to install from the internal package registry. You'll need it from Module 02 onward too, once you start running the agent itself, because Azure OpenAI is VPN-restricted. Langfuse and the seed publish in section 1.4 are the only steps that work without it.

The deployed URLs:

https://scoring-service.bravecliff-5469885d.swedencentral.azurecontainerapps.io
https://review-sync-service.bravecliff-5469885d.swedencentral.azurecontainerapps.io

Confirm both are reachable:

curl https://scoring-service.bravecliff-5469885d.swedencentral.azurecontainerapps.io/health
curl https://review-sync-service.bravecliff-5469885d.swedencentral.azurecontainerapps.io/health

Both return {"status": "ok"}. If a service has been idle, the first request can take 10-15 seconds while it cold-starts. That's normal, not a hang; a second request right after is fast.

Every caller of the two services has its own key, sent as an X-API-Key header. Yours is the workshop caller's key, one per service, shared by the attendees and rotated after each workshop. You need it to register your Langfuse project with the review-sync service (module 03) and to read the scoring service's evaluator-backend routes and submit a rubric (modules 06 and 09). Scoring a run and syncing reviews also send it, so the services identify the workshop; they still accept a call without one for now. Ask your facilitator for the two key values in advance, and set them in your .env file in section 1.3. Working through the modules on your own, ask Peter Ewald (on Teams or at peter.ewald@ramboll.com), the contact for access and keys, who also answers wherever a module says to ask the facilitator. A key that starts getting HTTP 401 has been rotated after a workshop; ask for the current one.

1.2 Choose and install a scaffold

Pick the framework you want to work in and download that starter scaffold from the portal's Get started page: Microsoft Agent Framework or LangChain. Both build the same support-assistant pod and both work for every module; choose whichever is closer to what you build day to day.

Extract the zip somewhere of your own and open a terminal in the extracted folder. Everything from here on happens inside it.

Authenticate to the package registry

The two shared SDKs, semantic-tracing and semantic-evaluation, are published to the internal Ramboll Artifactory registry rather than to public PyPI, so the install needs a credential and your VPN. Do this once per machine, not once per pod: create ~/.netrc with

machine artifactory.ramboll.com
  login    <your-email-or-empty-string>
  password <your-artifactory-token>

and generate the token in the Artifactory UI under Edit Profile → Authentication Settings. Both uv and pip read ~/.netrc on their own, and your pod already names the registry in its uv.toml, so there is nothing else to configure.

Skipping this step produces an HTTP 401 from artifactory.ramboll.com during the install below, not a missing-package error.

Install

uv (recommended)conda

uv sync

Each scaffold includes an environment.yml:

# Microsoft Agent Framework
conda env create -f environment.yml
conda activate eval-workshop-agent-framework

# LangChain
conda env create -f environment.yml
conda activate eval-workshop-langchain

Confirm the install with the pod's own test suite, which needs no credentials:

uv run pytest

Every command in this workshop is written as uv run ..., and every module assumes you run it from the pod directory you just installed. On the conda path, activate your environment and drop the uv run prefix throughout.

1.3 Configure credentials

Create your Langfuse project

Each attendee needs a separate Langfuse project. Sharing one project mixes traces, datasets, and annotation queues across everyone, which makes it hard to follow the exercises.

Go to cloud.langfuse.com and sign in.
Create a new project, named after your pod or project (e.g. eval-workshop-<your-name>).
Create new API Keys and create a new key pair. This gives you the LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY used below.

Fill in the environment file

Copy your pod's .env.workshop.template and fill in the values:

cp .env.workshop.template .env

Variable	Where to get it
`AZURE_OPENAI_ENDPOINT`	Azure portal → your Azure OpenAI resource
`AZURE_OPENAI_API_KEY`	Azure portal → Keys and Endpoint
`AZURE_OPENAI_DEPLOYMENT`	Azure AI Studio → Deployments
`LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY`	Your Langfuse project → Settings → API Keys
`LANGFUSE_BASE_URL`	`https://cloud.langfuse.com` or your self-hosted URL
`WORKSHOP_PRODUCT_ID`	Leave it at `northwind`, a fictional codename identifying this pod, not the demo app. It names the dataset review-sync writes reviewed items into, and the seed files shipped with your pod hardcode `northwind`, so changing it splits the two apart.
`OFFLINE_SCORING_BASE_URL` / `REVIEW_SYNC_SERVICE_BASE_URL`	Already set to the deployed services (section 1.1), no change needed
`SCORING_SERVICE_API_KEY` / `REVIEW_SYNC_API_KEY`	The workshop caller's keys: ask your workshop facilitator (section 1.1)

AZURE_OPENAI_ENDPOINT: two different URL shapes look almost identical

The Azure AI Foundry portal shows a per-deployment URL ending in /openai/v1/responses (the newer Responses API). agent/llm.py's AsyncAzureOpenAI(azure_endpoint=...) (or AzureChatOpenAI in the LangChain scaffold) is the classic client, which expects just the bare resource root:

https://{resource}.openai.azure.com/

and builds the full request path itself. Pasting the Foundry portal's Responses-API URL verbatim into .env produces a plain openai.NotFoundError: 404 with no hint about which URL shape is wrong.

A shell-exported value silently wins over .env, with no error

config.py/service_config.py both load .env via load_dotenv(..., override=False). A value already exported in your shell (e.g. from an earlier export AZURE_OPENAI_ENDPOINT=... test) silently takes priority over the correct .env value, and you will not see it anywhere in the file itself. If a call fails in a way that doesn't match what .env says, check the shell first:

env | grep AZURE_OPENAI

before spending time re-debugging the code.

LLM-judge scoring also needs a connection profile assigned to each judge backend on the scoring service: the model the judge calls. On the deployed service the platform team has registered and assigned them for the whole workshop, so you register none (registering a profile on the deployed service needs the platform-admin caller; on a service of your own you register one yourself). An assignment holds for every caller of the service, so do not change one (a workshop key cannot). The scoring service's Connection profiles page describes how they work.

When you run a scoring service of your own: a profile keeps its own endpoint

A connection profile stores its own endpoint field independently of .env, so fixing .env does not update a profile you already registered earlier. The same bare-resource-root vs. Responses-API URL distinction above applies here too, and there is no single file to check: inspect what is actually stored with curl GET /v1/connection-profiles/{name}. Re-POSTing the same profile name overwrites it cleanly (no conflict).

default_model must name a canonical model family, not a deployment name

The scoring service's OFFLINE_SCORING_DEFAULT_MODEL setting must be the canonical DeepEval-recognized model-family name (gpt-5-mini, gpt-4.1, …) matching whatever your connection profile's deployment field actually points at, not an arbitrary label, and not the deployment name itself if the two differ (a deployment named gpt-5-mini-DZ still needs OFFLINE_SCORING_DEFAULT_MODEL=gpt-5-mini). A mismatch doesn't fail loudly: DeepEval silently falls back to a generic model profile, producing symptoms like temperature values being rejected by the real API. This setting is read once at process startup, so a service restart is required after changing it.

1.4 Publish the seed dataset

Your pod ships a seed dataset: five hand-drafted customer emails and expected outputs, one file per evaluation scope, already in the local JSON shape the eval-dataset CLI expects.

evaluation/datasets/seed/intent_classification_dev.json
evaluation/datasets/seed/reply_generation_dev.json

Get familiar with the structure and one example item in each file. Notice that these datasets use custom fields (email_subject, email_body, intent, …) rather than one of the four canonical schema types. Module 08 explains why.

Before any traces exist, offline evaluation could already run against these seed datasets. Imagine you created the starter items yourself based on your intuition when the project just began, or your PO gave you initial data.

Now publish them to your Langfuse project. One eval-dataset push per scope, run from your pod directory:

uv run eval-dataset push \
  --name northwind_intent_classification_dev \
  --source evaluation/datasets/seed/intent_classification_dev.json \
  --version-label 1.0 --create-on-missing \
  --input-schema-json evaluation/datasets/schemas/support_email_input.schema.json \
  --expected-output-schema-json evaluation/datasets/schemas/intent_classification_expected_output.schema.json

uv run eval-dataset push \
  --name northwind_reply_generation_dev \
  --source evaluation/datasets/seed/reply_generation_dev.json \
  --version-label 1.0 --create-on-missing \
  --input-schema-json evaluation/datasets/schemas/support_email_input.schema.json \
  --expected-output-schema-json evaluation/datasets/schemas/reply_generation_expected_output.schema.json

--create-on-missing is what makes the dataset appear in Langfuse the first time; the two --*-schema-json flags attach the JSON Schemas that describe each scope's shape, and they only apply at creation. Both scopes share support_email_input.schema.json because the input to this pod is a customer email either way, and only the expected output differs. eval-dataset reads your .env, so the credentials from section 1.3 are picked up automatically.

Each command prints a summary; on a first run the counts look like this:

{
  "dataset_name": "northwind_intent_classification_dev",
  "version_label": "1.0",
  "dry_run": false,
  "created": ["northwind-WS-01-intent_classification", "..."],
  "updated": [],
  "backfilled": [],
  "would_delete": [],
  "unchanged_count": 0
}

Open your Langfuse project and go to Datasets → Items and confirm that two datasets appeared and they contain the expected columns (Item ID, Status, Created At, Input, Expected Output, and Metadata):

northwind_intent_classification_dev
northwind_reply_generation_dev

Item IDs read northwind-WS-01-intent_classification, not the bare WS-01 that names the underlying email. Langfuse enforces dataset-item-ID uniqueness project-wide across all datasets, not per dataset, and both scopes' items describe the same five emails, so bare IDs would collide the moment the second scope is pushed. The shipped files are written with the product and scope already in each ID for exactly that reason.

1.5 Verify services

Re-run the health checks from section 1.1 and confirm both still return {"status": "ok"} before continuing, and that you're still on VPN.

Continue to Module 02.