Skip to content

Workshop overview

This workshop teaches you to build an evaluation pipeline around an existing LLM-powered application. You work through the modules in order: each builds on the output of the previous one.

Modules

Module Title What you do
01 Setup Install dependencies, configure Langfuse and Azure OpenAI credentials, connect to the shared scoring and review-sync services, publish a starter golden dataset
02 Tracing Explore the example app, run it to produce traced responses, learn the tracing SDK's span model and local trace capture
03 Human Review Create annotation queues, score traces as a reviewer, configure review-sync to promote reviewed items into the golden dataset
04 Run Manifest Learn the RunManifest contract and build/extend one for your own metrics
05 Predictor Adapter Implement the PredictionRunner contract, including recovering a tool call's result from the trace
06 Metric Authoring Write a GEval rubric, tune it, and author a fully custom pod-local metric
07 Running Evaluations Run the full evaluation pipeline and read every output artifact
08 Dataset Lifecycle Use the eval-dataset CLI to pull, edit, push, promote, and archive golden datasets
09 Custom Metrics Submit a pod-local rubric for central, cross-pod registration

See the glossary for definitions of terms used throughout.

The two exercises

Almost all of the scaffold pod ships complete and runnable. Two functions are left deliberately unfinished, marked TUTORIAL TODO in the code and listed in the Tutorial exercise table in your pod's own README.md:

Exercise File Module What you build
TUTORIAL TODO 1 evaluation/review_to_dataset/main.py 03 _reply_expected_output_projection: the reply scope's expected-output projection
TUTORIAL TODO 2 evaluation/offline_eval/predictors/support_workflow_adapter.py 05 _build_prediction: map a workflow response onto a PredictionRecord, including recovering the lookup_account_status result from the trace

Both raise NotImplementedError until you implement them, so the module that first needs one stops there rather than working around it.

What the workshop is not

  • It is not a tutorial on building the agent itself: the example app is provided ready-made in the agent/ folder of the workshop scaffold you find for Microsoft Agent Framework and LangChain.
  • It does not walk you through deploying the scoring or review-sync services yourself. Both already run as deployed Azure Container Apps you connect to (Module 01); building or operating that infrastructure is not covered here.
  • It does not cover production monitoring, online evaluation, or CI/CD integration. Those are later chapters to be implemented in the broader evaluation architecture.

Key concepts you will encounter

Concept One-sentence explanation
Golden dataset A curated set of input/expected-output pairs that the offline evaluator measures the system against
Trace A structured log of one end-to-end workflow execution, captured automatically by the @step decorator
Local capture An in-process cache of the current trace's observations, readable via get_local_trace() without a round trip to Langfuse
Annotation queue A Langfuse feature where a human reviewer scores traced outputs for quality
Review sync A service that takes human-reviewed traces and promotes them into the golden dataset
Offline evaluation A batch process that reruns the workflow against every dataset item, scores the results with LLM-judge and statistical metrics, and checks a quality gate
Scoring service A shared HTTP service that executes evaluator metrics (DeepEval, Azure AI Evaluation, and pod-defined custom rubric evaluators) without exposing credentials to pod code
Run manifest A JSON file that declares which datasets, evaluators, thresholds, and gate rules define one evaluation run

Full definitions, including terms specific to one module, live in the glossary.

Prerequisites

  • Python 3.12+
  • uv package manager (recommended) or conda
  • Azure OpenAI access (with VPN)
  • A free Langfuse Cloud account
  • VS Code (recommended)