Skip to content

Module 03: human review

This module sets up annotation queues for human review and uses the review-sync service to promote reviewed items into the golden dataset, completing the primary way this workshop's dataset grows over time.

Run every command in this module from your scaffold pod directory, the one where you ran uv sync in Module 01 §1.2.

3.1 How traces become dataset items

In Module 01 you seeded the golden datasets from local JSON files via push_all(), a bootstrapping step for when no production data exists yet, or when you want to add SME-curated data from outside Langfuse (e.g. built manually by you or your PO based on a Teams/Email conversation with a domain expert or customer). Module 08 covers that mechanism, and the rest of the eval-dataset CLI, in full.

Going forward, the primary path to grow the golden dataset is human review:

graph TD
  A["Production trace"] --> B["Annotation queue"]
  B --> C["Human scores and corrects"]
  C --> D["Review sync"]
  D --> E["Golden dataset"]

This is what the rest of this module sets up.

3.2 Create annotation queues in Langfuse

In the Langfuse UI:

  1. Go to Human Annotation → Annotation Queues → New Queue.

  2. Create two queues, one per evaluation scope:

    1. Name them support-intent-review and support-reply-review.

    2. Add the two score definitions to each queue. The first time round you also have to create the score configs themselves, via Select → Manage Score Configs → Add new score config. Create one config per name, both with data type NUMERIC, minimum 1, maximum 5:

      • Answer Relevance
      • Answer Correctness

      The names have to match exactly. The bindings you register in section 3.5 set ReviewPolicy.required_score_names to these two strings, so a queue item missing either one is rejected by sync instead of being promoted with incomplete review data.

    3. No need to assign a user now. In a real pod you would assign the expert or experts who score the cases and write the corrected output.

  3. Add the child spans you produced in Module 02 §2.1 to the relevant queues. Open a trace in the Langfuse Tracing view and expand it to see the span hierarchy (support_workflow → classify_intent → generate_reply). Add classify_intent to the intent queue and generate_reply to the reply queue. Do not add the root support_workflow trace.

    Why spans, not the root trace? Langfuse records which object (trace or span) you added to the queue. When a root trace is queued, Langfuse assigns the queue item an object_id with a t- prefix. The review-sync service maps queue items to dataset records by fetching the queued object's input and output; for the intent step that is the classify_intent span's input/output, not the full trace. Adding the root trace also makes score attribution ambiguous when both scopes are queued from the same trace. The SDK has a fallback that handles root-trace queue items, but you will get the correct span-level data only when you queue the span itself.

    Exception: if you want to evaluate the full workflow end-to-end as a single scope (both intent and reply together), then queuing the root support_workflow trace is intentional; the binding's input/output projection should then reference trace-level fields.

You can add comments via 'Add Comment' in the trace view on each level (span/chain/agent/...). You can use that to give the reviewing expert hints, like pointing out what specifically looks not ideal in this agent response you want feedback for. Or what they should regard when reviewing, e.g. "Only score and potentially add corrected output if the intent field, not for rationale. Make sure to always provide scores for both Correctness and Relevance metrics from 1 to 5. If any value is 3 or below, add a corrected output". These comments are only shown to the expert, they are not propagated to the dataset later. Comments by the expert in the reviewing process, however, are appended to the dataset item as reviewer_comment.

Note each queue's Queue ID. Open the queue and read it out of the browser URL, which carries two different ids at once:

.../annotation-queues/<QUEUE_ID>/items/<ITEM_ID>

The first is the queue id that section 3.5 passes to --queue-id. The second identifies one item inside the queue and is not what any command in this module wants; pasting it is the most common cause of 404 Annotation queue not found.

3.3 Score traces as a reviewer

Pretend you are the expert now, and open each trace in the annotation queue (Langfuse UI → Annotation Queues → Process Queue) and provide scores:

Score name What to assess
Answer Relevance Does the intent/reply address the actual customer question?
Answer Correctness Is the intent label / reply content factually correct?

Where a score is low (≤ 3), add corrected output. This is the default policy and agreement between experts and pod engineers (the threshold value can be overwritten by pods). The corrected output is the "expert-in-the-loop" signal that turns production cases (raw traces) into high-quality offline evaluation dataset items.

Additionally, you can add a reviewer comment, for example, explaining what the model got wrong.

Scoring alone does not finish the item: mark it done

Review-sync only picks up queue items whose Langfuse status is explicitly COMPLETED. Adding score values alone does not set that status; after scoring (and correcting, if needed), explicitly mark each item complete/done in the annotation-queue UI. Skipping this step is not an error; sync in section 3.6 will simply report completed_reviewed_items: 0 with no further explanation.

3.4 Register your Langfuse connection profile

Before you can register a binding, the review-sync service needs to know how to reach your Langfuse project. This step is where you hand it those credentials.

Why this step exists

The review-sync service is a single shared process that handles sync jobs for all pods. Each pod uses its own Langfuse project, so when the service reads reviewed queue items on behalf of a binding, it must use that pod's project credentials to authenticate with Langfuse's API. Without a matching profile, the service looks for your queue in the wrong Langfuse project and returns 404 Annotation queue not found. That is one of three causes of the same error; section 3.6 lists all three.

A Langfuse connection profile is simply a named set of credentials (base URL, public key, secret key) that the service uses when running jobs for bindings that reference that profile. You register it once per project, then reference it by profile_id when you register bindings.

graph TD
  A["Your Langfuse project: credentials"] --> B["Review sync service stores profile: profile_id maps to public_key, secret_key, base_url"]
  B --> C["QueueToDatasetBinding.langfuse_profile_id"]
  C --> D["Reader authenticated against your Langfuse project"]

Register the profile

Find your Langfuse public key and secret key in the Langfuse UI under Settings → API Keys.

curl -s -X POST https://review-sync-service.bravecliff-5469885d.swedencentral.azurecontainerapps.io/v1/langfuse-profiles \
  -H "Content-Type: application/json" \
  -H "X-API-Key: <the workshop review-sync key>" \
  -d '{
    "profile_id": "<your-pod-id>",
    "base_url": "https://cloud.langfuse.com",
    "public_key": "<your-langfuse-public-key>",
    "secret_key": "<your-langfuse-secret-key>"
  }'

Use a stable, readable profile_id, for example workshop-pod-a or your name. You will reference the same string in the --langfuse-profile-id argument when registering bindings in section 3.5.

You can list registered profiles at any time:

curl -s https://review-sync-service.bravecliff-5469885d.swedencentral.azurecontainerapps.io/v1/langfuse-profiles \
  -H "X-API-Key: <the workshop review-sync key>" | uv run python -m json.tool

3.5 Register the review-sync binding

The review-sync service needs a binding per scope. A binding is a declarative recipe that tells the service which annotation queue to read, how to turn each reviewed queue item into an EvalCaseRecord, and which Langfuse dataset to write the result into.

What is in a queue item, and where does it go?

When a reviewer scores a trace, Langfuse stores the original request, the agent's selected output, the review scores, and any reviewer comments together as one queue item. A small slice of that payload looks like this:

{
  "record_id": "trace-2c1f...",
  "original_input": {
    "email_subject": "Refund request",
    "email_body": "Order #123 never arrived..."
  },
  "selected_output": {
    "intent": "refund_request",
    "rationale": "Customer asks for a refund on an undelivered order."
  },
  "review_metadata": {
    "workflow_version": "intent_v3",
    "selected_agent": "intent_classifier_v3"
  }
}

The binding projects the parts you care about onto the dataset record:

Binding side Reads from Lands on dataset record
input_projection original_input.* input.*
expected_output_projection selected_output.* (the corrected answer if the reviewer edited it) expected_output.*
metadata_mapping.workflow_version review_metadata.workflow_version metadata.workflow_version
metadata_mapping.selected_agent review_metadata.selected_agent metadata.selected_agent

Each projection is a list of ProjectionField (semantic_evaluation.orchestration.review_sync.service_contracts) entries. A field declares the dataset_field name and one or more ValueSelector (semantic_evaluation.orchestration.review_sync.service_contracts) entries that source it. A selector reads either a dotted queue_item_path out of the queue payload or a constant literal, and the optional coerce casts the raw value into the type the dataset schema expects. Langfuse stores queue-item payloads loosely, so a reviewer may type a number as a string or a boolean as "true", while the golden dataset schema is strict; the cast happens here, at the boundary between the two.

coerce takes a member of the ValueCoercion enum, not a bare string: ValueCoercion.STRING, ValueCoercion.NUMBER, ValueCoercion.INTEGER, ValueCoercion.BOOLEAN, or ValueCoercion.IDENTITY to keep the value exactly as it came in. evaluation/review_to_dataset/main.py imports it already.

When a field declares several selectors, they form an ordered fallback chain: the first one to yield a non-null value wins. This is what lets a binding survive a queue-payload rename: keep the old path first, add the new path as the second selector, and dataset writes never break. The trailing selector can be a literal to provide a default for fields that must always be populated.

One reference projection, one exercise

Open evaluation/review_to_dataset/main.py. It holds the two expected-output projections, and only one of them is finished.

_intent_expected_output_projection() is the complete, working reference. Read it as the pattern to follow for a scope of your own. Its selector list also shows the ordered-fallback-chain idea from above in practice: the extra selectors exist to tolerate a queue item whose selected_output came from a trace captured before a bug fix changed its shape, without needing to re-review it.

_reply_expected_output_projection() is TUTORIAL TODO 1, the first of the workshop's two exercises. In the scaffold it raises NotImplementedError; its own docstring names the dataset field, the queue payload path and the coercion to use. Implement it before you run the --scope reply_generation command below, otherwise that command stops on the exception instead of registering a binding.

Register a binding per scope

Register a binding for each scope. Each call sends the assembled binding to the review-sync service via _register_binding, which first calls ReviewSyncServiceClient.validate_binding to catch shape errors and then ReviewSyncServiceClient.upsert_binding to store it under binding_id.

The --langfuse-profile-id argument must match the profile_id you registered in section 3.4. The service stores the profile reference on the binding and uses it to look up your Langfuse credentials every time a sync job runs for this binding.

# Intent classification binding
uv run python evaluation/review_to_dataset/main.py register-binding \
  --queue-id <your-intent-queue-id> \
  --langfuse-profile-id <your-pod-id> \
  --scope intent_classification

# Reply generation binding
uv run python evaluation/review_to_dataset/main.py register-binding \
  --queue-id <your-reply-queue-id> \
  --langfuse-profile-id <your-pod-id> \
  --scope reply_generation

Each command prints the registered QueueToDatasetBinding as JSON. Skim the output to see the full assembled shape. The input_projection.fields, expected_output_projection.fields, and metadata_mapping blocks should match the projection functions in main.py. Check that dataset_target.name matches the Langfuse dataset names from Module 01 (northwind_intent_classification_dev / northwind_reply_generation_dev).

Review policy gating is configured on the same binding via ReviewPolicy. This workshop's bindings set only required_score_names: a queue item missing either Answer Relevance or Answer Correctness is rejected by sync with an explicit reason, rather than silently promoted with incomplete review data.

3.6 Run the sync: once configured, run forever

Trigger one sync per binding:

uv run python evaluation/review_to_dataset/main.py sync \
  --queue-id <your-intent-queue-id>

uv run python evaluation/review_to_dataset/main.py sync \
  --queue-id <your-reply-queue-id>

The review-sync service reads scored items from the queue, runs each one through the binding's review policy, projects it into a dataset record, and writes matching items into the Langfuse dataset.

404 Annotation queue not found has three different causes

The message is the same for all three, so check them in this order:

  • The queue item id was pasted instead of the queue id. Both sit in one Langfuse URL, .../annotation-queues/<QUEUE_ID>/items/<ITEM_ID>, and --queue-id wants the first one (section 3.2).
  • The binding names the wrong Langfuse connection profile, so the service authenticates against a project where that queue does not exist. Compare the binding's --langfuse-profile-id against the profile_id you registered in section 3.4.
  • The binding is stale, registered against a queue that has since been deleted and recreated. Langfuse issues a fresh id each time, so re-run register-binding with the current queue id; the upsert replaces the old binding rather than adding a second one.

Configuration is one-off, running is routine. Everything before this section (creating the queues, registering the binding, naming the dataset) only happens once per (queue, dataset) pair. After that, only the sync command needs to run, and it can run as often as you like:

  • Manually, the way you just did, when you want immediate promotion of a fresh batch of reviewed traces.
  • On a schedule via any cron-like cloud trigger (Azure Functions timer, Container Apps Job, GitHub Actions schedule, Kubernetes CronJob), typically every few hours; four times a day is a common cadence and keeps the dataset close to live without flooding the service with empty runs.

The same sync command serves both, so the schedule is just an operational choice and not a change to the workflow.

3.7 Verify in Langfuse

Open the two golden datasets and confirm that items with ground_truth_status = "reviewed" appeared alongside the seed items from Module 01. Reviewed items have non-null review_score values in their metadata.

You now have a dataset that mixes:

  • Seeded items: bootstrapped from static files or engineer judgment.
  • Reviewed items: production traces that passed through human quality control.

Modules 04 to 07 use this combined dataset for automated evaluation.

3.8 Why this scales

The cost of growing the dataset from here on is small and roughly fixed per reviewed item. A reviewer opens a trace in Langfuse, gives it a score, optionally edits the answer, and clicks save. The scheduled sync turns that click into a permanent dataset item the next time it runs.

A back-of-the-envelope picture: three reviewers each annotating only five traces per week is fifteen reviewed items per week, sixty per month, more than seven hundred per year, and the engineering effort to turn them into golden dataset items is exactly zero, because every addition flows through the same binding you registered once in section 3.5. A dataset that started with a handful of seed items in Module 01 becomes hundreds of curated, production-derived cases within a quarter, with no new code, no new scripts, and no manual JSON wrangling. The more reviewers you add and the more queues you point at, the more this compounds.

Continue to Module 04.