Module 02: Tracing

This module runs the support-assistant workflow, inspects traces in Langfuse, and learns the tracing SDK's span model and local trace capture. Human review (annotation queues, review-sync) is Module 03.

2.0 Explore the scaffold

Take a few minutes to read through the scaffold folder before running anything. Both scaffold (MAF/LangChain) options share the same folder layout:

<pod>/
├── agent/                            # The application under evaluation
│   ├── workflow.py                   #   Two-step pipeline: classify_intent → generate_reply
│   ├── models.py                     #   Shared Pydantic types (SupportEmail, SupportReply, …)
│   ├── prompts.py                    #   Prompt templates
│   ├── tools.py                      #   lookup_account_status: a toy tool call, see Module 05
│   └── llm.py                        #   LLM client and chain factories
│
└── evaluation/                       # Evaluation wiring: wraps around agent/, not inside it
    ├── trace_to_review/              #   Run workflow → write trace → (used in this module)
    │   └── main.py                   #     CLI entry point (run)
    ├── review_to_dataset/            #   Configure review-sync bindings + trigger sync (Module 03)
    │   └── main.py
    ├── offline_eval/                 #   Run evaluators against a dataset snapshot (Modules 04-07)
    │   └── main.py
    ├── evaluation_manifest.yaml      #   Every metric and the dataset of each scope (Module 04)
    └── runs/
        └── workshop_local.yaml       #   The evaluation run: its scopes and gate (Module 04)

Key relationship: agent/ is the system you are evaluating. evaluation/ is the scaffolding that connects your code to tracing, dataset management, and the scoring service.

Trace the call path through your chosen scaffold:

LangChainAgent Framework (MAF)

Open agent/workflow.py. run_support_workflow calls _classify_intent and then _generate_reply, two plain async functions, each decorated with @step(...) from the shared SDK, which is what makes them appear as spans in Langfuse. _generate_reply also calls lookup_account_status in agent/tools.py, its own @step(..., as_type="tool") observation, and the subject of Module 05's local-capture exercise.

Open agent/workflow.py. run_support_workflow builds a WorkflowBuilder wiring two Executor classes, IntentClassifier in agent/intent_classifier.py and ReplyGenerator in agent/reply_generator.py, and streams it to completion. Each executor's handle() method carries the same @step(...) decoration as the LangChain functions, which is what makes them appear as spans in Langfuse. ReplyGenerator also calls lookup_account_status in agent/tools.py, its own @step(..., as_type="tool") observation, and the subject of Module 05's local-capture exercise.

Then open the two files that configure the evaluation Modules 04 to 07 run: evaluation/evaluation_manifest.yaml and evaluation/runs/workshop_local.yaml.

2.1 Simulate production requests

In production, real users submit support emails through a front end or API. The workflow runs, answers the customer, and, because every step is wrapped with @step, writes a trace to Langfuse automatically. That trace is the raw material for the human-review flow built in Module 03.

The workshop has no deployed front end, so the run subcommand of evaluation/trace_to_review/main.py is a thin stand-in that calls the same workflow function a real request handler would call. The traces it produces in Langfuse are identical to what production traffic creates.

From your scaffold pod directory (VPN must be on), run several emails. Each also takes an --account-id (default acct-1001), the account lookup_account_status resolves during reply generation:

uv run python evaluation/trace_to_review/main.py run \
  --subject "Salesforce sync failing since Tuesday" \
  --body "Our Salesforce sync stopped working Tuesday morning. The log shows 'connection lost'. Nothing changed on our side." \
  --account-id acct-1003

uv run python evaluation/trace_to_review/main.py run \
  --subject "Can we get SSO support?" \
  --body "We'd like to use Azure AD single sign-on with your platform. Is that on the roadmap?" \
  --account-id acct-1002

uv run python evaluation/trace_to_review/main.py run \
  --subject "Billing discrepancy last month" \
  --body "We were charged for 12 seats but only have 8 active users. Please clarify." \
  --account-id acct-1001

Each command prints the SupportReply defined in agent/models.py as JSON and writes a trace to Langfuse.

Open the Langfuse UI and verify that traces named support_workflow, after run_support_workflow in agent/workflow.py, appeared with child spans classify_intent and generate_reply.

2.2 Understand the tracing structure

What the shared SDK provides

Pod engineers do not configure OpenTelemetry providers or Langfuse callbacks directly. The shared SDK semantic_tracing wraps that setup into a small surface:

SDK helper	What it does for the engineer
`@step(name, as_type=..., capture_output=...)` (`semantic_tracing.decorators`)	Decorates any async or sync function to emit an OTel span, with no boilerplate; works for both MAF executors and plain functions
`get_langchain_callback()` (`semantic_tracing.adapters`)	Returns a ready-made Langfuse `CallbackHandler` that LangChain LCEL chains accept directly
`get_observation_id()` (`semantic_tracing.ids`) / `get_trace_id()`	Returns the IDs the evaluation layer needs to link dataset items back to traced observations. Must be called inside the still-active `@step`-decorated function, not by the caller after it returns
`propagate(user_id=..., session_id=...)` (`semantic_tracing`)	Context manager that stamps every span inside its block with a user/session ID, so Langfuse's Sessions and Users views group related traces
`setup_tracing()`	One-liner process-wide init: sets the OTel exporter, Langfuse project, and sampling

This means every pod, regardless of framework, produces the same span structure and metadata fields. The downstream review-sync and offline-eval steps rely on this consistency without per-pod custom logic.

capture_output=True silently overrides an explicit update_current_step(output=...) call

@step's capture_output defaults to True, which auto-records the decorated function's return value as the observation's .output, even if the function body already called update_current_step(output=...) explicitly earlier. If a step function's return value is ever changed for a reason unrelated to what should appear in the trace (for example, returning an extra value purely for the caller's benefit), add capture_output=False to that @step(...) call, or the auto-capture silently wins with no error or warning.

Span hierarchy in the Langfuse UI

Open one of the traces you just created. Each span has an icon that indicates its as_type:

Type	Meaning	Example in this workflow
`chain`	A logical sequence that groups child steps	`support_workflow`
`agent`	A self-contained reasoning step	`classify_intent`, `generate_reply`
`generation`	A single LLM call (icon shows the provider, here Azure OpenAI)	The nested completions call

Other important observation types supported by the shared SDK but not used in this workflow:

Icon	Type	Meaning
	`trace`	Top-level container: one per request, created automatically
	`event`	Point-in-time marker with no duration (cache hit, decision point)
None	`tool`	A discrete tool/function call used by `lookup_account_status` (Module 05)
None	`span`	The generic type, used when none of the others applies. Langfuse's API reports the LangChain scaffold's `@step` observations as `SPAN` where this one's come back as `AGENT`

The intended hierarchy:

graph TD
  Root["support_workflow: chain"]
  Classify["classify_intent: agent"]
  Gen1["LLM generation"]
  Reply["generate_reply: agent"]
  Tool["lookup_account_status: tool, see Module 05"]
  Gen2["LLM generation"]

  Root --> Classify
  Classify --> Gen1
  Root --> Reply
  Reply --> Tool
  Reply --> Gen2

Why you don't see MAF's own workflow spans

agent_framework instruments its own machinery, so a MAF workflow also produces workflow.build, workflow.run, executor.process intent_classifier/reply_generator, edge_group.process SingleEdgeGroup and message.send. All six record how the framework moved data around rather than what the assistant did, and all six carry no input and no output, which makes them easy to mistake for a pod that failed to record anything.

The scaffold names them in telemetry/setup.py, which passes drop_span_names and drop_span_name_prefixes to setup_tracing(). Open that file: it is the only reason your trace matches the hierarchy above. Two of those spans, workflow.run and executor.process ..., sit directly between support_workflow and the two agent steps, and dropping them still leaves classify_intent and generate_reply nested where the diagram shows them, because each surviving span is reattached to the nearest ancestor that survived with it.

workflow_built is not part of that set. It is an event the pod itself records, at agent/workflow.py's event("workflow_built") call, so it stays in the trace and no drop list could remove it.

Each span carries an observation_id (semantic_tracing.ids). This is the link between the trace and the dataset item that the review-sync service writes; it connects "the output that was reviewed" to "the generation that produced it".

2.3 Local trace capture

Besides exporting to Langfuse, the SDK can keep a short-lived, in-process copy of the current trace's observations, readable synchronously via get_local_trace(trace_id), no network round trip required. This is enabled per pod with setup_tracing(..., capture_local_traces=True) (already set in both scaffolds' telemetry/setup.py).

Local capture matters whenever a prediction needs data that a workflow's own return value doesn't carry. The concrete example in this workshop is lookup_account_status's result, which never appears on SupportReply at all. Module 05 is the hands-on exercise that recovers it via get_local_trace() + find_observations(); this module only introduces that the mechanism exists.

One structural detail worth knowing ahead of Module 05: LocalObservation.output is read straight off the underlying OTel span attribute, which can only hold primitive types (str, int, float, bool, or arrays of those), so a function that returns a Pydantic model comes back JSON-encoded as a string, not as a dict. Module 05 is where this becomes practically relevant.

2.4 Sessions and users

propagate(user_id=..., session_id=...) is what populates Langfuse's Sessions and Users views. Without it, every trace is an isolated singleton regardless of what you configure in the UI. trace_to_review/main.py run wraps its run_support_workflow(...) call in exactly this context manager:

from semantic_tracing import propagate

with propagate(user_id=user_id, session_id=session_id):
    await run_support_workflow(email)

Pass --user-id/--session-id on the CLI:

uv run python evaluation/trace_to_review/main.py run \
  --subject "Invoice question" \
  --body "Our invoice was higher this month. Can you explain?" \
  --account-id acct-1001 \
  --user-id alice@example.com \
  --session-id session-42

Run two of the example emails from section 2.1 again with the same --session-id value, then open Langfuse's Sessions view. You should see one session grouping both traces, rather than three isolated ones.

Continue to Module 03.