Skip to content

Module 05: predictor adapter

This module covers the PredictionRunner contract, the piece that replays each dataset item through the live workflow and shapes the response into a PredictionRecord the scoring service can score. The hands-on exercise recovers a value that only exists in the trace, not in the workflow's return value at all.

5.1 The PredictionRunner contract

class PredictionRunner(Protocol):
    name: str
    async def predict_items(self, items: list[EvaluationItem]) -> list[PredictionRecord]: ...

The run file's predictor_adapter (Module 04) names this class as evaluation.offline_eval.predictors:SupportWorkflowPredictor. The offline-eval CLI imports it, builds it, and calls predict_items() once with every item across every requested scope.

Open evaluation/offline_eval/predictors/support_workflow_adapter.py; the code below is identical in both pods. predict_items() and _run_workflow() are given, not part of the exercise: the per-item loop and the response-caching logic (both scopes describe the same underlying email, so the workflow only needs to run once per (email_subject, email_body) pair) are identical for every scope and every pod; looping itself teaches nothing system-specific.

5.2 _build_prediction: the actual exercise

_build_prediction(item, response) is where a workflow's response becomes the scope-correct prediction dict inside a PredictionRecord:

if item.scope == INTENT_SCOPE:
    prediction = {"intent": response.intent, "rationale": response.rationale}
    observation_id = response.classifier_observation_id
else:
    prediction = {"reply_text": response.reply_text}
    observation_id = response.reply_observation_id
    # local-capture recovery: see 5.3

For reply_generation, prediction also needs account_status, the result of the lookup_account_status tool call, defined in agent/tools.py, that the reply-generation step makes. That value is never on SupportReply (the workflow's return type) at all; recovering it requires reading the trace.

5.3 Local-capture recovery

from semantic_tracing import find_observations, get_local_trace

local_trace = get_local_trace(response.trace_id)
tool_observations = find_observations(local_trace, name_in={"lookup_account_status"})
if tool_observations:
    output = tool_observations[0].output
    if isinstance(output, str):
        output = json.loads(output)
    prediction["account_status"] = output

get_local_trace(trace_id) returns the in-process copy of every observation recorded during that trace (Module 02 §2.3), no Langfuse round trip. find_observations(trace, name_in={...}) filters to the ones matching the given @step name(s).

The tool's result comes back JSON-encoded as a string, not a dict

lookup_account_status returns an AccountStatus Pydantic model. LocalObservation.output is reconstructed straight from the underlying OTel span attribute, and OTel span attributes can only hold primitive types (str, int, float, bool, or arrays of those); there is no "structured object" attribute type. So a Pydantic return value is stored as its JSON-serialized string form, not as a dict. The fix is the isinstance(output, str) + json.loads() guard above. Without it, prediction["account_status"] holds a string, and any downstream comparison against the dataset's expected_account_status dict (Module 06) fails every item, silently and without an obvious cause.

This is the concrete lesson of local capture: a PredictionRecord's content is not limited to whatever the workflow's top-level function happens to return. Anything traced as its own observation is recoverable, provided you know its exact captured shape.

5.4 capture_output and what a step records

Recall from Module 02: @step(..., capture_output=True) (the default) records a decorated function's return value as that observation's .output. lookup_account_status is a plain @step("lookup_account_status", as_type="tool") with the default capture_output=True; its return value (the AccountStatus instance) is exactly what ends up as .output, subject to the JSON-string caveat above. If you ever change a step function to return something different for a caller's convenience (an extra tuple element, for instance) without wanting that change reflected in the trace, pass capture_output=False and call update_current_step(output=...) explicitly instead; otherwise the auto-capture silently wins.

5.5 Try it: break the match on purpose

As a quick sanity check that this mechanism is real and not just plumbing that always happens to agree: temporarily change one seed item's expected_account_status in evaluation/datasets/seed/reply_generation_dev.json to a wrong value, re-push it (eval-dataset push, Module 08), and re-run the evaluation (Module 07). You should see account_status_match's pass rate drop and that one item's score fail, confirming the metric genuinely compares real values, not a tautology. Revert the change afterward.

Continue to Module 06.