Datasets: schema types

EvaluationItem.input and EvaluationItem.expected_output are open dict[str, Any] fields — the SDK imposes no shape beyond the EvaluationItem (semantic_evaluation.contracts) envelope itself. A pod evaluating structured code generation, tabular outputs, or any domain-specific I/O can use entirely custom shapes paired with custom evaluators.

Pre-defined schema types exist as conventions, not constraints. A pod that follows a schema type gets access to pre-built evaluators already registered in the scoring catalogue (relevancy, groundedness, tool-call matching, etc.) without writing custom metric code.

See 03-datasets-lifecycle.md for the naming convention and the operational workflow for pulling, editing, and pushing dataset items.

Normalizing single-key wrapped input

Adding an observation straight to a dataset via Langfuse's own "+ Add to dataset" action, rather than through eval-dataset push or review-sync promotion, captures input in the exact shape semantic_tracing's @step decorator recorded at the traced call site: a single top-level key named after the traced function's own argument, wrapping the real payload. For example, a call site whose one non-excluded argument is named request produces {"request": {"account_id": "...", "email_body": "..."}}, not the flat shape a pod's schema type or predictor expects.

normalize_dataset_item() (in orchestration/datasets/langfuse.py) detects and corrects this at the boundary, so no pod code has to special-case it:

Schema-aware, when a schema is registered for the scope (DatasetSourceConfig.input_schema for the offline-eval runner; SyncConfig.input_schema for pull/pull_all, see 03-scope-sync.md): input is unwrapped only when its single top-level key is not itself an expected field of the schema, and the wrapped dict's keys overlap the schema's fields. If a schema is registered but neither condition holds, input is left unchanged and a warning is logged. A registered schema is treated as positive information, not a hint to guess.
Bare heuristic, when no schema is registered for the scope: input is unwrapped whenever it has exactly one top-level key whose value is itself a dict, and an info line is logged naming the unwrapped key.

An input that already has the flat shape a schema type or pod expects is never touched by either mode.

Declaring a schema type

Pods declare which schema type they follow via two fields in raw_metadata of EvaluationItemMetadata (semantic_evaluation.contracts):

Field	Example	Purpose
`input_schema_type`	`"rag"`	Which pre-defined schema type applies
`input_schema_version`	`"1.0"`	Version of that schema type's field definition

Canonical schema type names

The four canonical values for input_schema_type are:

Schema type	`input_schema_type`	Pydantic models	`{schema_type}` in dataset name
QA	`"qa"`	`BaseInput`, `BaseExpectedOutput`	`qa` (or omitted — see naming convention)
RAG	`"rag"`	`RAGInput`, `RAGExpectedOutput`	`rag`
Agent	`"agent"`	`AgentInput`, `AgentExpectedOutput`	`agent`
RAG-Agent	`"rag_agent"`	`RAGAgentInput`, `RAGAgentExpectedOutput`	`rag_agent`

These same tokens are the {schema_type} segment in the dataset naming convention. The SDK does not enforce the value at the type level — raw_metadata is a free-form dict — so spelling matters.

These fields let evaluators identify the schema contract without parsing the data. Set them once when a dataset is first published. Bump input_schema_version whenever the field definitions for that schema type change — see 02-schema-versioning.md for the rules.

Schema type hierarchy

QA is the common root. RAG and Agent extend it independently. RAG-Agent combines both, inheriting all RAG and Agent fields.

flowchart TD
  Base["QA\ninput: messages\nexpected: response"]
  RAG["RAG\ninput: messages + context + document_ids\nexpected: response + citations + retrieval_ground_truth"]
  Agent["Agent\ninput: messages + available_tools\nexpected: response + tool_calls + handoff_to"]
  RAGAgent["RAG-Agent\ninput: messages + context + document_ids + available_tools\nexpected: response + citations + retrieval_ground_truth + tool_calls + handoff_to"]
  Base --> RAG
  Base --> Agent
  RAG --> RAGAgent
  Agent --> RAGAgent

All four schema types use messages: list[Message] as their required input field. Single-turn items use a one-element list; multi-turn items carry the full conversation history.

A. QA

The minimum for pre-built relevancy, coherence, and GEval-based evaluators. Suitable for any single-turn or multi-turn question-answering or text-generation scenario.

input:
  messages: list[{                              REQUIRED
    role:    "user" | "assistant" | "system"
    content: string
  }]

expected_output:
  response: string   REQUIRED

B. RAG — extends QA

Adds retrieval context to input and citation / retrieval ground truth to expected_output. Use for any system that grounds its response in retrieved documents.

input:
  messages:     list[Message]    REQUIRED  (see QA)
  context:      list[string]     OPTIONAL  retrieved text chunks passed to the LLM
  document_ids: list[string]     OPTIONAL  source references parallel to context

expected_output:
  response:                string           REQUIRED
  citations:               list[Citation]   OPTIONAL  expected citations in response
  retrieval_ground_truth:  list[{           OPTIONAL  for retrieval precision/recall
    document_id: string
    relevance:   0 | 1 | 2    (0 = not relevant, 1 = somewhat, 2 = highly)
  }]

Citation object — structured so provenance is machine-readable:

{
  "response_marker": "[1]",
  "document_id":     "doc_abc_123",
  "page_number":     null,
  "quotes":          ["..."]
}

Enables in addition to QA: groundedness (faithfulness to retrieved context), citation accuracy, retrieval precision/recall.

C. Agent — extends QA

For tool-using agents and orchestration workflows. Adds tool-call and routing expectations to expected_output. response is required — routing-only items carry the routing message as their response. For systems that also retrieve context, use Section D — RAG-Agent.

input:
  messages: list[{                              REQUIRED
    role:    "user" | "assistant" | "system"
    content: string
  }]
  available_tools: list[string]                 OPTIONAL  tool names in scope

expected_output:
  response:    string    REQUIRED  expected final assistant reply
  tool_calls:  list[ToolCall]   OPTIONAL  expected tool invocations, in order
  handoff_to:  string | null    OPTIONAL  expected routing / sub-agent target

ToolCall object in expected_output — args only, no result, because the expected tool output is not known at curation time:

{
  "name": "search_documents",
  "args": {"query": "...", "top_k": 5}
}

In the prediction, tool calls also carry the actual execution output:

{
  "name":   "search_documents",
  "args":   {"query": "...", "top_k": 5},
  "output": "..."
}

Where prediction data comes from

Evaluators compare expected_output.tool_calls against the tool_calls field in PredictionRecord.prediction (semantic_evaluation.contracts). Four patterns, ordered from most to least recommended:

A. Local trace capture (recommended) — predictor calls the live service, receives a trace_id, and reads tool calls from the in-process mirror immediately after the turn ends. No Langfuse polling, no ingestion-lag race, complete the instant all OTel spans have closed:

from semantic_tracing import get_local_trace
from semantic_tracing.trace_reader import find_observations, as_dict, coerce_text

# Requires setup_tracing(capture_local_traces=True) at service startup
trace = get_local_trace(trace_id)
tool_call_obs = find_observations(trace, names=["tool_a", "tool_b"])
reply_obs     = find_observations(trace, names=["final_reply"])
prediction = {
    "tool_calls": [
        {"name": obs.name, "args": as_dict(obs.input), "output": as_dict(obs.output)}
        for obs in tool_call_obs
    ],
    "response": coerce_text(reply_obs[0].output) if reply_obs else None,
}

See get_local_trace (semantic_tracing.local_capture), find_observations (semantic_tracing.trace_reader), as_dict (semantic_tracing.trace_reader).

B. Trace-fetching from Langfuse (fallback) — predictor fetches the trace after the turn ends. Adds an async round trip and can silently return a partial trace if Langfuse ingestion has not finished:

from semantic_tracing import fetch_trace, find_observations, as_dict

trace = await fetch_trace(trace_id, backoff=(...))
tool_call_obs = find_observations(trace, names=["tool_a", "tool_b"])
prediction = {
    "tool_calls": [
        {"name": obs.name, "args": as_dict(obs.input), "output": as_dict(obs.output)}
        for obs in tool_call_obs
    ],
    "response": ask["message"],
}

C. Direct collection (no tracing) — predictor calls the workflow function directly and maps its structured return value into the prediction dict. Simplest, but loses the full trace signal for inspection in Langfuse:

response = await run_workflow(item.input)
prediction = {
    "tool_calls": [{"name": tc.name, "args": tc.args} for tc in response.tool_calls],
    "handoff_to": response.routing.handoff_to,
    "response":   response.answer,
}

D. The platform's own transcript: for an agent on a platform that cannot be traced (Open WebUI, Copilot Studio), the predictor reads the answer and tool calls from the platform's transcript and fills prediction, measurements and source_transcript directly. See Predictions from a runtime without a Langfuse trace.

All four patterns produce the same PredictionRecord.prediction (semantic_evaluation.contracts) shape; evaluators are unaware of which collection method was used.

D. RAG-Agent — extends RAG

For systems that both retrieve context and invoke tools. Inherits all RAG fields and adds tool-call and routing expectations from the Agent schema. Use this type when a system needs both groundedness evaluation (citation accuracy, retrieval precision/recall) and agent evaluation (tool-call matching, routing).

input:
  messages:        list[Message]    REQUIRED  (see QA)
  context:         list[string]     OPTIONAL  retrieved text chunks passed to the LLM
  document_ids:    list[string]     OPTIONAL  source references parallel to context
  available_tools: list[string]     OPTIONAL  tool names in scope

expected_output:
  response:                string           REQUIRED
  citations:               list[Citation]   OPTIONAL  expected citations in response
  retrieval_ground_truth:  list[{           OPTIONAL  for retrieval precision/recall
    document_id: string
    relevance:   0 | 1 | 2    (0 = not relevant, 1 = somewhat, 2 = highly)
  }]
  tool_calls:  list[ToolCall]   OPTIONAL  expected tool invocations, in order
  handoff_to:  string | null    OPTIONAL  expected routing / sub-agent target

The Citation object format is the same as in Section B — RAG. The ToolCall object format is the same as in Section C — Agent.

Enables all evaluators from both RAG (groundedness, citation accuracy, retrieval precision/recall) and Agent (tool-call matching, routing).

Schema types as Pydantic models

The preferred way to validate that a dataset conforms to a schema type is to use Pydantic base classes from semantic_evaluation.schemas. These serve as the canonical field definition, integrate directly with eval-dataset validate --input-schema, and let JSON Schemas be auto-generated.

from semantic_evaluation.schemas import RAGInput, RAGExpectedOutput

# Extend with product-specific fields
class MyProductInput(RAGInput):
    department: str

class MyProductExpectedOutput(RAGExpectedOutput):
    confidence: float | None = None

# Validate a dataset against the extended model
# eval-dataset validate --source dataset.json \
#     --input-schema my_pod.evaluation.schemas:MyProductInput

# Generate JSON Schema (e.g. for documentation or catalogue registration)
print(MyProductInput.model_json_schema())

The following classes are exported from semantic_evaluation.schemas:

Class	Inherits from	Primary fields
`BaseInput`	`BaseModel`	`messages: list[Message]` (required)
`BaseExpectedOutput`	`BaseModel`	`response: str` (required)
`RAGInput`	`BaseInput`	`context: list[str] \\| None`, `document_ids: list[str] \\| None`
`RAGExpectedOutput`	`BaseExpectedOutput`	`citations: list[Citation] \\| None`, `retrieval_ground_truth: list[RetrievalGroundTruth] \\| None`
`AgentInput`	`BaseInput`	`available_tools: list[str] \\| None`
`AgentExpectedOutput`	`BaseExpectedOutput`	`response: str` (required, inherited), `tool_calls: list[ToolCall] \\| None`, `handoff_to: str \\| None`
`RAGAgentInput`	`RAGInput`	`available_tools: list[str] \\| None`
`RAGAgentExpectedOutput`	`RAGExpectedOutput`	`tool_calls: list[ToolCall] \\| None`, `handoff_to: str \\| None`
`Citation`	`BaseModel`	`response_marker`, `document_id`, `page_number`, `quotes`
`RetrievalGroundTruth`	`BaseModel`	`document_id`, `relevance: Literal[0, 1, 2]`
`Message`	`BaseModel`	`role: Literal["user", "assistant", "system"]`, `content`
`ToolCall`	`BaseModel`	`name`, `args: dict`

JSON Schema files

Committed JSON Schema files generated from the Pydantic models live in sdk/semantic_evaluation/schemas/:

base_input.schema.json — QA input
rag_input.schema.json — RAG input
agent_input.schema.json — Agent input
rag_agent_input.schema.json — RAG-Agent input

Regenerate after any model change:

cd sdk/semantic_evaluation && python -c "
import json, sys; sys.path.insert(0, 'src')
from semantic_evaluation.schemas import BaseInput, RAGInput, AgentInput, RAGAgentInput
for name, cls in [
    ('base_input', BaseInput), ('rag_input', RAGInput),
    ('agent_input', AgentInput), ('rag_agent_input', RAGAgentInput),
]:
    with open(f'schemas/{name}.schema.json', 'w') as f:
        json.dump(cls.model_json_schema(), f, indent=2); f.write('\n')
"

Version snapshots of each schema file are archived in schemas/archive/ whenever a major input_schema_version is cut. See 02-schema-versioning.md.

Unity Catalog

Spec for the Data Engineering team — not implemented in the SDK. The pod team promotes released dataset snapshots to Langfuse. The Data Engineering team owns Unity Catalog; they fetch those published snapshots, validate them, and register them in UC. The pod team has no UC access and no steps to take here.

See 03-datasets-lifecycle.md § Unity Catalog handover for the data-flow diagram and naming details.

Volume path convention

/Volumes/{catalog}/semantic_evaluation/datasets/{product_id}/{release_name}.json

{catalog} — enterprise catalog managed by the Data Engineering team
{product_id} and {release_name} are derived directly from the Langfuse dataset name
Example: /Volumes/ramboll_ai/semantic_evaluation/datasets/pathfinder/pathfinder_agent_e2e_v1.6.json

Validation rules

The Data Engineering team validates each released snapshot before listing it in UC.

Validation regime

Each item's input and expected_output are validated against the Pydantic model for the dataset's declared input_schema_type. Pod-specific fields beyond the base schema type's fields are allowed — the validator checks that required fields are present and correctly typed, not that no extra fields exist.

Rule	Detail
Validate against declared schema type	`"qa"` → `BaseInput` / `BaseExpectedOutput`; `"rag"` → `RAGInput` / `RAGExpectedOutput`; `"agent"` → `AgentInput` / `AgentExpectedOutput`; `"rag_agent"` → `RAGAgentInput` / `RAGAgentExpectedOutput`
Pod-specific fields are permitted	Fields beyond the schema type's own required and optional fields do not cause a failure in either `input` or `expected_output`

Checks

"Field key union" means the union of all keys present in the input (or expected_output) dict across every item in the dataset.

Check	New dataset	Minor version	Major version	Example
No duplicate `item_id`s	✅	✅	✅	Two items both have `"item_id": "q001"` → fail
Required fields present in every item	✅	✅	✅	RAG dataset with one item missing `messages` → fail
`input_schema_type` in metadata matches the validated schema type	✅	✅	✅	Metadata declares `"rag"` but items carry `available_tools` (Agent shape) → fail
Name-vs-metadata alignment	✅	✅	✅	`pathfinder_rag_retrieval_v1.6.json` must have `product_id: "pathfinder"`, `input_schema_type: "rag"`, scope `"retrieval"`, version `"1.6"` in metadata
Field key union identical to previous version's	—	✅	—	v1.0 union `{messages, response, context}`; v1.1 must have the same union → if it differs, a major bump is required
Field key union differs from previous version's	—	—	✅	v1.0 union `{messages, response}`; v2.0 union `{messages, response, context}` → field added, major correctly declared
Version strictly higher than previous	—	✅	✅	Previous is `v1.2`; proposed `v1.1` → fail