Skip to content

Module 08: dataset lifecycle

This module covers the eval-dataset CLI and the semantic_evaluation.datasets Python API, the standard mechanism for moving golden-dataset content between a local file and Langfuse. Module 01's seed publish and Module 03's review-sync both write to a Langfuse dataset; this module is where you learn the general-purpose tool for editing one directly yourself.

Run every command in this module from your scaffold pod directory, the one where you ran uv sync in Module 01 §1.2.

8.1 The local ↔ Langfuse sync model

Every dataset item, wherever it comes from, is a EvaluationItem:

class EvaluationItem(BaseModel):
    item_id: str
    scope: str
    dataset_name: str
    input: dict
    expected_output: dict
    metadata: EvaluationItemMetadata

A dev dataset ({product_id}_{scope}_dev) is mutable and grows over time: seeded items, reviewed items, and hand edits all land here. A release dataset is an immutable snapshot, created by promote() from a dev dataset's current state, used to pin what a CI gate or a reported evaluation run actually measured against.

graph TD
  A["Local JSON file"] -->|"push"| B["Dev dataset: mutable"]
  B -->|"pull"| A
  B -->|"promote"| C["Release dataset: immutable snapshot"]
  D["Seed publish and review sync"] --> B

8.2 pull / push

uv run eval-dataset pull --name northwind_intent_classification_dev \
  --out evaluation/datasets/incoming/intent_classification_dev.json
uv run eval-dataset push --name northwind_intent_classification_dev \
  --source evaluation/datasets/incoming/intent_classification_dev.json --version-label 1.1

evaluation/datasets/incoming/ is the pod-local scratch location for datasets you pull down to edit. Deliberately not the seed files from Module 01: a pull onto those would overwrite hand-drafted starter data with whatever is currently in Langfuse, and the following push would send it straight back.

pull writes the dataset's current remote state to a local JSON file in the shape {"dataset_name", "version", "items": [...]}. push reads that shape back and upserts by item_id: it never deletes a remote-only item, it only reports one under would_delete so you can review before doing anything destructive. Pass --dry-run to see the diff without writing anything, and --create-on-missing the first time you push to a dataset that doesn't exist yet in Langfuse, as Module 01 does for the seed data.

Credentials come from your .env, the same file the agent itself reads; anything already exported in your shell wins over it. Point the CLI somewhere else with --env-file.

Why item IDs carry a product and scope prefix

Upserting by item_id only behaves if those IDs are unique, and Langfuse enforces uniqueness project-wide across every dataset, not per dataset. Both scopes describe the same five underlying emails, so a bare ID like WS-01 reused for the intent and reply datasets would collide the moment the second dataset is pushed. Writing them as {product_id}-{item_id}-{scope}, e.g. northwind-WS-01-intent_classification and northwind-WS-01-reply_generation, keeps them globally unique while still reading as the "same" underlying case across both scopes to a human. Follow the same convention for any items you add by hand.

8.3 validate, diff, dedupe

  • validate runs structural and schema checks over a local file without touching Langfuse.
  • diff computes the three-way comparison (local vs. remote vs. last-synced) that push --dry-run uses internally.
  • dedupe removes duplicate items from a local file by content hash.

8.4 promote and archive

uv run eval-dataset promote --dev-name northwind_reply_generation_dev --version 1.0 \
  --scope reply_generation

Promotes a dev dataset's current state into an immutable release, here northwind_reply_generation_v1.0, enforcing version-bump rules based on the schema change detected since the previous release (SchemaChangeLevel.NONE/MINOR/MAJOR); a major schema change (a field removed or its type changed) requires a major version bump, not just an incrementing label.

Then file the release's snapshot, the one kind of dataset file a pod commits:

uv run eval-dataset archive --name northwind_reply_generation_v1.0 \
  --manifest evaluation/evaluation_manifest.yaml --scope reply_generation

It writes evaluation/datasets/archive/northwind_reply_generation_v1.0.json, refuses a dev dataset, and never replaces a snapshot that is already there.

promote-baseline is a different command and unrelated to first-time promotion: it snapshots a Langfuse run's scores into the baselines block of the pod's evaluation/evaluation_manifest.yaml, so later runs can be compared against a recorded result rather than against a fixed threshold.

8.5 Why custom fields here

The seed and reviewed datasets in this workshop use custom fields (email_subject, email_body, intent, reply_text, …) rather than one of semantic_evaluation's four canonical schema types (QA, RAG, Agent, RAG-Agent). That's a deliberate choice, not an oversight: canonical schema types unlock pre-built evaluators with zero custom metric code (an azure_ai_evaluation RelevanceEvaluator expects a query/response shape out of the box, for instance), but intent_classification's output is a classification label, not a natural-language response, so it genuinely doesn't fit any of the four shapes. reply_generation's output is a natural-language reply and would fit the QA shape if you wanted to adopt canonical fields there. This workshop keeps it on custom fields for consistency with intent_classification, but a real pod could reasonably choose the canonical route for that scope alone.

Continue to Module 09.