Datasets: lifecycle and sync

semantic_evaluation.datasets is the SDK's local-to-Langfuse dataset round-trip. It ensures every pod that maintains a curated golden dataset speaks one on-disk format, one canonical hash, one diff shape, and one set of CLI commands.

Langfuse is the operational source of truth for all golden dataset items. Local JSON files are short-lived working copies used for batch edits outside the Langfuse UI — they are not permanently stored in the pod repository. Archive snapshots (one per released version) are the only local files kept in git permanently. The SDK never deletes from Langfuse.

See 02-schema-types.md for the schema types (QA, RAG, Agent, RAG-Agent) and the versioning rules for field definitions. For multi-scope operations (SyncConfig, pull_all, push_all), see 03-scope-sync.md.

Naming convention

Pattern

{product_id}_{schema_type}_{scope}_{state}

where {state} is either dev (mutable, no version) or v{major}.{minor} (released, immutable):

{product_id}_{schema_type}_{scope}_dev          ← mutable dev dataset
{product_id}_{schema_type}_{scope}_v{major}.{minor}  ← released snapshot

Part	Allowed values	Notes
`product_id`	e.g. `product_001`, `pathfinder`	Short, stable, lower_snake; `shared` for cross-product benchmarks
`schema_type`	`qa`, `rag`, `agent`, `rag_agent`	Omit when the dataset uses the QA schema type (no extra fields beyond messages / response). These tokens are the canonical `input_schema_type` values — see 02-schema-types.md § Canonical schema type names
`scope`	`e2e`, `retrieval`, `generation`, `tool_selection`, `routing`, `intent`, `safety`	Which capability is being evaluated
`_dev`	literal suffix	Mutable dataset; never used in CI or scheduled evals
`_v{M.m}`	e.g. `_v1.0`, `_v1.1`, `_v2.0`	Content version of the released snapshot

Examples

Dataset name	Schema type	Evaluates
`product_001_rag_retrieval_v1.0`	rag	Retrieval precision/recall
`product_001_rag_generation_v1.0`	rag	Response quality + groundedness
`product_001_safety_v1.0`	qa	Safety baseline
`product_002_rag_e2e_v1.0`	rag	Full RAG pipeline end-to-end
`product_003_agent_tool_selection_v1.0`	agent	Tool call accuracy
`product_003_agent_e2e_v1.0`	agent	End-to-end agent workflow
`product_004_rag_agent_e2e_v1.0`	rag_agent	RAG retrieval + tool-use end-to-end
`shared_qa_safety_v1.0`	qa	Cross-product safety baseline

Mutable dev counterparts (never used in CI): product_001_rag_retrieval_dev, product_003_agent_e2e_dev, etc.

Dataset states

Every golden dataset lives in one of two states in Langfuse:

Mutable dev dataset


Langfuse name	`{product_id}_{schema_type}_{scope}_dev`
Purpose	Active curation: adding items, refining ground truths, iterating on field shapes
Who uses it	Pod developers, SMEs during annotation review
NOT used for	CI regression gates, scheduled dashboard runs, any reproducibility-sensitive evaluation
Characteristic	Items change freely; `changed_last_in_dataset_version` tags are advisory only

Released (immutable) snapshot


Langfuse name	`{product_id}_{schema_type}_{scope}_v{major}.{minor}`
Created by	`promote()` or manually pushing the dev items to a new Langfuse dataset
Local file	Filed with `eval-dataset archive` as `evaluation/datasets/archive/{name}_v{M.m}.json`, committed
Used for	CI regression gates; periodic automated offline eval runs; any reproducible benchmark
Characteristic	Never modified after creation. New items → new version.

graph TD
    DEV["Langfuse dev dataset\n{name}_dev\nmutable · active curation"]
    REL["Langfuse release\n{name}_v{M.m}\nimmutable snapshot"]
    GIT["Pod repo\narchive/{name}_v{M.m}.json\ncommitted to git"]
    CI["CI regression gate"]
    SCHED["Periodic offline eval"]
    DB["Dashboard feed"]
    UC["Unity Catalog\nData Engineering Team"]

    DEV -- "promote() when\nstable + reviewed" --> REL
    REL -- "eval-dataset archive\n→ commit" --> GIT
    REL --> CI
    REL --> SCHED
    REL --> DB
    REL -- "exported on release" --> UC

Why the split matters: a mutable dev dataset can change between two evaluation runs — making it impossible to tell whether a score change reflects a model improvement or a dataset change. Released snapshots guarantee that the same changed_last_in_dataset_version label always means the same items and ground truths, which is the prerequisite for regression detection and dashboard trending.

Promotion triggers — promote from dev to release when:

The schema has settled (no more field name changes expected for this scope)
An initial ground truth set has been SME-reviewed: DatasetRecordMetadata.ground_truth_status (semantic_evaluation.contracts) is "reviewed" or "promoted" (not just "seeded") for all intended items
A minimum item count has been reached (≥ 20 items per scope recommended)

DatasetRecordMetadata.review_outcome records whether a reviewer approved the output as-is or corrected it; tags is reserved for category labels (see what each metadata field means).

Curation workflow

A project maintains each dataset in one of two ways, and the way decides where an item is edited. In both, Langfuse holds every dataset, and releases and their archived snapshots work the same.

Way	Where an item comes from	Where it is edited
Curated in Langfuse	Traces, human review, hand-written items	In Langfuse, directly or through a short-lived working copy
Built from sources in the repository	Fixtures the repository already holds, by a committed recipe	In the source it is built from, never in Langfuse

Curated in Langfuse

Items can be curated via two paths: editing directly in the Langfuse UI, or pulling to a local file, editing in batch, and pushing back. Both paths write to the mutable dev dataset.

graph TD
    subgraph PathA["Path A — Langfuse UI"]
        UI["Edit / add / annotate items\ndirectly in Langfuse UI"]
    end
    subgraph PathB["Path B — local batch edit"]
        PULL["eval-dataset pull\n(working copy, gitignored)"]
        EDIT["Edit JSON in editor"]
        VALIDATE["eval-dataset validate"]
        PUSH["eval-dataset push --version-label"]
    end
    LF_DEV["Langfuse\n{name}_dev\nmutable · source of truth"]
    LF_REL["Langfuse\n{name}_v{M.m}\nimmutable snapshot"]
    ARCHIVE["Pod repo\narchive/{name}_v{M.m}.json"]
    CONSUMERS["CI · scheduled evals · dashboards"]

    UI --> LF_DEV
    LF_DEV -.-> PULL
    PULL --> EDIT --> VALIDATE --> PUSH --> LF_DEV
    LF_DEV -- "eval-dataset promote\n--version M.m" --> LF_REL
    LF_REL -- "eval-dataset archive\n→ commit" --> ARCHIVE
    LF_REL --> CONSUMERS

    style PathA fill:#e5f4fd,stroke:#0098eb,stroke-width:2px
    style PathB fill:#e9ebec,stroke:#273943,stroke-width:2px

An item added via Path A carries input in whatever shape the traced observation captured it, and joins the dev dataset with no version stamp and, for a plain trace, no ground_truth_status. push() (Path B) closes both gaps the first time such an item is touched by the standard workflow: it backfills a missing changed_last_in_dataset_version on any item regardless of whether its content changed (reported under PushResult.would_backfill/backfilled), and it always re-supplies the item's source_trace_id/source_observation_id on every upsert so a pull-edit-push cycle never silently drops the link back to the originating trace. EvaluationItemMetadata.added_via_langfuse_ui (semantic_evaluation.contracts) reflects which path an item came through: it's True for an item with a trace link that was never touched by this SDK's own seed or promote tooling (Path A), and False once an item carries ground_truth_status (set by seeding or promotion) or has no trace link at all (a hand-authored Path B item).

Built from sources in the repository

Some agents are tested on inputs the repository already holds: a drawing, a scene, a set of documents, each with its expected outcome. Then the items are built from those sources by a recipe that is committed with them, and the project's own command writes the working copy and pushes it to the dev dataset. The runner may check a prediction against the source itself, so the source, not Langfuse, is where an item is true.

Edit an item where its source lives. An expected outcome changes in the fixture and the item is rebuilt and pushed, never edited in Langfuse, where the next push would overwrite it. A project working this way keeps a short table next to its datasets that names, for each kind of change, which dataset it touches:

You changed	So update
A fixture's expected outcome	Rebuild and push the dev dataset of every suite that uses the fixture
A suite's recipe (which fixtures, which prompts)	Rebuild and push that suite's dev dataset; release it when the change is meant to be judged on
The shape of an item's `input` or `expected_output`	Rebuild and push, then release with a new major version (see Schema change classification)

Promotion, release names and the archived snapshot of each release are the same as for a curated dataset.

The coverage map

Either way, a project keeps evaluation/coverage.md, its coverage map: one row per behaviour the agent must keep, with the items that prove it and the metrics that check it, the datasets and their releases, and the project's "you changed X, so update Y" table. It is updated in the same change that changes the agent, the way a unit test is written with a change to deterministic code, and it is where the next person, or coding agent, finds which items a change touches.

Promotion

promote() atomically transitions a mutable dev dataset to an immutable release snapshot in Langfuse, with schema-change detection and version-bump enforcement built in.

What `promote()` does

Pull dev items — fails fast if the dev dataset is empty.
Diff against previous release — if previous_release_name is supplied, pulls that dataset and blocks if zero content changes exist (nothing new to release).
Detect schema changes — schema_change_report() inspects which field keys were added, removed, or modified across input and expected_output dicts.
Enforce version-bump rules — validates that the proposed M.m version is consistent with the detected change level (see below).
Push to release dataset — creates {base}_v{M.m} in Langfuse (create_on_missing=True) and writes provenance metadata (major, minor, stage, promoted_from, promoted_at, schema_change_level, sdk_version) to the Langfuse dataset object. A release whose sdk_version is 1.0.0 was promoted by semantic_evaluation 1.0.0, 1.0.1 or 1.1.0, which all reported that number; from 1.2.0 the value is the exact installed version.
Return PromoteResult — reports items promoted, release name, schema change level, and the underlying PushResult.

Version-bump enforcement

The proposed version M.m is validated against the previous release version (derived from previous_release_name) and the detected schema change level:

Schema change	Proposed bump	Result
NONE	Minor bump (same major, higher minor)	✅ OK
MINOR or MAJOR	Major bump (higher major)	✅ OK
MINOR or MAJOR	Minor bump	❌ `VersionBumpError` — any shape change requires a major version
NONE	Major bump	❌ `VersionBumpError` — no schema changes detected
Any	Same or lower version	❌ `VersionBumpError` — always blocked

Pass force=True to override the schema-rule blocks (MAJOR+minor and NONE+major). Same-or-lower version is always blocked regardless of force.

The eval-dataset promote CLI catches VersionBumpError interactively and prompts for confirmation before re-running with force=True.

Schema change classification

schema_change_report(new_items, old_items) returns a SchemaChangeReport with level NONE, MINOR, or MAJOR based on field-key differences in the input and expected_output dicts across all items. Fields are prefixed "input." or "expected_output." to distinguish the two sub-schemas.

Level	Trigger
NONE	Identical field sets; no change.
MINOR	New field present in some but not all new items — the field is absent from at least one item, making it optional across the dataset.
MAJOR	Field removed from new items, or new field present in all new items (effectively required).

Pre-promotion schema alignment check

Before promoting, run schema_change_report() manually to verify that the dev items' field shapes still match the schema type models in 02-schema-types.md. This is most useful when the pod's Pydantic models have evolved since the last release and you want to confirm the change level before deciding which version number to use.

from semantic_evaluation.datasets import pull, schema_change_report

dev_items  = pull("product_001_rag_retrieval_dev")
rel_items  = pull("product_001_rag_retrieval_v1.0")

report = schema_change_report(dev_items, rel_items)
print(report.level)                   # SchemaChangeLevel.MINOR
print(report.added_optional_fields)   # ['input.document_ids']
print(report.added_required_fields)   # []
print(report.removed_fields)          # []

promote() runs the same check internally, but calling it up front lets you see the detected changes before committing to a version number. If the level is unexpected (for example MAJOR when you only added an optional field), investigate the field keys in the new items before proceeding.

Python API

from semantic_evaluation.datasets import promote, VersionBumpError

# First release (no previous version to compare against)
result = promote(
    "pathfinder_rag_end_to_end_dev",
    version="1.0",
)
print(result.release_name)      # pathfinder_rag_end_to_end_v1.0
print(result.items_promoted)    # e.g. 42

# Subsequent release with schema change detection
try:
    result = promote(
        "pathfinder_rag_end_to_end_dev",
        version="1.1",
        previous_release_name="pathfinder_rag_end_to_end_v1.0",
    )
except VersionBumpError as e:
    print(e)  # e.g. "MAJOR changes detected but proposed version 1.1 is a minor bump"
    # Re-run with the correct major version
    result = promote(
        "pathfinder_rag_end_to_end_dev",
        version="2.0",
        previous_release_name="pathfinder_rag_end_to_end_v1.0",
    )

# Dry-run: check what would be created without writing
result = promote("pathfinder_rag_end_to_end_dev", version="1.1", dry_run=True)

CLI

# First release
eval-dataset promote \
  --dev-name pathfinder_rag_end_to_end_dev \
  --version 1.0

# Subsequent release with version enforcement
eval-dataset promote \
  --dev-name pathfinder_rag_end_to_end_dev \
  --version 1.1 \
  --previous-release-name pathfinder_rag_end_to_end_v1.0

# Dry-run to preview
eval-dataset promote \
  --dev-name pathfinder_rag_end_to_end_dev \
  --version 1.1 \
  --previous-release-name pathfinder_rag_end_to_end_v1.0 \
  --dry-run

# Force override (skips schema-rule blocks; still blocks on same/lower version)
eval-dataset promote \
  --dev-name pathfinder_rag_end_to_end_dev \
  --version 2.0 \
  --previous-release-name pathfinder_rag_end_to_end_v1.0 \
  --force

Output is JSON with dev_name, release_name, version, items_promoted, schema_change, dry_run, and a nested push summary.

Archiving superseded dev items

After promotion, items that have been captured in the release snapshot can be soft-deleted in the dev dataset by setting their Langfuse status to ARCHIVED:

from semantic_evaluation.datasets import archive_item_status, pull

# Archive all current dev items (they are now in the release snapshot)
dev_items = pull("pathfinder_rag_end_to_end_dev")
archive_item_status(
    "pathfinder_rag_end_to_end_dev",
    [it.item_id for it in dev_items],
)

Archived items are excluded from live evaluation runs by default in Langfuse. This step is optional — the dev dataset remains usable for the next development cycle without archiving.

Pod file layout

Path	Role	Tracked in git?
`evaluation/datasets/incoming/{scope}_dev.json`	Short-lived working copy for local batch edits, written by `pull-all` (`dataset_sync.scoped_dir`). Delete after push.	no
`evaluation/datasets/archive/{name}_v{M.m}.json`	The snapshot of one released version, filed with `eval-dataset archive`. Never edited.	yes
`evaluation/evaluation_manifest.yaml` (`dataset_sync` block)	Links scopes to dataset names, and names the working-copy directory, the archive directory and optional input schemas. Paths are relative to the manifest.	yes
`evaluation/coverage.md`	The coverage map: behaviours, the items that prove them, the metrics that check them.	yes

archive/ holds release snapshots and nothing else; push and push-all write nothing locally, because Langfuse keeps every dev version.

Releasing a new version:

Curate items (Langfuse UI or local pull → validate → push), or rebuild them from their sources
Run eval-dataset promote --dev-name {name}_dev --version {M.m} to create the immutable Langfuse snapshot
Run eval-dataset archive --name {name}_v{M.m} --manifest evaluation/evaluation_manifest.yaml and commit the file it writes

Full cycle walkthrough

A concrete end-to-end example using the CLI. This assumes a multi-scope pod with two scopes (routing, grounding) configured in evaluation/evaluation_manifest.yaml with scoped_dir: datasets/incoming. Each step maps to one of the sections in this file or in 03-scope-sync.md — follow the links for full option reference.

1. Bootstrap (first time only)

Create the Langfuse dev datasets and push an initial set of items:

Set create_on_missing: true in dataset_sync for this first push only:

eval-dataset push-all \
  --manifest evaluation/evaluation_manifest.yaml \
  --version-label 0.0

After this succeeds, set create_on_missing: false and commit. All subsequent pushes will fail loudly on any unknown dataset name.

2. Pull a working copy

eval-dataset pull-all --manifest evaluation/evaluation_manifest.yaml
# → writes evaluation/datasets/incoming/routing_dev.json
#           evaluation/datasets/incoming/grounding_dev.json  (Pattern B)
#   or the one merged_path file                              (Pattern A)

Working copies are gitignored. Delete them after pushing back.

3. Edit and validate

Edit items in the JSON files, then validate structure before pushing:

eval-dataset validate --source evaluation/datasets/incoming/routing_dev.json
eval-dataset validate --source evaluation/datasets/incoming/grounding_dev.json

Optionally diff against the remote to see what will change:

eval-dataset diff \
  --name product_001_qa_routing_dev \
  --source evaluation/datasets/incoming/routing_dev.json

4. Dry-run push

Check what would be written without actually writing:

eval-dataset push-all \
  --manifest evaluation/evaluation_manifest.yaml \
  --version-label 0.1 \
  --dry-run

5. Push

eval-dataset push-all \
  --manifest evaluation/evaluation_manifest.yaml \
  --version-label 0.1

push is idempotent — re-running with the same items produces zero writes.

6. Pre-promotion schema check (optional)

Before choosing a version number, inspect the schema change level:

# Python — no CLI command for schema_change_report
python - <<'EOF'
from semantic_evaluation.datasets import pull, schema_change_report
dev = pull("product_001_qa_routing_dev")
rel = pull("product_001_qa_routing_v1.0")   # skip if no prior release
report = schema_change_report(dev, rel)
print(report.level, report.added_optional_fields, report.removed_fields)
EOF

7. Promote to release

# First release — no previous version to compare against
eval-dataset promote \
  --dev-name product_001_qa_routing_dev \
  --version 1.0

# Subsequent release — pass previous version for schema-change enforcement
eval-dataset promote \
  --dev-name product_001_qa_routing_dev \
  --version 1.1 \
  --previous-release-name product_001_qa_routing_v1.0

Promotion creates the immutable product_001_qa_routing_v1.1 dataset in Langfuse. See Version-bump enforcement for the rules on which version bumps are allowed.

8. File the release snapshot and commit

eval-dataset archive \
  --name product_001_qa_routing_v1.1 \
  --manifest evaluation/evaluation_manifest.yaml \
  --scope routing
# → writes evaluation/datasets/archive/product_001_qa_routing_v1.1.json

git add evaluation/datasets/archive/product_001_qa_routing_v1.1.json
git commit -m "eval: archive routing dataset v1.1"

Archive snapshots are the only dataset files committed to the pod repository. Working copies (*_dev.json) stay gitignored.

9. Archive promoted items in the dev dataset (optional)

Soft-delete the items now captured in the release so the dev dataset starts clean for the next development cycle:

from semantic_evaluation.datasets import archive_item_status, pull

dev_items = pull("product_001_qa_routing_dev")
archive_item_status("product_001_qa_routing_dev", [it.item_id for it in dev_items])

Archived items are excluded from Langfuse evaluation runs by default. Skip this step if you prefer to keep the dev dataset intact.

Unity Catalog handover

This section is a specification for the Data Engineering team — the pod team has no steps here. After eval-dataset promote creates an immutable released snapshot in Langfuse, the Data Engineering team fetches that snapshot, validates it, and registers it in Unity Catalog. Dev datasets (name ends in _dev) are always skipped.

Identifying datasets to fetch

Langfuse dataset name	State	Data engineer action
`{product_id}_{schema_type}_{scope}_dev`	mutable, actively curated	skip
`{product_id}_{schema_type}_{scope}_v{M.m}`	released, immutable	fetch → validate → list in UC

Data-flow diagram

flowchart TD
  POD["Pod\neval-dataset promote"]
  LF["Langfuse\nreleased dataset\nname_v{M.m}"]
  BLOB["Blob Storage\nreleased snapshot JSON"]
  VAL["Validator\n(Data Engineering)"]
  UC["Unity Catalog Volume\n/semantic_evaluation/datasets/{product_id}/{name}.json"]
  API["UC API\nexposed to CI runners"]

  POD --> LF
  LF -- "Data Engineering:\nfetch released datasets" --> BLOB
  BLOB --> VAL
  VAL -- "on pass" --> UC
  UC --> API

Volume path convention

/Volumes/{catalog}/semantic_evaluation/datasets/{product_id}/{release_name}.json

{catalog} — enterprise catalog managed by the Data Engineering team
{product_id} and {release_name} are derived directly from the Langfuse dataset name
Example: /Volumes/ramboll_ai/semantic_evaluation/datasets/pathfinder/pathfinder_agent_e2e_v1.6.json

Dataset hierarchy

flowchart TD
  P1["pathfinder"]
  P2["product_001"]
  PDOT["..."]

  P1 --> P1_QA["qa"]
  P1 --> P1_RAG["rag"]
  P1 --> P1_AGT["agent"]

  P2 --> P2_QA["qa"]
  P2 --> P2_RAG["rag"]

  P1_RAG --> SC_E2E["e2e"]
  P1_RAG --> SC_RET["retrieval"]
  P1_RAG --> SC_GEN["generation"]
  P1_RAG --> SC_DOT["..."]

  SC_RET --> V10["v1.0"]
  SC_RET --> V11["v1.1"]
  SC_RET --> V20["v2.0"]
  SC_RET --> VDOT["..."]

Each pod may use any combination of schema types (qa, rag, agent) and any number of scopes; each scope produces its own independent version sequence.

For the validation rules the Data Engineering team applies on each incoming JSON, see 02-schema-types.md § Unity Catalog.