Skip to content

Datasets: eval-dataset CLI

The eval-dataset CLI is a thin wrapper over the public Python API. All commands exit 0 on success and 1 on validation/diff/error failure. JSON summaries are written to stdout so the CLI composes cleanly into scripts.

See 03-datasets-lifecycle.md for the full lifecycle workflow and 03-scope-sync.md for multi-scope pull-all / push-all operations.

Credentials

Every command that reaches Langfuse needs LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY and LANGFUSE_BASE_URL in the environment. Before running a command the CLI reads ./.env and then ./.env.local, so a pod that already keeps those values in .env needs no extra step:

cd path/to/your/pod
eval-dataset pull --name my_scope_dev --out evaluation/datasets/incoming/my_scope_dev.json

Values already exported in the shell take priority over the file, which means a stale export silently wins; check the shell first when a command behaves in a way .env does not explain. Use --env-file PATH, before the subcommand, to read one specific file instead of the two defaults:

eval-dataset --env-file ci/langfuse.env push --name my_scope_dev --source items.json --version-label 1.1

Calling the Python helpers in semantic_evaluation.datasets directly reads no dotenv file at all. Loading one is an entry point's decision, and importing a library must not change its host's environment.

eval-dataset CLI reference

Command Purpose Required args Notable flags
pull Download a single Langfuse dataset to a JSON file. --name, --out --version <label> filters on raw_metadata.changed_last_in_dataset_version; --dry-run. Prints the absolute path it wrote.
push Upsert a single local JSON file into a Langfuse dataset. --name, --source, --version-label --dry-run; --create-on-missing.
promote Promote the dev dataset to an immutable release snapshot. Prints the archive command to run next. --dev-name, --version --previous-release-name, --scope, --force, --dry-run.
pull-all Pull all scopes defined in a config and write local files. Relative paths in the config resolve against its own directory. --manifest or --config --scope, --version, --dry-run. Prints the absolute path it wrote.
push-all Read local files, push all scopes. Writes nothing locally. --manifest or --config, --version-label --dry-run, --scope.
promote-baseline Snapshot Langfuse run scores into the manifest baselines block. --run-name, --manifest --update-gates, --dry-run, --yes / -y.
resolve-run Print a run file fully resolved against the project's evaluator catalogue and scope-to-dataset map. --run-file, --manifest
schemas Write the JSON Schemas of evaluation_manifest.yaml and the run files, from the installed SDK, for editor validation. Re-run after an upgrade. --dir (default evaluation).
validate Structural + optional input-schema check. --source --input-schema dotted.module:ClassName (Pydantic) or --input-schema-json path/to/schema.json.
diff Compare a local file against a Langfuse dataset. --name, --source
dedupe Rewrite a local file in place, keeping one item per canonical_hash. --source
archive File the snapshot of a released dataset as <archive directory>/<name>.json, to commit. Refuses a dev dataset and a snapshot that is already there. --name --manifest (use its dataset_sync.archive_dir) or --dir (default evaluation/datasets/archive); --scope.

promote-baseline reference

promote-baseline reads the dataset_sync.scope_to_dataset mapping from a manifest YAML, finds the named run among the dataset's experiments (Langfuse's name for a dataset run; legacy dataset runs are included), reads its run-level aggregate scores (the whole-run means, the scored metrics' per-group means named {scope}.{metric}.by.{attr}.{value}, and the agreement scores; spread statistics and trace-metric aggregates are left out), and writes them into the baselines block of that same manifest file. gate_passed is recorded as 1.0 or 0.0. A dataset that cannot be fetched is logged as a warning and skipped. Baselines are keyed by run name, so multiple snapshots can coexist and be compared over time.

eval-dataset promote-baseline \
  --run-name pathfinder-local-all-scopes_baseline_20260702T143200Z \
  --manifest evaluation/evaluation_manifest.yaml

# Preview without writing
eval-dataset promote-baseline \
  --run-name pathfinder-local-all-scopes_baseline_20260702T143200Z \
  --manifest evaluation/evaluation_manifest.yaml \
  --dry-run

# Also update gated evaluator thresholds to match the baseline scores
eval-dataset promote-baseline \
  --run-name pathfinder-local-all-scopes_baseline_20260702T143200Z \
  --manifest evaluation/evaluation_manifest.yaml \
  --update-gates \
  --yes
Flag Description
--run-name Exact execution name of the Langfuse experiment (dataset run) to snapshot.
--manifest Path to evaluation_manifest.yaml.
--update-gates When set, updates each gated evaluator's threshold to match the baseline score; the per-group and agreement scores are kept in the baseline but never change a threshold. Only evaluators that already have a threshold field are touched; non-gated metrics are never promoted to gated by this command. When a metric is present in multiple scopes, the most conservative (minimum) value across scopes is used.
--dry-run Print the diff without writing any changes.
--yes / -y Skip the interactive confirmation prompt.

The command uses ruamel.yaml for round-trip YAML rewriting so that inline comments on evaluator criteria and threshold rationale are preserved.

The working copy

pull, pull-all and dump_local_items() write, and push, push-all and validate read, one JSON shape: the dataset's name, the version label, and its items, each an EvaluationItem.

{
  "dataset_name": "product_001_agent_end_to_end_dev",
  "version": "0.1",
  "items": [
    {
      "item_id": "product_001-end_to_end-001",
      "scope": "end_to_end",
      "dataset_name": "product_001_agent_end_to_end_dev",
      "input": {"message": "Refund order #1001, please."},
      "expected_output": {"expected_tool_calls": ["lookup_order", "issue_refund"]},
      "metadata": {"product_id": "product_001", "raw_metadata": {"ground_truth_status": "seeded"}}
    }
  ]
}

item_id is unique across the whole Langfuse project, not only within one dataset, so it carries the product and the scope. A new item has no changed_last_in_dataset_version yet; push stamps it with the push's version label. Write the first items of a new dataset into such a file by hand, validate it, and push it.

Idempotency

push is upsert-by-item_id: Langfuse's create_dataset_item(id=...) replaces an existing item rather than creating a duplicate. The SDK short-circuits items whose canonical_hash and tags already match the remote copy, so repeated push calls converge to zero writes once the local file stops changing. An edit that only changes an item's tags is pushed like any other edit, and an explicit "tags": [] clears them in Langfuse.

push will never delete a remote item that is missing locally. Items in that bucket are reported under would_delete for visibility only. Delete operations are intentionally out of scope; perform them through the Langfuse UI after a deliberate review.

promote is safe to re-run with the same version — the release dataset already exists and the push will find all items unchanged, resulting in zero writes.

Version semantics

The SDK maintains two distinct versioning indicators on dataset objects in Langfuse. Understanding the difference avoids confusion when items appear to lag behind the version label declared in a push.

changed_last_in_dataset_version — per-item content label

raw_metadata.changed_last_in_dataset_version is stamped on each item that is created or updated during a push:

  • push --version-label vX.Y stamps changed_last_in_dataset_version: "vX.Y" on every item whose content changed or that did not previously exist in Langfuse.
  • Items whose canonical hash and tags already matched the remote copy are left untouched — they retain the version label from the last push that actually modified them.
  • pull --version vX.Y filters on this field, returning only items whose content was last changed at that specific label.
  • derive_version(items) returns the maximum changed_last_in_dataset_version across a list of items.

After a push that declared version 0.2 but found all items identical to their remote counterparts, every item still carries its original changed_last_in_dataset_version (e.g. "0.0"). This is correct — no content changed.

dev_version — dataset-object declared version

dev_version is a metadata field on the Langfuse dataset object itself (not on individual items). push() stamps it on every push, whether or not any items were upserted:

  • After push_all(..., version_label="0.2"), the dataset object carries metadata.dev_version: "0.2" even when all items were already up to date.
  • pull_all() reads dev_version from the dataset object and writes it as the local file envelope version — so the local file correctly reports "version": "0.2" even though no individual item carries that label.
  • The "Current version in Langfuse" message from the CLI reflects this dev_version, not the max per-item changed_last_in_dataset_version.

At a glance

After pushing at version 0.2 with no content changes (35 items last changed at 0.0, 1 item last changed at 0.1):

Location Field Value
Each unchanged item raw_metadata.changed_last_in_dataset_version "0.0" or "0.1"
Dataset object metadata.dev_version "0.2"
Local file envelope version "0.2"
CLI "Current version" message — "0.2"

derive_version(items) returns "0.1" (max per-item label). Use it for content-change comparisons; use dev_version to answer "what version was last declared on this dataset?"

See 02-schema-versioning.md — Versioning: schema vs. content for the full versioning strategy: what to bump, when to bump it, and how schema type versioning and dataset-content versioning interact.

Programmatic access

The CLI is a thin wrapper over the public Python API:

from semantic_evaluation.datasets import (
    load_local_items, dump_local_items, push, validate, promote,
    schema_change_report, VersionBumpError, diff,
)

# Validate a local file
_, _, items = load_local_items(
    "evaluation/datasets/archive/product_001_rag_retrieval_v1.2.json"
)
errors = validate(items)
assert not errors, errors

# Dry-run push
result = push(
    "product_001_rag_retrieval_dev",
    items,
    version_label="1.2.0",
    dry_run=True,
)
print(result.would_create, result.would_update, result.would_delete)

# Write items back to a local file in the standard on-disk shape
dump_local_items(
    "evaluation/datasets/product_001_rag_retrieval_dev.json",
    items,
    dataset_name="product_001_rag_retrieval_dev",
    version="1.2.0",
)

# Compare local vs remote without writing
from semantic_evaluation.datasets import pull
remote_items = pull("product_001_rag_retrieval_dev")
result = diff(items, remote_items)
print(result.added, result.changed, result.removed, result.unchanged)

# Inspect schema changes before promoting
dev_items = pull("product_001_rag_retrieval_dev")
rel_items = pull("product_001_rag_retrieval_v1.1")
report = schema_change_report(dev_items, rel_items)
print(report.level)                  # SchemaChangeLevel.MINOR
print(report.added_optional_fields)  # ['input.document_ids']

Pods can script bespoke flows (CI-time validation, scheduled archives, etc.) on top of these functions without invoking a subprocess.