Skip to content

Datasets: multi-scope sync

Most pods split evaluation into several independent concerns — routing, grounding, playbook adherence, and so on. Each concern maps to its own Langfuse dataset. Working with them one at a time via eval-dataset pull and eval-dataset push works, but requires looping over each scope manually. The multi-scope helpers automate that loop.

See 03-datasets-lifecycle.md for the naming convention, dataset states, promotion, and the full cycle walkthrough. See 03-datasets-cli.md for the full eval-dataset CLI reference and programmatic access.

SyncConfig

SyncConfig is the central configuration object. It captures:

  • Which scopes exist and which Langfuse dataset each maps to (scope_to_dataset).
  • Whether to auto-create missing Langfuse datasets on first push (create_on_missing).
  • How local files are laid out — Pattern A or Pattern B.
  • Optional input_schema and expected_output_schema: a Pydantic model class (its .model_json_schema() is called at push time) or a pre-built dict. Written to the Langfuse dataset when it is first created. input_schema is also consulted on pull/pull_all to detect and unwrap a single-key-wrapped input — see Normalizing single-key wrapped input.

Pattern A — one merged file (set merged_path + canonical_name):

All scope items are merged into a single local JSON. Useful when reviewing the whole dataset in one editor session or when a pod is small enough that a single file is manageable.

from pathlib import Path
from semantic_evaluation.datasets import SyncConfig

config = SyncConfig(
    scope_to_dataset={
        "routing":   "product_001_qa_routing_dev",
        "grounding": "product_001_rag_grounding_dev",
    },
    merged_path=Path("evaluation/datasets/incoming/product_001_dev.json"),
    canonical_name="product_001",
)

Pattern B — one file per scope (set scoped_dir):

Each scope is written to {scoped_dir}/{scope}_dev.json. Useful when scopes are large, independently maintained, or need separate code reviews.

config = SyncConfig(
    scope_to_dataset={
        "routing":   "product_001_qa_routing_dev",
        "grounding": "product_001_rag_grounding_dev",
    },
    scoped_dir=Path("evaluation/datasets/incoming"),
)
# produces: evaluation/datasets/incoming/routing_dev.json
#           evaluation/datasets/incoming/grounding_dev.json

Two equivalent workflows: Python API vs CLI

Both paths call the same underlying functions and produce identical results.

Python API

from semantic_evaluation.datasets import pull_all, push_all, derive_version

# Pull all scopes from Langfuse → write local files
pulled = pull_all(
    config,
    on_scope_pulled=lambda sc, ds, n: print(f"{sc} ← {ds} ... {n} items"),
)
version = derive_version(pulled)
print(f"Current version: {version} — bump before pushing changes.")

# Edit local files, then push back
results = push_all(
    config,
    version_label="1.6.0",
    dry_run=False,
    on_scope_pushed=lambda sc, ds, r: print(f"{sc}: created={len(r.created)} updated={len(r.updated)}"),
)

CLI

Add the scope_to_dataset block and layout fields to the dataset_sync section of evaluation/evaluation_manifest.yaml. Paths are relative to the manifest's own directory, so the same file works from the project root and from evaluation/:

dataset_sync:
  scope_to_dataset:
    routing:   product_001_qa_routing_dev
    grounding: product_001_rag_grounding_dev
  merged_path: datasets/incoming/product_001_dev.json  # Pattern A
  canonical_name: product_001                           # Pattern A
  archive_dir: datasets/archive                         # where `eval-dataset archive` files releases
  create_on_missing: false
  # Optional: Pydantic class reference (module:ClassName) or JSON file path.
  # The SDK calls .model_json_schema() at push time — no committed JSON file needed.
  input_schema: evaluation.schemas.my_input:MyAgentInput

Then run:

# Pull all scopes
eval-dataset pull-all --manifest evaluation/evaluation_manifest.yaml

# Dry-run push to check what would change
eval-dataset push-all --manifest evaluation/evaluation_manifest.yaml --version-label 1.6.0 --dry-run

# Real push
eval-dataset push-all --manifest evaluation/evaluation_manifest.yaml --version-label 1.6.0

A standalone JSON config file works too (same field names, all at the root level, paths relative to that file): pass --config sync_config.json instead of --manifest.

Each scope's push returns a PushResult: beyond created/ updated/unchanged, it also reports backfilled — items that needed no content change but still got a missing version label stamped, which is how an item added straight to Langfuse (rather than through this sync workflow) ends up covered by version-filtered pulls. Dry-run mode previews the same information as would_create/would_update/ would_backfill, with no writes performed.

First push (bootstrapping)

When pushing to a pod that has no Langfuse datasets yet, pass create_on_missing=True so the datasets are created automatically:

config = SyncConfig(..., create_on_missing=True)
push_all(config, version_label="0.0")

The version label on the initial push is arbitrary — it stamps raw_metadata.changed_last_in_dataset_version on each item and is used by derive_version to report the current dataset version. 0.0 or 0.0.0 are conventional pre-release labels; the first formal promotion to 1.0 will record the stable baseline.

Dev version labels and release versions are independent. Dev labels are operational markers for filtering and watermarking. Release versions (1.0, 1.1, 2.0) are the consumer-facing contract that CI gates and dashboards pin to. promote() re-stamps changed_last_in_dataset_version to the release version on every item it writes into the snapshot, so items in the release dataset always carry the release version regardless of what label they had in dev.

Why create_on_missing defaults to False after bootstrap: create_on_missing=True is only needed the first time. After the datasets exist, the flag's only effect is to silently create a new dataset if a name in scope_to_dataset doesn't match what's in Langfuse — which is exactly what a typo produces. With False, a misspelled dataset name raises an immediate error; with True, the push succeeds and the real dataset is untouched, leaving nothing to indicate something went wrong. Commit create_on_missing: false to evaluation_manifest.yaml and set true only for the one bootstrapping push.

Recovery after dataset deletion: pull_all always overwrites the local working-copy file with the current contents of Langfuse. If the Langfuse dev datasets are deleted and pull is run before pushing, the local file is replaced with {"items": []}, and the curated items disappear locally too. Before re-bootstrapping, restore the working copy from the snapshot of the most recent release, which holds every item as it was released:

cp evaluation/datasets/archive/{name}_v{M.m}.json \
   evaluation/datasets/incoming/{scope}_dev.json

Then run push_all with create_on_missing=True to recreate the Langfuse datasets and repopulate them from the restored file. Items curated after that release exist only where a working copy of them still does.

derive_version

derive_version(items) returns the highest changed_last_in_dataset_version label found across a list of items, using semantic-version comparison. This gives the correct answer even when only a subset of items has been bumped — the most recent version wins, not the most frequent one.

from semantic_evaluation.datasets import derive_version

pulled = pull_all(config)
print(derive_version(pulled))  # e.g. "1.5.1"

Partial pulls and pushes

Both pull_all and push_all accept an optional scope argument to operate on a single scope while leaving others untouched:

# Re-pull only the routing scope
pull_all(config, scope="routing")

# Push only the grounding scope
push_all(config, version_label="1.6.0", scope="grounding")

The CLI equivalents are --scope routing / --scope grounding.