Skip to content

Datasets: schema type versioning

Three distinct versioning concerns are tracked independently — see below.

Versioning: schema vs. content

Concern Field Owner Bumped when
Input/output field definitions (schema type) raw_metadata.input_schema_type + raw_metadata.input_schema_version semantic-evaluation team (base) / Pod (extensions) Schema type field definitions change
Dataset content (items, ground truths) raw_metadata.changed_last_in_dataset_version (and file envelope version) Pod Items are added/changed, ground truths updated
EvaluationItem (semantic_evaluation.contracts) envelope (SDK model) Implicit in SDK release semantic-evaluation team SDK-level structural changes (rare)

Keeping these separate means a pod can add 50 new items (bump dataset content version) without touching the schema type version, and the semantic-evaluation team can refine a field definition (bump input_schema_version) without forcing every existing dataset to be re-versioned.

Dataset content versioning — semantic versioning (major.minor)

Use major.minor semver for changed_last_in_dataset_version. Owned by the Pod.

Bump When
Major Any schema shape change: field added, removed, or renamed within the schema type; adoption of a breaking input_schema_version; change of input_schema_type is a new dataset entirely (not a version bump)
Minor Content-only changes: new items added, ground truths improved, no field definitions changed

Version numbers must be strictly monotonically increasing within both the minor and major tracks — a dataset at v1.3 cannot be followed by v1.2 or by another v1.3.

Patch is not used — metadata corrections do not warrant a new version label in an evaluation context.

Example progression:

v1.0  initial items: messages + response ground truth
v1.1  +40 new items; improved ground truths for existing items
v2.0  adopt citations field (structured Citation objects) → shape change

Schema type change: switching a dataset from QA (input_schema_type: "qa") to RAG (input_schema_type: "rag") is not a version bump — create a new dataset with a new name and start its version at v1.0.

Schema type versioning (input_schema_version)

Schema type definitions are owned by the semantic-evaluation team and live in sdk/semantic_evaluation/src/semantic_evaluation/schemas/ (as Pydantic models; JSON Schemas generated from them).

  • Any schema shape change (field added, removed, or renamed) → major bump (1.x → 2.0)
  • No shape changes in the current release warrant a minor bump

When a new input_schema_version introduces a breaking change, any dataset that migrates to it must also bump its own changed_last_in_dataset_version major.

Setting input_schema_version in items

When an item's raw_metadata includes input_schema_type, it must also include input_schema_version. validate() returns a ValidationError per offending item, and promote() raises ValueError for the same condition (pass force=True to override).

What value to set: for the current SDK schemas (QA, RAG, Agent, RAG-Agent), set "1.0" — these are the first stable snapshots, archived under schemas/archive/.

When to bump: only when the SDK team releases a new major version that changes the field definitions for that schema type. The bump is SDK-team-driven, not pod-driven — a pod adding 50 new items or improving ground truths does not change input_schema_version.

Independence from dataset content version: input_schema_version tracks the schema type definition; changed_last_in_dataset_version tracks item content. Both live in raw_metadata but advance on separate clocks and for separate reasons.

SDK team process — schema type field changes

When a standard schema type's field definitions change in a new SDK major release:

  1. Update schemas/_models.py with the field change.
  2. Regenerate the JSON Schema for the affected type: call ModelClass.model_json_schema() and write the result to schemas/{type}_input.schema.json.
  3. Archive the previous version before overwriting: copy schemas/{type}_input.schema.json → schemas/archive/{type}_input_v{old_version}.schema.json (the archive/ directory contains qa_input_v1.0, rag_input_v1.0, and agent_input_v1.0 as the initial snapshots).
  4. Add a CHANGELOG entry under the new SDK major version.
  5. Inform pods: datasets using the affected schema type must update raw_metadata.input_schema_version on their items before their next promote().