Skip to content

Changelog

[Unreleased]

[3.4.0] - 2026-11-13

An Open WebUI evaluation can measure each conversation from the Langfuse traces its chat writes instead of from LiteLLM's spend log. Conversations then run in parallel. Cached input is counted on every model call, cost is Langfuse's own at its model prices, and tool calls are counted by name from the trace. The spend log stays the default and keeps working unchanged. It remains the choice for an instance that does not trace every chat.

Added

  • OpenWebUIConfig.measure_from: "spend_log" (default) or "traces". With "traces", a round reads each chat's traces by session id and matches each turn to its trace by the assistant message_id in the trace metadata. The deployment must write one trace per user message.
  • OpenWebUIConfig.parallel (default 4): how many conversations run at once when measuring from traces. A conversation hook's next_message is then called from several threads at once.
  • OPENWEBUI_LANGFUSE_BASE_URL_<INSTANCE>, OPENWEBUI_LANGFUSE_PUBLIC_KEY_<INSTANCE> and OPENWEBUI_LANGFUSE_SECRET_KEY_<INSTANCE> name the Langfuse project the chats trace to, when the evaluation writes to another one. Either key may be a Key Vault secret URI. With none set, the evaluation's own Langfuse project is read.
  • The Open WebUI candidate description records measured_from in the run config.

Changed

  • Trace metrics count the usage key input_cache_read as cached input, alongside input_cached_tokens and cache_read_input_tokens.
  • Each turn in the Open WebUI driver's turn log, saved with the transcript, records its message_id.

[3.3.0] - 2026-11-12

Code that reads a platform's transcripts, during a run or from a stored export, has public readers for them, and a conversation hook gets the parsed tool calls on the outcome. Which modules of semantic_evaluation.platforms are public is now stated: the rest is internal, stays importable until 4.0.0, and has a named replacement in 09-platform-predictors.md. The runner identifies a project to the scoring service from SCORING_SERVICE_API_KEY in the environment, with no code change.

Added

  • Public in semantic_evaluation.platforms: ToolCall, open_webui_active_path, open_webui_tool_calls, open_webui_whole_file_sources, copilot_tool_calls and delivered_text. They read a transcript live from ConversationOutcome.transcript or one stored earlier.
  • ConversationOutcome.tool_calls: the tool calls the platform read from the transcript, the ones measurements.tool_calls counts. Empty for an errored outcome.

Changed

  • execute_offline_evaluation() sends SCORING_SERVICE_API_KEY from the environment when no scoring_service_api_key is passed. A passed key still wins.
  • active_path and whole_file_sources are public as open_webui_active_path and open_webui_whole_file_sources. The old names stay importable from semantic_evaluation.platforms.transcripts until 4.0.0.
  • 09-platform-predictors.md lists the internal modules of semantic_evaluation.platforms (open_webui.client, .chat, .litellm_spend, copilot_studio.workflow, .dataverse, documents.docx_text) with the public replacement for each. They keep their import paths until 4.0.0, and their behaviour may change in a minor release.

[3.2.0] - 2026-11-11

Agents that run on Open WebUI or on Copilot Studio can now be evaluated with a predictor the SDK ships. semantic_evaluation.platforms holds real conversations with a candidate copy of the agent, reads the platform's own transcript back and records what the platform measures. A project writes only a conversation hook: which items make up one conversation, what to say next, and what the transcript predicts. A run file can name the predictor, the platform configuration and the hook, with no entry point of the project's own. A conversation that produced no transcript no longer counts as a wrong answer: its predictions carry an error, are left out of scoring and are counted against a tolerance, by default zero. Upgrading changes nothing for a predictor that never sets error.

Added

  • semantic_evaluation.platforms: PlatformPredictor, the ConversationHook and Platform protocols, Conversation and ConversationOutcome, OpenWebUIConfig, OpenWebUIPlatform and SpendRates (cost from the LiteLLM spend log, at the configured rates or at LiteLLM's own), CopilotStudioConfig and CopilotStudioPlatform (transcripts from Dataverse; no cost, which Copilot Studio does not report per conversation). See 09-platform-predictors.md.
  • PredictionRecord.error: why the agent produced no answer to read. An errored prediction is saved, published with a created trace carrying offline_eval.error, and left out of scoring, trace metrics, grouped aggregates and agreement.
  • max_error_fraction in the run file: the largest share of a scope's predictions that may be errored before the gate fails. Defaults to 0. A project that commits run_file.schema.json regenerates it with eval-dataset schemas after upgrading, or the editor flags the key.
  • errored_prediction_count in run_summary.json; eval-report shows errored: n of N beside a run's items and an Errored row when compared runs have errors.
  • The pdf extra (semantic-evaluation[pdf], pypdf), needed only to send a PDF as text to Copilot Studio.

Changed

  • When the runner builds the predictor from predictor_adapter, it passes run_config and repeats to a constructor that accepts them, and calls describe_candidate() when the predictor defines it, merging the result into the run's run_config before items load. A predictor with a no-argument constructor is built as before; one passed in by the caller is not described.
  • A resumed run describes a built predictor's candidate again and is refused when the description differs from the saved run's.

[3.1.0] - 2026-11-09

eval-dataset push now writes back every edit to an item's metadata. A change that only touches an item's tags is pushed, so category labels can be corrected through pull, edit, push. That includes stripping the approved-as-is / explicit-correction strings that items promoted before 3.0.0 still carry. Typed metadata fields such as product_id reach Langfuse even when the local file does not repeat them under raw_metadata, which is the shape of a hand-written seed file. pull and pull-all print the absolute path they wrote and accept --dry-run. The first push after upgrading may update items whose stored tags or typed fields differ from the local file. Before this release, those differences were silently skipped.

Added

  • eval-dataset pull --dry-run and eval-dataset pull-all --dry-run report what would be written without writing a file. pull_all() takes dry_run=True for the same.

Changed

  • diff() puts an item in changed when its content (canonical_hash) or its tags differ, so push re-upserts a tag-only edit and promote accepts a release whose only change is to tags. canonical_hash still leaves tags out, so dedupe treats items that differ only in tags as duplicates.
  • eval-dataset pull and pull-all print the resolved absolute path of the file or directory they wrote.

Fixed

  • push wrote only raw_metadata, the version and non-empty tags, so the typed fields product_id, created_by, created_date, workflow_version and selected_agent were lost when they were not also present in raw_metadata. They are now written, and a typed field set in the file overrides the same key in raw_metadata. canonical_hash covers them, so editing one counts as a change.
  • An explicit "tags": [] in a local file clears the item's tags in Langfuse. Before this fix, loading the file restored the old tags from raw_metadata, and push skipped the empty list. raw_metadata["tags"] is lifted onto tags only when the file has no tags key.

[3.0.0] - 2026-11-07

What an offline evaluation run writes onto a prediction trace now describes itself, and a dataset item's tags mean one thing. Per-item scores are named offline_eval.<metric>, and their comment names the run and the dataset version that produced them. Review provenance has moved out of tags into a typed field. No Python signature changes. It is a major because the score names and the trace tags this package writes to Langfuse change. Before upgrading, move any Langfuse score config, dashboard or script that reads a per-item score by its bare metric name to the prefixed name. Move any query that filters dataset items by the tags approved-as-is or explicit-correction to review_outcome. The reasons are recorded in project-context/adr/0002-offline-evaluation-writes-onto-prediction-traces.md in the source repository.

Added

  • DatasetRecordMetadata.review_outcome: approved_as_is when the reviewer accepted the agent's output, explicit_correction when they supplied a corrected one. Review-sync promotion sets it.
  • Docs: what each dataset item metadata field means for a prediction trace, and a recommended set of run_config keys that describe the candidate the same way on every runtime (platform, candidate, model, delivery, content_sha256, plus the reserved instructions_version, knowledge_version and release_id, which the SDK never fills). See eval-docs show evaluation/04-offline-evaluation.

Changed

  • Per-item scores on the experiment item span, both metrics and trace metrics, are named offline_eval.<metric> (for example offline_eval.answer_correctness, offline_eval.wall_time_s). Their comment starts [<execution name> | dataset <version>], or dataset unversioned, followed by the judge's justification. Run scores on the experiment ({scope}.{metric}, spread, per-group, agreement, gate_passed) keep their names, so manifest baselines and promote-baseline are unaffected.

Removed

  • Review-sync promotion no longer appends approved-as-is or explicit-correction to a record's tags, so these strings no longer become trace tags on prediction traces. Items promoted by an earlier version keep them until they are promoted again. The first sync that promotes such an item rewrites it once, because its stored metadata lacks review_outcome.

[2.5.0] - 2026-11-06

The scoring and review-sync clients call the services' versioned API, /v1. Products change nothing: base URLs stay host-only and every method keeps its signature. The unversioned routes older versions call are deprecated and removed no earlier than 4 February 2027, so upgrade before then. A service you run locally has to be updated together with the SDK; see eval-docs show evaluation/08-service-api-versions.

Added

  • Both clients send User-Agent: semantic-evaluation/<version>, which the services log with the API version.
  • A clear error when a service does not serve /v1: a 404 with the route-not-found body raises RuntimeError naming the base URL and saying the service is older than the SDK. The client does not fall back to unversioned paths.
  • One logged warning per client instance when a service announces the retirement of the API major the client calls (Deprecation header), with its Sunset date.
  • Docs page 08-service-api-versions: which API major each SDK major calls, what counts as breaking, and the retirement window.

Changed

  • ScoringServiceClient and ReviewSyncServiceClient call every route under /v1.

[2.4.0] - 2026-11-05

A run whose agent predicts for an hour on an external platform no longer loses those predictions when the scoring service or the network drops out afterwards. The runner saves the predictions before anything is scored, the scoring client retries transient failures, and a stopped run can be scored and published from what it saved. Run files also keep a bare YES or NO as the string the author wrote, instead of turning it into a boolean.

Added

  • execute_offline_evaluation(resume_from=<run directory>), and run --resume-from in the pod CLI templates: scores and publishes the predictions a stopped run saved, without calling predict_items, and writes into the same directory. It refuses a resume whose run file name, predictor_adapter, run_config, repeats, scopes or run label differ from the stopped run, and saved records that do not fit the loaded items.
  • predict_items may take a keyword repeat: int (from 1); the runner passes it when the predictor's signature accepts it. Predictors without it work as before.
  • ScoringServiceClient(retry_attempts=6, retry_initial_backoff_seconds=2.0): transient failures are retried with a doubling backoff. Status polls retry on any transport error and HTTP 502/503/504; the job submission only on a connection error, connect timeout or HTTP 503, so a job is never submitted twice.
  • A warning when agreement.run_shift is set and the run has exactly 2 repeats: a unit that differs then marks both repeats, so it cannot say which run moved.

Changed

  • effective_config.json and predictions.jsonl are written as soon as predicting ends, before trace metrics and scoring; the other artefacts follow after scoring. effective_config.json also records the run label. When a step after predicting fails, the runner logs the run directory to resume from and re-raises the error.

Fixed

  • Run files, evaluation_manifest.yaml and review binding files are read with YAML 1.2 booleans: only true and false are booleans. An unquoted yes, no, on or off loads as a string and logs a warning naming the file and line. Before, one_of: [YES, REVIEW] loaded as [True, "REVIEW"] and FieldEquals silently matched only REVIEW. The scoring service now refuses a FieldEquals whose one_of mixes booleans and strings.

[2.3.0] - 2026-11-05

An offline evaluation can now report each metric per group of dataset items as well as over the whole run. A screening agent, for example, gets its share of risks caught over the items expected YES and its false-flag rate over the items expected NO, two figures one whole-run mean cannot give. A run with repeats can also measure how its repeats agree on one prediction field: per item, how many repeats give the most frequent value; per run and group, how many items are unanimous; and, optionally, which repeat of one source document moved off the other repeats' majority as a whole. All of it is opt-in in the run file. Nothing changes for a run that does not set the new fields.

Added

  • Run file aggregate_by: [attribute, ...]: every metric, scored or trace, is also aggregated per value of each named item metadata attribute. Grouped aggregates are written to grouped_aggregates.json and published as run scores {scope}.{metric}.by.{attribute}.{value}, with spread statistics under the existing score_publication switches. They never enter the gate.
  • Run file agreement: {field, run_shift: {per, min_share}}: agreement across repeats on a dotted prediction field, for a run with 2 or more repeats. It is written to agreement.json and published as {scope}.agreement.{field}.unanimous_share, .mean_modal_share (whole run and per group) and .run_shifts, with score source offline_evaluation_agreement.
  • AggregateMetricResult.group_by and group_value, and the contracts ItemAgreement, AgreementSummary, RunShift and AgreementResult.
  • eval-report shows a table per grouping attribute, the agreement table and the run-shift table, and compares groups and agreement across runs.
  • promote-baseline keeps the grouped means of scored metrics and the agreement scores in the manifest's baselines; --update-gates ignores them.

Changed

  • aggregate_metric_results moved from the scoring service into semantic_evaluation.scoring.aggregation, and the dotted-path lookup the field evaluators use into semantic_evaluation.scoring.fields, so the service and the runner share one definition of each. The service's aggregates and gate are unchanged.

[2.2.1] - 2026-11-05

Fixed

  • The 2.2.0 wheel and sdist contained semantic_evaluation/.boost/, a working cache an agent tool had written into the source tree: local paths and shell commands from the release session, nothing the package uses. The build excludes it now. The code is the same as 2.2.0; install 2.2.1 instead.

[2.2.0] - 2026-11-05

Langfuse Cloud stops serving its deprecated public API on 16 November 2026, and this package published runs, promoted baselines and read reviews through it. 2.2.0 moves to Langfuse Python SDK 5 and Langfuse's current APIs: an offline evaluation publishes as a Langfuse experiment, which is what Langfuse now calls a dataset run. Upgrade before 16 November; on an older release publication, promote-baseline and review sync stop working that day. Score names, run names and the manifest's baselines block are unchanged.

Agents that run on a platform semantic_tracing cannot instrument, such as Open WebUI or Copilot Studio, can also be evaluated offline like any other: the predictor reads the answer, tool calls, timing and spend from the platform's own transcript, and the runner publishes the experiment, per-item and aggregate scores, and trace metrics without a Langfuse trace from the agent. And a scoring service can offer Microsoft Decision-1 as a fixed-option judge through the shared contract.

Changed

  • Requires langfuse>=5.1.0,<6 (now declared directly) and semantic-tracing>=0.10.0,<1.
  • Publication links each prediction to its run by adding one span, offline_eval_item, to the prediction trace the agent wrote. The span is in Langfuse's experiment environment sdk-experiment and carries the experiment, the dataset item, the item's tags as trace tags, and the run metadata (offline_eval.*, run_config.<key>, prediction_metadata) as trace metadata. The agent's own observations are no longer modified after the run.
  • Per-item scores, trace metrics included, attach to that span, where Langfuse's experiment view reads them, instead of to the prediction's observation. Run scores (aggregates, spread, gate_passed) attach to the experiment as before. Langfuse's own latency and cost columns in the experiment view describe the span; wall_time_s and cost_usd describe the agent's run.
  • Publishing needs Langfuse tracing enabled, because experiment items are spans; with it disabled, publication raises RuntimeError before writing anything.
  • promote-baseline finds the run among the dataset's experiments, older dataset runs included, and records gate_passed as 1.0 or 0.0. A dataset it cannot read is logged as a warning and skipped.
  • Review sync reads a reviewed trace's input, output and metadata from its root observation, ignoring an offline evaluation's offline_eval_item span, and reads annotation and correction scores through Langfuse's v3 scores API. Redeploy review_sync_app with this release.
  • measure_trace() returns PredictionMeasurements, which replaces the TraceMeasurements dataclass in semantic_evaluation.orchestration.runner.
  • The missing trace_id error names prediction_traces: create_when_missing as the alternative to tracing the agent call.

Added

  • The decision_llm_judge evaluator backend and the foundry_decision scoring connection-profile kind, so a scoring service can expose Microsoft Decision-1 evaluators through the shared SDK contract.
  • PredictionRecord.measurements (PredictionMeasurements): wall time, tokens, cost and tool calls the predictor measured itself. When set, the run's trace_metrics are taken from it and no trace is read; the score's justification says "supplied by the predictor." and details.measured_from is predictor.
  • prediction_traces: create_when_missing in a run file: publication writes a thin prediction trace, named offline_eval_prediction/<predictor>, for every record without a trace_id. Its one span is the experiment item: the item's identifiers as input, the prediction as output, the run metadata and the item's tags, never the item input, and the per-item scores. run_summary.json reports created_trace_count. The default, required, keeps refusing a record without a trace_id.
  • PredictionRecord.source_transcript (TranscriptReference): where the platform keeps the run's transcript, written to the prediction trace as offline_eval.source_transcript. A reference only; the transcript never goes to Langfuse.
  • PredictionMeasurements and TranscriptReference are exported from semantic_evaluation.prediction.

The scoring service request is unchanged for a service on an older release: it ignores the two new PredictionRecord fields.

[2.1.1] - 2026-10-22

Fixed

  • eval-report shows costs in USD to six decimals instead of four, so a cheap model's cost per prediction no longer reads 0.0000.

[2.1.0] - 2026-10-22

Added

  • eval-report: a Markdown report of one or several offline evaluation runs, read from the artifacts in their run folders, so it needs no network. Per run: the label and run_config, items and repeats, the gate with each failing metric, every metric's mean, median, p90, standard deviation, pass rate and threshold, the trace metrics with their totals, and a Langfuse link per dataset run. Several runs start with a comparison table, one column per run. A run with no recorded cost is flagged, because an unpriced model reads as free. load_run_report() and render_markdown() in semantic_evaluation.orchestration.reporting.run_report do the same from Python.
  • eval-review: registers a project's Langfuse profile (register-profile, profiles) and its review bindings (validate, register-binding, bindings) with the review-sync service, and runs a binding's sync (sync, with --dry-run). A binding is a QueueToDatasetBinding kept as a YAML or JSON file, so a project needs no code of its own for review.
  • ReviewSyncServiceClient.register_langfuse_profile() and list_langfuse_profiles(), which send the client's api_key.
  • 04-offline-evaluation: the run folder and its six artifact files; how a configuration reaches the agent (the project builds the predictor with run_config; a key the predictor does not use only labels the run); eval-report.
  • 06-review-sync: setting up queues and score configs in Langfuse, by UI or API, and which paths of a reviewed item a projection can read; the eval-review command with a binding file; what to do when a sync fails, the three causes of 404 Annotation queue not found among them.

[2.0.1] - 2026-10-21

Documentation only. No code changed, so nothing in a project changes with the upgrade except what eval-docs shows.

Fixed

  • The install instructions now set prerelease = "allow" in uv.toml, above the first [[index]] table, and pass --prerelease allow to uv add. Without it, a new project's uv add semantic-evaluation fails with No solution found, because the SDK pins pre-releases of Microsoft Agent Framework. The index is named ramtech-pypi-internal, as the repository it points at is.

Added

  • 03-datasets-cli: the working copy's JSON shape, which pull writes and push and validate read, with an example to write the first items of a new dataset from.
  • 03-datasets-lifecycle and the glossary: the coverage map, evaluation/coverage.md, the project file naming each behaviour the agent must keep with the items that prove it and the metrics that check it.
  • 04-run-config: how a metric passes the gate, and that a gate means something only on a released dataset.
  • The scoring service's evaluator catalogue page, shipped in this package: when a behaviour gets a deterministic check, when an LLM judge, and when a unit test instead, and how to set each one's threshold.

[2.0.0] - 2026-10-21

Every project now configures its evaluation the same way. There is one evaluation/evaluation_manifest.yaml per project, declaring each metric and each scope's dataset once. Then there is one YAML file per run under evaluation/runs/ that names its scopes and what it changes. The editor checks both files against schemas the SDK writes into the project. The run's predictor_adapter now decides which predictor runs, and archive/ holds only release snapshots. The changes below that break a 1.x project are each one edit: name the predictor as module:Class, write dataset_sync paths relative to the manifest, and file release snapshots with eval-dataset archive --name.

Added

  • RunFile, the shape of a run file, and EvaluationManifestFile with DatasetSyncConfig, the shape of evaluation_manifest.yaml. Both refuse an unknown key, so a misspelled field fails when the file is loaded. resolve_run_manifest() validates its input as a RunFile, and load_evaluation_manifest_file() returns the whole project manifest.
  • eval-dataset schemas [--dir evaluation] writes evaluation/evaluation_manifest.schema.json and evaluation/runs/run_file.schema.json from the installed SDK, for the # yaml-language-server: $schema=... line at the top of each file. The committed schemas/ gain run_file.schema.json.
  • resolve_predictor_class(manifest) imports the class predictor_adapter names.
  • eval-dataset archive --name <release> pulls a released dataset into <archive directory>/<release>.json. The directory is dataset_sync.archive_dir with --manifest, or --dir. It refuses a dev dataset and a snapshot that is already there. archive_release() is the same in Python. promote prints the command to run next.

Changed

  • Breaking: predictor_adapter must name a class as module.path:ClassName. A bare label such as support_workflow is refused when the manifest is loaded. execute_offline_evaluation() now uses it: predictor is optional, and without it the runner builds the named class with no arguments. A predictor passed in must be an instance of that class, or the run stops with a TypeError.
  • Breaking: relative paths in dataset_sync (scoped_dir, merged_path, archive_dir, schema files) resolve against the directory of the manifest, or of the --config JSON, that holds them, not against the working directory.
  • Breaking: evaluation_manifest.schema.json now describes the file (evaluators, dataset_sync, baselines) rather than the loaded EvaluationManifest. It and run_file.schema.json set additionalProperties: false.
  • The PredictionRunner docstring says what the runner does: one predict_items() call per repeat, with the items of every requested scope.

Removed

  • Breaking: push and push-all no longer copy each pushed working file into the archive directory, and SyncConfig.archive_dir is gone. Langfuse keeps every dev version.
  • Breaking: archive(source, archive_dir, label=...) and eval-dataset archive --source are replaced by archive_release() and eval-dataset archive --name.

[1.5.0] - 2026-10-21

The wheel now carries the documentation: this SDK's, both shared services', and the repository glossary, read with the new eval-docs command. A run manifest can also choose which scores a run writes to Langfuse, which bills each one, so a repeated configuration writes its aggregates once and leaves out the spread statistics and per-trace measurements that say nothing. Everything is additive: a manifest that sets none of the new fields writes what 1.4.0 wrote.

Added

  • The wheel carries the documentation: this SDK's docs and changelog, both shared services' docs, and the repository glossary, under semantic_evaluation/_docs/, as they were when the release was built.
  • eval-docs console script to read them: list, show <page>, search <regex>, path and version. It reads semantic-tracing's shipped docs too, and rewrites links in show output as page:<name> (another shipped page) or repo:<path> (a source file that is not shipped).
  • docs/07-access-and-credentials.md: every credential a project needs, where it comes from, whether it is self-service, and whom to ask.
  • RunManifest.score_publication (ScorePublicationConfig): which scores a run writes to Langfuse, which bills each one as a unit. aggregates_on: last_repeat puts the aggregates and gate_passed on the last repeat's dataset run only; spread_statistics names which of median, p90 and stdev are written; informative_spread_only writes no spread for a binary metric or an aggregate over one value; trace_metric_item_scores: false writes trace metrics as aggregates only. The same four are keyword arguments of publish_offline_evaluation_dataset_runs. A manifest that sets none of them writes exactly what 1.4.0 wrote. The setting is recorded in each dataset run's metadata, and run_manifest.schema.json carries it.

[1.4.0] - 2026-10-20

An offline evaluation run can now compare one agent across configurations, on time and cost as well as on its scores. The configuration under test has a structured place to go and becomes filterable in Langfuse; each item can be predicted several times, so a comparison has a spread rather than one value; and what each prediction cost is read off its trace into metric results. Every aggregate also reports a median, a p90 and a standard deviation. Everything is additive: a manifest that sets none of the new fields runs as before.

Added

  • RunManifest.run_config and execute_offline_evaluation(run_config=...): flat key-value pairs written into every dataset run's metadata and onto every prediction trace as top-level run_config.<key> metadata. The argument is merged over the manifest's value.
  • RunManifest.repeats and execute_offline_evaluation(repeats=...): each item is predicted that many times and all predictions are scored in one job. Each repeat is its own Langfuse dataset run, {execution_name}_r{n}, because Langfuse keeps one run item per dataset item in a dataset run.
  • RunManifest.trace_metrics: wall_time_s, input_tokens, output_tokens, cached_tokens, reasoning_tokens, cost_usd, tool_calls and tool_calls_by_name, read per prediction from its trace and reported as metric results and aggregates with raw_scale set to their unit. They never fail the gate. A predictor may define fetch_trace(trace_id) to supply the trace, such as from local capture; otherwise it is read from Langfuse.
  • PredictionRecord.repeat and MetricResult.repeat, so a result can be matched to the prediction it scored.
  • AggregateMetricResult.median_score, p90_score, stdev_score and raw_scale, and semantic_evaluation.scoring.stats.spread(), which the scoring service now uses for its aggregates. In Langfuse each metric gets {scope}.{metric}.median, .p90 and .stdev run scores beside the mean.
  • PredictionRecord.metadata, never forwarded before, is written onto the prediction trace as prediction_metadata, with offline_eval.execution_name, offline_eval.dataset_run_name and offline_eval.repeat.

Changed

  • eval-dataset promote-baseline skips the spread scores and the trace-metric aggregates a run now publishes; neither is a metric's baseline.

Compatibility

  • repeats above 1 needs the scoring service built from this release, which copies each prediction's repeat onto its results. Against an older service a repeated run stops after scoring rather than publish scores on the wrong traces. With repeats: 1 either service works.

[1.3.0] - 2026-08-23

EvaluatorBackendKind now names all three evaluator backends the scoring service supports. It was missing custom_rubric_llm_judge, so any code annotating a backend with this type was rejecting a value the service accepts, and the published contract advertised a backend set one short of reality.

Fixed

  • contracts.py: EvaluatorBackendKind gains custom_rubric_llm_judge. The scoring service now derives its own accepted backend set from this Literal instead of restating it, so the two can no longer drift.

[1.2.0] - 2026-08-22

eval-dataset now reads credentials out of a project's .env, so the commands in the documentation work as written from a pod directory instead of only after exporting LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY by hand. This makes the CLI usable as the pod's bootstrap tool: publishing a starter dataset is two push --create-on-missing invocations rather than a bespoke script. Also stops semantic_evaluation.__version__ reporting a stale number, which mattered because it is stamped into the provenance of every promoted dataset.

Added

  • eval-dataset: reads ./.env and ./.env.local before running a command, without overriding values already exported in the shell. A new top-level --env-file PATH names a single file instead, and reports an error if it does not exist. Calling the semantic_evaluation.datasets Python helpers directly is unchanged and still touches no dotenv file, because a library must not alter its host's environment on import.

Fixed

  • semantic_evaluation.__version__ reported 1.0.0 for the 1.1.0 release. It is now read from the installed package metadata, so it cannot drift from the released version again. Datasets promoted by 1.1.0 carry 1.0.0 in their provenance metadata.

[1.1.0] - 2026-07-29

A Langfuse dataset item added straight from a traced observation, via Langfuse's own "+ Add to dataset" action, now works end to end with this SDK: its input is auto-normalized to the shape a pod's predictor expects, it joins the version-tracking scheme on its next push instead of being silently excluded forever, its trace link survives a pull-edit-push cycle instead of being silently dropped, and callers can tell it apart from an item produced by this SDK's own seed or promote tooling. Also removes a dead publish path that had no remaining callers. Separately, scoring a batch of predictions is now a background job instead of one blocking HTTP call: ScoringServiceClient submits and polls automatically, so a batch scored against a slow LLM judge can no longer time out regardless of size, and judge calls within a batch now run concurrently instead of one at a time.

Added

  • orchestration/datasets/langfuse.py: normalize_dataset_item() accepts an optional input_schema and auto-unwraps a single-key-wrapped input (the shape Langfuse's own "+ Add to dataset" action produces from a traced observation), schema-aware when a schema is registered for the scope and via a bare structural heuristic when it isn't; new shared fetch_and_normalize_items() used by both dataset-read paths.
  • contracts.py: DatasetSourceConfig.input_schema — a dotted module.path:ClassName reference, resolved by LangfuseDatasetProvider.load_items() to drive the same schema-aware unwrap for the offline-eval runner path.
  • contracts.py: EvaluationItemMetadata.added_via_langfuse_ui — a derived, read-only property identifying items whose ground truth came directly from Langfuse's own "+ Add to dataset" action rather than this SDK's seed or promote tooling, computed from a populated trace_id combined with the absence of ground_truth_status in raw_metadata — no new tagging convention required.
  • _schema_resolution.py: new shared module (resolve_dotted_class(), to_json_schema()) consolidating dotted-reference and JSON-schema resolution previously duplicated across datasets/cli.py and datasets/_multi.py.
  • datasets/_sync.py: PushResult gains would_backfill/backfilled, mirroring the existing would_create/created pairing.
  • contracts.py: ScoreJobStatus, ScoreJobSubmitted, ScoreJobResult — the scoring-job contracts returned by the scoring service's POST /score-jobs/GET /score-jobs/{job_id}.
  • scoring/client.py: ScoringServiceClient.submit_score()/get_score_job() — submit a scoring job and check its status directly, for callers that want manual control instead of score()'s submit-and-wait convenience.
  • scoring/client.py: ScoringServiceClient constructor gains poll_interval_seconds (default 3.0) and max_wait_seconds (default None, waits indefinitely); score() gains an optional on_progress callback invoked on every poll with the current ScoreJobResult.

Changed

  • datasets/_sync.py: pull() accepts input_schema and shares its dataset-fetch loop with the offline-eval runner via fetch_and_normalize_items(); datasets/_multi.py's pull_all() resolves SyncConfig.input_schema and threads it through to pull().
  • datasets/_sync.py: push() now backfills a missing changed_last_in_dataset_version onto any item regardless of which diff bucket it falls into — previously only added/changed items were stamped, so an unchanged, UI-added item sat outside version-filtered pulls indefinitely.
  • datasets/_sync.py: push() now always re-supplies source_trace_id/ source_observation_id on every upsert. Previously these were omitted entirely; the installed Langfuse SDK sends an explicit JSON null for omitted fields rather than dropping the key, so a pull → edit → push cycle risked silently clearing a UI-added item's trace link even when its content never changed.
  • datasets/cli.py: push, push-all, and promote's printed summaries now report would_backfill/backfilled alongside the existing would_create/created etc.
  • scoring/client.py: ScoringServiceClient.score() now submits the request as a job and polls it to completion instead of making one blocking HTTP call. Its signature and return type are unchanged, so existing callers work without changes; the scoring service's HTTP contract underneath moves from a single POST /score to POST /score-jobs + GET /score-jobs/{job_id}.

Removed

  • orchestration/staging/: publish_seeded_eval_cases(), its private helper _ensure_langfuse_dataset(), and io.py's write_eval_cases()/load_eval_cases() are gone — all had zero production callers left anywhere in this repo, the one real caller having already moved to the standard eval-dataset push CLI path in an earlier wave. LangfuseDatasetDefinition is retained; review_sync's publish path still depends on it.

[1.0.1] - 2026-07-27

Fixes review sync's Langfuse artifact reader, which was silently broken against the currently-installed Langfuse Python SDK: two methods it called had been renamed or moved. Any pod running a review-sync job would hit these calls whenever fetching an observation or its annotation scores, so upgrading is recommended for every consumer of review sync.

Fixed

  • orchestration/review_sync/langfuse.py: LangfuseReviewArtifactReader now calls self.langfuse.api.legacy.observations_v1.get(...) instead of the removed self.langfuse.api.observations.get(...), and self.langfuse.api.scores.get_many(...) instead of the renamed self.langfuse.api.scores.get(...).

[1.0.0] - 2026-07-21

Unifies all schema types to use messages: list[Message] as the only input form, adds the RAG-Agent combined schema type for systems that retrieve context and use tools, and makes AgentExpectedOutput.response required. promote() now enforces that any schema shape change (including optional fields added to only some items) requires a major version bump.

Added

  • RAGAgentInput — extends RAGInput with available_tools: list[str] | None; for systems that retrieve context and invoke tools
  • RAGAgentExpectedOutput — extends RAGExpectedOutput with tool_calls and handoff_to; inherits mandatory response: str
  • schemas/rag_agent_input.schema.json — JSON Schema generated from RAGAgentInput
  • schemas/archive/ — qa_input_v1.0, rag_input_v1.0, agent_input_v1.0 snapshots of the initial schema type definitions
  • validate() returns a ValidationError for each item that declares raw_metadata.input_schema_type without raw_metadata.input_schema_version
  • promote() raises ValueError for the same condition when force=False; pass force=True to override

Changed

  • BaseInput: query: str replaced by messages: list[Message] (required)
  • AgentInput: reparented from BaseModel to BaseInput; messages is now inherited. isinstance(x, BaseInput) returns True for agent inputs.
  • AgentExpectedOutput: reparented from BaseModel to BaseExpectedOutput; response is now str (required), inherited from BaseExpectedOutput. Callers that omit response raise ValidationError.
  • RAGInput: inherits messages from BaseInput (no direct field change to RAGInput itself)
  • promote() version enforcement: SchemaChangeLevel.MINOR (optional field added to some items) now requires a major bump; previously it allowed a minor bump

[0.7.0] - 2026-07-08

Pods can now declare each evaluator once in a shared per-metric catalogue and have run YAML files inherit dataset names and evaluator definitions from it, instead of hand-copying full evaluator/dataset definitions into every run file.

Added

  • EvaluationManifest — pod-level canonical config model: evaluators: dict[str, CatalogueMetricConfig] (a metric-name-keyed catalogue, each entry tagged with the scopes that use it) and scope_to_dataset: dict[scope, str].
  • CatalogueMetricConfig — extends MetricConfig with a scopes: list[str] field.
  • load_evaluation_manifest(path) — loads evaluators and dataset_sync.scope_to_dataset from a pod's evaluation_manifest.yaml.
  • resolve_run_manifest(run_source, pod) and load_resolved_run_manifest(run_manifest_path, evaluation_manifest_path) — resolve a partial run YAML (scopes: plus optional overrides) into a fully-specified RunManifest, filling in dataset names and evaluator definitions from the pod's EvaluationManifest. Supports per-metric field overrides (with a shallow merge on evaluator_params), scope narrowing via evaluators.{scope}.only: [...], and ad hoc metrics not yet in the catalogue.
  • eval-dataset resolve-run --run-file --manifest — prints a run YAML fully resolved against a pod manifest, for inspecting exactly what a run will execute.
  • schemas/evaluation_manifest.schema.json — generated JSON Schema for EvaluationManifest.

Changed

  • eval-dataset promote-baseline --update-gates now recognises the new dict-keyed evaluators catalogue shape in addition to the pre-catalogue flat list, so it keeps working unchanged for pods that haven't migrated yet.

RunManifest and execute_offline_evaluation() are unchanged, and adopting load_resolved_run_manifest() is opt-in — pods that keep hand-writing fully inline run YAML are unaffected.

[0.6.2] - 2026-07-03

promote() now writes major, minor, and stage fields to the Langfuse release dataset metadata so data engineers can filter and query datasets by version components without parsing the dataset name.

Added

  • promote(): writes major: int, minor: int, and stage: "release" to the Langfuse dataset object metadata alongside the existing provenance fields (promoted_from, promoted_at, schema_change_level, sdk_version).

[0.6.1] - 2026-07-03

Fixes promote() writing only {"dev_version": ...} to promoted dataset metadata, and the dev_version stamp incorrectly firing during dry-run pushes.

Fixed

  • push(): add stamp_dev_version: bool = True parameter. When False, the dev_version stamp on the Langfuse dataset object is skipped entirely. This prevents promote() from clobbering its own provenance metadata (promoted_from, promoted_at, schema_change_level, sdk_version) with a bare {"dev_version": "..."} after writing it.
  • push(): the dev_version stamp is now also skipped during dry-run, fixing a latent bug where a dry-run push would still write to Langfuse.
  • promote(): passes stamp_dev_version=False to push(). Release datasets carry only provenance metadata — dev_version is a dev-dataset concept.

[0.6.0] - 2026-07-03

Renames the per-item version label field from dataset_version to changed_last_in_dataset_version to clarify that the field reflects the version at which a specific item's content was last changed or first pushed — not a global snapshot version that advances on every push. The dataset-level dev_version field (on the Langfuse dataset object) continues to reflect the most recently declared push label.

Breaking

  • EvaluationItemMetadata.dataset_version → EvaluationItemMetadata.changed_last_in_dataset_version
  • raw_metadata["dataset_version"] key in Langfuse → "changed_last_in_dataset_version". Existing items carry the old key; re-push after deleting dev datasets to migrate.
  • pull --version X.Y and pull_all --version X.Y now filter on changed_last_in_dataset_version.
  • derive_version(items) now reads raw_metadata["changed_last_in_dataset_version"].

Not renamed

dataset_version on QueueToDatasetBinding, EvalCaseRecord, and build_promoted_records in the review-sync subsystem are unchanged — those fields have a distinct semantic and are part of a separate service API.

[0.5.4] - 2026-07-03

pull-all now reports the correct current dev-dataset version even when a push declared a new version label but no item content changed. Previously, the "Current version in Langfuse" message and the local file envelope version were derived from the highest per-item dataset_version, which only advances when items are actually upserted. Now pull_all reads the dev_version field from the Langfuse dataset object (stamped by push on every push) and uses it as the file envelope version.

Changed

  • pull_all(): after pulling items, calls _get_dataset_dev_version() on one dataset object and passes the result to _write_merged / _write_scoped.
  • _write_merged() and _write_scoped(): prefer dev_version over derive_version(pulled) when writing the local file envelope version.
  • _cmd_pull_all CLI: reads the envelope version from the just-written merged file instead of recomputing derive_version(pulled), so the terminal message matches the file.

[0.5.3] - 2026-07-03

push() now stamps the current version_label on the Langfuse dataset object metadata (dev_version field) on every push, regardless of whether any items were upserted. This makes the dev dataset version visible at the dataset level in Langfuse — previously it was only visible per-item (buried in item metadata).

Changed

  • push() in _sync.py: after upserting items, calls create_dataset(name=name, metadata={"dev_version": version_label}) to update the dataset-level metadata. No-op in dry-run mode. Existing dataset metadata other than dev_version is preserved (Langfuse merges the provided fields).

[0.5.2] - 2026-07-02

Pods no longer maintain a separate committed JSON schema file. SyncConfig now accepts Pydantic model classes directly for input_schema and expected_output_schema; push_all resolves them to dicts at push time. The manifest's dataset_sync.input_schema field can now be a dotted class reference (module.path:ClassName) as well as a JSON file path — _load_sync_config resolves it automatically when building SyncConfig.

Added

  • SyncConfig.input_schema and SyncConfig.expected_output_schema: optional fields that accept a Pydantic model class or a pre-built schema dict. When a class is passed, .model_json_schema() is called at push time.
  • SyncConfig schema passthrough in push_all(): both _push_merged and _push_scoped now resolve and forward schemas to the per-scope push() call, so Langfuse datasets created via eval-dataset push-all --manifest carry the correct schema metadata.
  • _resolve_schema_string() in cli.py: resolves a manifest schema value (dotted class reference or JSON file path) to a Pydantic class or dict, then stores it in SyncConfig. Reuses the existing _load_input_schema() logic.

[0.5.1] - 2026-07-02

Adds eval-dataset promote-baseline to close the local-tuning loop: after a baseline run lands in Langfuse, engineers can snapshot its aggregate scores into evaluation_manifest.yaml without manually transcribing numbers.

Added

  • eval-dataset promote-baseline subcommand: fetches run-level aggregate scores from Langfuse for a named dataset run and writes them into the baselines block of an evaluation manifest YAML file (keyed by run name, so multiple snapshots coexist).
  • ruamel.yaml dependency: required for comment-preserving round-trip rewriting of evaluation_manifest.yaml (plain yaml.safe_dump would destroy inline comments on evaluator criteria and threshold rationale).
  • --update-gates flag: when set, also updates each gated evaluator's threshold to the corresponding baseline score. Only evaluators that already have a threshold field are touched — non-gated metrics are never promoted to gated by this command.

[0.5.0] - 2026-07-02

Pods can now label evaluation runs with a free-text experiment description and designate specific runs as baselines, enabling structured local-tuning comparisons in Langfuse.

Added

  • RunManifest.run_label: str | None — experiment label baked into the Langfuse run name ({manifest.name}_{run_label}_{timestamp}). Set in the YAML run config; overridable per invocation via the pod's --run-label CLI flag.
  • RunManifest.baseline: bool — when True, marks the run as a baseline in Langfuse metadata (run_type = "baseline", is_baseline: True), and defaults run_label to "baseline" when no label is provided.

Changed

  • execute_offline_evaluation(): parameter renamed output_label → run_label. The effective label is resolved as run_label or manifest.run_label, so pods can set a default in the YAML and override it per invocation.
  • build_offline_evaluation_run_identity(): parameter renamed output_label → run_label; new baseline: bool = False parameter.
  • ArtifactWriter.write_run_artifacts(): parameter renamed output_label → run_label.
  • OfflineEvaluationRunIdentity.output_label → .run_label.
  • OfflineEvaluationRunPublication.output_label → .run_label.
  • Langfuse run name separator changed from __ to _. Existing run names in Langfuse are not migrated; the new format applies to all runs created from this version onwards.

[0.4.1] - 2026-07-02

Added

  • datasets.push() accepts input_schema and expected_output_schema parameters (paths to JSON Schema files). When a dataset is created via create_on_missing=True, these schemas are written to the Langfuse dataset object.
  • datasets.push() accepts a scope_ids: bool parameter. When True, item IDs are derived as sha256(dataset_name + item_id)[:32] and the original logical ID is stored in raw_metadata["_item_id"]. This is required for Langfuse 4.x, which enforces project-level uniqueness on item IDs and rejects the same ID across two datasets. pull() restores the logical ID transparently on every fetch.
  • datasets.promote() accepts the same input_schema and expected_output_schema parameters; passes scope_ids=True to push() when promoting to a release dataset.
  • eval-dataset push CLI subcommand gains --create-on-missing, --input-schema-json, and --expected-output-schema-json flags.
  • eval-dataset promote CLI subcommand gains --input-schema-json and --expected-output-schema-json flags.

[0.4.0] - 2026-07-01

Pods can now promote a mutable dev dataset to an immutable release snapshot in a single call, with schema-change detection and version-bump enforcement built in.

Added

  • datasets.promote(dev_name, *, version, ...) — pull dev items, detect schema changes vs. the previous release, enforce version-bump rules, and push to a new {base}_v{M.m} Langfuse dataset (creating it automatically). Provenance metadata (promoted_from, promoted_at, schema_change_level, sdk_version) is written to the Langfuse dataset object.
  • datasets.schema_change_report(new_items, old_items) — compare field-key sets across two item lists and return a SchemaChangeReport with level NONE, MINOR, or MAJOR. MAJOR is triggered by removed fields or new fields present in all items; MINOR by new fields present only in some items.
  • datasets.SchemaChangeLevel — enum with values NONE, MINOR, MAJOR.
  • datasets.SchemaChangeReport — frozen dataclass with level, removed_fields, added_required_fields, added_optional_fields.
  • datasets.VersionBumpError — exception raised when the proposed version is inconsistent with the detected schema changes. Carries change_level, proposed_version, and previous_version attributes.
  • datasets.PromoteResult — frozen dataclass summarising a promote() call: dev_name, release_name, version, items_promoted, schema_change, dry_run, push_result.
  • datasets.archive_item_status(name, item_ids) — set Langfuse item status to ARCHIVED for the given IDs (soft-delete after promotion).
  • eval-dataset promote CLI subcommand — thin wrapper over promote() with --dev-name, --version, --previous-release-name, --scope, --force, --dry-run. Interactive y/N confirmation on schema-rule violations.

Changed

  • datasets.push() accepts a new optional dataset_metadata: dict | None parameter. When provided and a dataset is created via create_on_missing=True, the metadata is written to the Langfuse dataset object.

[0.3.0] - 2026-07-01

Pods can now validate and extend the pre-defined evaluation schema types using Pydantic base classes from semantic_evaluation.schemas, instead of defining their own models from scratch.

Added

  • semantic_evaluation.schemas — new module with Pydantic base classes for every schema type:
  • BaseInput, BaseExpectedOutput — QA schema type
  • RAGInput, RAGExpectedOutput — RAG schema type (extends QA)
  • AgentInput, AgentExpectedOutput — Agent schema type
  • Shared nested types: Citation, RetrievalGroundTruth, Message, ToolCall
  • All classes export a JSON Schema via .model_json_schema().
  • All classes integrate directly with eval-dataset validate --input-schema my_module:MyClass.

[0.2.0] - 2026-06-30

Pods with multiple evaluation scopes — each backed by its own Langfuse dataset — can now orchestrate pull, push, and archive in a single call rather than scripting the loop themselves.

Added

  • datasets.SyncConfig — dataclass capturing a scope→dataset mapping plus local file layout. Supports Pattern A (all scopes merged into one file) and Pattern B (one {scope}_dev.json per scope). Select by setting either merged_path+canonical_name or scoped_dir.
  • datasets.pull_all(config) — pull every scope, write local files, return pulled items. Accepts an optional on_scope_pulled progress callback.
  • datasets.push_all(config, version_label) — read local files, upsert each scope into Langfuse, archive. Accepts an optional on_scope_pushed callback. Archives the merged file once (Pattern A) or each scope file after its push (Pattern B).
  • datasets.derive_version(items) — return the highest dataset_version across a list of items using semantic-version ordering. Useful for surfacing the current Langfuse version after a pull.
  • datasets.push(..., create_on_missing=False) — new keyword argument; when True, a Langfuse 404 on the first push auto-creates the dataset. Defaults to False so typos fail loudly.
  • eval-dataset pull-all CLI subcommand — reads the dataset_sync block from manifest.yaml (--manifest) or a standalone JSON config (--config), then calls pull_all().
  • eval-dataset push-all CLI subcommand — same config sources, adds --version-label and --dry-run.
  • pyyaml>=6.0 runtime dependency, used by the CLI to parse manifest.yaml.

[0.1.0] - 2026-06-23

First release under the semantic_evaluation namespace. The evaluation stack previously shipped as part of the monolithic eval_framework package; it is now a standalone package that depends on semantic-tracing for the tracing layer. Pods implementing a PredictionRunner and running offline evaluations against the shared scoring service use this package.

Added

  • contracts — shared data models: EvaluationItem, PredictionRecord, PredictionRunner Protocol, and RunManifest for manifest-based run configuration.
  • connections — secret resolution helpers including Azure Key Vault support (optional azure extra).
  • datasets — golden dataset sync tooling: canonical hashing, three-way diff, idempotent push, version-filtered pull, validation, and deduplication. Drives the eval-dataset CLI (eval-dataset push/pull/diff/validate).
  • orchestration.config — shared manifest, metric config, and gate contracts.
  • orchestration.datasets — Langfuse dataset loading and normalized item retrieval.
  • orchestration.runner — execute_offline_evaluation(): end-to-end offline evaluation runner (dataset load → prediction → scoring → Langfuse publish).
  • orchestration.staging — candidate-record serialization and seeded dataset publication.
  • orchestration.review_sync — review-sync contracts, client, promotion, and publication helpers.
  • orchestration.reporting — Langfuse dataset-run publication for offline evaluation; propagates EvaluationItem.metadata.tags onto linked traces.
  • prediction — PredictionRunner Protocol and related contracts.
  • scoring — ScoringServiceClient with score() and list_evaluators() methods; full request/response contracts.
  • schemas/run_manifest.schema.json — generated JSON Schema for run manifests (validated by scripts/generate_manifest_schema.py --check).

Changed

  • Top-level import namespace: eval_framework.* → semantic_evaluation.*.
  • semantic-tracing is now a separate runtime dependency; all tracing helpers import from semantic_tracing.* rather than from this package.