Overview

sdk/semantic_evaluation/ is the shared package for evaluation data models, clients, and reusable helpers. It is the only package intentionally imported by all runtime surfaces: reference_pod_implementations/{agent-framework-pod,langchain-pod}/, review_sync_app/, and scoring_app/.

Definition: transport-neutral helper A transport-neutral helper works on normalised Python objects instead of one specific HTTP or CLI entrypoint. In this repository that lets the same SDK helper support pod code and shared services. See transport-neutral helper.

Responsibilities

Area	Owned here
Shared contracts	Datasets, manifests, predictions, metrics, gates, review-sync bindings, and scoring profile references
Shared clients	HTTP clients for scoring_app and review_sync_app
Shared data access	Langfuse dataset loading and reviewed-artifact normalisation
Shared lifecycle helpers	Trace-to-review staging, review-sync promotion and publication helpers, reviewer-comment preservation in promoted dataset metadata, human_review_scores preservation, and the generic offline-evaluation runner
Shared reporting	Offline-evaluation Langfuse dataset-run publication and run-summary helpers

Non-responsibilities

Area	Owned elsewhere
Evaluator execution and aggregation	scoring_app/
Provider secret resolution for evaluator backends	scoring_app/
Review-sync binding persistence and job execution state	review_sync_app/
Pod-specific workflow execution and predictor adapters	reference_pod_implementations/{agent-framework-pod,langchain-pod}/
Pod-specific scope semantics and dataset schemas	pod-local packages
Tracing helpers, decorators, and OTel/Langfuse setup	semantic-tracing package

Package map

Module area	Purpose
semantic_evaluation.datasets	Local ↔ Langfuse dataset sync: canonical hashing, validation, three-way diff, idempotent push, version-filtered pull, schema-change detection, dev → release promotion, multi-scope helpers (SyncConfig, pull_all, push_all), plus the eval-dataset CLI
semantic_evaluation.orchestration.config	The two files a project writes (EvaluationManifestFile for evaluation_manifest.yaml, RunFile for each run file), the resolved run (RunManifest), metric and gate contracts, the resolver (resolve_run_manifest(), load_resolved_run_manifest()) that fills in a run file's datasets and evaluators from the catalogue, and the schema writer behind eval-dataset schemas
semantic_evaluation.orchestration.datasets	Langfuse dataset loading and normalised dataset items
semantic_evaluation.orchestration.staging	LangfuseDatasetDefinition, the shared dataset-publication descriptor used by review-sync's promotion path
semantic_evaluation.orchestration.review_sync	Review-sync contracts, client, normalisation, promotion, publication, and artifact helpers, plus the eval-review CLI, which registers a project's bindings and runs its syncs
semantic_evaluation.orchestration.reporting	Langfuse dataset-run publication for offline evaluation, plus the eval-report CLI, which turns runs' local artifacts into a Markdown report
semantic_evaluation.orchestration.runner	Shared offline-evaluation control flow
semantic_evaluation.platforms	Platform predictors: PlatformPredictor, ConversationHook, and the Open WebUI and Copilot Studio platforms; see Platform predictors
semantic_evaluation.prediction	Shared predictor-side contracts including PredictionRunner Protocol
semantic_evaluation.scoring	Shared scoring client (ScoringServiceClient: job submission, polling, and score-and-wait convenience), request/response and scoring-job contracts, and evaluator catalogue response models
semantic_evaluation.docs	The eval-docs CLI, which reads the documentation shipped in the wheel (_docs/, staged at build time by hatch_build.py): this SDK's docs, both services' docs, the repository glossary, and semantic-tracing's docs
schemas/run_file.schema.json, schemas/evaluation_manifest.schema.json, schemas/run_manifest.schema.json	Generated JSON Schemas of a run file, the project manifest and the resolved run (from RunFile, EvaluationManifestFile and RunManifest; regenerated by scripts/generate_manifest_schema.py). Projects write their own copies with eval-dataset schemas

Runtime architecture

graph TD
    subgraph Pod["reference_pod_implementations (MAF + LangChain)"]
        A["Predictor adapters and CLIs"]
    end
    subgraph Tracing["semantic-tracing"]
        T["TracerProvider setup, step/span/event, trace_reader"]
    end
    subgraph SDK["semantic-evaluation"]
        B["Shared contracts"]
        C["Shared clients"]
        D["Shared orchestration"]
        E["Shared reporting"]
    end
    subgraph Services["Shared services"]
        F["review_sync_app"]
        G["scoring_app"]
    end
    subgraph Langfuse["Langfuse"]
        H["Datasets, queues, traces, dataset runs"]
    end
    A --> B
    A --> C
    A --> D
    A --> T
    D --> G
    D --> E
    E --> H
    C --> F
    C --> G
    F --> B
    F --> D
    T --> H
    G --> B
    style Pod fill:#e9ebec,stroke:#273943,stroke-width:2px
    style Tracing fill:#e5f4fd,stroke:#0098eb,stroke-width:2px
    style SDK fill:#e5f4fd,stroke:#0098eb,stroke-width:2px
    style Services fill:#e7eeec,stroke:#125a40,stroke-width:2px
    style Langfuse fill:#efe9ed,stroke:#62294b,stroke-width:2px

Design rules

• Keep provider-specific connection secrets out of shared request contracts.
• Keep pod-local scope names and payload semantics opaque inside shared flows.
• Keep service runtime state, binding persistence, and evaluator execution outside the SDK.
• Keep reviewer comments as a transport-neutral passthrough from reviewed records into dataset metadata instead of adding binding-specific selector rules for them.
• Prefer adding shared transport-neutral helpers only when at least two runtime surfaces need them.

Publishing

The package is published to the internal Ramboll Artifactory PyPI registry as semantic-evaluation.

• Upload endpoint (publishing): https://artifactory.ramboll.com/artifactory/api/pypi/ramtech-pypi-internal
• Install endpoint (pip / uv resolution): https://artifactory.ramboll.com/artifactory/api/pypi/ramtech-pypi-internal/simple

How to publish a new version

1. Bump the version field in pyproject.toml.
2. Build and publish in one step using the index alias declared in uv.toml:

# From sdk/semantic_evaluation/
uv publish --index ramtech-pypi -u "" -p <your-artifactory-token>

-u "" sets an empty username (Artifactory token auth does not use a username). Generate a personal access token in the Artifactory UI under Edit Profile → Authentication Settings.

The uv.toml at the root of sdk/semantic_evaluation/ declares the index once:

[[index]]
name        = "ramtech-pypi"
publish-url = "https://artifactory.ramboll.com/artifactory/api/pypi/ramtech-pypi-internal"
url         = "https://artifactory.ramboll.com/artifactory/api/pypi/ramtech-pypi/simple"
default     = true

Artifactory has two PyPI repositories, and this file uses both:

Repository	Holds	Used here as
ramtech-pypi	A mirror of public PyPI. It does not hold the SDKs	url, so uv sync resolves the SDK's own runtime dependencies through Artifactory
ramtech-pypi-internal	Ramboll's own packages, both SDKs included	publish-url, where every release is uploaded

A consumer needs ramtech-pypi-internal to install the SDKs; whether its default index is the mirror or public PyPI is its own choice. The reference pods use the mirror.

How consumers install the package

Consumers need the index URL in their uv.toml and credentials in ~/.netrc.

~/.netrc (local development and CI):

machine artifactory.ramboll.com
  login    <your-email-or-empty-string>
  password <your-artifactory-token>

uv.toml (or [tool.uv.index] in pyproject.toml):

prerelease = "allow"

[[index]]
name = "ramtech-pypi-internal"
url  = "https://artifactory.ramboll.com/artifactory/api/pypi/ramtech-pypi-internal/simple"

prerelease = "allow" is needed because semantic-evaluation pins pre-releases of Microsoft Agent Framework, and it must come before the first [[index]] table.

Omitting default = true keeps PyPI as the default for all other dependencies.

uv add semantic-evaluation

The importable top-level name is semantic_evaluation.

Version policy

semantic-evaluation follows semantic versioning. Backward compatibility is guaranteed within a major version.

Release	Contract
PATCH (x.y.z → x.y.z+1)	Bug fix, no API change — drop-in replacement
MINOR (x.y → x.y+1)	New capability added — existing callers work without changes
MAJOR (x → x+1)	Breaking change — documented in CHANGELOG; callers must update

The public API for this guarantee is every symbol exported from the top-level semantic_evaluation package and every interface documented in this docs/ folder. Internal modules and undocumented helpers carry no compatibility guarantee.

Declare a version constraint in pyproject.toml to receive patch and minor updates automatically while being protected from breaking major-version changes:

[project]
dependencies = [
    "semantic-evaluation>=0.2.0,<1",
]

Reading order

File	What it covers
00-evaluation-overview.md ← you are here	Package architecture, module map, design rules, publishing
01-getting-started.md	Install, first dataset item, first predictor, first eval run
02-schema-types.md	Schema types (QA, RAG, Agent, RAG-Agent), hierarchy diagram, Pydantic models, JSON schemas, UC validation
02-schema-versioning.md	input_schema_version, bump rules, schema type vs content versioning
03-datasets-lifecycle.md	Naming convention, dataset states, curation workflow, promotion, pod file layout, full cycle walkthrough, UC handover
03-scope-sync.md	Multi-scope sync: SyncConfig, pull_all/push_all, bootstrapping, recovery
03-datasets-cli.md	eval-dataset CLI reference, idempotency, version semantics (changed_last_in_dataset_version vs dev_version), programmatic access
04-run-config.md	RunManifest, EvaluationManifest and resolve_run_manifest(), MetricConfig, GateConfig, evaluator catalogue and override rules
04-offline-evaluation.md	PredictionRunner, execute_offline_evaluation(), pipeline, run artifacts, comparing configurations, eval-report, local tuning workflow
09-platform-predictors.md	semantic_evaluation.platforms: predictors for agents on Open WebUI and Copilot Studio, ConversationHook, errored conversations
05-scoring.md	MetricConfig, MetricResult, ScoringServiceClient, normalisation scale
06-review-sync.md	Setting up queues and score configs in Langfuse, QueueToDatasetBinding, ReviewPolicy, Projection, ValueSelector, ReviewSyncServiceClient, the eval-review command, job execution, checkpointing, when a sync fails
06-review-sync-pod-helpers.md	Pod-side promotion helpers: LangfuseReviewArtifactReader, build_promoted_records(), publish_promoted_records()
07-access-and-credentials.md	Every credential and kind of access a project needs, where each comes from, and whom to ask
glossary.md	Definitions for all package-local terms