Skip to content

Getting started — semantic-evaluation

Install

Packages are hosted on the internal Ramboll Artifactory registry. Authenticate once via ~/.netrc:

machine artifactory.ramboll.com
  login    <your-email-or-empty-string>
  password <your-artifactory-token>

Generate a personal access token in the Artifactory UI under Edit Profile → Authentication Settings.

Declare the Artifactory index in your project's uv.toml (uv reads ~/.netrc automatically). Omitting default = true keeps PyPI as the default for all other dependencies. semantic-evaluation pins pre-releases of Microsoft Agent Framework, which uv installs only with prerelease = "allow"; the setting must come before the first [[index]] table, or TOML reads it as part of that table and the install fails with No solution found:

prerelease = "allow"

[[index]]
name = "ramtech-pypi-internal"
url  = "https://artifactory.ramboll.com/artifactory/api/pypi/ramtech-pypi-internal/simple"

Then add the package:

uv add semantic-evaluation

semantic-tracing is installed automatically as a transitive dependency.

Or install without modifying uv.toml:

# uv
uv add semantic-evaluation --prerelease allow \
  --extra-index-url https://artifactory.ramboll.com/artifactory/api/pypi/ramtech-pypi-internal/simple

# pip
pip install semantic-evaluation \
  --extra-index-url https://artifactory.ramboll.com/artifactory/api/pypi/ramtech-pypi-internal/simple

What an evaluation item looks like

EvaluationItem (semantic_evaluation.contracts) is the unit of evaluation. Each item carries an input dict (what the system receives), an expected_output dict (what it should produce), and metadata (dataset version, trace ID, tags, and other bookkeeping):

from semantic_evaluation.contracts import EvaluationItem, EvaluationItemMetadata

item = EvaluationItem(
    item_id="item-001",
    input={"query": "What is the vacation policy for Ramboll Denmark employees?"},
    expected_output={"response": "Ramboll Denmark employees receive 25 vacation days per year."},
    metadata=EvaluationItemMetadata(
        product_id="pathfinder",
        changed_last_in_dataset_version="v1.0",
    ),
)

Items live in Langfuse. The eval-dataset CLI manages the local ↔ Langfuse sync. For multi-scope projects (the common case), use pull-all / push-all with the project manifest:

eval-dataset pull-all --manifest evaluation/evaluation_manifest.yaml
eval-dataset push-all --manifest evaluation/evaluation_manifest.yaml --version-label 1.0.0
eval-dataset diff --name product_001_agent_e2e_dev --source evaluation/datasets/incoming/e2e_dev.json

See 03-datasets-lifecycle.md for the full naming convention, the dev → release lifecycle, and the complete CLI reference.

Implement a predictor

A predictor wraps your system under test. Implement PredictionRunner.predict_items() (semantic_evaluation.prediction):

from semantic_evaluation.contracts import EvaluationItem, PredictionRecord
from semantic_evaluation.prediction import PredictionRunner

class MyPredictor(PredictionRunner):
    async def predict_items(self, items: list[EvaluationItem]) -> list[PredictionRecord]:
        results = []
        for item in items:
            response = await my_system(item.input["query"])
            results.append(
                PredictionRecord(
                    item_id=item.item_id,
                    prediction={"response": response},
                    trace_id=get_trace_id(),  # links prediction to a Langfuse trace
                )
            )
        return results

Each PredictionRecord carries:

  • item_id — links the prediction back to its source EvaluationItem
  • prediction — the system's output (keys match what the scoring evaluators expect)
  • trace_id — the Langfuse trace for this prediction (used to link scores in the UI)

See 04-offline-evaluation.md for the full PredictionRunner protocol.

Configure the evaluation

Two YAML files under evaluation/ configure it. The project manifest, evaluation/evaluation_manifest.yaml, declares every metric once with the scopes it scores, and the dataset of each scope:

# yaml-language-server: $schema=evaluation_manifest.schema.json
evaluators:
  groundedness:
    framework: azure_ai_evaluation
    evaluator: GroundednessEvaluator
    threshold: 4.0
    scopes: [e2e]
dataset_sync:
  scope_to_dataset:
    e2e: my_product_agent_e2e_v1.0
  scoped_dir: datasets/incoming
  archive_dir: datasets/archive

Each run file under evaluation/runs/ names one run's scopes and inherits the rest. predictor_adapter is your predictor class as module.path:ClassName. The optional run_label sets a default experiment label in the Langfuse run name ({name}_{run_label}_{timestamp}); baseline: true marks the run as a reference point for local-tuning comparisons:

# evaluation/runs/regression.yaml
# yaml-language-server: $schema=run_file.schema.json
name: my-eval-run
predictor_adapter: evaluation.predictor:MyPredictor
run_label: null    # set to e.g. "top_k_8" to label this experiment
baseline: false    # set to true to mark as a reference baseline
scopes: [e2e]

eval-dataset schemas writes the two JSON Schemas the first line of each file names, so VS Code checks both files as you type. 04-run-config.md covers every field, how a run file overrides the catalogue, and the gate.

Run an offline evaluation

Resolve the run file against the project manifest and pass the result to execute_offline_evaluation():

import asyncio
from pathlib import Path
from semantic_evaluation.orchestration.config import load_resolved_run_manifest
from semantic_evaluation.orchestration.runner import execute_offline_evaluation

async def main() -> None:
    manifest, run_path = load_resolved_run_manifest(
        Path("evaluation/runs/regression.yaml"),
        Path("evaluation/evaluation_manifest.yaml"),
    )
    result = await execute_offline_evaluation(
        manifest=manifest,
        manifest_path=run_path,
        scope_override=None,
        run_label=None,
        scoring_service_base_url="http://localhost:8001",
        scoring_service_timeout_seconds=120,
        outputs_dir=Path("evaluation/outputs"),
    )
    print("gate passed:", result.gate_passed)
    print("summary:", result.summary)

asyncio.run(main())

execute_offline_evaluation() handles the full pipeline:

  1. Builds the predictor the run's predictor_adapter names
  2. Loads dataset items from Langfuse
  3. Calls predict_items() on your predictor
  4. Sends predictions to the scoring service
  5. Publishes results back to Langfuse as an experiment (a dataset run)
  6. Returns an OfflineEvaluationExecutionResult with gate pass/fail and summary

eval-dataset resolve-run --run-file evaluation/runs/regression.yaml --manifest evaluation/evaluation_manifest.yaml prints what the run will execute.

Reading these docs from the installed package

The package carries these pages, both shared services' docs and the repository glossary, as they were when the installed version was built. eval-docs reads them without a network or a checkout of this repository, which is how a coding agent in your project finds them:

uv run eval-docs list                    # every page, by name and title
uv run eval-docs show run-config         # one page
uv run eval-docs search ToolCallMatch    # matching lines, as page:line: text
uv run eval-docs path                    # where the files are
uv run eval-docs version                 # installed versions and the docs' commit

It reads semantic-tracing's pages too. In show output, a link to page:<name> is another shipped page and repo:<path> is a file in the SDK's source repository that is not shipped.

Next steps

Topic Where to read
Credentials, keys, VPN, and whom to ask 07-access-and-credentials.md
Dataset naming convention and lifecycle 03-datasets-lifecycle.md
Schema types (QA, RAG, Agent, RAG-Agent) 02-schema-types.md
RunManifest, MetricConfig, gate config 04-run-config.md
PredictionRunner, execute_offline_evaluation(), local tuning 04-offline-evaluation.md
Metric contracts and scoring client 05-scoring.md
Term definitions glossary.md
Package architecture and publishing 00-evaluation-overview.md