Platform predictors

semantic_evaluation.platforms holds a ready-made predictor for agents that run on a chat platform and emit no Langfuse trace the evaluation can read: Open WebUI and Copilot Studio. The predictor holds real conversations with a candidate copy of the agent, reads the platform's own transcript of each one back, and turns it into PredictionRecords with the measurements the platform can give. A project writes only the part that knows its agent: what to ask, how to reply, and what the answer predicts.

See 04-offline-evaluation.md for the records and traces such a run produces, and 04-run-config.md for how a run file names the predictor.

The three parts

Part	Who writes it	What it does
Platform	The SDK: OpenWebUIPlatform, CopilotStudioPlatform	Describes the candidate, runs every conversation of a round, reads each transcript back
ConversationHook	The project	Turns items into conversations and a conversation's outcome into predictions
PlatformPredictor	The SDK	Joins the two for the runner and implements PredictionRunner

A ConversationHook has four methods:

Method	Returns
conversations(items)	The Conversations of the run. One conversation may answer several items: it carries them in items, with the first message and an optional document
next_message(conversation, reply, turn)	The next user message, given the agent's last reply and the turn number, or None to end the conversation
predictions(conversation, outcome)	One prediction dict per item of conversation.items, in the same order, read from outcome.transcript; outcome.tool_calls holds its parsed tool calls
describe(run_config)	Run-config values the project adds to the candidate's description; may return {}

predictions is called only for an outcome without an error; an errored conversation gets an empty prediction for each of its items, see Errored conversations. The predictor fills the rest of each record: the item's identifiers, source_transcript (a reference to the platform's transcript), measurements and error. It leaves trace_id empty, so the run file sets prediction_traces: create_when_missing.

A conversation has a key that groups items into it, and a title that names it in logs, chat titles and the files a failed run leaves behind.

Configuring a run

A platform is configured by one frozen dataclass, defined in the project and named from the run file. Open WebUI:

Field	Default	Meaning
platform	required	The platform value written to the run config, for example support_assistant
candidate_prefix	required	Only assistants whose id starts with this are run
default_candidate	required	The assistant used when the run config names no candidate
live_runs	required	Folder for transcripts and candidate snapshots; files go under live_runs/open_webui/
instance	"STAGING"	The only OPENWEBUI_INSTANCE a round may run on
max_turns	5	Most turns one conversation takes
measure_from	"spend_log"	Where a conversation's tokens, cost and tool calls come from: "spend_log" or "traces", see Measurements and cost
parallel	4	Conversations run at once; used only with measure_from="traces"
pause	30	Spend log only: seconds between conversations
settle	60	Spend log only: seconds after the last conversation before the spend log is read
spend_rates	None	Spend log only: USD per million tokens by model name, as SpendRates; None prices at LiteLLM's own spend

Copilot Studio:

Field	Default	Meaning
platform	required	The platform value written to the run config
org	required	The Dataverse organisation URL
publisher	required	The schema-name prefix of the publisher
candidate_prefix	required	The agent name prefix a candidate must carry
default_candidate	required	The agent name used when the run config names no candidate
workflow	required	The HTTP workflow that calls the candidate
live_runs	required	Folder for transcripts, snapshots and failed runs; files go under live_runs/copilot_studio/
max_turns	5	Most turns one conversation takes
pause	30	Seconds between conversations
poll_from	29 minutes	When transcripts are first read, counted from the latest conversation's last turn
poll_until	75 minutes	When the search for transcripts stops, counted the same way
poll_every	60	Seconds between transcript reads
first_turn_attempts	2	Attempts at a first turn that fails on the connection

measure_from and parallel are checked before the round: an unknown source or a parallel that is not a positive integer raises ValueError.

A minimal project module defines the configuration and the hook:

# evaluation/platform.py
from pathlib import Path

from semantic_evaluation.platforms import OpenWebUIConfig

SUPPORT_STAGING = OpenWebUIConfig(
    platform="support_assistant",
    candidate_prefix="eval-",
    default_candidate="eval-support",
    live_runs=Path("evaluation/live_runs"),
)

# evaluation/hook.py
from semantic_evaluation.contracts import EvaluationItem
from semantic_evaluation.platforms import Conversation, ConversationOutcome


class SupportConversations:
    def conversations(self, items: list[EvaluationItem]) -> list[Conversation]:
        return [
            Conversation(
                key=item.item_id,
                title=item.item_id,
                items=[item],
                message=item.input["question"],
                document=None,
            )
            for item in items
        ]

    def next_message(self, conversation, reply, turn):
        return None  # one question, one answer

    def predictions(
        self, conversation: Conversation, outcome: ConversationOutcome
    ) -> list[dict]:
        history = outcome.transcript["chat"]["chat"]["history"]
        # read the final answer from the platform's chat JSON
        return [{"response": final_answer(history)}]

    def describe(self, run_config):
        return {}

The run file names both, by module:attr:

predictor_adapter: semantic_evaluation.platforms:PlatformPredictor
prediction_traces: create_when_missing
run_config:
  platform_config: evaluation.platform:SUPPORT_STAGING
  conversation_hook: evaluation.hook:SupportConversations
  candidate: eval-support
  max_conversations: 40

To measure from traces, set measure_from="traces" in the configuration, and parallel to the number of conversations to run at once.

platform_config names an OpenWebUIConfig or a CopilotStudioConfig instance; conversation_hook names a class the predictor builds with no arguments. The runner builds the predictor with the run's run_config and repeats, so the same file runs from the pod CLI unchanged.

A project whose hook needs arguments builds the predictor itself. Passed to the runner, the predictor is not asked to describe the candidate, so the project calls describe_candidate() first and passes the merged mapping as run_config:

from semantic_evaluation.platforms import PlatformPredictor

predictor = PlatformPredictor(
    run_config, repeats, platform=OpenWebUIPlatform(SUPPORT_STAGING), hook=hook
)
run_config = predictor.describe_candidate()
result = await execute_offline_evaluation(
    manifest=manifest, ..., run_config=run_config, repeats=repeats, predictor=predictor
)

A project's own predictor that wraps PlatformPredictor and is named in predictor_adapter is built by the runner, which describes it through its describe_candidate(). The wrapper must define that method and return the wrapped predictor's describe_candidate(). Without it the runner has nothing to call, the candidate goes undescribed, and the platform's guards never run. A wrapper the caller builds and passes in is described by the caller, as above.

Size cap

A run holds conversations * repeats conversations, and each one costs a platform run. run_config.max_conversations is required, as a positive integer. A run that needs more is refused before any conversation starts; --config max_conversations=N raises the cap.

Run-config keys

Each platform reads these keys from the run config:

Key	Platform	Meaning
max_conversations	both	The size cap above, read by the predictor
candidate	both	The candidate to run; the configuration's default_candidate when absent
upload_mode	Open WebUI	default (the platform injects retrieved chunks of an uploaded document) or whole (the "Using Entire Document" toggle); default default
workflow	Copilot Studio	The HTTP workflow, replacing the configuration's workflow for this run

Before the round, the predictor's describe_candidate() merges what the platform and then the hook record about the candidate into the run config. That description labels the run's artifacts, dataset runs and traces, as in Keys that describe the candidate:

Key	Open WebUI	Copilot Studio
platform	the configuration's platform	the configuration's platform
candidate	the assistant id	the agent's schema name
model	the assistant's base model, left out when the export names none	the model series the agent's configuration names, left out when it names none
delivery	attachment	text
content_sha256	hash of the saved candidate snapshot	hash of the saved candidate snapshot
upload_mode	default or whole	not written
measured_from	spend_log or traces, from measure_from	not written
file_context	whether uploaded documents are injected into the context	not written
bot_id	not written	the agent's Dataverse bot id
workflow	not written	the workflow that calls the candidate

The three reserved keys, instructions_version, knowledge_version and release_id, are never written here. The snapshot behind content_sha256 is saved as live_runs/<store>/candidate-<first 12 characters of the hash>.json, so two runs on the same content carry the same hash and the content can be read afterwards.

Environment and access

	Open WebUI	Copilot Studio
Variables	OPENWEBUI_INSTANCE (default DEV), OPENWEBUI_URL_<INSTANCE>, OPENWEBUI_API_KEY_<INSTANCE>; for the spend log also LITELLM_URL_<INSTANCE>, LITELLM_API_KEY_<INSTANCE>; for traces OPENWEBUI_LANGFUSE_BASE_URL_<INSTANCE>, OPENWEBUI_LANGFUSE_PUBLIC_KEY_<INSTANCE>, OPENWEBUI_LANGFUSE_SECRET_KEY_<INSTANCE>	COPILOT_WORKFLOW_URL, the HTTP URL of the workflow that calls the candidate
Login	The API key's account owns the candidate	az login: the Flow and Dataverse tokens come from az account get-access-token, with no app registration
Guard on the instance	A round runs only when OPENWEBUI_INSTANCE equals the configuration's instance	Not applicable
Guard on the candidate	The assistant id must start with candidate_prefix, and the assistant must exist	The agent's schema name must start with the prefix, the agent must exist, and the workflow's agent node must call it

With measure_from="traces" no LiteLLM variable is needed. The three OPENWEBUI_LANGFUSE_* variables name the Langfuse project the instance writes its traces to; either key may be a Key Vault secret URI. With none of them set, the traces are read from the evaluation's own Langfuse project (the LANGFUSE_* variables). describe_candidate() checks these credentials before the round, and raises ValueError when only some of the three are set.

Both guards raise ValueError from describe_candidate(), before any conversation, so a round never runs on the live agent or on the wrong instance. Create the candidate as a copy of the live assistant or agent under the prefix first. On Open WebUI the copy includes its skills and belongs to the API key's account, because the server offers a conversation only the skills its caller can read.

How a round runs

The predictor runs one round for the whole run: the first predict_items call runs every conversation of every repeat, and each repeat is served from the result.

Open WebUI's round depends on measure_from. Background tasks (title, tags, follow-ups) are off in both modes, so the measured work is the conversation's own model calls. Each chat is read back and saved with the driver's turn log (each turn records its message_id) as live_runs/open_webui/<chat id>.json; then the chat and its upload are deleted.

	measure_from="spend_log"	measure_from="traces"
Conversations	One after another, repeat by repeat, with pause seconds between them	parallel at a time, with no pause
Why	The spend log names no chat, so a conversation's calls are matched to it by time, and overlapping conversations could not be told apart	Every trace names its chat, so conversations do not need to be kept apart
After the round	Waits settle seconds, reads the spend log for the round's time window, matches rows to each conversation's turns	Reads each chat's traces and waits until they stop changing, because Langfuse ingests spans and computes cost a while after a turn ends

With measure_from="traces" and parallel above 1, the hook's next_message is called from several threads at once, so it must be thread-safe: keep no state shared between conversations that a call changes, or guard it with a lock. conversations, predictions and describe are called from the main thread.

Copilot Studio also runs the conversations one after another with pause between them, but writes a conversation's Dataverse conversationtranscripts row only when its session times out, long after the last turn. The round drives every conversation first. From poll_from after the latest conversation's last turn it reads the candidate's transcripts every poll_every seconds until each is found or poll_until has passed. A conversation's transcript is the row whose name is the workflow's conversation id, an underscore and the agent's bot id. A first turn that fails on the connection (no HTTP status, or a 5xx) opened no conversation, so it is made again, after pause, up to first_turn_attempts attempts in all. Copilot Studio keeps transcripts for 30 days by default, so read them within that window. Each transcript is saved with the driver's turns as live_runs/copilot_studio/<transcript id>.json.

Measurements and cost

Open WebUI measures a conversation from one of two sources, chosen by measure_from:

Measurement	From traces	From the spend log
wall_time_s	the sum of the conversation's turn times, timed by the runner	the same
Tokens	summed over the chat's trace observations, cached input included	summed over the spend rows matched to the conversation's turns by time
Cached input	counted from the usage details, under any of the names the evaluation recognises	counted from the rows' cache details; some rows carry none, and the notes say so
cost_usd	Langfuse's computed cost; the notes name a model Langfuse has no price for	spend_rates when set, else the rows' spend, priced at LiteLLM's own rates
tool_calls, tool_calls_by_name	observations of type TOOL and RETRIEVER	from the chat's assistant messages
Notes	a turn with no trace, and a model with no price	no matching spend row, a model with no rate, and a spend log that cannot be read

Copilot Studio measures only wall time (the sum of the conversation's turn times) and tool calls, from the transcript's ToolCallTrace activities.

From traces, a conversation's turns are matched to traces by message_id, the id of the assistant message a turn produced, read from the trace metadata. The deployment must write one Langfuse trace per user message, with the chat id as the session id and message_id in the trace metadata. A turn with no trace leaves its share of the measurements out, and the notes name it. An instance that traces only a share of its chats leaves the other conversations unmeasured, so use the spend log there; the spend log works on any instance.

With spend_rates set, each row is priced at the rates of the first entry whose key is a substring of the row's model name; a row with no match is left unpriced, and the notes name its model. With spend_rates=None, the cost is the sum of the rows' spend column, priced at LiteLLM's own rates, and the notes say so. When no spend row matches a conversation, the measurements carry no tokens and the notes say that. A spend log that cannot be read leaves the conversations' transcripts intact and says so in the notes.

Copilot Studio reports no tokens and no credits per conversation to a program: credits are read by hand from the agent's Activity page, and every outcome's notes say so.

Errored conversations

A conversation is errored when the platform left no transcript to read:

• no conversation was opened, for example the first turn failed or the workflow returned no conversation id;
• no transcript appeared within poll_until, or an Open WebUI run raised;
• the transcript cannot be read.

An answer that was read but is wrong, empty or invalid is not an error; it is a prediction, and the evaluators score it. An errored conversation does not stop the round. Its outcome carries the reason as error and no transcript, its notes repeat the reason, and the failure is logged as a warning. Each of its items becomes a PredictionRecord with an empty prediction and that error. Errored predictions are not scored: they are left out of the scoring request, the trace metrics, the group aggregates and the agreement. The run file's max_error_fraction (default 0) is the largest share of a scope's predictions that may be errored before the gate fails, and the run report prints errored: n of N. See Errored predictions for what the runner and the publication do with them.

Delivering a document

A conversation's document reaches the agent in the way its platform takes files.

	Delivery
Open WebUI	Uploaded as an attachment and sent with the first message; optional. Deleted after the run
Copilot Studio	Sent as text after the first message: a blank line, then the text of the .docx or the PDF. Any other type, and a PDF without a text layer (a scan), is refused before the round

A .docx is read with the standard library: a line per paragraph, a line per table row with the cells joined by |; headers and footers are not read. Reading a PDF needs the pdf extra:

uv add "semantic-evaluation[pdf]"

Reading transcripts, and what is public

The readers below take a platform's own transcript, live from ConversationOutcome.transcript or stored (a chat export, a Dataverse row saved earlier). All are exported from semantic_evaluation.platforms.

Name	Returns
ToolCall	One tool call: name, target, status (completed, failed or no result), duration_ms, error, result_text
open_webui_active_path(history)	The messages from the first user message to the active leaf; history is transcript["chat"]["history"] of a chat as Open WebUI returns or exports it
open_webui_tool_calls(message)	The tool calls of one assistant message
open_webui_whole_file_sources(message)	{file name: characters} of the whole files the platform put in the model's context, not retrieved chunks
copilot_tool_calls(activities)	The tool calls of a Copilot Studio transcript; activities is the list in the content JSON of a Dataverse transcript row
delivered_text(path)	The text a .docx or PDF is sent as

ConversationOutcome.tool_calls holds the tool calls the platform read from the transcript, the same ones measurements.tool_calls counts. It is empty for an errored outcome. active_path and whole_file_sources in semantic_evaluation.platforms.transcripts are the older names of the first and third readers; they stay importable until 4.0.0 but are not public.

The modules below are internal. They stay importable at their current path and name until 4.0.0, and their behaviour may change in any minor release. Use the replacement:

Module	Replacement
platforms.open_webui.client	None in the SDK: a script that calls Open WebUI routes the platform does not use keeps its own REST client
platforms.open_webui.chat	OpenWebUIPlatform through PlatformPredictor: it runs the conversation, uploads the document and reads the chat back
platforms.open_webui.litellm_spend, except SpendRates, which is public	ConversationOutcome.measurements: tokens and cost_usd per conversation from OpenWebUIPlatform with measure_from="spend_log"
platforms.open_webui.chat_traces	ConversationOutcome.measurements: tokens, cost_usd and tool calls per conversation from OpenWebUIPlatform with measure_from="traces"
platforms.copilot_studio.workflow, platforms.copilot_studio.dataverse	CopilotStudioPlatform through PlatformPredictor
platforms.documents.docx_text, pdf_text	delivered_text, which returns the same text
Other helpers in platforms.transcripts (parameters, copilot_target, save_transcript)	The readers above