Evaluator catalogue

The evaluator catalogue is a frozen tuple of CatalogueEntry objects in scoring_service/scoring/catalogue.py. It fixes the generic evaluator types this service supports — every (framework, evaluator) pair, its dataset scope, its native score scale, and the parameter names it accepts under evaluator_params — not any pod-specific metric. A pod's actual metric definitions (metric name, threshold, evaluator_params values, ...) never live in the catalogue; they travel inline inside every ScoreRequest, built pod-side from whatever run manifest that pod maintains. See 00-scoring-overview.md — Where evaluator definitions live for the full picture.

A run manifest names a metric by its (framework, evaluator) pair; the catalogue is what turns that pair into a running evaluator instance for one logical metric. Naming a pair the catalogue does not know is a hard failure that lists every supported pair.

The same catalogue backs the discovery endpoint, GET /v1/evaluators — see 04-api-reference.md. Manifest authors and the SDK's list_evaluators() helper use it to find out what is available without reading this service's source.

Each entry declares:

Field	Purpose
framework	Which evaluator framework the evaluator belongs to
evaluator	The concrete evaluator class or function name within that framework
scope	Which dataset scope the evaluator targets (routing or end_to_end)
raw_scale	The framework-native scale (binary, 0-1, likert_1_5, or categorical) — see Score normalization
supported_params	Parameter names accepted under MetricConfig.evaluator_params for this evaluator; anything else is rejected before construction
factory	The callable that builds one evaluator instance for one logical metric

Scoring pipeline

A submitted scoring job flows through four stages before its result is available via GET /v1/score-jobs/{job_id}:

graph TD
  A["ScoreRequest items, predictions, evaluators, gate config"]
  B["build_evaluators per metric per scope"]
  C["connection profile resolved for LLM-judge frameworks"]
  D["evaluator instances one per logical metric"]
  E["run each evaluator against every item concurrently"]
  F["MetricResult raw score and normalized score per item"]
  G["aggregate_metric_results per scope and metric pair"]
  H["AggregateMetricResult mean, pass rate, passed flag"]
  I["build_gate_result minimum item count and threshold checks"]
  J["GateResult passed or failed with failure list"]

  A --> B
  B --> C
  C --> D
  D --> E
  E --> F
  F --> G
  G --> H
  H --> I
  I --> J

  style A fill:#e5f4fd,stroke:#0098eb,stroke-width:2px
  style J fill:#e7eeec,stroke:#125a40,stroke-width:2px

Stages B–C run synchronously inside POST /v1/score-jobs, before any job is created. If any evaluator cannot be built — for example, because a backend has no connection profile assigned — the submission fails with HTTP 400 and no job exists to poll. Stages D–J run as the job's background task.

Framework families

Framework	Backend key	What it is
builtin	none	Deterministic exact-match for the routing scope. No LLM call.
builtin_deterministic	none	Generic deterministic checks for the end_to_end scope — tool-call shape, order and output, prediction fields against a bound or an expected value, URL allow-lists, message-content patterns, router-mode comparison. No LLM call.
deepeval	deepeval_llm_judge	Reflective adapter that instantiates any allow-listed DeepEval metric class by name
azure_ai_evaluation	azure_ai_evaluation_llm_judge	Reflective adapter that instantiates Azure AI Evaluation SDK evaluator classes by name
custom_rubric	custom_rubric_llm_judge	Pod-defined LLM-judge, in two forms: inline (rubric embedded in the manifest) and {name} (rubric committed to custom_evaluators/accepted/ and reviewed by the eval core team) — see 03-custom-rubric-and-registration.md
decision_judge	decision_llm_judge	One fixed-option question about free text, answered by Microsoft Decision-1 without generative reasoning — see 03-decision-evaluator.md

Adding a metric that belongs to an already-supported framework needs one catalogue entry — the deepeval and azure_ai_evaluation wrappers instantiate metric classes reflectively by name, so no new adapter code is required.

deepeval, azure_ai_evaluation, custom_rubric, and decision_judge entries all resolve a connection profile for their backend key before building the evaluator — including statistical azure_ai_evaluation evaluators such as F1ScoreEvaluator that do not themselves use an LLM. The catalogue's factory resolves the backend unconditionally for every entry in that framework, so azure_ai_evaluation_llm_judge must have an assigned profile before any azure_ai_evaluation metric — statistical or LLM-judge — can run. See 02-connection-profiles.md — Resolution behavior.

Choosing a deterministic check or a judge

An offline evaluation measures the agent, not the code around it. That decides which kind of metric a behaviour gets:

What is being checked	Metric
A decision the agent made about a tool: called or not called, with which arguments, in which order, what it returned, which URLs and policies it used	builtin_deterministic. It is exact, costs no model call, and gives the same score every time
What the agent said, and the answer is one fixed option such as yes/no or a named outcome	decision_judge/inline, with a Decision profile. It agrees with human labels as often as a generative judge, gives the same verdict every run, and is about five times faster and a hundred times cheaper; it returns no reasoning text (comparison)
What the agent said, and assessing it needs a graded scale, open-ended reasoning or a written reason	A generative LLM judge (custom_rubric, deepeval, azure_ai_evaluation), with a connection profile
Something only deterministic code decides: parsing, arithmetic, formatting, a tool's own implementation	Neither. It is a unit test in the project's test suite, which runs in milliseconds and fails on the line that broke

A deterministic evaluator scores 1 or 5, so its threshold is 5.0 and the gate passes only when every item passes. A judge's score varies a little from one run to the next on the same answer, so its threshold leaves margin below the score a good answer gets (4.0 on a rubric whose good answers score 5), and a change is read from the metric's aggregate over many items, not from one item's score.

The full catalogue

builtinbuiltin_deterministicdeepevalazure_ai_evaluationcustom_rubricdecision_judge

Framework	Evaluator	Scope	Raw scale	Notes
builtin	ExactMatch	routing	binary	Deterministic string match between prediction["handoff_to"] and expected_output["handoff_to"]

No connection profile required. Works with an empty profile registry.

Framework	Evaluator	Scope	Raw scale	Notes
builtin_deterministic	ToolCallMatch	end_to_end	binary	tool_name, arg_name, expected_field, match_mode_field, empty_means_not_called
builtin_deterministic	ToolCallAbsent	end_to_end	binary	tool_name, arg_filters
builtin_deterministic	ToolCallOrdering	end_to_end	binary	before, after, match_args
builtin_deterministic	ToolCallOutputMatch	end_to_end	binary	tool_name, expected_field, fields, success_field, non_empty_fields, prefix_fields
builtin_deterministic	ToolCallPayloadMatch	end_to_end	binary	tool_names, expected_field, scalar_args, dict_args
builtin_deterministic	FieldWithinThreshold	end_to_end	binary	field, and either max/min or expected_field with tolerance
builtin_deterministic	FieldEquals	end_to_end	binary	field, and one of value, one_of, expected_field, expected_one_of_field
builtin_deterministic	StructuredFieldMatch	end_to_end	binary	field, expected_field
builtin_deterministic	URLAllowList	end_to_end	binary	mode, extra_allowed_field, static_allowed, include_citation_sources
builtin_deterministic	PatternPresent	end_to_end	binary	source, kind, must_include_field, must_not_include_field, static_patterns, static_not_patterns, case_sensitive, skip_when_field_equals
builtin_deterministic	RouterModeMatch	end_to_end	binary	check_mode, check_playbook_id, check_step_id, check_divergence
builtin_deterministic	PolicyIdAllowList	end_to_end	binary	allowed_policy_ids, tool_name, arg_name

No connection profile required. Works with an empty profile registry.

Evaluator detail — what each check does

No evaluator here names a tool: the manifest's evaluator_params name the tools and the prediction fields. The tool-call evaluators read prediction["tool_calls"], a list of {"name", "args", "output"} entries. Where an evaluator does not apply to an item, it skips it and leaves no result, as noted per evaluator.

Evaluator	What it checks
ToolCallMatch	The set of values a named tool received for a named argument matches an expected set from the item, in either "exact" or "superset" mode. When empty_means_not_called is true and the expected set is empty, the check inverts to "the tool was not called at all."
ToolCallAbsent	A named tool was not invoked with a specific argument shape (arg_filters).
ToolCallOrdering	Every call to the after tool is preceded by a call to the before tool whose match_args arguments have the same values. Without match_args it checks order alone. Items that never call after are skipped.
ToolCallOutputMatch	The last call to tool_name returned what the item's expected_output[expected_field] object expects; a string output is parsed as JSON. fields are compared for equality (default: every key of the expected object); non_empty_fields must be present and non-empty; prefix_fields ({output_field: expected_key}) must start with the prefix the expected object holds. When success_field is set and the expected object sets it false, only fields are compared. Items without the expected object are skipped; a tool never called fails.
ToolCallPayloadMatch	The arguments of the last call to each tool in tool_names match the item's expected_output[expected_field] object, one result per tool called, named {metric}_{tool}. scalar_args must be equal; dict_args must contain the expected keys with the expected values. Without either list, keys holding an object are dict arguments and the rest are scalars. Items without the expected object are skipped; if no listed tool was called, the item fails.
FieldWithinThreshold	A number at the dotted path field of the prediction (drawing.max_invert_error_m, pipes.0.slope) is within the inclusive max/min bounds, or within tolerance of the number at expected_field in the item's expected output. Items without that expected value are skipped. A missing or non-numeric field fails, with the reason. Every result's reason carries the measured value and the bound.
FieldEquals	The value at the dotted path field equals value, is in one_of, equals the value at expected_field in the item's expected output, or is in the list at expected_one_of_field in the item's expected output. Meant for booleans and enumerations; true does not equal 1. Quote string values in a YAML file ("YES"), because unquoted yes, no, on and off load as strings and true, false as booleans; a one_of that mixes booleans and strings is refused. Items without that expected value are skipped; a value at expected_one_of_field that is not a non-empty list fails, with the reason; a missing field fails, with the reason.
StructuredFieldMatch	A top-level prediction field equals a top-level expected-output field (a missing one compares as null).
URLAllowList	Two modes: every URL in the reply must come from a configured allow-list (all_urls_allowed), or at least one expected URL must appear (expected_present).
PatternPresent	Required substrings/regex patterns appear and forbidden ones do not, in either message or closing_line. Supports per-item skip via skip_when_field_equals.
RouterModeMatch	The router-classified mode/playbook/step/divergence matches the item's expectations, with independent per-field toggles.
PolicyIdAllowList	Every value passed as arg_name to tool_name exists in a snapshot allow-list (allowed_policy_ids) of legitimate documents. Regression canary — should never fire if prompt rules hold.

The table above plus each evaluator's supported_params (listed in the entry table) is the complete reference for configuring any of these evaluators.

Pathfinder's tool names are filled in when a manifest names no tool

The manifests of Pathfinder's proof of concept, the HR assistant these evaluators were written for, predate the parameters that name its tools, so the catalogue fills them in for a metric that names none, and any parameter the manifest does set wins. Setting the marker parameter (after, tool_name or tool_names) turns the preset off for that metric entirely; for PolicyIdAllowList each of its two parameters is filled on its own. The preset is _PATHFINDER_PRESETS in scoring/catalogue.py, and it goes once those manifests carry these parameters or the proof of concept is retired, which reproduce the old behaviour exactly:

Evaluator	Parameters filled in
ToolCallOrdering	before: preview_ticket, after: create_ticket, match_args: [service_key, opened_for_email, subject_email, fields]
ToolCallOutputMatch	tool_name: create_ticket, expected_field: expected_ticket_outcome, fields: [ok], success_field: ok, non_empty_fields: [ticket_number], prefix_fields: {ticket_url: expected_ticket_url_prefix}
ToolCallPayloadMatch	tool_names: [preview_ticket, create_ticket], expected_field: expected_ticket_call, scalar_args: [service_key, opened_for_email, subject_email, opened_for_name, subject_name], dict_args: [fields]
PolicyIdAllowList	tool_name: read_policy, arg_name: policy_id

Example manifest excerpt: a figure measured on the drawing after the run, against a fixed bound and against the item's value; the order of two tools; and whether a write happened:

{
  "max_invert_error": {
    "framework": "builtin_deterministic",
    "evaluator": "FieldWithinThreshold",
    "threshold": 5.0,
    "evaluator_params": {"field": "drawing.max_invert_error_m", "max": 0.005}
  },
  "outlet_invert": {
    "framework": "builtin_deterministic",
    "evaluator": "FieldWithinThreshold",
    "threshold": 5.0,
    "evaluator_params": {
      "field": "drawing.outlet_invert_m",
      "expected_field": "outlet_invert_m",
      "tolerance": 0.01
    }
  },
  "plan_before_apply": {
    "framework": "builtin_deterministic",
    "evaluator": "ToolCallOrdering",
    "threshold": 5.0,
    "evaluator_params": {
      "before": "plan_minimum_slope",
      "after": "apply_elevation_plan",
      "match_args": ["network"]
    }
  },
  "wrote_drawing": {
    "framework": "builtin_deterministic",
    "evaluator": "FieldEquals",
    "threshold": 5.0,
    "evaluator_params": {"field": "outcome.wrote", "expected_field": "should_write"}
  }
}

Framework	Evaluator	Scope	Raw scale	Notes
deepeval	GEval	end_to_end	0-1	Generic evaluator; accepts name, criteria, evaluation_steps, evaluation_params, rubric, context_key, skip_when_field_equals
deepeval	AnswerRelevancyMetric	end_to_end	0-1	Common DeepEval params only
deepeval	FaithfulnessMetric	end_to_end	0-1	Common DeepEval params only
deepeval	ContextualRecallMetric	end_to_end	0-1	Common DeepEval params only
deepeval	HallucinationMetric	end_to_end	0-1	Common DeepEval params only
deepeval	BiasMetric	end_to_end	0-1	Common DeepEval params only

Requires backend key deepeval_llm_judge to be assigned before any deepeval metric can run. See 02-connection-profiles.md.

All DeepEval evaluators additionally accept the common parameters threshold, include_reason, strict_mode, async_mode, verbose_mode, input_key, actual_output_key, and expected_output_key (folded into "Common DeepEval params only" above). skip_when_field_equals is implemented generically in the DeepEval wrapper, but only GEval's catalogue entry currently lists it as a supported parameter — passing it to any other DeepEval metric is rejected as an unknown parameter.

Framework	Evaluator	Scope	Raw scale	Notes
azure_ai_evaluation	GroundednessEvaluator	end_to_end	likert_1_5	LLM-judge via Azure AI Evaluation SDK
azure_ai_evaluation	RelevanceEvaluator	end_to_end	likert_1_5	LLM-judge
azure_ai_evaluation	CoherenceEvaluator	end_to_end	likert_1_5	LLM-judge
azure_ai_evaluation	F1ScoreEvaluator	end_to_end	0-1	Statistical — no LLM call, but still resolves a connection profile, see Framework families above

Requires backend key azure_ai_evaluation_llm_judge to be assigned before any azure_ai_evaluation metric can run — including the statistical F1ScoreEvaluator. See 02-connection-profiles.md.

Framework	Evaluator	Scope	Raw scale	Notes
custom_rubric	inline	end_to_end	likert_1_5	name, criteria, scale, input_fields, output_fields, expected_output_fields, model — see 03-custom-rubric-and-registration.md
custom_rubric	{name}	end_to_end	likert_1_5	One entry per file in custom_evaluators/accepted/, loaded at startup; only model may be overridden

Requires backend key custom_rubric_llm_judge to be assigned. See 03-custom-rubric-and-registration.md for the pod-driven registration lifecycle.

Framework	Evaluator	Scope	Raw scale	Notes
decision_judge	inline	end_to_end	categorical	One noul, choice, or score question. type, instructions, criteria, scores, abstain_option, input_fields, output_fields, expected_output_fields, min_confidence, and model are accepted.

Requires backend key decision_llm_judge to be assigned to a foundry_decision profile. See 03-decision-evaluator.md for the complete question contract.

Evaluator parameters

Definition: evaluator parameters Evaluator parameters (evaluator_params) are the per-metric keyword arguments a manifest author declares for one metric. They flow through the matching evaluator wrapper and, for deepeval and azure_ai_evaluation metrics, into the underlying framework's constructor or call signature. Only names listed in the catalogue entry's supported_params are accepted. See evaluator parameters.

Example manifest excerpt (GEval with custom evaluation steps):

{
  "correctness": {
    "framework": "deepeval",
    "evaluator": "GEval",
    "enabled": true,
    "threshold": 3.8,
    "evaluator_params": {
      "name": "Correctness",
      "evaluation_steps": [
        "Compare the factual claims in the response against the expected output.",
        "Check that no critical facts are missing.",
        "Check that no facts are invented."
      ],
      "evaluation_params": ["ACTUAL_OUTPUT", "EXPECTED_OUTPUT"]
    }
  }
}

DeepEval field-name routing

By default, the DeepEval wrapper reads input["query"], prediction["response"], and expected_output["response"] to build the LLMTestCase. Manifests that use different field names (for example, email_body / intent / reply_text) override this with:

evaluator_params key	Default	Purpose
input_key	"query"	Key in item.input passed as LLMTestCase(input=...)
actual_output_key	"response"	Key in prediction.prediction passed as LLMTestCase(actual_output=...)
expected_output_key	"response"	Key in item.expected_output passed as LLMTestCase(expected_output=...)
context_key	(none)	Key in prediction.prediction extracted and passed as LLMTestCase(retrieval_context=[...]). Populated as a single-element list when the field is non-empty; None otherwise. Use with "RETRIEVAL_CONTEXT" in evaluation_params to expose the retrieved context to the GEval judge.

These keys configure the wrapper only. They are never forwarded to the underlying DeepEval metric constructor.

azure_ai_evaluation input pool

The azure_ai_evaluation wrapper builds an input pool from multiple sources and passes the evaluator only the subset of keys it declares in its own __call__ signature:

Key	Primary source	Fallback
query	item.input["query"]	prediction.prediction["query"]
response	prediction.prediction["response"]	(none)
context	item.input["context"] or item.expected_output["context"]	prediction.prediction["context"]
ground_truth	item.expected_output["response"]	(none)

The fallback for query and context to prediction.prediction accommodates pods whose datasets do not flatten these into item.input but carry them in the prediction payload instead. Pods that do provide item.input["query"] take precedence. response is always read from prediction.prediction — pods must include it there under that key.

Score normalization

Every evaluator emits its result on a single 1-to-5 normalized score.

Field on MetricResult	Meaning
normalized_score	Float in [1.0, 5.0]. The only scale used for thresholds, gate decisions, and aggregates.
raw_score	The raw score — the framework-native value, preserved for inspection.
raw_scale	Tag describing the raw scale ("binary", "0-1", "likert_1_5", or "categorical").
justification	Optional reviewer-facing explanation.

Mapping rules by raw scale:

Raw scale	Normalization	Examples
binary	match to 5.0, mismatch to 1.0	builtin/ExactMatch, every builtin_deterministic evaluator
0-1	normalized = 1 + clamp(raw, 0, 1) * 4	DeepEval metrics, F1ScoreEvaluator
likert_1_5	identity (clamped to [1, 5])	Azure AI Evaluation LLM-judge evaluators, custom_rubric
categorical	score assigned to the most likely configured option; the option probabilities remain in details	decision_judge/inline

Manifest thresholds (MetricConfig.threshold) are always expressed on the normalized scale, never the raw one.

An abstaining Decision result has normalized_score: null and passed: null. It remains in the item count but not the scored count, so it cannot make an aggregate look better or worse by inventing a numeric value.

Aggregation and gating

After every item is scored, the service aggregates results per (scope, metric_name) pair into one AggregateMetricResult: item count, scored count, mean/min/max/median/p90/standard deviation of normalized_score, pass rate, and an aggregate passed flag (mean score compared against the metric's threshold and direction). The aggregation is semantic_evaluation.scoring.aggregation.aggregate_metric_results, which the SDK's runner also calls per group of items for a run that sets aggregate_by; the gate itself is scoring/gating/policies.py. The statistics come from semantic_evaluation.scoring.stats.spread, which the SDK's runner also uses for trace metrics.

A request may carry several predictions for one item, one per repeat of a run that sets repeats. Each is scored on its own, every result is stamped with its prediction's repeat, and the aggregates and the gate count all of them; item_count_by_scope and the gate's minimum item count still count items.

The gate (build_gate_result) then combines two kinds of failure into one GateResult:

• Minimum item count — any requested scope that loaded fewer items than GateConfig.minimum_item_count fails the gate, independent of scores.
• Aggregate threshold — any AggregateMetricResult whose passed flag is False fails the gate.

GateConfig.mode is currently always "all_metrics": every enabled, thresholded metric must pass for the run to pass. GateResult.passed is True only when there are zero failures.