Decision evaluator

decision_judge/inline evaluates one fixed-option question about a prediction with Microsoft Decision-1. It is for a judgement a deterministic evaluator cannot read from the workflow path, but whose answer is still a small, predefined set: whether the reply asked for a missing value, or which named outcome occurred.

It is not a generative judge. The service sends a System One request to the Foundry Decision endpoint and receives a probability for each configured option. The model returns no explanation text.

Use a deterministic evaluator first when a tool call, field, or URL proves the behaviour. Use this evaluator when free text must be read but the answer is fixed: it agrees with human labels as often as a generative judge, and is steadier, faster and far cheaper (see How it compares with a generative judge). Use a generative judge such as custom_rubric only when evaluation needs a graded scale, open-ended reasoning, or the written reason itself.

Connection profile

The service owner must register a foundry_decision profile and assign it to decision_llm_judge before a metric can run. The profile endpoint is the Foundry resource endpoint ending in /providers/microsoft, or that resource's /v1/systemone endpoint, not a Foundry project endpoint. The evaluator adds the System One path only when it is absent. It uses the same literal API key or Key Vault secret URI model as other profiles. See 02-connection-profiles.md.

Metric definition

The question belongs in a pod's evaluator_params, so it is versioned with the pod's evaluation configuration:

asked_for_order_number:
  framework: decision_judge
  evaluator: inline
  threshold: 4.0
  evaluator_params:
    type: noul
    instructions: Did the reply ask the customer for their order number?
    input_fields: [email_body]
    output_fields: [reply_text]

The evaluator builds state as a JSON object with input, output, and expected_output objects. input_fields defaults to ["query"], output_fields defaults to ["response"], and expected_output_fields defaults to []. Include only fields the question needs.

Parameter	Required	Meaning
type	Yes	noul, choice, or score
instructions	Yes	The question Decision-1 answers
criteria	For choice and score	Named option descriptions for choice, or an ordered level list for score
scores	For choice; optional otherwise	Mapping from option label to the normalized score 1 through 5
abstain_option	No	A choice or score option which deliberately produces no numeric score
input_fields	No	Keys read from item.input
output_fields	No	Keys read from prediction.prediction
expected_output_fields	No	Keys read from item.expected_output
min_confidence	No	A value from 0 to 1; lower-confidence results are marked for review
model	No	Deployment override; otherwise the profile deployment is used

The evaluator rejects unknown parameters, fewer than two options, scores outside 1 through 5, missing non-abstaining scores, and score names not in the question before making any model call.

Question types and score mappings

noul

Use noul for a yes/no question. It needs no criteria. Its default map is yes: 5 and no: 1; provide scores to reverse the meaning for a question about something that should not happen. It does not support abstention.

choice

Use choice for named, unordered outcomes. criteria maps every option label to the description the model should use, and scores must map every non-abstaining option to a normalized score:

type: choice
instructions: Which team should handle this request?
criteria:
  billing: Payment and invoice issues
  technical: Software and integration issues
scores:
  billing: 5
  technical: 1

score

Use score only when the answer is an ordered set of levels. Its criteria is a list indexed from zero by the API. With no explicit scores, the evaluator spreads the first and last levels evenly across 1 through 5; three levels therefore map to 1, 3, and 5. A fixed-option choice is preferred when it can express the question, because Decision-1 is usually more confident about it.

Results and review

The most likely option determines normalized_score; raw probability never does. This makes repeated runs stable when probability values vary slightly but the selected option does not. raw_scale is "categorical" and raw_score is null because the native result is a label, not one numeric measurement.

details keeps the selected label, configured option order, every option probability, model/deployment labels, usage, latency_ms (the duration of the model call alone, in milliseconds), and score's probability-weighted expected level. It also contains low_confidence: true when the model's confidence is below min_confidence; that flag marks the item for human review but does not change the numeric score.

An abstention produces normalized_score: null and passed: null. It counts as an evaluated item but not as a scored item, so aggregates and gates do not turn uncertainty into a passing or failing numeric value.

How it compares with a generative judge

The same three closed questions about customer support replies were asked of this evaluator (Decision-1) and of custom_rubric (gpt-5.1, Data Zone, default reasoning effort): 30 fixed replies with human labels, three repeats each.

Measure	Decision judge	Generative judge
Agreement with the labels	0.90 to 0.97	0.90 to 0.97
Items whose verdict changed between repeats	0 of 90	1 of 90
Labels changed by reversing the option order	0 of 30	not applicable
Median latency of one call	about 0.3 s	about 1.5 s
Cost per 100 items	about 0.001 USD	about 0.1 USD

The two judges agree with the labels equally often but fail on different items, so neither is a strict improvement in judgement. The difference is in stability, latency and cost. Decision-1 bills input tokens only and answers with a single label.

The setup, the per-item disagreements and the prices used are recorded in the experiment folder scoring_app/experiments/2026-11-09-decision-judge-comparison/.