Custom rubrics and pod-driven registration

custom_rubric is the framework pods use for an LLM-judge rubric that no existing DeepEval or Azure AI Evaluation metric covers. It makes a direct Azure OpenAI chat-completion call with a pod-defined prompt — no third-party evaluation framework is involved. It comes in two forms: inline, defined entirely inside the requesting manifest, and {name}, a rubric committed to source and reviewed by the eval core team.

This inline vs. {name} split is specifically about rubric reuse — whether the prose criteria and scale are repeated in every pod's manifest or committed once and shared by name. It is a different axis from the fact that every framework's MetricConfig (threshold, evaluator_params, ...) always travels inline inside the ScoreRequest, regardless of framework — see 00-scoring-overview.md — Where evaluator definitions live. Even for a custom_rubric/{name} metric, the metric name and threshold are still request-local; only the rubric's prose criteria and scale move server-side once accepted.

The rubric definition

Both forms validate against the same CustomRubricDefinition shape:

Field	Required	Notes
name	Yes (named form only)	Must match the filename stem, e.g. intent_quality.json → "intent_quality"
criteria	Yes	Prose description of what the metric measures
scale	Yes	Exactly the integer keys 1–5, one description per level
input_fields	No	Keys from item.input; defaults to ["query"]
output_fields	No	Keys from prediction.prediction; defaults to ["response"]
expected_output_fields	No	Keys from item.expected_output; defaults to []
model	No	Azure deployment name override; falls back to the service default model if omitted

At evaluation time the service builds a judge prompt from the listed fields, calls Azure OpenAI in JSON mode, and expects {"score": <integer 1-5>, "reason": "<string>"} back. The score is clamped to [1, 5] and used as normalized_score directly — custom_rubric results are likert_1_5, so no further normalization is applied (see 01-evaluator-catalogue.md — Score normalization).

Inline vs. named

	custom_rubric/inline	custom_rubric/{name}
Where the definition lives	The requesting manifest's evaluator_params	A JSON file committed to custom_evaluators/accepted/
Who can change it	Whoever owns the manifest, on every run	Only the eval core team, by committing a new file and redeploying
Overridable fields per request	All of them, since the caller owns the whole definition	Only model, via evaluator_params.model
Visibility to other pods	None — it lives in one pod's manifest	Global — any pod can reference it by name once accepted

Both forms need a connection profile assigned to the custom_rubric_llm_judge backend before they can run — see 02-connection-profiles.md.

Model resolution order

For a custom_rubric/{name} evaluator, the deployment name used for the judge call is resolved in this order:

1. evaluator_params.model (the calling manifest's override for this run)
2. model in the accepted JSON file (set by the submitting pod)
3. The assigned connection profile's deployment
4. OFFLINE_SCORING_DEFAULT_MODEL (service-wide fallback)

Result details

The judge's reason becomes the metric result's reason. details holds:

Key	Meaning
evaluation_model	The deployment that answered, after the resolution order above
connection_profile	The profile the backend resolved to
framework, evaluator_name	custom_rubric and the rubric's name
latency_ms	Duration of the model call alone, in milliseconds
usage	input_tokens and output_tokens, plus reasoning_tokens when the model reports them (already counted in output_tokens); null when the response has no usage

The registration lifecycle

Pods register a named rubric so other pods can reference it without embedding the full definition inline:

Stage	Action
Develop	The pod validates the rubric using custom_rubric/inline in its own manifest
Submit	The pod calls POST /v1/evaluators/requests with the rubric definition and a submitted_by identifier
Review	The eval core team inspects pending submissions via GET /v1/evaluators/requests
Accept	The core team commits the JSON file to custom_evaluators/accepted/{name}.json and redeploys the service
Use	Any pod references it as framework: custom_rubric, evaluator: {name}

Submissions are stored as pending files under OFFLINE_SCORING_STATE_DIR/custom_rubric_requests/ until a core-team member commits them to custom_evaluators/accepted/; the pending directory and the accepted directory are never the same location.

Submission rules

• Re-submitting a pending name overwrites the pending file — no error.
• inline is reserved. POST /v1/evaluators/requests returns HTTP 400 for that name.
• Already-accepted names are immutable through this endpoint. POST /v1/evaluators/requests returns HTTP 409; corrections go to the eval core team directly.
• A malformed file inside custom_evaluators/accepted/ is skipped with a warning at startup — it does not stop the service from starting, but it also never becomes a catalogue entry until fixed.

On the deployed service POST /v1/evaluators/requests accepts any known caller key in the X-API-Key header, while listing pending submissions with GET /v1/evaluators/requests needs the platform-admin caller. See 02-connection-profiles.md — Authentication.

File format and step-by-step submission

scoring_service/custom_evaluators/README.md is the file-level reference: the exact accepted-evaluator JSON layout, a worked intent_quality example, and the full submission curl walkthrough. This page covers the concept and the HTTP contract; that README covers the directory itself.