Module 06: metric authoring

This module covers two ways to write an LLM-judge metric without writing any evaluator code: tuning a pre-built GEval metric's rubric, and defining a fully custom rubric via custom_rubric/inline.

6.1 GEval: a config-only customization surface

GEval is a pre-built evaluator: you do not write a class, you write a criteria string in evaluator_params. This is simpler than either azure_ai_evaluation's fixed-criteria evaluators (no customization at all, just fixed rubrics behind a name) or custom_rubric/inline (fully free-form, covered below): with GEval you get a real 5-point scale calibrated by DeepEval's own chain-of-thought scoring, and you only ever touch the prose.

The current intent_correctness rubric:

"criteria": "Score 1.0 if the predicted intent exactly matches the expected intent. Score 0.5 if the general category is correct (e.g. billing vs. technical) but the specific label differs. Score 0.0 if the intent label is wrong in a way that would lead to a different response strategy."

A vague rubric produces noisy scores; a concrete one (as above) reduces variance between runs of the same underlying LLM judge.

6.2 Hands-on: tune a threshold or rubric, then diff the results

Run the evaluation once as-is (Module 07) and note aggregate_metrics.json's mean_score for reply_correctness.
Change either the threshold or the criteria text for reply_correctness in evaluation/evaluation_manifest.yaml; for example, tighten the rubric to explicitly penalize a reply that doesn't reference the customer's account data when it's relevant.
Re-run and diff the new aggregate_metrics.json against the previous run's output (each run writes to its own timestamped folder; Module 07 covers the exact layout).

This makes the tune-and-rerun loop something you actually do once, rather than something you only read about. Expect scores to shift, sometimes non-trivially, from a rubric wording change alone.

LLM-based metrics are non-deterministic

GEval results vary between runs because the evaluation LLM samples with non-zero temperature. Use aggregate pass rates across several runs rather than a single-run pass/fail when judging whether a rubric change actually helped.

6.3 `custom_rubric/inline`: a fully custom metric, still no code

custom_rubric/inline lets you describe what to measure and how to interpret each score level; the scoring service builds the judge prompt for you. Use it instead of GEval when you want a domain-specific signal with full control over the prompt structure, not just the criteria text.

Situation	Recommended approach
You want a standard quality signal (relevance, coherence, faithfulness)	Use a built-in `azure_ai_evaluation` evaluator
You want a domain-specific signal used only by your pod	Use `custom_rubric/inline`: no registration required, works in all environments
You want a custom signal other pods can reference by name	Validate inline first, then submit for central registration (Module 09)

custom_rubric/inline does not require server-side registration for pod-local use. The rubric lives in evaluator_params inside the manifest, which is committed to the pod's repository. The scoring service receives the rubric definition once, at job submission, and builds the judge prompt on the fly; nothing is stored on the server beyond the job's own record. The same manifest therefore produces identical results on a developer laptop, a CI runner, and a scheduled cloud batch job without any per-environment registration step.

The only per-environment requirement is a custom_rubric_llm_judge connection profile that provides Azure OpenAI credentials, the same setup required for any LLM-judge metric, because credentials must stay out of the committed manifest.

Required fields

Add a metric block with framework: custom_rubric and evaluator: inline. All five scoring levels must be described. Field names below match this workshop's actual data shape, intent/rationale, not a placeholder name:

"intent_quality": {
  "framework": "custom_rubric",
  "evaluator": "inline",
  "threshold": 3.5,
  "evaluator_params": {
    "name": "Intent Classification Quality",
    "criteria": "Score how correctly the system identified the user's intent given the email subject and body. Consider whether the predicted intent would lead to the right response strategy.",
    "scale": {
      "5": "Intent label is exact and unambiguous; the system chose the most specific correct label.",
      "4": "Intent label is correct but a more specific label existed and was not chosen.",
      "3": "Intent category is right (e.g. billing) but the specific label is wrong.",
      "2": "Intent is partially related to the actual need but would lead to a partially wrong response.",
      "1": "Intent is completely wrong and would lead to a response that does not address the user's need."
    },
    "input_fields": ["email_subject", "email_body"],
    "output_fields": ["intent"],
    "expected_output_fields": ["intent"]
  }
}

Field	Required	Default	Notes
`name`	Yes	None	Label shown in Langfuse and `MetricResult.details`
`criteria`	Yes	None	Prose description of what the metric measures
`scale`	Yes	None	String keys `"1"`–`"5"`, one description per level
`input_fields`	No	`["query"]`	Keys from `EvaluationItem.input` included in the prompt
`output_fields`	No	`["response"]`	Keys from `PredictionRecord.prediction` included in the prompt
`expected_output_fields`	No	`[]`	Keys from `EvaluationItem.expected_output`: add when the rubric needs to compare against a reference answer
`model`	No	Service default	Azure deployment name override for this metric

output_fields/expected_output_fields both read "intent" here because this pod's prediction dict and dataset's expected_output both use that key directly; check your own scope's actual field names (Module 04) before copying this example verbatim into a different metric.

Connection profile

custom_rubric/inline uses the custom_rubric_llm_judge backend, which needs a connection profile assigned to it. On the deployed scoring service the platform team has assigned one to every judge backend. Check that custom_rubric_llm_judge is in the map:

curl https://scoring-service.bravecliff-5469885d.swedencentral.azurecontainerapps.io/v1/evaluator-backends \
  -H "X-API-Key: <the workshop scoring-service key>"

You cannot assign a different profile yourself: assigning a backend needs the platform-admin caller, because an assignment holds for every caller of the service and would change the judge model for every attendee at once. If the backend is missing from the map, ask your facilitator. The scoring service's Connection profiles page carries the full route reference and the list of backend keys.

Reading the results

Custom rubric scores appear in Langfuse the same way as any other metric. The MetricResult includes:

normalized_score: float 1.0–5.0 (the LLM's integer score cast directly to float)
justification: the LLM's one-to-two sentence reasoning
details.evaluator_name: the name field from your evaluator_params
details.evaluation_model: the Azure deployment used

Continue to Module 07.