Module 09: custom metrics

This module covers submitting a pod-local custom_rubric/inline rubric (Module 06) for central registration, so other pods can reference it by name without copying the full definition into their own manifest.

9.1 Submit a rubric for central registration

Once a pod-local rubric has proven stable, submit it:

curl -X POST https://scoring-service.bravecliff-5469885d.swedencentral.azurecontainerapps.io/v1/evaluators/requests \
  -H "Content-Type: application/json" \
  -H "X-API-Key: <the workshop scoring-service key>" \
  -d '{
    "name": "intent_quality",
    "criteria": "Score how correctly the system identified the user intent.",
    "scale": {
      "1": "Intent completely wrong.",
      "2": "Intent partially related but would lead to a wrong response.",
      "3": "Intent category right but specific label wrong.",
      "4": "Intent correct but a more specific label existed.",
      "5": "Intent label exact and unambiguous."
    },
    "input_fields": ["email_subject", "email_body"],
    "output_fields": ["intent"],
    "expected_output_fields": ["intent"],
    "submitted_by": "your_name"
  }'

The submission is written to the service's pending submissions directory. The eval core team reviews pending submissions via GET /v1/evaluators/requests and commits accepted rubrics to scoring_service/custom_evaluators/accepted/. After the next service deployment, any pod can reference the evaluator as framework: custom_rubric, evaluator: intent_quality; no need to copy the criteria/scale/field-mapping definition into their own manifest at all.

graph TD
  A["Pod submits rubric"] --> B["Pending submissions directory"]
  B --> C["Eval core team reviews"]
  C --> D["Accepted evaluators folder"]
  D --> E["Any pod references it by name"]

Re-submitting the same name while it is still pending overwrites the file. Submitting a name that is already accepted returns 409; corrections go through the core team.

9.2 Checking submission status

The 201 response to your submission returns the stored record, which confirms the service received it. Listing every pending submission (GET /v1/evaluators/requests) needs the platform-admin caller, which a workshop key is not: ask your facilitator to check the queue if you want to see it there before the core team's review.

See scoring_service/custom_evaluators/README.md for the full schema and lifecycle rules.

9.3 When central registration is worth it

Situation	Central registration?
One pod, stable rubric, no other pod needs it	No: custom_rubric/inline in your own manifest is enough
Several pods measuring the same underlying quality (e.g. "routing correctness" across different intent taxonomies)	Yes: one canonical definition instead of N slightly-diverging copies
Rubric is still being iterated on	No: iterate inline first; submit once it has stopped changing between runs

This is the end of the workshop's evaluation-pipeline content. From here, review the semantic-evaluation documentation for production guidance on dataset versioning, CI integration, and scaling the review workflow.