Testing
uv run pytest scoring_service/tests/ -q
The suite has four layers, each with a different job when something upstream changes.
| Layer | Files | Purpose |
|---|---|---|
| Package API contract tests | test_deepeval_api_contract.py, test_azure_ai_eval_api_contract.py |
Pin the exact import paths, constructor signatures, attribute names, and return-value keys of the installed deepeval and azure-ai-evaluation packages. These fail immediately on a package upgrade if any API surface this service depends on changed, and point at exactly which wrapper code must adapt. |
| Evaluator wrapper tests | test_scoring_deepeval.py, test_scoring_azure_ai_evaluation.py, test_custom_rubric_evaluator.py, test_decision_judge_evaluator.py, test_d_series_evaluators.py |
Test evaluator wrapper behavior — field routing, normalization, error handling, direction gating, metadata, and the deterministic pattern-matching logic — using mock stubs or mocked HTTP calls. These tests never call an LLM. |
| Catalogue and app integration tests | test_catalogue.py, test_app.py, test_profiles.py |
Test catalogue dispatch, parameter validation, connection-profile resolution, and the HTTP contract end to end through TestClient. |
| API version tests | test_api_versions.py, test_openapi_snapshot.py |
Every /v1 operation has a deprecated unversioned twin; retirement headers appear only on a scheduled mount; the request log carries api_version and user_agent; and the committed openapi/v1.json matches the app. |
| Evaluator registration tests | test_evaluator_requests.py |
Test the pending-submission registry, accepted-entry catalogue loading, and the POST/GET /v1/evaluators/requests HTTP contract. |
Shared setup lives in conftest.py.
Upgrade workflow
When upgrading deepeval or azure-ai-evaluation:
- Run the test suite. Contract tests show exactly which API surfaces changed.
- Update the wrapper code in
scoring_service/scoring/evaluators/to adapt to the new API. - Update the contract tests to reflect the new pinned API surface.
- Confirm the wrapper tests still pass — from a caller's perspective, nothing should change.
Shared test fixtures
conftest.py exposes reusable helpers so individual test files do not each
reimplement DeepEval stubs or EvaluationItem/PredictionRecord
boilerplate:
| Helper | Purpose |
|---|---|
make_deepeval_stubs() |
Builds a fake deepeval module surface (AzureOpenAIModel, LLMTestCase, LLMTestCaseParams) for wrapper tests that must not call a real LLM |
make_evaluator_config() |
Builds a stand-in AzureOpenAIEvaluatorConfig |
make_eval_item() |
Builds a minimal valid EvaluationItem |
make_prediction() |
Builds a minimal valid PredictionRecord |
RecordingMetric, FailingMetric |
Stub DeepEval metric classes that record their inputs or raise, for exercising the wrapper's success and error paths |
StubLLMTestCase, StubLLMTestCaseParams, StubAzureOpenAIModel |
Stand-ins for the corresponding DeepEval classes |
clear_settings_cache |
Autouse fixture that resets the lru_cache-backed get_settings() singleton between tests |
configure_profiles_state(tmp_path) |
Writes a connection-profile state JSON file to a temp directory and points OFFLINE_SCORING_STATE_DIR at it, for tests that need a pre-populated profile registry |