Review sync — overview

review_sync_app/ is the shared service for the workspace's review-to-dataset step. It is a single FastAPI application (review_sync_service/) that reads completed items from a Langfuse annotation queue, checks them against a configurable promotion policy, maps the reviewed fields into a target schema, and writes the approved records into a Langfuse golden dataset.

Definition: review sync Review sync is the step that reads completed human reviews from a Langfuse annotation queue, normalizes the reviewed data, maps it into a target schema, and writes approved records into a Langfuse golden dataset. It does not include reviewer assignment, queue authoring, or queue cleanup in the Langfuse UI. See review sync.

The problem this service solves

Every pod that runs human review needs the same handful of things: a place to poll a Langfuse annotation queue for completed items, a way to decide whether a reviewed item is good enough to promote, a way to turn a queue item's loose JSON shape into a dataset record with a strict schema, and a durable record of which reviewed items have already been handled so re-running sync never double-promotes. Building that once per pod means duplicated queue-polling code, duplicated Langfuse credentials scattered across pod codebases, and inconsistent promotion rules.

review_sync_app/ centralizes all of it behind one HTTP boundary. A pod registers a declarative review binding describing one (annotation queue, golden dataset, scope) triple, then triggers a sync job whenever it wants the service to poll that queue and publish anything promotable. Langfuse project credentials live only inside this service's Langfuse connection profiles — pod code never carries them.

Current responsibilities

Area	Owned here
Binding registration	Queue-to-dataset bindings, dataset targets, review policy, and declarative field projections, stored in a file-backed registry
Langfuse connection	Named Langfuse connection profiles, one per Langfuse project, resolved at sync-job execution time
Sync execution	Completed queue reads, reviewed-record normalization, review-policy enforcement, projection-driven mapping, publication, and binding-scoped queue cleanup
Job tracking	Machine-readable sync-job results, handled-version checkpoints, cleanup retry state, cleanup audit records, and local debug artifacts persisted per job
HTTP boundary	FastAPI service surface for local and cloud execution

Current non-responsibilities

Area	Owned elsewhere
Shared review-sync contracts (QueueToDatasetBinding, ReviewPolicy, JobRequest/JobResult, ...) and the pod-side HTTP client (ReviewSyncServiceClient)	sdk/semantic_evaluation/ — see 06-review-sync.md
Reviewer workflow, queue authoring, and reviewer assignment	Langfuse UI and Langfuse queue APIs
Prediction execution	Pod-specific cloud or local workflow runtime, for example reference_pod_implementations/{agent-framework-pod,langchain-pod}/
Scoring execution	scoring_app/
Pod-side CLI trigger for manual or scheduled sync runs	reference_pod_implementations/{agent-framework-pod,langchain-pod}/evaluation/review_to_dataset/

This service does not import from reference_pod_implementations/ or scoring_app/.

Architecture

graph TD
  subgraph Pod["Pod side caller"]
    A["ReviewSyncServiceClient\nregister binding and trigger sync"]
  end
  subgraph Service["review_sync_app review_sync_service"]
    B["FastAPI app"]
    C["Binding store"]
    D["Langfuse profile registry"]
    E["Sync executor"]
    F["Checkpoint and cleanup audit stores"]
  end
  subgraph Shared["sdk semantic_evaluation"]
    G["Review sync contracts, reader, promotion, and publisher helpers"]
  end
  subgraph Langfuse["Langfuse"]
    H["Annotation queue"]
    I["Golden dataset"]
  end
  A --> B
  B --> C
  B --> D
  B --> E
  E --> F
  E --> G
  G --> H
  G --> I
  style Pod fill:#e9ebec,stroke:#273943,stroke-width:2px
  style Service fill:#e5f4fd,stroke:#0098eb,stroke-width:2px
  style Shared fill:#e7eeec,stroke:#125a40,stroke-width:2px
  style Langfuse fill:#efe9ed,stroke:#62294b,stroke-width:2px

A pod calls ReviewSyncServiceClient to register one binding and, later, to trigger a sync job. The FastAPI app looks up the binding in the binding store and the referenced Langfuse connection profile in the profile registry, then hands both to the executor. The executor uses shared sdk/semantic_evaluation helpers to read completed queue items, promote the ones that pass the review policy, and publish them into the golden dataset — while the checkpoint and cleanup-audit stores record what the job did so re-runs and queue cleanup stay safe.

Module map

Path	Purpose
review_sync_service/app.py	FastAPI app: lifespan, request-logging middleware (logs api_version and user_agent, adds retirement headers), /health, and the v1 router mounted under /v1 and unversioned
review_sync_service/api/v1.py	Every route of API v1 and the API-key dependency
review_sync_service/api/versioning.py	SUPPORTED_MAJORS, api_version_for_path(), and RETIREMENT_SCHEDULE with the Deprecation/Sunset/Link headers it produces
review_sync_service/openapi_snapshot.py	Writes openapi/<major>.json; the root test gate fails when the committed file is stale, and the PR pipeline diffs it for breaking changes
review_sync_service/main.py	Local entrypoint (uv run python -m review_sync_service.main)
review_sync_service/service_config.py	Environment-backed settings, .env loading, state-path resolution
review_sync_service/binding_store.py	File-backed binding store
review_sync_service/langfuse_profiles.py	LangfuseConnectionProfile model and file-backed profile registry, including Key Vault secret resolution
review_sync_service/mapping.py	Declarative projection resolution and binding-level semantic validation
review_sync_service/policy.py	Review-policy enforcement for one reviewed record
review_sync_service/executor.py	Sync-job execution: ordering, promotion, publication, checkpoint advancement, and queue cleanup
review_sync_service/checkpoint_store.py	File-backed checkpoint state per binding
review_sync_service/cleanup_audit_store.py	File-backed cleanup audit record storage per binding and job
review_sync_service/scripts/reset_local_state.py	Deletes local state/ and outputs/ for a clean local run
review_sync_service/tests/	Service test suite — see 05-testing.md

Design rules

• A review binding is data only. It never runs by itself — a sync job reads it and executes against it.
• Langfuse project credentials never travel with a binding. Each binding references a Langfuse connection profile by ID, and the service resolves the actual credentials only at sync-job execution time.
• policy_skipped and skipped_unchanged are non-blocking outcomes — later promotable records in the same job still run. failed_publish is the only blocking outcome; every later pending record in that job becomes deferred. See 02-sync-execution.md.
• Checkpoint state only advances on a non-dry-run job, and only when that job actually handled a record or changed pending cleanup state. Dry runs never advance the checkpoint and never delete queue items.
• Queue cleanup is disabled by default and binding-scoped. When enabled, only items handled as published or skipped_unchanged become cleanup candidates, and failed deletes are retried on later jobs rather than dropped.

Reading order

File	What it covers
getting-started.md	Install, run locally, register a Langfuse profile, register a binding, trigger a first sync job
00-review-sync-overview.md ← you are here	What this service is, why it exists, boundary, architecture, module map
01-bindings.md	The binding model, review policy, field projections, and binding validation
02-sync-execution.md	Sync-job execution flow, checkpoints, promotion outcomes, reviewer-comment and score passthrough, artifacts
03-langfuse-profiles-and-cleanup.md	The Langfuse connection profile model and binding-scoped queue cleanup
04-api-reference.md	The full HTTP API surface
05-testing.md	Test-suite layers
glossary.md	Definitions for terms local to this service