Langfuse profiles and queue cleanup

The Langfuse connection profile model

Definition: Langfuse connection profile A named set of Langfuse project credentials registered with the review-sync service. Each profile stores a profile_id, base_url, public_key, and secret_key. The service resolves profiles at sync-job execution time so pod code never carries credentials directly. See Langfuse connection profile.

Unlike the binding contract, the Langfuse connection profile is owned entirely by this service (review_sync_service/langfuse_profiles.py) — it has no equivalent in sdk/semantic_evaluation. Each pod (or, in a workshop setting, each participant) registers one profile pointing at their own Langfuse project, then references that profile's profile_id from the langfuse_profile_id field of every binding that reads from that project.

Field	Meaning
profile_id	Unique name for this profile, chosen by whoever registers it
base_url	Langfuse project URL, e.g. https://cloud.langfuse.com
public_key	Langfuse public API key for the target project
secret_key	Langfuse secret API key

public_key and secret_key each accept either a literal string (local development) or an Azure Key Vault secret URI of the form https://{vault}.vault.azure.net/secrets/{name}. The registry resolves Key Vault URIs via azure-identity's DefaultAzureCredential (managed identity in Azure, developer credentials locally) only when a sync job actually reads the profile's credentials (LangfuseProfileRegistry.get()) — GET /v1/langfuse-profiles and GET /v1/langfuse-profiles/{profile_id} return the stored value (literal or URI) as registered, never a resolved secret.

Profiles persist to a JSON file (REVIEW_SYNC_LANGFUSE_PROFILES_FILE, review_sync_service/state/langfuse_profiles.json by default) so they survive service restarts. All four /v1/langfuse-profiles routes, reads included, require a known caller key in the X-API-Key header on the deployed service, so every pod can register its own Langfuse project. See 04-api-reference.md — Authentication.

curl -s -X POST http://127.0.0.1:8002/v1/langfuse-profiles \
  -H "Content-Type: application/json" \
  -d '{
    "profile_id": "reference-pod",
    "base_url": "https://cloud.langfuse.com",
    "public_key": "pk-lf-...",
    "secret_key": "sk-lf-..."
  }'

A binding that references a profile_id that is not yet registered can still be stored — validation is schema-shape only (see 01-bindings.md — Binding validation) — but any sync job for that binding fails at credential resolution until the profile exists.

Queue cleanup

Queue cleanup is opt-in, binding-scoped (binding.queue_cleanup.enabled, default false), and only ever removes queue items that a sync job has already handled safely.

Behavior	Rule
Cleanup candidates	Only items with published or skipped_unchanged outcomes. Items with policy_skipped or behind a failed_publish block are never deleted.
Retry across jobs	A failed delete is kept as pending_cleanup_items in the binding's checkpoint state and retried on the next sync job, merged with any new candidates that later job produces.
Dry runs	Never delete. Cleanup retry state is not advanced regardless of queue_cleanup.enabled.
Audit records	Every non-dry-run job with candidates or cleanup outcomes writes a cleanup audit record to REVIEW_SYNC_CLEANUP_AUDITS_DIR/{binding_id}/{job_id}.json, recording the candidate lists, delete outcomes, and resulting pending-retry set. Separate from and independent of the persist_artifacts artifacts in 02-sync-execution.md — Artifacts.
Job summary	JobSummary.cleanup_summary reports per job: enabled status, current-job vs. retry candidate counts, and how many deletes succeeded, were already absent, or remain pending. See CleanupSummary for the field shape.

Enable cleanup on a binding by setting:

"queue_cleanup": {"enabled": true}