Overview
One shared backbone
The system is built around one shared backbone: Langfuse. Every loop reads from it and writes back to it.
This covers the agentic and LLM parts of your application specifically, the pieces you need to debug and evaluate model behavior. Your backend and frontend keep sending their usual telemetry to your normal tooling, e.g. Application Insights; only what's relevant to evaluation gets traced to Langfuse.
Your own application code still lives in your pod. That's the part that actually calls a model and produces an answer, what we call prediction. Prediction stays pod-specific: each team keeps its own agent or workflow code.
Scoring and Human Review to Golden Dataset Sync are shared services every pod calls into instead. Langfuse holds everything in one place: traces, annotation queues, golden datasets, and published evaluation runs. The next section walks through what happens to a trace once it exists.
How traces become better systems
End-to-end evaluation cycle
Every workflow run becomes a trace automatically. That's true whether it happens in production, in a local experiment, or in a CI pipeline.
That trace is the raw material for two things. Offline Evaluation is deliberate, developer-driven improvement, run locally or in CI. Expert-in-the-loop review is how the golden dataset grows from real cases, including production ones, when someone chooses to review them.
The two reinforce each other. A better dataset means a more reliable gate. A better gate means fewer bad outputs reaching review.
What's not yet built is automatic production monitoring: routing low-scoring outputs into review on their own, without someone choosing to look. That doesn't mean every production issue reaches a domain expert unfiltered, though: engineers already triage what they see first, resolving anything traceable to a code or context bug directly, and escalating only the genuinely ambiguous cases. See how review-queue items actually reach a human today, right below.
Two paths to the annotation queue
Manual inspection and automatic routing
Today, a trace reaches the annotation queue when a product owner or engineer manually scans production traces in Langfuse and flags a case. Automatically scoring every production trace and routing low-scoring cases is planned. Manual scanning stays valuable even after that exists, especially for investigating edge cases and building domain expertise.
How teams use the system
One shared backbone, three loops
You just saw how cases reach the annotation queue. That's one of three loops covering how teams actually use the system day to day. Each loop draws on the four foundations below.
Local tuning
Developer runs experiments
Optimize code, prompts, and configuration against golden datasets, then publish comparison runs back to Langfuse.
Regression gating
PR / CI pipeline
Validate changes through the same central scoring service and release-facing gate policy.
Expert review & production monitoring
Manual review in the Langfuse UI, or staged via the SDK
Curate corrections, monitor production quality, and grow the golden dataset.
What makes it work
The four pillars
Observability & tracing
Every run becomes a structured trace automatically, via the semantic-tracing SDK. It doesn't matter whether that run happens in production, in a local experiment, or in CI. Nothing else below works without this first.
Quality, measured against a shared standard
The shared scoring service gives every pod the same reusable yardstick: the same evaluators run against each pod's own golden dataset, checked by hand during local tuning or automatically on every PR.
Human-in-the-loop, made durable
Product owner and domain expert judgment gets captured once and stays reusable from then on, instead of getting lost in a Teams thread. That judgment might define what "good" means up front, or correct what's wrong afterward. Either way, it flows through the Human Review to Golden Dataset Sync service.
Efficiency & scale, compounding over time
Pillars 1 through 3 are shared infrastructure, not per-pod scripts: config templates, SDK clients, shared evaluators, and shared servers that every pod reuses instead of rebuilding. That's what makes the system cheaper to run the more it's used. Each pod's own golden dataset keeps growing, checks run automatically many times a day at near-zero marginal cost, and a new pod starts from real infrastructure instead of rebuilding from zero.
Architecture
Full system walkthrough
A more complete walkthrough of the target architecture is available as a downloadable slide deck. It also covers the online-monitoring pillar in more depth, though that part stays high-level for now.
Semantic Evaluation System Architecture Overview