Get started: Product Owner
Why this matters
Evaluation is how your pod knows whether an LLM-powered agent or application is actually doing its job, not only the day it ships, but also every sprint after. Model behavior isn't testable at a glance Why this isn't testable at a glance Model output depends heavily on context: the exact wording of a user's message, an update to the underlying model, or a change in the data it's grounded in can all shift the result. Testing it by eye means trying a handful of prompts and scenarios, never all of them, so there's no way to see how every other situation's response changed. With that many possible cases, a regression can slip through silently. Only a repeatable test, run again after every change, checks the full set instead of a sample. the way a button or a form is.
Without an explicit standard and repeatable test, quality quietly becomes a matter of guesswork. You don't see when a change made things worse. It's also what lets a pod treat model quality as a release gate Release gate The same way a failing unit test already blocks a deployment pipeline from shipping, an evaluation gate blocks a release when model quality drops below the standard your criteria set: an automated check wired into that same pipeline, just testing model behavior instead of code behavior. : "should we ship this" becomes a checkable decision instead of a judgment call.
What you get back: user stories and acceptance criteria that actually hold up, engineering quality that keeps improving instead of quietly slipping, domain experts and end users who are happier with the result, and a real dashboard Dashboard A live view of evaluation results and trends over time, built automatically from the same dataset and traces, not assembled by hand for a specific meeting. of quality metrics.
How this connects to your metrics
Product metric: what you report
A lagging indicator: it moves slowly, and only indirectly.
↓ you can't move it without
Example: user engagement
Eval metric: what the engineer moves
A leading indicator, checked on every run.
↓ can't be computed without
Example: the share of responses your criteria mark as good
Ground truth
The expected answers behind your acceptance criteria. Left to engineer judgment, this drifts; owned by you, the whole chain holds.
What we'll ask of you
In plain terms
The honest version: this costs real time, defining goals, reading through traces Trace A structured record of one run through your agent or app: what it was asked, what it decided, and what it answered. Langfuse captures this automatically for every request. now and then, finding the right domain experts and coordinating with them. Here's exactly what that looks like, and what you get back for each piece of it.
| Task | What you get |
|---|---|
| Define acceptance criteria before work is built. | The engineer can check their own work against a standard, instead of asking you repeatedly for the same feedback. |
| Occasionally read through traces Trace A structured record of one run through your agent or app: what it was asked, what it decided, and what it answered. Langfuse captures this automatically for every request. yourself. | Quality issues get caught early, most without needing deep domain knowledge. |
| Help find and schedule domain experts. | The reviews that truly need expert judgment get it, without you becoming the bottleneck. |
| Write feedback into Langfuse Langfuse The shared platform every pod uses to hold traces, datasets, and reviewed feedback in one place, structured and visible to the whole team instead of scattered across chat messages and documents. directly. | Your feedback becomes a reusable check, instead of a Teams message that scrolls away. |
From vague to testable
Acceptance criteria are often specific enough to sound concrete without being testable. "Our agent shouldn't go off-topic" says something real, but it doesn't tell anyone what to test.
In practice the gap shows up two ways: sometimes there's no acceptance criterion beyond "it should work well," and sometimes a criterion exists but nobody has paired it with an expected answer to check against. Either way, there's nothing yet an engineer, or a gate, can actually test.
This is usually smaller than it sounds. One or two well-chosen examples per case often generalize Why one example is usually enough A well-chosen example rarely covers just itself. One off-topic redirect case stands in for the whole category of off-topic questions, not only that literal question. Covering the different situations that come up matters more than collecting many examples of the same situation. The true variety of how real users actually ask will show up later anyway, once the dataset starts growing from production traces. . Building a good dataset Dataset The evolving collection of real cases engineers check new work against. Each entry pairs a case (an input, like a user question) with the expected answer for it, so a change either matches the standard or it doesn't. is about covering the different situations that come up, not accumulating endless examples of each one.
Example
Vague
"The assistant shouldn't go off-topic."
Testable
"Asked about the weather, don't answer, redirect to support topics (>= 9 out of 10)."
Example
Vague
"Answers shouldn't be too long."
Testable
"Responses should stay under 40 words unless the user explicitly asks for more detail."
How we collaborate
Two ways feedback turns into data
Both cycles shape how product owners, engineers, and domain experts work together around a shared quality standard. Data-first runs alongside product iterations and development; Trace-to-data runs once something is already in production, growing the dataset Dataset The evolving collection of real cases engineers check new work against. Each entry pairs a case (an input, like a user question) with the expected answer for it, so a change either matches the standard or it doesn't. from real user interactions instead of intended ones.
Data-first
When a new product, proof of concept, Feature, or User Story is on the table, agree the acceptance criteria and at least one concrete example before the engineering work. This applies at any scale, whether you're starting from zero or iterating on something already live: a single criterion like "keep answers short" and a full proof of concept with ten criteria defined upfront both follow the same pattern.
Drafting is flexible. Write it however you already write acceptance criteria, plain language or Given-When-Then both work. Product owner and engineer can write examples together, a domain expert can help where the judgment calls for it, one drafts and the other reviews, or the engineer builds the dataset Dataset The evolving collection of real cases engineers check new work against. Each entry pairs a case (an input, like a user question) with the expected answer for it, so a change either matches the standard or it doesn't. entirely if the product owner provides criteria that are precise and measurable enough to work from directly. Any of that beats building first and discovering the bar was wrong afterward.
Trace-to-data
Once something is running, don't leave feedback as a Teams message or a comment in a document. Write it down once, directly on the trace Trace A structured record of one run through your agent or app: what it was asked, what it decided, and what it answered. Langfuse captures this automatically for every request. in Langfuse Langfuse The shared platform every pod uses to hold traces, datasets, and reviewed feedback in one place, structured and visible to the whole team instead of scattered across chat messages and documents. , where it's structured, persistent, and everyone can see it.
This is also how a dataset Dataset The evolving collection of real cases engineers check new work against. Each entry pairs a case (an input, like a user question) with the expected answer for it, so a change either matches the standard or it doesn't. that started with one or two examples per situation grows to cover the real variety of how users actually ask things. Each trace that catches a missed case and didn't produce a good result becomes the next dataset entry. The initial examples define the standard. Production traces fill in the coverage.
Why bother with a dataset
It's the cheaper option, not the extra one
The reactive approach of occasional manual testing of an app or agent, answering "was this good?" with unstructured comments about the findings in Teams or a document review, feels lighter than writing it down. But every future change re-opens the same conversation. An engineer who wants to iterate quickly can't ask that question ten times a day.
Put the answer in a dataset Dataset The evolving collection of real cases engineers check new work against. Each entry pairs a case (an input, like a user question) with the expected answer for it, so a change either matches the standard or it doesn't. once, and it becomes a standard the engineer can check against on their own, as often as needed. Your own time goes into occasionally adding a few targeted new cases: defining something new before it's built, or creating ground truth Ground truth The expected, correct answer for a specific case. Pairing a criterion with ground truth is what turns it from a description into something an engineer, or an automated check, can actually test against. for a production trace Trace A structured record of one run through your agent or app: what it was asked, what it decided, and what it answered. Langfuse captures this automatically for every request. that surfaced a gap.
Where you start
No pod adopts this all at once. And that's fine.
You don't have to reach that full picture on day one. Here's a realistic path to it, plus one practice worth starting this week no matter where the rest of your work stands.
Start immediately, never stop
Define before you build
You commit to writing down what "good" means for each new capability, and review or help build an example of it, before the engineering work. This alone is a real win: it gives engineers something concrete to build and test against, instead of only after-the-fact "this isn't right" feedback. No pod outgrows this: even a team running fully on the two steps below still starts every new feature here, running side by side rather than one after the other.
Step 1
A dataset exists
Once acceptance criteria are written down, an engineer can build a first dataset Dataset The evolving collection of real cases engineers check new work against. Each entry pairs a case (an input, like a user question) with the expected answer for it, so a change either matches the standard or it doesn't. from them, manually or synthetically, no production traffic or Langfuse Langfuse The shared platform every pod uses to hold traces, datasets, and reviewed feedback in one place, structured and visible to the whole team instead of scattered across chat messages and documents. required. That dataset is what makes everything else possible: it's what the engineer checks new work against automatically, as often as needed, without asking you each time. The setup behind that is an engineering concern to coordinate, not something you need to run yourself.
Step 2
Trace-to-data in Langfuse
The same Trace-to-data pattern from above, once a feature is live: instead of a Teams message, feedback goes directly onto the trace Trace A structured record of one run through your agent or app: what it was asked, what it decided, and what it answered. Langfuse captures this automatically for every request. in Langfuse Langfuse The shared platform every pod uses to hold traces, datasets, and reviewed feedback in one place, structured and visible to the whole team instead of scattered across chat messages and documents. . Recurring review time, light coordination support from you, and a dashboard Dashboard A live view of evaluation results and trends over time, built automatically from the same dataset and traces, not assembled by hand for a specific meeting. for you to report from.
Learn more
For the conceptual background behind all of this, see the Overview page.