Agent Skills

What the skills do

With the skills, your coding agent sets up evaluation for your product, maintains its datasets and metrics, runs experiments, organises expert reviews and configures Langfuse. They hold what it needs for that: the documentation of the SDKs and the shared services, the commands, and the practices this evaluation system is built on.

1. 1

   Set up tracing and evaluation

   In a new or existing repo.

   Example

   Set up tracing and regression evaluation for this agent.

   The agent run

   It asks three things back: the product id, whether the datasets are kept in the repository or in Langfuse, and which Langfuse project.

   semeval-setup-tracingsemeval-model-pricingsemeval-setup-evaluation

   1. Reads the agent and the SDKs' documentation.
   2. Adds tracing and checks one real trace in Langfuse, cost included.
   3. Spots a 10% price gap in Langfuse and asks before fixing it.
   4. Writes the manifest, run file, predictor and two metrics.
   5. Builds a 14-item dataset and pushes it to Langfuse.
   6. Runs and publishes the first evaluation, installs the always-on rule.

   You get

   • Every turn traced in Langfuse, with cost
   • evaluation/ with manifest, run file, predictor and coverage map
   • A dataset and a first passing run in Langfuse
   • The always-on rule for both tools

   Both set up about ten minutes later.

   From the real run: the files it wrote

   CLAUDE.md                               the always-on rule, Claude Code
   .github/instructions/eval.instructions.md   the same rule, Copilot
   .env.example                            variable names, no values
   refund_agent/telemetry.py               tracing set up once, at startup
   evaluation/
     evaluation_manifest.yaml              metrics, dataset sync, baselines
     runs/regression.yaml                  the regression run
     predictor.py                          calls the agent for each item
     fixtures/end_to_end.yaml              the 14 cases, edited in the repository
     build_dataset.py                      pushes them to Langfuse
     run.py                                runs, scores, gates, publishes
     coverage.md                           what is covered, what to update on a change

   From the real run: the first evaluation, as the agent showed it

   Metric	Mean	Median	P90	Stdev	Pass rate	Threshold	Result
   end_to_end.refunds_the_right_order	5.00	5.00	5.00	0.00	100%	5.0	pass
   end_to_end.takes_the_expected_path	5.00	5.00	5.00	0.00	100%	5.0	pass

   14 items, gate passed, published in Langfuse next to the dataset.

2. 2

   Change the agent, and its evaluation follows

   Ordinary feature work. Evaluation is never mentioned.

   Example

   Change the agent so that refunds over 100 EUR are escalated to a human instead of being issued.

   The agent run

   semeval-maintain

   1. Escalates orders over 100 EUR, with the limit as one named constant.
   2. Rewrites the three dataset items the change made wrong.
   3. Keeps exactly 100 EUR as a refund, the boundary case.
   4. Adds a non-English refund case and pushes a new dataset version.
   5. Runs the regression evaluation before calling it done.

   You get

   • The feature
   • The dataset items that prove it, in the same change
   • A passing regression run in Langfuse

   Done in under two minutes.

   From the real run: the dataset items it changed

    - id: "002"-  behaviour: refund_delivered+  behaviour: refund_over_limit_escalates   message: Please refund A1004, the jacket didn't fit.-  tools: [lookup_order, issue_refund]-  refund: A1004+  tools: [lookup_order, escalate_to_human]+  refund: null - id: "004"   behaviour: refund_delivered+  # A1003 is exactly 100.00 EUR: at the limit, not over it, so still refunded.   message: can u refund me for A1003 pls   tools: [lookup_order, issue_refund]   refund: A1003 ...+- id: "015"+  behaviour: refund_delivered+  message: Bitte erstatten Sie mir Bestellung A1001, die Schuhe passen nicht.+  tools: [lookup_order, issue_refund]+  refund: A1001

3. 3

   Run experiments and decide

   Compare configurations on the same dataset, with repeats.

   Example

   Compare reasoning effort low and high for the intent call, 3 repeats each, and tell me which one we should use.

   The agent run

   It asks before committing the experiment record.

   semeval-experimentsemeval-model-pricing

   1. Checks the model's pricing before comparing on cost.
   2. Writes a run file: both settings, 3 repeats, cost, tokens and time.
   3. Proves with one traced turn that each setting reaches the model.
   4. Runs each setting 3 times on 15 items, all published to Langfuse.
   5. Compares them with eval-report and recommends one.
   6. Commits the experiment record on a branch.

   You get

   • A recommendation with the comparison table
   • An experiment record in evaluation/experiments/
   • Every run side by side in Langfuse

   90 predictions compared and written up in twelve minutes.

   From the real run: the comparison table

   Metric	effort-low	effort-high
   refunds_the_right_order	5.00 ± 0.00 (pass)	5.00 ± 0.00 (pass)
   takes_the_expected_path	5.00 ± 0.00 (pass)	5.00 ± 0.00 (pass)
   cost_usd per item	0.000043 ± 0.000001	0.000174 ± 0.000043
   output_tokens per item	10.8 ± 0.4	76.2 ± 21.5
   wall_time_s per item	0.98 ± 0.19	1.35 ± 0.20
   Gate	passed	passed

   Written by eval-report, a semantic-evaluation command that turns finished runs into a Markdown comparison. Verdict: use low, the same quality at a quarter of the cost and 0.37 s faster per turn.

   From the real run: the experiment record it committed

   evaluation/experiments/2026-10-22-intent-reasoning-effort.md
   
   # Reasoning effort of the intent call: low or high
   ## Question
   ## Configurations      the two settings and their run names in Langfuse
   ## Results             the comparison table above
   ## What it means
   ## Decision

4. 4

   Manage expert in the loop reviews

   Reviewed and corrected answers become dataset items.

   Example

   Set up a review queue so our support leads can check and correct the agent's answers, and get those into the dataset.

   Later

   The support leads have finished reviewing. Bring their reviews into the evaluation dataset.

   The agent run

   semeval-review-loop

   1. Creates the annotation queue and score configs in Langfuse.
   2. Registers the project and binding with the review-sync service.
   3. Dry-runs a sync on a real trace.
   4. Writes a short guide for the reviewers.
   5. Later, syncs the reviewed traces into the dev dataset and flags duplicates.

   You get

   • A review queue in Langfuse for your experts
   • Corrected answers as dataset items, with who reviewed them
   • A sync to run again, or on a schedule in CI

   A reviewed correction synced into the dev dataset.

5. 5

   Price models as Azure bills them

   So cost comparisons and budgets are right.

   Example

   Our Langfuse traces show tokens but no cost for the intent call. Fix that.

   The agent run

   It asks before writing prices into the shared Langfuse project.

   semeval-model-pricing

   1. Finds model calls with tokens but no cost.
   2. Reads the deployment's type and region, and Azure's prices for it.
   3. Writes the price definition to Langfuse.
   4. Checks one new turn against Azure's prices.
   5. Flags earlier experiments that are not comparable on cost.

   You get

   • Correct cost on every new trace
   • A committed pricing record

   Every new trace priced as Azure bills it.

6. 6

   Manage datasets in Langfuse

   Langfuse holds the one copy every run reads.

   Example

   Pull the evaluation dataset so I can edit some items locally, then push it back.

   Or

   Is our evaluation dataset ready to be released? We have 8 items.

   Or

   Release the end-to-end dev dataset as version 1.0 and file the snapshot.

   The agent run

   Each prompt loaded semeval-datasets in testing; the steps are what the skill does.

   semeval-datasets

   1. Pulls a working copy, validates it and diffs it against Langfuse.
   2. Pushes the edit as a new dev version, and never deletes an item on its own.
   3. Checks release readiness: settled shape, reviewed ground truth, enough items.
   4. Releases an immutable version, checks the version bump, files the snapshot.

   You get

   • Versioned datasets in Langfuse
   • Releases that gates and experiments read, never the dev copy
   • A snapshot of each release in the repository