Evals before features
Five to fifteen named cases beat another tool call. On an internal agent review workflow at work, I ship the evaluation loop first — then grow the prompt.
| Published | |
|---|---|
| Length | 8 min |
| Stack | Claude · internal review harness |
| Status | Active at work |
Pass/fail before the prompt grows
I added a clarifying step to an internal agent review workflow at work. The demo looked sharper. Two weeks later nobody could say whether intake-ambiguous still asked one question or had started inventing fields again. We were arguing from vibes.
That is the expensive pattern: when the agent feels thin, people reach for a tool, a step, or a longer prompt. Failures get harder to pin. Every feature without a named check becomes something you can only debate.
Write the evaluation loop first. Lock what pass and fail look like on a small case set. Grow the prompt, the tools, and the autonomy only after that. A bigger agent with no check is still a vibe — louder, and harder to kill.
(01)
Write the cases
Pull 5–15 real inputs the workflow already sees, including the ones that bite you. Skip anything without an expected outcome — that is a story, not a case.
(02)
Name pass and fail
For each case, write what must be true to pass and what counts as fail. Prefer wording a second person can score without pinging you for intent.
(03)
Freeze the harness
Run the same cases the same way every time the prompt or a feature moves. If the harness drifts with the feature, you relocated the vibe. You did not evaluate.
What the shape actually is
Four moves, always in this order. A feature may enter only after a failing case has a written expectation. Most teams skip freezing the score before they touch the agent. That skip is the whole bug.
Turn the agent workflow in this conversation into an eval-first loop. Before proposing any new feature, tool, or prompt expansion: 1. Draft 5–15 representative cases from the real inputs described. 2. For each case, write a pass condition and a fail condition a second reviewer could apply without asking me. 3. Describe a minimal harness: how each case is run, what evidence is recorded, and how pass/fail is decided. 4. Only after that, list at most three feature changes — each tied to a case it would improve, and each with the re-run plan. Return: case table → harness sketch → feature proposals gated by failing cases. If you cannot name pass/fail for a case, drop the case or ask one clarifying question — do not invent metrics.
The quality gate. A good result returns scored cases and a harness you can run twice. Feature lists first, or metrics nobody defined, mean the prompt failed — fix the cases, not the feature count in the instructions.
Each case costs authoring time up front and attention on every re-run. That is why a tiny frozen set beats a fat set you never re-score. Skip the re-run after a change and you wrote documentation. You did not build a loop.
| Kind | Cost | Catches | Fails when |
|---|---|---|---|
| frozen case set | Medium to write, cheap to re-run | Regressions you already named | The failure mode is new |
| ad-hoc spot check | Low once, high forever | Whatever you happened to open | You are tired or biased |
| feature with no eval | Zero now | Nothing | The first silent wrong answer |
evals:
harness: fixed case set, same runner, same scorer
cases:
- id: intake-ambiguous
expect: asks one clarifying question; does not invent facts
on_fail: tighten the refusal / clarify rule before adding tools
- id: known-good-path
expect: produces the required artefact fields without extras
on_fail: treat as regression; do not ship the latest prompt change
rule: no new feature lands unless it maps to a failing case
An eval that moves with the feature proves nothing
Rewrite the expected outcome so the new prompt “passes,” and you negotiated. You did not evaluate. Keep the case set and the pass/fail text stable across changes. When reality shifts, add a new case with a new expectation. Do not edit history to flatter the model.
Happy-path-only sets also break the loop. At work, the review loop earned trust only after the set included ambiguous, partial, and adversarial inputs the agent already mishandled — intake-ambiguous style cases, not just known-good-path. A green board of easy cases is still a vibe.
- Method is first-party evals practice from building and revising an internal agent workflow — not a vendor playbook or a product guarantee.
- The prompt and harness sketch are my own implementation shape; adapt the case count and scorer to your domain.