Build Your First Eval Set
Before/after score demo; three tasks: collect ten real cases, write pass criteria and run a baseline, let the numbers talk
THE QUESTION THIS PAGE ANSWERS
ANSWER FIRSTWhat is the key idea behind “Build Your First Eval Set”?
Before/after score demo; three tasks: collect ten real cases, write pass criteria and run a baseline, let the numbers talk
Follow the handoffs, not the demo. A system becomes dependable at the boundaries between model, tools, state, permissions, and people. Read each handoff as a place where you can observe, test, and recover.
Name the input, owner, approval, and recovery action for one automated step.
A successful run that cannot explain what happened or be safely repeated.
An eval set isn’t mysterious. It’s a pile of fixed inputs, each with a “what counts as pass” criterion. The magic is the moment you score: what you thought was an across-the-board win is often three up, two down. Without an eval set, those two drops stay invisible forever.
Same cases, before and after a prompt change
Three eval pitfalls this chapter already covered
First, quietly editing cases while you score. Change a case and the before/after scores aren’t comparable. Adding cases is fine; changing cases means re-running the full baseline.
Second, looking only at the total. Is 4/5 better than 3/5? First check whether the failing case used to pass.
Third, writing the criterion as “high-quality answer”. That can’t be judged. Write “includes next week’s plan” or “numbers match the source” — sentences you can decide at a glance. Only when you can’t judge that way does LLM-as-Judge come in.
This Chapter's Hands-on Checklist
0 / 3 done
Collect 10 real cases
15 min EveryonePick 10 inputs you’ve actually used and save them in a file, one per line. The mix matters: seven everyday, three nasty. Pull the nasty ones from the M1 five-run streak and the M2 stuck records — the inputs that already crashed it are the most valuable.
What counts as done
Write a pass criterion for each, run a baseline
1 hour If you want changes to be evidence-basedEach case gets one “what counts as pass,” written so you can decide at a glance: “output is exactly three sections,” “the conclusion includes XX,” “no numbers invented beyond the source.” Then run all 10 and write down the first baseline score.
What counts as done
Change one prompt version, let the score talk
Half a day If you want to keep iteratingChange the prompt against the failing cases. Don’t touch a single case. Re-run the full set and compare. Watch two things: how much the total moved, and whether any case that used to pass now fails. A regression isn’t shameful — write down which case and why. You’re already doing the real engineering Anthropic talks about.
What counts as done
How “See it first · One change, five cases, five different fates” becomes executable
“An eval set isn’t mysterious.” is not about a magic phrase. It is about giving the model enough information to know who the work is for, what must be done, and what counts as acceptable.
Background sets direction; constraints set the boundary
“First, quietly editing cases while you score .” shows why a useful request separates the task, audience, source material, output format, and constraints. Without background, the model guesses. Without acceptance criteria, fluent text is not evidence that the task is complete.
More words do not guarantee a better result
Turn “For M3, record: where the eval set lives, what the baseline is, and what the first comparison run concluded.” into a small experiment: change only one of background, requirements, or constraints while keeping the rest fixed, then observe which layer actually changes the output.
From “See it first · One change, five cases, five different fates” to “Hands-on checklist · Pick one, start it, tick it off”
“See it first · One change, five cases, five different fates” grounds the problem in “An eval set isn’t mysterious. It’s a pile of fixed inputs, each with a “what counts as pass” criterion . The magic is the moment you score: what you thought was an across-the-board win is often three up, two do…”. “Hands-on checklist · Pick one, start it, tick it off” then moves it toward “Pick 10 inputs you’ve actually used and save them in a file, one per line. The mix matters: seven everyday, three nasty . Pull the nasty ones from the M1 five-run streak and the M2 stuck records — the inputs th…”. Together, they show that the lesson is not just a conclusion to remember, but a claim with conditions.
Carry the judgment into the next situation
Build a request layer by layer: task and audience first, material and output rules next, constraints and acceptance checks last. Change one layer at a time so you know what actually helped.
- “See it first · One change, five cases, five different fates”: An eval set isn’t mysterious. It’s a pile of fixed inputs, each with a “what counts as pass” criterion . The magic is the moment you score: what you thought was an across-the-board win is often three up, two do…
- “Hands-on checklist · Pick one, start it, tick it off”: Pick 10 inputs you’ve actually used and save them in a file, one per line. The mix matters: seven everyday, three nasty . Pull the nasty ones from the M1 five-run streak and the M2 stuck records — the inputs th…
- “The closing point”: For M3, record: where the eval set lives, what the baseline is, and what the first comparison run concluded. From this square on, when you say “it got better,” you have a number in hand
The final “The closing point” brings the discussion to “For M3, record: where the eval set lives, what the baseline is, and what the first comparison run concluded. From this square on, when you say “it got better,” you have a number in hand”. The useful thing to carry forward is knowing which judgments must be revisited when input, scale, or risk changes.
I turned one judgment from this article into a small experiment I could run today. Knowing what to observe next is more useful than simply remembering the conclusion.
After reading this, I first looked for the conditions behind the idea instead of copying the method into a project. That order made the later trade-offs much clearer.
When this judgment reaches real work, which constraint should be added first? I am curious which step matters most between reading and the first practical attempt.
No discussion on this article yet.