Three Graders: Code, Model, Human
Static assertions vs LLM-as-Judge vs human calibration — which fits which scenario
THE QUESTION THIS PAGE ANSWERS
ANSWER FIRSTWhat is the key idea behind “Three Graders: Code, Model, Human”?
Static assertions vs LLM-as-Judge vs human calibration — which fits which scenario
Make the claim earn its place. Use this page as a decision aid, not a definition to memorize. Connect the idea to one real task, one observable result, and one failure that would change your mind.
Write one question you could answer with evidence after trying this idea.
A conclusion that sounds complete but leaves the key assumption untested.
Code Grader
| Dimension | Details |
|---|---|
| Methods | String matching, regex, static analysis (AST), unit test pass/fail, tool-call validation (correct API and parameters) |
| Pros | Fast (millisecond-level), near-zero cost, fully objective and reproducible, ideal for CI/CD automation |
| Cons | Too strict on valid variants (e.g., fails on different variable names), lacks semantic understanding, cannot evaluate subjective quality |
| Best for | Tasks with clear correct answers: code compilation, API return value checks, file format compliance, mathematical accuracy |
Model Grader (LLM-as-Judge)
| Dimension | Details |
|---|---|
| Methods | Feed the Agent's output and a scoring Rubric to a judge LLM, which returns a score and rationale based on predefined criteria |
| Pros | Can evaluate subjective quality (writing style, logic, creativity); understands intent beyond literal matching; flexibly adapts to different task types |
| Cons | Higher cost (each evaluation consumes tokens); potential scoring bias; requires carefully designed rubrics; results not fully reproducible |
| Best for | Open-ended tasks: research report quality, code style assessment, conversational naturalness, summary completeness and accuracy |
0 = Completely fails to answer the question, or contains serious factual errors
0.3 = Answers the question but misses critical information
0.7 = Complete and accurate, but poorly organized or redundant
1 = Complete, accurate, concise, and clearly structured
Human Grader
| Dimension | Details |
|---|---|
| When to use | Calibrating scoring criteria early in evaluation; when LLM judgment shows clear blind spots; high-stakes domains requiring expert judgment (medical, legal, financial) |
| Pros | Highest-quality feedback; can identify blind spots in automated methods; provides deep insights for improvement |
| Cons | Not scalable (human time is limited); slow (hours to days); expensive; inter-rater disagreement |
| Best for | Periodic spot-checks to calibrate Model Grader accuracy; establishing evaluation criteria for new domains; final validation for critical product decisions |
- Static Analysis (Code Grader): checks whether generated code compiles and passes lint
- Browser Agent Testing (Code Grader): automatically opens the generated page to verify the UI matches expectations
- LLM Judge (Model Grader): evaluates code quality, readability, and adherence to best practices
How “Code Grader” changes an answer
“Static assertions vs LLM-as-Judge vs human calibration — which fits which scenario” shows that a model does not process the “word count” we see. It processes Token pieces. Tokenization affects input length, how much context fits, and how much computation a request consumes.
Length, information, and context are different
As “Static assertions vs LLM-as-Judge vs human calibration — which fits which scenario” grows, separate three questions: how many Tokens the text becomes, which pieces can change the current decision, and whether older material has fallen outside the context window. Removing repetition is often more useful than simply making the window larger.
- Static Analysis (Code Grader): checks whether generated code compiles and passes lint
- Browser Agent Testing (Code Grader): automatically opens the generated page to verify the UI matches expectations
- LLM Judge (Model Grader): evaluates code quality, readability, and adherence to best practices
Keep what can change the decision
Use “Static assertions vs LLM-as-Judge vs human calibration — which fits which scenario” as an A/B test: keep the same question while removing repeated background, compressing format, and trimming irrelevant history. Compare answer quality, latency, and Token count.
From “Code Grader” to “Model Grader (LLM-as-Judge)”
“Code Grader” grounds the problem in “Automatically evaluates correctness using programmatic logic Dimension Details Methods String matching, regex, static analysis (AST), unit test pass/fail, tool-call validation (correct API and parameters) Pros…”. “Model Grader (LLM-as-Judge)” then moves it toward “Uses another LLM to score against a rubric Dimension Details Methods Feed the Agent's output and a scoring Rubric to a judge LLM, which returns a score and rationale based on predefined criteria Pros Can evalua…”. Together, they show that the lesson is not just a conclusion to remember, but a claim with conditions.
Carry the judgment into the next situation
For long text, keep what can change the conclusion before compressing format and history. A larger context is worth its cost only when the added information is useful.
- “Code Grader”: Automatically evaluates correctness using programmatic logic Dimension Details Methods String matching, regex, static analysis (AST), unit test pass/fail, tool-call validation (correct API and parameters) Pros…
- “Model Grader (LLM-as-Judge)”: Uses another LLM to score against a rubric Dimension Details Methods Feed the Agent's output and a scoring Rubric to a judge LLM, which returns a score and rationale based on predefined criteria Pros Can evalua…
- “The closing point”: LLM Judge (Model Grader): evaluates code quality, readability, and adherence to best practices
The final “The closing point” brings the discussion to “LLM Judge (Model Grader): evaluates code quality, readability, and adherence to best practices”. The useful thing to carry forward is knowing which judgments must be revisited when input, scale, or risk changes.
I turned one judgment from this article into a small experiment I could run today. Knowing what to observe next is more useful than simply remembering the conclusion.
After reading this, I first looked for the conditions behind the idea instead of copying the method into a project. That order made the later trade-offs much clearer.
When this judgment reaches real work, which constraint should be added first? I am curious which step matters most between reading and the first practical attempt.
No discussion on this article yet.