The smallest useful mental model: next-token prediction
Use a simple sentence-completion experiment to build a durable model of what an LLM is doing. It is not the whole story, but it explains why context, wording, probability, and fluent mistakes matter.
THE QUESTION THIS PAGE ANSWERS
ANSWER FIRSTWhat is the key idea behind “The smallest useful mental model: next-token prediction”?
Use a simple sentence-completion experiment to build a durable model of what an LLM is doing. It is not the whole story, but it explains why context, wording, probability, and fluent mistakes matter.
Simple does not mean simplistic. Next-token prediction is a practical debugging lens. When an answer surprises you, inspect which continuation the prompt made likely instead of assuming the model followed your intention.
Rewrite one request so the desired next step is unambiguous.
Treating a fluent continuation as proof that the model understood your goal.
Forget AI for a second. The three sentences below are all unfinished. Tap each one open and see whether the ending that pops into your head matches what most people say.
Pick an opening line and click "Let the AI continue". The box below is the multiple-choice question in its head at every step, slowed down for you. Two things to watch. First, it does not write one word at a time: "wonton" arrives as won + ton, "check-up" as check + - + up. That unit is called a token — sometimes a whole word, sometimes just a fragment of one; a later lesson covers how the pieces are cut. Second, the length of the bars: at some steps the preceding text all but forces its hand, at others you can watch it waver. Run the same opening a few times — where the bars are close, it doesn't always take the longest one.
You might ask: I get the sentence-finishing part, but the AI and I are having a back-and-forth conversation, aren't we? Actually, chat is just sentence-finishing in new packaging —
Why can it write essays?
It just finishes sentences at length, fluently. Give it an opening (a topic and requirements), and it continues token by token until a whole piece comes out.
Why does it spout nonsense?
It only cares whether a continuation sounds right, not whether it is right. A smooth-sounding falsehood beats a clumsy truth in its selection. This matters a lot — it gets its own lesson later.
Why different answers to the same question?
In the demo above it didn't always take the longest bar — where a few candidates are close in likelihood, it rolls a die weighted by those odds and wanders down a different fork of the sentence. At a step where one bar reaches 99%, any roll gives the same result, so the differences only ever show up where it was already wavering.
You may have heard that "AI gets even elementary-school math wrong". Now that you know about finishing sentences, you can solve this mystery yourself — it was never "calculating"; it's finishing sentences. Send it both of these problems and see how different the candidates in its head look.
How “Step 1 · You were born knowing this game” changes an answer
“Forget AI for a second.” shows that a model does not process the “word count” we see. It processes Token pieces. Tokenization affects input length, how much context fits, and how much computation a request consumes.
Length, information, and context are different
As “Pick an opening line and click "Let the AI continue".” grows, separate three questions: how many Tokens the text becomes, which pieces can change the current decision, and whether older material has fallen outside the context window. Removing repetition is often more useful than simply making the window larger.
- AI = a super sentence-finishing machine : all it does is endlessly guess "what's the next token"
- Chat is just sentence-finishing repackaged : your message is the first half; it supplies the second half
- It has no thoughts of its own : what it has are language patterns distilled from oceans of text
Keep what can change the decision
Use “You may have heard that "AI gets even elementary-school math wrong".” as an A/B test: keep the same question while removing repeated background, compressing format, and trimming irrelevant history. Compare answer quality, latency, and Token count.
From “Step 1 · You were born knowing this game” to “Step 2 · Watch the AI finish a sentence with your own eyes”
“Step 1 · You were born knowing this game” grounds the problem in “Forget AI for a second. The three sentences below are all unfinished. Tap each one open and see whether the ending that pops into your head matches what most people say”. “Step 2 · Watch the AI finish a sentence with your own eyes” then moves it toward “Pick an opening line and click "Let the AI continue". The box below is the multiple-choice question in its head at every step , slowed down for you. Two things to watch. First, it does not write one word at a t…”. Together, they show that the lesson is not just a conclusion to remember, but a claim with conditions.
Carry the judgment into the next situation
For long text, keep what can change the conclusion before compressing format and history. A larger context is worth its cost only when the added information is useful.
- “Step 1 · You were born knowing this game”: Forget AI for a second. The three sentences below are all unfinished. Tap each one open and see whether the ending that pops into your head matches what most people say
- “Step 2 · Watch the AI finish a sentence with your own eyes”: Pick an opening line and click "Let the AI continue". The box below is the multiple-choice question in its head at every step , slowed down for you. Two things to watch. First, it does not write one word at a t…
- “The closing point”: Sounding right ≠ being right : the single most useful thing to keep in mind when using AI — next lesson digs in
The final “The closing point” brings the discussion to “Sounding right ≠ being right : the single most useful thing to keep in mind when using AI — next lesson digs in”. The useful thing to carry forward is knowing which judgments must be revisited when input, scale, or risk changes.
✅ What this lesson wants to share with you
- AI = a super sentence-finishing machine: all it does is endlessly guess "what's the next token"
- Chat is just sentence-finishing repackaged: your message is the first half; it supplies the second half
- It has no thoughts of its own: what it has are language patterns distilled from oceans of text
- Its math relies on "memory", not "calculation": common problems are memorized cold; unseen ones get a plausible-looking guess
- Sounding right ≠ being right: the single most useful thing to keep in mind when using AI — next lesson digs in
Once I pictured a model choosing the next token from its current context, many “why doesn't it know?” questions became “did I give it usable clues?” That shift has been surprisingly helpful.
For someone new to models, is it better to explain tokens with a Chinese segmentation example or show the difference between English and code first? The intuition seems different across languages.
I would love a small experiment where two equivalent sentences use noticeably different token counts. For anyone budgeting API costs, that would land better than an abstract explanation.
No discussion on this article yet.