Vocabulary & Trie: How Tokenizers Cut Words
You saw tokenization in the LLM fundamentals part—now the underbelly: how a prefix tree recognizes “五花肉” (streaky pork) as one chunk. Walk a Trie and tokenize by hand
THE QUESTION THIS PAGE ANSWERS
ANSWER FIRSTWhat is the key idea behind “Vocabulary & Trie: How Tokenizers Cut Words”?
You saw tokenization in the LLM fundamentals part—now the underbelly: how a prefix tree recognizes “五花肉” (streaky pork) as one chunk. Walk a Trie and tokenize by hand
Make the claim earn its place. Use this page as a decision aid, not a definition to memorize. Connect the idea to one real task, one observable result, and one failure that would change your mind.
Write one question you could answer with evidence after trying this idea.
A conclusion that sounds complete but leaves the key assumption untested.
Suppose the vocabulary has: 五, 五月, 五花肉, 今天, 天, 天气, 吃, 花, 好. Hang them by first character, then second… shared prefixes share branches—“五月” and “五花肉” crowd the same “五” branch. A green ✓ means “a complete word ends here”; note “五→花” has no ✓—it’s only a waypoint (“五花” isn’t a word).
🧩 BPE: repeatedly merge the “most co-occurring character pairs” into chunks
Real tokenizers (GPT and DeepSeek both use BPE) build vocabularies more wildly: shatter text into tiny fragments, count which two fragments sit next to each other most, glue them into the vocab; count again, glue again, tens of thousands of times. Common combos “grow” into big tokens—same idea as Trie’s “common words stored whole.”
So the phenomena you’ve seen make sense: “的” and “，” always cost one token; rare characters get split into several. That’s also why Chinese usually costs more tokens than English—most vocabularies train on English-heavy corpora, so English common words earn whole tokens while Chinese gets fewer, forcing more cuts. Same sentence, Chinese bill often higher—roots in who that “dictionary” vocabulary organized.
How “First, the organizing · how a vocabulary hangs as a tree” changes an answer
“Suppose the vocabulary has: 五 , 五月 , 五花肉 , 今天 , 天 , 天气 , 吃 , 花 , 好 .” shows that a model does not process the “word count” we see. It processes Token pieces. Tokenization affects input length, how much context fits, and how much computation a request consumes.
Length, information, and context are different
As “Real tokenizers (GPT and DeepSeek both use BPE ) build vocabularies more wildly: shatter text into tiny fragments, count which two fragments sit next to each other most , glue them…” grows, separate three questions: how many Tokens the text becomes, which pieces can change the current decision, and whether older material has fallen outside the context window. Removing repetition is often more useful than simply making the window larger.
- A vocabulary is a dictionary : it organizes all tokens; tokenization is “look up the dict and cut the sentence into chunks”
- Trie organizes by shared prefixes : longest-match becomes one lookup per step—no backtracking the whole list
- Greedy longest match : go as deep as you can; when stuck, back up to the nearest word end and cut
Keep what can change the decision
Use “So the phenomena you’ve seen make sense: “的” and “，” always cost one token;” as an A/B test: keep the same question while removing repeated background, compressing format, and trimming irrelevant history. Compare answer quality, latency, and Token count.
From “First, the organizing · how a vocabulary hangs as a tree” to “Under the hood · real LLMs use BPE—same idea”
“First, the organizing · how a vocabulary hangs as a tree” grounds the problem in “Suppose the vocabulary has: 五 , 五月 , 五花肉 , 今天 , 天 , 天气 , 吃 , 花 , 好 . Hang them by first character, then second… shared prefixes share branches—“五月” and “五花肉” crowd the same “五” branch. A green ✓ means “a comple…”. “Under the hood · real LLMs use BPE—same idea” then moves it toward “Real tokenizers (GPT and DeepSeek both use BPE ) build vocabularies more wildly: shatter text into tiny fragments, count which two fragments sit next to each other most , glue them into the vocab; count again…”. Together, they show that the lesson is not just a conclusion to remember, but a claim with conditions.
Carry the judgment into the next situation
For long text, keep what can change the conclusion before compressing format and history. A larger context is worth its cost only when the added information is useful.
- “First, the organizing · how a vocabulary hangs as a tree”: Suppose the vocabulary has: 五 , 五月 , 五花肉 , 今天 , 天 , 天气 , 吃 , 花 , 好 . Hang them by first character, then second… shared prefixes share branches—“五月” and “五花肉” crowd the same “五” branch. A green ✓ means “a comple…
- “Under the hood · real LLMs use BPE—same idea”: Real tokenizers (GPT and DeepSeek both use BPE ) build vocabularies more wildly: shatter text into tiny fragments, count which two fragments sit next to each other most , glue them into the vocab; count again…
- “The closing point”: Why Chinese costs more tokens : English-leaning vocabularies, fewer whole Chinese tokens, so more cuts
The final “The closing point” brings the discussion to “Why Chinese costs more tokens : English-leaning vocabularies, fewer whole Chinese tokens, so more cuts”. The useful thing to carry forward is knowing which judgments must be revisited when input, scale, or risk changes.
What this lesson wants to share
- A vocabulary is a dictionary: it organizes all tokens; tokenization is “look up the dict and cut the sentence into chunks”
- Trie organizes by shared prefixes: longest-match becomes one lookup per step—no backtracking the whole list
- Greedy longest match: go as deep as you can; when stuck, back up to the nearest word end and cut
- BPE shares the same idea: repeatedly merge the most co-occurring pairs into chunks—common stays cheap, rare gets shattered
- Why Chinese costs more tokens: English-leaning vocabularies, fewer whole Chinese tokens, so more cuts
I turned one judgment from this article into a small experiment I could run today. Knowing what to observe next is more useful than simply remembering the conclusion.
After reading this, I first looked for the conditions behind the idea instead of copying the method into a project. That order made the later trade-offs much clearer.
When this judgment reaches real work, which constraint should be added first? I am curious which step matters most between reading and the first practical attempt.
No discussion on this article yet.