AI's Food: Training Data
What does 15T Tokens look like? Corpus composition visualization + data-scale intuition slider
THE QUESTION THIS PAGE ANSWERS
ANSWER FIRSTWhat is the key idea behind “AI's Food: Training Data”?
What does 15T Tokens look like? Corpus composition visualization + data-scale intuition slider
Inspect what the model is being shown. The practical move is to separate instructions, source material, history, tools, and output rules. Once the context is visible, the right fix is usually easier to choose.
Draw the input and output of one small workflow before changing its prompt or model.
Adding more text when the real issue is relevance, ordering, or a missing boundary.
The quality and diversity of training data set the upper bound of a model's worldview.
How “Data Scale & Composition” changes an answer
“What does 15T Tokens look like?” shows that a model does not process the “word count” we see. It processes Token pieces. Tokenization affects input length, how much context fits, and how much computation a request consumes.
Length, information, and context are different
As “What does 15T Tokens look like?” grows, separate three questions: how many Tokens the text becomes, which pieces can change the current decision, and whether older material has fallen outside the context window. Removing repetition is often more useful than simply making the window larger.
Keep what can change the decision
Use “What does 15T Tokens look like?” as an A/B test: keep the same question while removing repeated background, compressing format, and trimming irrelevant history. Compare answer quality, latency, and Token count.
From “Data Scale & Composition” to “Drag the slider · Feel the capability gap across model sizes”
“Data Scale & Composition” grounds the problem in “~1B Tokens · Training data scale for a very small model What a model has read is the ceiling of what it can say. The quality and diversity of training data set the upper bound of a model's worldview”. “Drag the slider · Feel the capability gap across model sizes” then moves it toward “270M Hallucination 270M 0.6B 1.8B 30B 70B 120B 235B 1T 5T+ 270M Gemma-4-e2b Google · Local inference Hallucination Q: Who is Lee Ji-eun (IU)? Please describe her life and key works”. Together, they show that the lesson is not just a conclusion to remember, but a claim with conditions.
Carry the judgment into the next situation
For long text, keep what can change the conclusion before compressing format and history. A larger context is worth its cost only when the added information is useful.
- “Data Scale & Composition”: ~1B Tokens · Training data scale for a very small model What a model has read is the ceiling of what it can say. The quality and diversity of training data set the upper bound of a model's worldview
- “Drag the slider · Feel the capability gap across model sizes”: 270M Hallucination 270M 0.6B 1.8B 30B 70B 120B 235B 1T 5T+ 270M Gemma-4-e2b Google · Local inference Hallucination Q: Who is Lee Ji-eun (IU)? Please describe her life and key works
The final “Finish by testing the claim” brings the discussion to “What does 15T Tokens look like”. The useful thing to carry forward is knowing which judgments must be revisited when input, scale, or risk changes.
I turned one judgment from this article into a small experiment I could run today. Knowing what to observe next is more useful than simply remembering the conclusion.
After reading this, I first looked for the conditions behind the idea instead of copying the method into a project. That order made the later trade-offs much clearer.
When this judgment reaches real work, which constraint should be added first? I am curious which step matters most between reading and the first practical attempt.
No discussion on this article yet.