Why Can't Some AIs See Pictures?
"Can talk" and "can see" are two different skills. Send the same cat photo to two models, see the gap, then unpack how the "eyes" work
THE QUESTION THIS PAGE ANSWERS
ANSWER FIRSTWhy Can't Some AIs See Pictures?
"Can talk" and "can see" are two different skills. Send the same cat photo to two models, see the gap, then unpack how the "eyes" work
Make the claim earn its place. Use this page as a decision aid, not a definition to memorize. Connect the idea to one real task, one observable result, and one failure that would change your mind.
Write one question you could answer with evidence after trying this idea.
A conclusion that sounds complete but leaves the key assumption untested.
Because "speaking" and "seeing" are two independent skills. A language model natively understands only text; to make it see, you have to fit it with an extra pair of "eyes." The ones that have them are called multimodal models — the ones that don't can only apologize to your photo.
"Sorry, I can't view images. If you describe what's in the picture, I'll be happy to help analyze it."
"An orange cat is lounging on a bay window in the sun, right front paw resting on a ball of yarn. Judging by the physique… this one might need to watch its diet 😄"
Step 1: chop the image up
The "eyes" (a vision encoder) slice the image into many small squares, translating each into "words" the model understands — a picture becomes a special kind of "text."
Step 2: continue it like a sentence
Once translated, an image is no different from a passage of text to the model, and it "finishes the sentence" as usual. So under the hood, image understanding is still that same sentence-finishing machine.
Why not fit eyes on everything?
Fitting eyes means retraining on massive amounts of paired image-and-text data — expensive, and the model gets bigger and pricier. Many use cases never need vision, and text-only models are faster and cheaper.
Put “A Little Experiment · Send the Same Cat Photo to Two Models” back into its constraints
“Because "speaking" and "seeing" are two independent skills .” shows that a model, license, access route, or leaderboard is information—not an answer outside context. The real choice depends on task, data boundary, latency, quality floor, and operating cost.
Write elimination criteria before chasing the top score
The comparison in “The "eyes" (a vision encoder) slice the image into many small squares, translating each into "words" the model understands — a picture becomes a special kind of "text."” should use the same real inputs while observing correctness, failure behavior, response time, and cost. A model leading a public leaderboard may still fail your license, privacy, or peak-latency constraints.
- Speaking ≠ seeing : vision requires extra "eyes" (a vision encoder)
- How the eyes work : chop the image up, translate it into "special text," then finish the sentence as usual
- No eyes ≠ outdated : faster and cheaper, and plenty for many use cases
Without a test set, there is no reliable winner
Start with “Fitting eyes means retraining on massive amounts of paired image-and-text data — expensive, and the model gets bigger and pricier .”: choose inputs that could genuinely change the decision and write down one counterexample that would reverse your choice. That is more useful than memorizing a single ranking.
From “A Little Experiment · Send the Same Cat Photo to Two Models” to “How It Works · How the "Eyes" Get Fitted”
“A Little Experiment · Send the Same Cat Photo to Two Models” grounds the problem in “📤 Send the same cat photo to both models Model A · Text-only model (no eyes fitted) 🖼️ cat-photo.jpg "Sorry, I can't view images. If you describe what's in the picture, I'll be happy to help analyze it." Mode…”. “How It Works · How the "Eyes" Get Fitted” then moves it toward “The "eyes" (a vision encoder) slice the image into many small squares, translating each into "words" the model understands — a picture becomes a special kind of "text."”. Together, they show that the lesson is not just a conclusion to remember, but a claim with conditions.
Carry the judgment into the next situation
For model selection, write non-negotiable constraints from the real task first. Compare quality, failure behavior, latency, licensing, and cost on the same inputs; use a leaderboard only as a starting point.
- “A Little Experiment · Send the Same Cat Photo to Two Models”: 📤 Send the same cat photo to both models Model A · Text-only model (no eyes fitted) 🖼️ cat-photo.jpg "Sorry, I can't view images. If you describe what's in the picture, I'll be happy to help analyze it." Mode…
- “How It Works · How the "Eyes" Get Fitted”: The "eyes" (a vision encoder) slice the image into many small squares, translating each into "words" the model understands — a picture becomes a special kind of "text."
- “The closing point”: Seeing and drawing are two different things : an all-in-one app is really several models dividing the work
The final “The closing point” brings the discussion to “Seeing and drawing are two different things : an all-in-one app is really several models dividing the work”. The useful thing to carry forward is knowing which judgments must be revisited when input, scale, or risk changes.
✅ What this page wants to share with you
- Speaking ≠ seeing: vision requires extra "eyes" (a vision encoder)
- How the eyes work: chop the image up, translate it into "special text," then finish the sentence as usual
- No eyes ≠ outdated: faster and cheaper, and plenty for many use cases
- Seeing and drawing are two different things: an all-in-one app is really several models dividing the work
I turned one judgment from this article into a small experiment I could run today. Knowing what to observe next is more useful than simply remembering the conclusion.
After reading this, I first looked for the conditions behind the idea instead of copying the method into a project. That order made the later trade-offs much clearer.
When this judgment reaches real work, which constraint should be added first? I am curious which step matters most between reading and the first practical attempt.
No discussion on this article yet.