Special Topic · AI Product Psychology: Design the Feeling

Mental Models: Users Brought the Wrong Manual

Treat AI as a search engine, a database, a learning apprentice, or a calculator—four mismatches, four kinds of bad reviews. Spot-the-difference across four dialogues, then three correction moves: empty-state examples, boundary-first, and memory visibility

THE QUESTION THIS PAGE ANSWERS

ANSWER FIRST

What is the key idea behind “Mental Models: Users Brought the Wrong Manual”?

Treat AI as a search engine, a database, a learning apprentice, or a calculator—four mismatches, four kinds of bad reviews. Spot-the-difference across four dialogues, then three correction moves: empty-state examples, boundary-first, and memory visibility

DECISION RULE

Turn taste into a behavior the product can repeat. The useful outcome is not a nice opinion. It is a visible rule, a small example, and a way to tell when the experience falls below the bar.

TRY NEXT

Capture one before-and-after example that shows the quality bar without extra explanation.

WATCH FOR

Polish that improves the surface while leaving the user's uncertainty untouched.

Four wrong manuals · wholesale source of bad reviews

The chat box looks like a search box, answers look like an encyclopedia, the UI looks like a patient support agent. Users match four old manuals by appearance—and each one manufactures a precise class of bad review.

Wrong manual ①
Used as a search engine
ExpectResults have sources; information is real-time
RealityAnswers from memory, knowledge has a cutoff, no links. Chapter Zero already said “search gives you shelves; AI gives you conclusions”—but users never took that class
Review“The links you gave me don’t open” · “Can’t even look up today’s news”
Wrong manual ②
Used as a database
ExpectAsk and look up; if it’s missing, say so
RealityA probability-completion machine—when it can’t find something, it invents a plausible answer
Review“I asked for our Q3 revenue and it just blurted a number—scared me to death”
Wrong manual ③
Used as a learning apprentice
ExpectTell it once and it remembers; the more you use it, the more it gets you
RealityParameters are frozen; a new session means amnesia—“gets me better over time” is just notes stuffed back into context
Review“Told it eight hundred times not to use tables—turns around and forgets. Attitude problem”
Wrong manual ④
Used as a calculator
ExpectMath is supposed to be correct, period
RealityLanguage models emit numbers by token probability; long formulas and multi-line sums often fail. The right move is to have it write code to compute
Review“Can’t even sum 37 numbers—and this is ‘intelligent’?”
Spot-the-difference drill · name the manual in their hands

Four dialogues—the accident has already happened. First decide which manual the user is holding, then read the explain: each one includes a product fix. Diagnosis is only the start.

Four dialogues, four misdiagnosis scenes 0 / 4 judged
The clues are in the chat: see what the user wants, then what the AI actually gave.
Correction move 1 · turn the empty state into a classroom

You can’t fix a mental model with a help doc—nobody reads it. You teach in the cracks of the usage path, and the first scene is the blank first screen of a new session: the user hasn’t typed a word yet, the manual isn’t open—what this screen says decides which book they pick up. Three first-screen schemes—flip them yourself, and watch the UI on the right and first-question quality below change.

Empty-state designer Flip it
New-user first-question quality
24
Your AI assistant · New session
Message the AI…
Correction move 2 · let users see “Remembered” with their own eyes

The apprentice-manual review phrase is “I told you and you didn’t listen.” Both products below agreed to the user’s request—the difference is how they agreed. Tap the version you think can kill that review line.

“Remembered” visibility duel Pick one
Both versions’ users said the same sentence. Scroll down to see what happened three days later.
Version A
From now on don’t reply with tables—I paste straight into WeChat groups.
Got it.
Three days later · new session
Help me compare these three projectors.
Here’s a comparison table:
ModelBrightnessPrice
X1800 lumens2299
X21200 lumens3499
Version B ✓ Saved preference: avoid tables
From now on don’t reply with tables—I paste straight into WeChat groups.
Got it—I’ll use bullet lists instead.
Three days later · new session
Help me compare these three projectors.
Per your preference · no tables · X2 is brightest—clear even in a daytime living room; · X1 is cheapest, wins on light and easy storage; · X3 has the smoothest system—pick it for binge-watching.
Correction move 3 · state the boundary before the mistake

Time-sensitive questions are a hot zone for the search-engine manual—spot-the-difference dialogue one crashed exactly that way. Both AIs say the same boundary text, word for word; the only difference is when they say it. Pick the version that better protects the mental model.

Boundary-first duel Pick one
Same question, same knowledge cutoff, two timings.
Version A · explain after the miss
What’s today’s USD/CNY exchange rate?
USD/CNY is about 7.24.
User checks the bank app: actual 7.09
That’s wrong—why such a gap?
Sorry. My knowledge cuts off at June 2025—I can’t get live rates. What I gave was an old value from training data.
Version B · declare before answering
What’s today’s USD/CNY exchange rate?
This is a live-data question: my knowledge cuts off at June 2025—for this kind of ask, turn on web search. Web searchOn
Looked it up live: today’s USD/CNY midpoint is 7.09 (source: China Foreign Exchange Trade System, updated 10:15).
The calculator manual gets the same prescription: sums, stats, date math—route them to a code interpreter, and label the UI “computed with code.” Error rates drop, and users also see the point: the trustworthy path is calling tools; the barehanded arithmetic was never meant to be trusted.
Sources and further reading: The mental-model idea comes from Craik (1943); Norman brought it into product design in The Design of Everyday Things: users act on their own model—wrong model, wrong action. Memory visibility maps to the first of Nielsen’s ten usability heuristics—“visibility of system status”; putting examples in the empty state also stacks Cialdini’s social proof from Influence: The Psychology of Persuasion: others ask this way, so I will too. Evidence that a first miss triggers abandonment is in Lesson 6 on algorithm aversion (Dietvorst et al., 2015).

Turn the feeling in “Four wrong manuals · wholesale source of bad reviews” into a judgment

“The chat box looks like a search box, answers look like an encyclopedia, the UI looks like a patient support agent.” points out that AI has lowered the bar for making something usable. The skill readers need is noticing what is wrong and turning that feeling into an actionable requirement.

Watch the user's next action, not just the surface

Turn “Four dialogues—the accident has already happened.” into observable questions: does the user know what happened, what to do next, and how to recover from an empty or failed state? Does the hierarchy make the important information visible first?

  • Bad reviews: check the mental model first : users run AI on an old product’s manual—name the book in their hands before you talk repair
  • Don’t write a welcome on the empty state : put three examples that model the right ask, plus one boundary line
  • Memory must be visible : make “Remembered” a UI object you can see and delete—verbal assent won’t fix apprentice mismatch

Pretty is not the same as usable

Apply “Time-sensitive questions are a hot zone for the search-engine manual—spot-the-difference dialogue one crashed exactly that way.” to a second screen or flow. Record one moment of hesitation and the user action after the change; observable behavior is stronger evidence than polish alone.

From “Four wrong manuals · wholesale source of bad reviews” to “Spot-the-difference drill · name the manual in their hands”

“Four wrong manuals · wholesale source of bad reviews” grounds the problem in “The chat box looks like a search box, answers look like an encyclopedia, the UI looks like a patient support agent. Users match four old manuals by appearance—and each one manufactures a precise class of bad re…”. “Spot-the-difference drill · name the manual in their hands” then moves it toward “Four dialogues—the accident has already happened. First decide which manual the user is holding, then read the explain: each one includes a product fix. Diagnosis is only the start”. Together, they show that the lesson is not just a conclusion to remember, but a claim with conditions.

Carry the judgment into the next situation

For experience work, turn abstract impressions into user actions: did the person understand the state, find the next step, recover from an error, and want to continue?

  • “Four wrong manuals · wholesale source of bad reviews”: The chat box looks like a search box, answers look like an encyclopedia, the UI looks like a patient support agent. Users match four old manuals by appearance—and each one manufactures a precise class of bad re…
  • “Spot-the-difference drill · name the manual in their hands”: Four dialogues—the accident has already happened. First decide which manual the user is holding, then read the explain: each one includes a product fix. Diagnosis is only the start
  • “The closing point”: State boundaries before the miss : surface weak spots like freshness and arithmetic early, and give a tool switch as the way out

The final “The closing point” brings the discussion to “State boundaries before the miss : surface weak spots like freshness and arithmetic early, and give a tool switch as the way out”. The useful thing to carry forward is knowing which judgments must be revisited when input, scale, or risk changes.

✅ What this lesson wants to share

  • Bad reviews: check the mental model first: users run AI on an old product’s manual—name the book in their hands before you talk repair
  • Don’t write a welcome on the empty state: put three examples that model the right ask, plus one boundary line
  • Memory must be visible: make “Remembered” a UI object you can see and delete—verbal assent won’t fix apprentice mismatch
  • State boundaries before the miss: surface weak spots like freshness and arithmetic early, and give a tool switch as the way out
Mark as learned Your reading progress updates automatically
← PreviousNext →

Keep reading

The next useful article in the thread.

ARTICLE DISCUSSION

Leave one useful thought here.

Keep the idea that clicked, the question that stayed open, or a small note for the next learner.

Discussing Mental Models: Users Brought the Wrong Manual AI Product Psychology: Design the Feeling
3discussionsArticle discussion · synced with the Circle
View in the learning circle
AM
Asha MorganContent editor
INSIGHTField note

I turned one judgment from this article into a small experiment I could run today. Knowing what to observe next is more useful than simply remembering the conclusion.

ARTICLE DISCUSSION7 helpful
LH
Lin HarperIndie developer
INSIGHTInsight

After reading this, I first looked for the conditions behind the idea instead of copying the method into a project. That order made the later trade-offs much clearer.

ARTICLE DISCUSSION5 helpful
KM
Kiki MooreProduct operations
QUESTIONQuestion

When this judgment reaches real work, which constraint should be added first? I am curious which step matters most between reading and the first practical attempt.

ARTICLE DISCUSSION4 helpful