Free · structured · practical
Search 中文 Start at lesson 1
Courses/They Will Quiz You/LLM Fundamentals · 30 Tough Questions
Part 1 · The Model Under the Product

LLM Fundamentals · 30 Tough Questions

Each with intent, framework, and bonus points: probability prediction / message list / hallucination explanation / RAG vs retraining / Temperature / context window

THE QUESTION THIS PAGE ANSWERS

ANSWER FIRST

What is the key idea behind “LLM Fundamentals · 30 Tough Questions”?

Each with intent, framework, and bonus points: probability prediction / message list / hallucination explanation / RAG vs retraining / Temperature / context window

DECISION RULE

Make the claim earn its place. Use this page as a decision aid, not a definition to memorize. Connect the idea to one real task, one observable result, and one failure that would change your mind.

TRY NEXT

Write one question you could answer with evidence after trying this idea.

WATCH FOR

A conclusion that sounds complete but leaves the key assumption untested.

How to use this page
Each question shows who's asking. They're testing the same knowledge, but they each want to hear something different.
🎙 InterviewerWants to verify whether you truly understand or are just reciting buzzwords
👔 BossWants explanations and commitments
🛠 Tech ColleagueIs probing whether you're worth trusting
Each question has three layers: what they're assessing → answer framework → bonus points. For any part you can't answer, click the linked lesson pages at the bottom to review.
Q1Interviewer
"In your own words, how does ChatGPT and similar LLMs generate responses? Is it actually thinking?"
🎯 What they're assessing
The opening question — it sets the depth of everything that follows. It tests whether you can explain probabilistic next-Token prediction in plain language. Someone who just memorized terms will pile on "Transformer, attention mechanism"; someone who truly understands will walk through the mechanism with a clear example.
🧭 Answer framework
  1. Lead with the essence: An LLM is a massive probabilistic prediction machine. It does one thing at a time: given all existing Tokens, it predicts the probability distribution for the next Token and generates tokens one at a time.
  2. Distinguish training from inference: Training is learning statistical patterns from enormous text data. When you're chatting with it, the parameters are frozen — it's not learning, it's computing.
  3. Address the thinking question directly: It doesn't think in the human sense, yet at sufficient scale it demonstrably exhibits reasoning-like behavior. So neither deify it nor dismiss it as a simple autocomplete.
  4. Ground it with an example: Given "The weather today is really," the model outputs: good 62% / bad 18% / cold 9% — and samples from that distribution. A full response is this action repeated hundreds of times.
⭐ Bonus point Proactively point out that chatting with the model ≠ learning. Many PMs assume the model continuously learns from user conversations. Correcting this misconception shows you've understood it properly.
Organize your answer using these lesson pages → Base Model: Token by Token Training vs. Inference Vocabulary & Training
Q2Interviewer
"It's clearly a chat product — so why is the OpenAI API called chat/completions (completion)? How does multi-turn dialogue actually work?"
🎯 What they're assessing
This is the litmus test for whether you've actually worked with the API. Someone who has only used the ChatGPT web interface can't answer this. Understanding the message list mechanism means you'll be able to handle follow-up questions about context, cost, and Agents.
🧭 Answer framework
  1. Reveal the core: The model is fundamentally a text-completion machine. "Dialogue" means packaging the conversation history in chat-log format and having the model complete the assistant's next turn.
  2. The truth about multi-turn: The model has no memory. Every turn sends the entire message list — system + all previous user/assistant turns + the current question — from scratch.
  3. Add the engineering layer: The model understands conversation format because of Chat Template + SFT instruction tuning — the key step that teaches the base model to "speak."
  4. Elevate one level: Every AI Harness operation (RAG, memory, Agent) is fundamentally about manipulating this message list. Understand it and you've found the entry point to every solution.
⭐ Bonus point Naturally transition to the cost implication: because the full history is resent every turn, multi-turn conversations get progressively more expensive. All downstream cost-optimization solutions are aimed at this.
Organize your answer using these lesson pages → The Mystery of chat/completions Simulating Chat History Chat Template + SFT Why Build Fundamentals
Q3Boss
"Our AI customer service just made up another non-existent refund policy. Explain to me why this happens, and when can we fix it once and for all?"
🎯 What they're assessing
This question isn't testing knowledge — it's testing expectation management. Do you have the nerve to say "we can't fix it completely"? And after saying that, can you immediately offer a reassuring plan with quantifiable commitments? Saying "I'll ask the engineers to look into it" is the worst possible answer.
🧭 Answer framework
  1. Lead with the conclusion, no hedging: Hallucination is an inevitable byproduct of probabilistic prediction. It cannot be fully eliminated, but engineering measures can compress it to a level acceptable for the business.
  2. Then explain the root causes: Two sources: incorrect parametric knowledge (training data was wrong or outdated), and context misinterpretation (the model always picks the most likely continuation, and "most likely" ≠ "most accurate").
  3. Offer a combined solution: RAG to inject real refund policy documents + Prompt constraint "answer only based on the document" + lower Temperature + evaluation harness and human review as backstop.
  4. Give a quantifiable commitment: Define a hallucination rate metric (sampled review), report convergence weekly, and turn "when will it be fixed?" into "when does the metric reach X?"
⭐ Bonus point Add: "Any vendor claiming they can completely eliminate hallucination is overpromising." This helps your boss develop realistic expectations for the entire industry — a core value of an AI PM.
Organize your answer using these lesson pages → LLM Hallucination Demo Summary: Hallucination Types & Root Causes Evaluation + Human Review
Q4Interviewer
"What methods do you know for mitigating hallucination? If resources are limited and you can only deploy one first, which one do you choose and why?"
🎯 What they're assessing
The first half tests breadth of knowledge; the second half is the real test: decision-making ability and cost awareness. There's no standard answer. Someone who jumps straight to an answer fails; someone who first asks "what's the scenario?" passes.
🧭 Answer framework
  1. List all four methods: Prompt constraints (cheapest), RAG (most effective for knowledge hallucination but has cost), lower Temperature (reduces randomness only, doesn't fill knowledge gaps), evaluation + human review (external backstop).
  2. Ask about the scenario first: Is the hallucination mainly fabricating facts, or is the expression unstable? Does the knowledge base change? What's the budget?
  3. Give a conditional answer: Knowledge hallucination (fabricating policies, inventing data) → RAG. Unstable expression → Prompt constraints + low Temperature, nearly zero cost, deploy first.
  4. Add one principle: Regardless of which you choose, establish evaluation first. Without evaluation you can't measure effectiveness — it's wasted effort.
⭐ Bonus point Point out that in real projects all four must be used in combination, staged over time — multiple-choice exists only in interviews. Being able to describe a tempo like "use Prompt and Temperature to stop the bleeding first, then deploy RAG to address the root cause" is rare and impressive.
Organize your answer using these lesson pages → Mitigation 1: Prompt Mitigation 2: RAG Mitigation 3: Temperature Mitigation 4: Evaluation Summary: Decision Framework
Q5Tech Colleague
"The product needs to integrate our latest internal product manual. You're not actually thinking of having us retrain the model, are you?"
🎯 What they're assessing
A half-joking challenge from a tech colleague — actually probing whether you understand the boundary between training and inference. Answer incorrectly (e.g., "Yeah, isn't retraining fine?") and your tech team's trust in you drops immediately. Answer correctly and the collaboration ahead goes much smoother.
🧭 Answer framework
  1. Pick up the joke: No retraining needed. Frozen parameters don't mean knowledge can't get in — the context window is the knowledge entry point.
  2. Give the solution: Use RAG. Chunk the manual, build a vector index, retrieve relevant passages and inject them into context when users ask. Updating the manual only requires rebuilding the index — active in a day, costs orders of magnitude less than retraining.
  3. Clarify the correct use of fine-tuning: Fine-tuning changes behavioral style (tone, format, domain vocabulary) — it's not the right tool for injecting time-sensitive knowledge. Once knowledge enters the weights, every update requires another training run.
  4. Show cost awareness: RAG also has costs: each request uses more tokens, latency goes up. It needs to be paired with caching, routing, and precise chunking for optimization.
⭐ Bonus point Being able to state RAG's trade-offs and optimization strategies (semantic caching, keyword triggers, model routing) shows you've actually done the math. "Use RAG" is something everyone can recite.
Organize your answer using these lesson pages → Training vs. Inference RAG Retrieval Augmentation RAG Trade-offs & Optimization
Q6Interviewer
"What are Temperature and Top-P? How would you set them for your product?"
🎯 What they're assessing
Testing whether you've actually tuned these parameters. The second half — "your product" — is waiting for a scenario-specific answer. Someone who rattles off a fixed value ("just set 0.7") almost certainly has never tuned them in practice.
🧭 Answer framework
  1. Explain the mechanics: Temperature controls how steep the probability distribution is — lower means more deterministic, higher means more spread out. Top-P controls the sampling candidate pool size. Together they determine output randomness.
  2. Give scenario-specific settings: Customer service / factual Q&A / data extraction → lower (0–0.3). Creative copy / brainstorming → higher (above 0.7).
  3. State the boundary: This only mitigates expressive randomness — it doesn't fix knowledge gaps. Even with low Temperature, the model will still confidently make things up.
  4. Provide a validation method: Parameter values should be validated with A/B testing on an evaluation set. Picking one value for the entire product by gut feeling is a bad practice.
⭐ Bonus point Proactively distinguish random hallucination from knowledge hallucination: Temperature only treats the former. Many people can't articulate this boundary.
Organize your answer using these lesson pages → Temperature & Top-P Interactive Demo
Q7Interviewer
"What is a context window? Models are now up to 1M Tokens — is bigger always better?"
🎯 What they're assessing
The first half is a conceptual question; the second half is a trap. Agreeing that "of course bigger is better" falls right into it. This tests cost awareness and whether you know the real limitations of long contexts.
🧭 Answer framework
  1. Clarify the concept: The window = the total number of Tokens the model can see in one pass (input + output). Anything beyond that is truncated — as far as the model is concerned, it never existed.
  2. Expose the trap: A larger window means a larger bill. Tokens are charged by volume. Stuffing everything in makes costs rise linearly.
  3. Add the technical limitation: Long contexts suffer from "lost in the middle": the more information, the more diluted the attention, and recall of content in the middle sections drops significantly.
  4. Give the right approach: A large window is just a capability ceiling. The correct practice is context engineering: retrieve, compress, filter — put only what belongs in the window.
⭐ Bonus point Citing mainstream model window specs (Qwen 1M / Claude 200K / GPT 128K) and noting that window size affects architecture decisions signals that you know the industry well.
Organize your answer using these lesson pages → The Context Window is Key Further Reading (Chapter 2): AI Working Memory
Q8Interviewer
"What exactly is a Token? How is it different from a character? Why should product people even care about this?"
🎯 What they're assessing
Looks like a basics question; what they're really testing is cost sensitivity. Plenty of PMs can say the word Token, but can't estimate what a Prompt actually costs. The second half is waiting for an answer that ties back to money and the window.
🧭 Answer framework
  1. Start with the definition: A Token is the basic unit the model uses to process text. The first step of training is Tokenization — slicing continuous text into units from the vocabulary. One Chinese character is roughly 1 to 3 Tokens; one English word is roughly 1 Token.
  2. Connect it to cost: Inference is billed by Token — input and output both cost money. A Prompt template that looks like 500 characters can actually eat 1,500 Tokens. Multi-turn chats resend the history every turn, so the bill piles up.
  3. Connect it to the window: The context window is also measured in Tokens; overflow gets truncated. The more bloated the template, the less room left for the actual content and history.
  4. Land on an action: Before launch, measure the real Token count of your key Prompts with a Tokenizer. Don't guess from character count.
⭐ Bonus point Point out that Token count is the key to both cost and quality: System Prompt, history, and retrieved content all get concatenated into one blob for the model. Every Token in the window is competing with everything else for space.
Organize your answer using these lesson pages → Summary: Tokens & Context Window Real Token-Counting Examples Billed by Token The Mystery of chat/completions
Q9Interviewer
"What actually goes into an LLM's training data? What's the real relationship between data and model capability?"
🎯 What they're assessing
Testing whether you get "it can only say as much as it has read." Few people can name the corpus mix and the order of magnitude; even fewer can connect data coverage to hallucination.
🧭 Answer framework
  1. Give the mix: Pretraining corpus is mostly web text at about 70%, books about 12%, code about 8%, academic papers and dialogue data about 5% each.
  2. Give a feel for scale: Training a small model on about 1B Tokens is like 750,000 novels — a small library. 15T Tokens is like reading the entire internet two or three times.
  3. Give the conclusion: The quality and diversity of training data set the ceiling of the model's worldview. In domains the data never covered, the model either refuses or makes things up.
  4. Land on a real test: Same obscure-person question: a 270M model fabricates an identity outright, a 1.8B model can only refuse, and only 30B+ gets it right. Knowledge gets covered correctly only when data and parameters are enough.
⭐ Bonus point Add that huge parameter counts do not mean no hallucination: in the course's live test, a 70B Llama still got an obscure person wrong, while a 30B Qwen got it right. Coverage matters more than scale.
Organize your answer using these lesson pages → AI's Food: Training Data
Q10Tech Colleague
"The boss wants to just swap in the biggest-parameter model to fix hallucination. You think that works too? It's just an API change anyway."
🎯 What they're assessing
Probing whether you'll just echo the boss, or you have your own technical judgment. "More parameters = more capability" is the most common outsider intuition. Get this wrong and your tech colleague files you under "faking it."
🧭 Answer framework
  1. Lead with a counterexample: In the course's live test, Llama-3.3 70B mistook singer Lee Ji-eun for an actress of the same name — birth year off by 6 years, representative works all wrong. A 30B Qwen got it right. Bigger parameters do not mean no hallucination.
  2. Name the root cause: The knowledge boundary is set by training-data coverage. For obscure facts the data never covered, even a huge parameter count can only invent.
  3. Give the right approach: Knowledge errors get fixed by injecting real material via RAG. Swapping models does not fix "it isn't in the parameters."
  4. Add the cost math: A bigger model means every Token is more expensive and latency is higher. Attribute the error type first, then pick a plan.
⭐ Bonus point Say "the model has no concept of not knowing — confidence does not drop when the information is wrong." That shows you understand hallucination's nature: no model swap removes this.
Organize your answer using these lesson pages → Parameter-Scale Ladder Tests LLM Hallucination Demo RAG Retrieval Augmentation
Q11Interviewer
"What's the difference between a Base model and a Chat model? What happens if you put a Base model straight into customer service?"
🎯 What they're assessing
Testing whether you understand a model's factory form. Anyone who has touched open-source models can answer immediately. Anyone who has only used the ChatGPT website will stall at "what's a Base model?"
🧭 Answer framework
  1. Define Base: What you get after pretraining is a Token-predicts-Token machine. It only predicts the next most likely Token from the preceding text. It does not understand the question, think through an answer, or look anything up.
  2. Give the consequence: Ask a Base model "Who is Zixia Fairy (紫霞仙子)?" and it will continue in corpus style with something like "Zixia Fairy is gege, gege do you like me" — it is not answering the question at all.
  3. Explain the jump: Base to Chat takes two steps: first agree on a Chat Template dialogue format, then run SFT instruction tuning on a huge set of formatted conversations, so the model learns to answer as an assistant.
  4. Name the essence: After SFT the model is still doing Token prediction. What changed is the training data — formatted dialogue plus high-quality answers.
⭐ Bonus point Tell the history: the original idea was to fake a chat log, leave the assistant slot blank, and let the completion machine fill it in. Knowing that dialogue ability was constructed will light the interviewer up.
Organize your answer using these lesson pages → Base Model: Token by Token Chat Template + SFT Simulating Chat History
Q12Interviewer
"Before GPT we already had CNN, RNN, BERT. Why did GPT end up being the one that broke out?"
🎯 What they're assessing
Depth of understanding of the technical evolution. Reciting a timeline is useless. The interviewer wants to hear where each generation of architecture got stuck, and why the PreTraining paradigm was a qualitative leap.
🧭 Answer framework
  1. Name the old path: Before GPT, models were task-specific — pick the task first, then train. Change the task, change the model.
  2. Point out each limit: CNN can only see words inside a sliding window — relations outside the window are invisible in one step. RNN memory decays turn by turn; information from the start of a long text is almost gone by the end. BERT's bidirectional attention is strong at understanding, but the pretraining objective is fill-in-the-blank, so it is weak at generation.
  3. Give GPT's answer: Causal pretraining's objective is literally "predict the next word," which matches generation naturally. No special task data required, and the larger the scale, the more startling the emergent abilities.
  4. Lift it to a paradigm: PreTraining does Token prediction on almost all text — grammar, common sense, facts, and logic are all byproducts. Train once, transfer the capability to any task. That is the core idea of a Foundation Model.
⭐ Bonus point Add "switching tasks went from retraining to rewriting the Prompt," and tie the architecture story to the context-window lesson as one thread. That shows you see the whole map.
Organize your answer using these lesson pages → GPT's Leap The Context Window is Key
Q13Boss
"Users are complaining the AI support forgets what they said earlier in the chat. You keep saying how smart it is — how does it not even have a memory?"
🎯 What they're assessing
Can you translate "the model has no memory" into an explanation a boss will accept, and immediately give a product-level fix. Saying "that's just how models are" is not an answer.
🧭 Answer framework
  1. Explain the mechanism first: Parameters are completely frozen during a conversation — the model itself has no memory. So-called multi-turn dialogue is the system resending the full history every time so the model can continue it.
  2. Locate this incident: Once history exceeds the context window it gets truncated. Truncated content, to the model, never existed. What the user feels is "it forgot."
  3. Give a product plan: Control history length, keep key facts inside the context, and stop a bloated System Prompt and the history from crowding each other out of the window.
  4. Give the shape of a commitment: Quantify the turn count and character boundary that trigger truncation, and design specifically for extra-long chats. Don't promise "it will never forget again."
⭐ Bonus point Casually correct a deeper misconception: what users say does not get learned into the model. Tell it today "your name is Xiao Ming," reopen the chat tomorrow, and it still doesn't remember. Dialogue only lives in the context.
Organize your answer using these lesson pages → Training vs. Inference Token Truncation Visualization The Truth About Multi-Turn Dialogue
Q14Interviewer
"What's the difference between SFT and pretraining? What are those <|im_start|> markers in the Chat Template for?"
🎯 What they're assessing
A classic follow-up, usually after you mention SFT. Testing whether you have actually looked at the dialogue format, or you just say "just fine-tune it."
🧭 Answer framework
  1. Separate the two stages: Pretraining learns statistical patterns from a huge general corpus. SFT keeps training on formatted dialogue data so the model learns to act as an assistant inside that format.
  2. Explain the Template: Borrowing Jinja-style templating, special tokens like <|im_start|> and <|im_end|> wrap each message and distinguish system, user, and assistant roles.
  3. Explain where generation starts: All messages are concatenated into one text blob in that format. The model starts completing after <|im_start|>assistant. What SFT trains is producing a decent answer at that position.
  4. Name the essence: SFT is still Token prediction. The only change is the training data — formatted dialogue plus high-quality answers.
⭐ Bonus point Say "the messages array you pass to the API eventually gets assembled by the Chat Template into one long string of special tokens and sent to the model." Connecting the API layer to the training format is a layer few PMs reach.
Organize your answer using these lesson pages → Chat Template + SFT The Mystery of chat/completions Simulating Chat History
Q15Interviewer
"Why can most AI product needs be met just by writing Prompts these days? Where's the boundary of that?"
🎯 What they're assessing
The first half tests the principle; the second half tests sense of boundary. People who only chant "Prompt engineering is important" can't say why it works, and definitely can't say when it fails.
🧭 Answer framework
  1. Give the causal chain: The model predicts what comes next from what came before. A Prompt is high-quality preceding text — better prefix, better continuation. After PreTraining you don't retrain to switch tasks; you just feed in the task description.
  2. Name the vehicle: All of this rides on a large context window: task instructions, rules, and Few-Shot examples all go into the prefix, with an effect comparable to specialized training.
  3. Give the boundary: Prompt is enough only when the knowledge is already in the training data — format, style, tone. Private knowledge, real-time data, and anything after the knowledge cutoff need RAG. A fixed domain style or reasoning paradigm needs fine-tuning.
  4. Give a debugging habit: When a Prompt change does nothing, first check whether the System Prompt got truncated or history filled the window. The problem is often context management, not how well the Prompt was written.
⭐ Bonus point Quote the decision-matrix line: "Retraining to fix errors is a misconception — try Prompt first, 100× cheaper." That shows you rank decisions by cost.
Organize your answer using these lesson pages → The Context Window is Key Summary: Decision Framework Why Build Fundamentals
Q16Boss
"A competitor's launch event said their new model has already solved hallucination. They solved it — so why are we still messing up every day?"
🎯 What they're assessing
Testing industry judgment and managing up. You have to help the boss see through the hype without putting down your own team, and still offer a plan for "how we prove our own level."
🧭 Answer framework
  1. Lead with the judgment: Hallucination is an inevitable byproduct of probabilistic prediction. It cannot be fully eliminated. Any claim of a complete fix is worth doubting — that's industry consensus, and every vendor faces the same boundary.
  2. Give a way to puncture it: Ask for their evaluation protocol: what's the hallucination rate, what scenarios does the test set cover? A "solved" with no numbers is just marketing copy.
  3. Give a way to prove ourselves: Bring our own evaluation baseline. Quantify hallucination rate on a test set with known answers, and compare with the competitor under the same protocol.
  4. Give an action commitment: Treat hallucination rate as a weekly quantifiable metric and write it into the PRD as an acceptance criterion — manage it like "load time under 2 seconds."
⭐ Bonus point Add this chapter's key insight: the right move is to mitigate with engineering, not wait for a smarter model to make hallucination vanish on its own. Helping the boss set a long-term expectation is the value of an AI PM.
Organize your answer using these lesson pages → Summary: Hallucination Cannot Be Eliminated LLM Hallucination Demo Evaluation + Human Review
Q17Interviewer
"What types of hallucination are there? Don't recite definitions — give me the one concrete example that stuck with you the most."
🎯 What they're assessing
Testing the granularity of your knowledge. Plenty of people can recite the four types; few can land on a concrete example. The second half is there to filter out the people who only memorized.
🧭 Answer framework
  1. List all four types: Factual hallucination invents facts and numbers that don't exist. Source hallucination cites papers, links, or authors that don't exist. Reasoning hallucination starts from a correct premise but botches the steps. Code hallucination calls APIs or functions that don't exist.
  2. Give a sharp example: Ask the model to sort a pandas DataFrame by multiple columns, and it will wrongly apply numpy.sort's stable argument to sort_values. The syntax looks fine; run it and you get a TypeError.
  3. Say why it's dangerous: Code hallucination's damage is that it looks real. The logic is wrong, small test data won't show it, and it blows up in production.
  4. Close on the root cause: The model only has a rough impression of the API and stitches together the most plausible-looking call. Probabilistically reasonable, factually wrong.
⭐ Bonus point Add one more boundary: hallucination is not inevitable. When the Prompt and the training data line up completely, the output is accurate. The problem is mixing sources. That distinction separates you from people who just yell "AI makes stuff up."
Organize your answer using these lesson pages → Summary: Four Types of Hallucination Code Hallucination Case
Q18Tech Colleague
"Your PRD says 'have the model learn our knowledge base.' When I implement this, does the model actually learn anything? You tell me."
🎯 What they're assessing
Nitpicking the wording is really nitpicking the mental model. The word "learn" gives away that you think the model can be changed at runtime. Flub this, and your tech colleague will put a question mark on every PRD you write after that.
🧭 Answer framework
  1. Own the wording: Parameters are completely frozen at inference. The model learns nothing. RAG temporarily injects retrieved documents into context at runtime, then throws them away.
  2. Give an accurate analogy: Parameters are an encyclopedia sealed after it was written. Context is the reference material sitting on the desk. Knowledge-base content goes into the latter.
  3. Spell out the engineering meaning: Because it's a temporary injection, updating the knowledge base only requires rebuilding the index — you don't touch the model. That's exactly RAG's advantage.
  4. Give the correction: Change the PRD to "retrieval injection," and write down how many Tokens get injected per request and what the fallback is when retrieval fails.
⭐ Bonus point Voluntarily quote the course line: "RAG is an open-book exam — the model didn't get smarter, it just got a better starting point." A tech colleague who hears that knows you actually get it.
Organize your answer using these lesson pages → Frozen Parameters Summary: Common Misconceptions RAG Retrieval Augmentation
Q19Interviewer
"Walk me through the full RAG pipeline. How would you decide the chunking granularity?"
🎯 What they're assessing
The pipeline question tests completeness; the chunking follow-up tests hands-on experience. Not many people can separate the build stage from the query stage; even fewer can name a Token count for chunks.
🧭 Answer framework
  1. Split the two stages first: Knowledge-base construction is a one-time offline job — documents get chunked, run through an Embedding model into vectors, and stored in a vector database. The query stage runs in real time on every conversation.
  2. Walk the query path: The user question is vectorized with the same Embedding model, Top-K related chunks are retrieved by cosine similarity, stitched into the Prompt as reference, and the model generates a sourced answer from the injected documents.
  3. Answer the chunking question: Granularity directly hits retrieval quality: too large injects redundant Tokens and wastes money; too small loses context. Best practice is about 512 to 800 Tokens per chunk, using titles and paragraphs as boundaries so the semantics stay intact.
  4. Add a critical detail: Question vectors and document vectors must come from the same Embedding model, or you can't compare similarity in the same semantic space.
⭐ Bonus point Point out that vector retrieval matches on meaning: "how do I return this" and "refund policy" can still match even with different keywords. That's the fundamental reason it beats keyword search.
Organize your answer using these lesson pages → RAG's 7-Step Process Precise Chunking Strategy
Q20Boss
"Didn't you say RAG would fix it? Why did this month's API bill jump instead?"
🎯 What they're assessing
Testing understanding of the cost structure. What RAG buys is accuracy; the price is money. If you can't explain why the bill went up, you never did the math when you shipped RAG.
🧭 Answer framework
  1. Name the cost driver: The jump is the bigger Prompt: each query injects an extra 500 to 2,000 Tokens, so LLM cost multiplies. Vectorization and retrieval themselves are cheap — question embedding is about 0.1 yuan per million Tokens.
  2. Give the most important optimization: Do intent recognition first to decide whether to retrieve at all. About 70% of conversations don't need a document lookup — answering directly is faster and cheaper.
  3. Give the combo: Keyword triggers can skip 30% to 70% of queries. Route simple questions to a small model and overall LLM spend drops 60% to 80%. Similar questions go through a semantic cache — cosine similarity above 0.95 reuses the result directly.
  4. Give a management action: Put RAG trigger rate and cost per conversation on a dashboard and watch them converge weekly.
⭐ Bonus point One sentence that cuts through: the core of production RAG is "when not to use RAG." Filtering and routing are what actually save money. That line will make the boss trust your cost sense.
Organize your answer using these lesson pages → RAG Trade-offs & Optimization RAG Retrieval Augmentation
Q21Tech Colleague
"You keep saying RAG quality is bad and we need to optimize. If I actually handed you the debug, do you know which stages to look at?"
🎯 What they're assessing
Probing whether your grip on the RAG pipeline is real. Someone who can name concrete stages and metrics is someone tech will debug with. Someone who can't is only there to nag about the timeline.
🧭 Answer framework
  1. Check retrieval first: Chunking granularity, Embedding model, similarity threshold — the three most common break points. Watch retrieval hit rate. In the course case, 78% against a target of 85% is a clear miss.
  2. Then check the knowledge base itself: RAG's quality ceiling is the knowledge base. Expired docs and contradictory content will produce wrong answers no matter how accurate the retrieval.
  3. Then look at generation: Injected content is limited, so the model fills in the gaps from training memory — a RAG-plus-hallucination mix that's harder to spot than pure hallucination.
  4. Give a debug handle: Force-display citation sources so every answer traces back to a specific chunk. A bad case then tells you immediately whether retrieval missed or generation drifted.
⭐ Bonus point Flag a counterintuitive risk: when retrieval pulls the wrong document, hallucination doesn't shrink — it just puts on a coat of authoritative citation, and users trust it even more.
Organize your answer using these lesson pages → RAG's Limitations RAG Trade-offs & Optimization Where You'll Get Stuck Without Fundamentals
Q22Interviewer
"Set Temperature to 0 and the output is the same every time. Does that mean the answer is correct?"
🎯 What they're assessing
A trap question, testing the difference between determinism and correctness. Anyone who just nods only understands sampling as "lower is more stable."
🧭 Answer framework
  1. Split the mechanism first: Temperature scales the logits by a constant before softmax. The smaller T, the sharper the distribution. At 0, every step locks onto the highest-probability word.
  2. Spring the trap: Low temperature locks "most likely," and most likely ≠ most correct. If that answer was wrong in the training data, the model will be extremely stably wrong.
  3. Give the boundary: Temperature only affects how each step is sampled. It does not move the knowledge boundary. Things the model doesn't know, it will still invent at low T — just more consistently.
  4. Give the conclusion: Parameter tuning is the cheapest first line of defense. It solves stability of expression. High-stakes factual accuracy still needs RAG or human review.
⭐ Bonus point Point out that being stably wrong is more dangerous than being randomly wrong: reproducible output makes the team think the answer is reliable, so they drop their guard.
Organize your answer using these lesson pages → Temperature's Limitations Summary: Common Misconceptions
Q23Tech Colleague
"Product wants every question to hit the knowledge base first. Latency just jumped two seconds. Does every single one really need a lookup?"
🎯 What they're assessing
Half complaint, half handing you a step. They want you to volunteer to cut unnecessary retrieval. It's also a test of whether you treat RAG as a silver bullet or as a tool.
🧭 Answer framework
  1. Catch it first: Right, we shouldn't retrieve on everything. About 70% of conversations don't need a document lookup. "What's today's date" can just be answered.
  2. Give a filter: Put an intent classifier or simple keyword-trigger rules in front. Only things like "our refund policy" go through RAG — that can skip 30% to 70% of queries.
  3. Give a cache: High-frequency similar questions go through a semantic cache. Similarity above 0.95 returns the cached result and skips the whole retrieval path — latency and cost both drop by about half.
  4. Give the scenario boundary: Small talk and creative scenes were never a fit for RAG. Injecting retrieval just makes the answers stiff.
⭐ Bonus point State the full decision frame: knowledge is time-sensitive, it's a private knowledge base, and being wrong is expensive — only then is the latency of retrieval worth paying. That call is the PM's job, and it belongs in the plan.
Organize your answer using these lesson pages → Keyword Triggers & Semantic Caching When You Must Use RAG
Q24Boss
"The AI quarterly report cited an industry report that doesn't exist. It almost went straight to the board. How do we make sure this never happens again?"
🎯 What they're assessing
An incident-review question. They're listening for whether you dare say "we can't eliminate it, but we can catch it" — and whether your catch-net is specific enough.
🧭 Answer framework
  1. Name it first: Classic source hallucination. Inventing numbers and inventing report titles use the same probabilistic continuation. Telling it "don't make things up" will not fix this.
  2. Give the first line of defense: System Prompt constraint: specific numbers, report names, and institution names that the user didn't provide must be marked "needs verification" — better to leave a blank for a human to fill.
  3. Give the second line: Tiered review. External materials are high-risk: AI does the first draft, and a human must sign off before it goes out.
  4. Close the loop: This bad case goes into the eval set and through attribution → Prompt iteration → regression. That's how recurrence of the same class actually drops.
⭐ Bonus point Give the boss a policy sentence they can ship: every AI-assisted external piece carries a "needs verification" checklist, and it isn't done until the reviewer signs. In high-risk scenes, a human backstop is itself part of the product design.
Organize your answer using these lesson pages → Number Hallucination in Writing Tiered Review & Feedback Loop
Q25Interviewer
"If the System Prompt says 'if you're not sure, say you don't know' — does that actually work? What's the mechanism?"
🎯 What they're assessing
Depth of understanding of Prompt constraints. "It works" or "it doesn't" both fail. The interviewer wants a mechanism-level explanation and the failure conditions.
🧭 Answer framework
  1. Give the mechanism: When the model predicts each Token, every Token in the context is shaping the probability distribution. Constraint words are high-quality prefix: they raise the probability of the "admit I don't know" sequence and suppress the fabrication sequence.
  2. Give when it works: Most effective when the model itself is uncertain about the question — out-of-knowledge questions, vague queries, time-sensitive information.
  3. Give when it fails: When the model is highly confident in a wrong answer, the top candidate is already wrong and constraint words can't intervene. Systematic errors in the training data, and outdated knowledge treated as fact, all sit here.
  4. Give the pairing: In domains that are "highly confident but possibly wrong," Prompt constraints aren't enough. Pair them with RAG to inject real knowledge, or a human-review backstop.
⭐ Bonus point One sentence on the root: the model does not know that it doesn't know. Constraint instructions change the probability distribution; they cannot give it self-awareness. That's the ceiling of Prompt methods.
Organize your answer using these lesson pages → Why Prompt Constraints Work
Q26Interviewer
"You're in charge of an AI Q&A product. How do you quantify its hallucination level? What do you do before launch vs. after?"
🎯 What they're assessing
Testing systems thinking. Scattered "test more" won't cut it. The interviewer wants the full structure — pre-launch baseline, post-launch review, long-term loop — plus concrete metrics.
🧭 Answer framework
  1. Build a baseline before launch: Use a set of questions with known correct answers as a hallucination test set, quantify the rate, and set a gate. Course-case protocol: factual accuracy 94.2%, hallucination rate 5.8% against a target under 3% — miss the target, don't ship.
  2. Tiered review after launch: Auto-route by risk: low-risk goes out, high-risk waits for a human. In finance and healthcare, AI only produces a first draft.
  3. Run a long-term feedback loop: Hallucinations found in review become bad cases and go through attribution → Prompt iteration → regression. In the course case, discovery-to-fix averaged 3.2 days.
  4. Put the metric in the PRD: "Hallucination rate under 3%" should be a quantifiable acceptance criterion, same as "load time under 2 seconds."
⭐ Bonus point Voluntarily name evaluation's blind spot: the test set covers known risk points. Passing eval does not mean comprehensively reliable. Long-tail issues have to be fed back into the test set from live review.
Organize your answer using these lesson pages → Evaluation + Human Review
Q27Tech Colleague
"We only shipped after the eval set went all green, and this week another dozen bad cases popped up. Was that eval a waste of time?"
🎯 What they're assessing
An emotional challenge. Can you explain the natural limits of evaluation and turn bad cases into process assets, instead of getting dragged into a defensive crouch by "waste of time"?
🧭 Answer framework
  1. Answer it head-on: Eval guarantees known risk points don't recur. Long-tail issues outside the test set can still hallucinate. Passing eval has never meant comprehensively reliable.
  2. Give a data view: Look at loop speed, not just this week's new count. Course-case rhythm: 12 new bad cases this week, 47 already fixed and archived, regression pass rate 96.8%.
  3. Give the process: Every bad case goes through four steps: discover, attribute, iterate the Prompt, regress. After the fix it joins the test set, and the eval set thickens as it rolls.
  4. Give the root-cause explanation: The model has no self-check. One wrong Token becomes the base for the next, and error accumulates. Eval plus review is an external correction layer on the output — it catches results, it does not control the generation process itself.
⭐ Bonus point Say "the first three methods change probability; this one intercepts the result." That layering tells a tech colleague immediately that you know where each of the four methods sits.
Organize your answer using these lesson pages → Error Feedback Loop
Q28Interviewer
"The model got a question wrong. How do you decide whether to change the Prompt or do fine-tuning?"
🎯 What they're assessing
A classic debug question, testing attribution. The two fix paths differ by two orders of magnitude in cost. A team that picks the wrong direction burns weeks. The interviewer is watching whether you have an order of operations.
🧭 Answer framework
  1. Split the error type first: Was the context given wrong, or is this knowledge simply not in the parameters? The two fixes are completely different, and the wrong direction wastes a lot of time.
  2. Give the debug order: Try Prompt first: complete the task description, rules, and examples, then retest. If the knowledge is in the training data and the issue is format, style, or tone, Prompt can fix it — 100× cheaper than retraining.
  3. Decide whether you need RAG: Private knowledge, real-time data, anything after the knowledge cutoff — it isn't in the parameters. More Prompt won't help; you need retrieval injection.
  4. Fine-tuning last: A fixed professional-domain style, or a need to change the reasoning paradigm — those are worth fine-tuning. Fine-tuning changes behavioral style; it is the wrong tool for injecting time-sensitive knowledge.
⭐ Bonus point Add a common pit: many "Prompt change did nothing" cases are actually a truncated System Prompt or history filling the window. The problem is context management — you don't need either fix path.
Organize your answer using these lesson pages → Summary: Solution Selection Matrix Prompt or Fine-tune The Context Window is Key
Q29Boss
"A user asked the AI what it had for dinner. It said it cooked tomato-and-egg noodles, and the screenshot is all over the internet. It's a program — why is it lying?"
🎯 What they're assessing
Sounds like a rant; it's actually asking about the nature of hallucination. Answer well and you help the boss build the right mental model of generative AI. Answer "it's a bug, we'll fix it" and you've buried yourself.
🧭 Answer framework
  1. Explain the mechanism first: It has no motive to lie — it's doing probabilistic continuation. After "did you eat?", "I ate" is the highest-frequency reply pattern in the training corpus. When pressed, it invents concrete details by contextual probability.
  2. Name the alignment boundary: Only when the user explicitly challenges it does RLHF alignment training make it admit it has no body. If the user doesn't push, it will keep performing, and confidence does not drop because the content is false.
  3. Give the key insight: Hallucination and creativity share a source. The vivid tomato-and-egg-noodle detail and the copy it writes use the exact same capability. Kill hallucination and you kill creativity with it.
  4. Give a product action: Set strategy by scene: role-play in small talk is harmless; factual scenes need RAG and review to suppress it. One global knife cut does both jobs badly.
⭐ Bonus point Use "two books" to give the boss a mental model: parameters are a sealed encyclopedia; context is the reference on the desk. It is always looking for the most similar continuation in those two books, and if it can't find one it still won't say it doesn't know.
Organize your answer using these lesson pages → Everyday Conversation Hallucination
Q30Interviewer
"You've learned all this LLM material. From what you've seen, what cognitive traps do AI product managers fall into most easily?"
🎯 What they're assessing
A closing open question, testing how internalized the knowledge is. Anyone can list misconceptions. The interviewer is listening for whether you can connect them to the underlying mechanism — and for your self-awareness.
🧭 Answer framework
  1. List the high-frequency traps: Raising Temperature makes it smarter — actually just more random. RAG made the model learn the docs — actually just a temporary runtime injection. The model is calling an API — actually it only output formatted text. It got it wrong so retrain — actually try Prompt first, 100× cheaper.
  2. Dig the shared root: All four come from not separating training from inference. Once parameters are frozen, every runtime method is operating on context.
  3. Lift it to a method: Every engineering operation is, at bottom, manipulation of the message list. Look at any new proposal through that lens and you can tell which layer it acts on and where it stops.
  4. Give a self-positioning: Close with the Dunning-Kruger curve: the most dangerous moment is right after you learn the nouns, when you feel like you understand everything. Keep building, keep getting punched in the face, and you walk from the Peak of Mount Stupid onto the plateau.
⭐ Bonus point Close with the course's own line: weakness and ignorance have never been obstacles to survival — arrogance has. Naturally bringing the conversation to the cognitive layer scores more than reciting ten extra terms.
Organize your answer using these lesson pages → Summary: Common Misconceptions The Message-List Perspective The Dunning-Kruger Effect
One final recommendation
The correct way to use these 30 questions is to say them out loud — to a colleague, a friend, or a recording. Just reading them doesn't count. Wherever you stumble is where you think you understand but don't — click the linked lesson pages and go back to review.

Why “LLM Fundamentals · 30 Tough Questions” can find relevant content

“Each with intent, framework, and bonus points: probability prediction / message list / hallucination explanation / RAG vs retraining / Temperature / context window” moves retrieval beyond storing material: the real question is how to find what is relevant. That decision shapes the input quality of RAG, recommendation, and image-search systems.

Similarity is not the answer

In the flow described by “Each with intent, framework, and bonus points: probability prediction / message list / hallucination explanation / RAG vs retraining / Temperature / context window”, embeddings place items in a comparable semantic space and a neighbor index narrows the search. The final answer still depends on whether the retrieved chunks cover the question, whether the distance metric fits, and whether the evidence is current.

  • Lead with the essence: An LLM is a massive probabilistic prediction machine. It does one thing at a time: given all existing Tokens, it predicts the probability distribution for th…
  • Distinguish training from inference: Training is learning statistical patterns from enormous text data. When you're chatting with it, the parameters are frozen — it's not learning…
  • Address the thinking question directly: It doesn't think in the human sense, yet at sufficient scale it demonstrably exhibits reasoning-like behavior. So neither deify it nor dismi…

Separate findable from relevant

Turn “Each with intent, framework, and bonus points: probability prediction / message list / hallucination explanation / RAG vs retraining / Temperature / context window” into a small test: prepare queries with known answers, record relevance, misses, and distractors, then decide whether chunking, the index, or reranking needs to change.

Take the example one step further

The lesson starts with “Lead with the essence: An LLM is a massive probabilistic prediction machine. It does one thing at a time: given all existing Tokens, it predicts the probability distribution for the next Token and generates tok…” and then moves to “Distinguish training from inference: Training is learning statistical patterns from enormous text data. When you're chatting with it, the parameters are frozen — it's not learning, it's computing”. Reading those two pieces together makes the distinction clearer: which points are facts in the lesson, and which judgments depend on their conditions.

Carry the judgment into the next situation

The same logic applies to retrieval: define what counts as relevant, check whether recall covers the question, and then inspect whether ranking, chunking, or freshness pushed useful evidence out.

  • “LLM Fundamentals · 30 Tough Questions”: Lead with the essence: An LLM is a massive probabilistic prediction machine. It does one thing at a time: given all existing Tokens, it predicts the probability distribution for the next Token and generates tok…
  • “Take it further”: Distinguish training from inference: Training is learning statistical patterns from enormous text data. When you're chatting with it, the parameters are frozen — it's not learning, it's computing
  • “The closing point”: Reveal the core: The model is fundamentally a text-completion machine. "Dialogue" means packaging the conversation history in chat-log format and having the model complete the assistant's next turn

The final “The closing point” brings the discussion to “Reveal the core: The model is fundamentally a text-completion machine. "Dialogue" means packaging the conversation history in chat-log format and having the model complete the assistant's next turn”. The useful thing to carry forward is knowing which judgments must be revisited when input, scale, or risk changes.

Mark as learned Your reading progress updates automatically
← PreviousNext →

Keep reading

The next useful article in the thread.

What You Can Do Now · Hands-onLock That One Job DownFour-step demo of tightening a requirement; three tasks: write a four-line spec, try it on five real inputs, draw a human–AI boundaryPart Recap · RecapRecap (Part B) · Mitigation Strategies + Decision FrameworkFour mitigation strategies compared / common misconceptions / solution selection matrixPart Recap · RecapRecap (Part A) · What LLMs Are + HallucinationsTraining essence / Token / Base→SFT→Chat / four hallucination types and root causesHallucinations & Four Mitigations · ConceptMitigation 4: Evaluation + Human ReviewExternal correction layer — a cold-start fallback strategy (HITL)

ARTICLE DISCUSSION

Leave one useful thought here.

Keep the idea that clicked, the question that stayed open, or a small note for the next learner.

↗
Discussing LLM Fundamentals · 30 Tough Questions The Model Under the Product
3discussionsArticle discussion · synced with the Circle
View in the learning circle
AM
Asha MorganContent editor
INSIGHTField note

I turned one judgment from this article into a small experiment I could run today. Knowing what to observe next is more useful than simply remembering the conclusion.

ARTICLE DISCUSSION7 helpful
LH
Lin HarperIndie developer
INSIGHTInsight

After reading this, I first looked for the conditions behind the idea instead of copying the method into a project. That order made the later trade-offs much clearer.

ARTICLE DISCUSSION5 helpful
KM
Kiki MooreProduct operations
QUESTIONQuestion

When this judgment reaches real work, which constraint should be added first? I am curious which step matters most between reading and the first practical attempt.

ARTICLE DISCUSSION4 helpful