Movement II · What It Actually Is — Lesson 6
Chat: Its Uses and Limitations
Almost nobody explains hallucination correctly. It is not a glitch, not a mystery, and not something the next model release removes. It is what you get when you train a system to be graded like a student sitting an exam.
Learning objective
By the end of this lesson you can state a working theory of why models produce confident falsehoods — naming both the training-side and the evaluation-side cause — and you have a three-rule verification habit you can apply the same day.
6.1 What chat is genuinely excellent at
Start with the capability, because a lesson that opens on limitations trains cynicism rather than judgement. Chat models are very good at drafting, restructuring, explaining, translating between registers, summarising material you supply, and rubber-ducking — talking a problem through until you understand it yourself.
Those tasks share one property, and it is the property that makes them safe:
Rewriting a paragraph in a plainer register: you read it, and you know instantly whether it is right. Recalling last quarter's actual revenue figure: you cannot verify it by reading it, because a wrong number looks exactly like a right one. The first is a good use. The second is the worst possible use, and §6.2 explains precisely why.
6.2 Hallucination, explained properly
A hallucination is plausible-sounding but factually incorrect output.1 That definition is easy. The mechanism is where nearly all popular explanations go wrong, usually by treating it as a mysterious defect that better engineering will eventually remove.
The clearest published account argues that hallucinations are not mysterious at all: they are natural statistical errors arising from how models are trained and — decisively — how they are evaluated.2 The argument has two halves, and the second is the one that matters.
The training half
During pretraining the model learns to predict text. It is not shown a labelled boundary between true statements and false ones, because within the training data, incorrect statements cannot be reliably distinguished from facts.2 What the model acquires is a sense of what text looks like. Plausibility is learnable; truth, from this signal alone, is not. Hallucination rates are correspondingly worse wherever the training data was sparse, contradictory, or low quality — which is exactly where you are most likely to be asking a specialist question.1
The evaluation half — the part almost nobody says out loud
Models are optimised as test-takers, and the tests are scored the way a multiple-choice exam is scored: a correct answer earns a point, a wrong answer earns nothing, and “I don't know” also earns nothing. Under that scoring, guessing is strictly better than abstaining. The published characterisation of the field's benchmarking culture is blunt — an epidemic of penalising uncertain responses.2
The exam analogy, which is the paper's own
Think of a student facing a question they half-remember on an exam with no penalty for wrong answers. Writing something plausible has positive expected value; leaving it blank has exactly zero. A rational student guesses. We then train models on precisely that incentive structure, at enormous scale, and express surprise when they bluff.
The proposed remedy is correspondingly unglamorous and points at us rather than at the models: rather than inventing more hallucination-specific benchmarks, change how the dominant existing benchmarks are scored, so that admitting uncertainty is not penalised.2
The takeaway to carry out of this lesson: an “I don't know” is expensive for a model to produce, because nothing in its training or its grading ever rewarded one. Confidence is not evidence of knowledge. It is evidence of training.
The scoring table that makes bluffing rational
A simple 2×3 payoff table, drawn as an exam mark scheme. Columns: Confident and right, Confident and wrong, Says "I don't know". Row one, Score under standard benchmarks: +1, 0, 0. Row two, What we actually want: +1, −1, 0. Set the "Confident and wrong / 0" cell in the caution colour and draw the eye to it — that single zero is the entire mechanism. Below, one line of caption-weight text: abstaining and being wrong score identically, so guessing dominates.
Aspect 4:3 · 2400×1800 px · Brand palette: navy #00040c, silver-blue #a8c0d8, column gold #a87848, caution #8f3a2a · Style: clean information graphic
6.3 The verification habit
Three rules. They cost very little and they catch nearly everything.
- Never accept a number, a citation, or a proper name without checking it. These are the three categories where a plausible fabrication is indistinguishable from the truth by inspection. Prose you can judge by reading. A page number you cannot.
- Ask for the reasoning separately from the answer. Requesting the working before the conclusion gives you something checkable, and stops the conclusion from anchoring your judgement of the reasoning.
- Give the model the source rather than asking it to recall the source. The single highest-leverage rule in this course.
Rule three deserves its own paragraph, because it is the bridge to everything in Movement IV. A model asked to recall a fact must reconstruct it from statistical regularities. A model handed the document and asked to read it is doing comprehension — a task from §6.1's safe list, where you can verify the answer by looking at the passage it came from. The technique of retrieving relevant documents and supplying them as context has a name, retrieval-augmented generation, and a founding paper from 2020.3 Lesson 15 has you build one.
A caution about the fix
Grounding reduces hallucination; it does not eliminate it. A model given a document can still misread it, over-generalise from it, or blend it with prior training. When you build the pipeline in Lesson 15, the citation back to the source line is not a decoration — it is the mechanism that makes the answer checkable.
6.4 Where this hurts most
Combine §6.1's test with §6.2's mechanism and you can predict your own exposure. Risk is highest where all three of these are true at once:
- Verification is expensive. You cannot tell right from wrong by reading.
- The material is thin in training data. Internal documents, niche regulations, recent events, specialist sub-fields.
- Being wrong is costly. Legal, medical, financial, or safety consequences.
All three together describes a great deal of professional work, which is why the answer is not “don't use it.” The answer is grounding plus verification: supply the source, demand the citation, and check the citation. That is the workflow, and it is available today.
Demonstration protocol
- Ask for citations on a deliberately niche topic — narrow enough that training data is thin. Ask for four sources with authors, titles, and dates.
- Check them live, on camera. Search each one. Some will be real. Some will be plausible and wrong — a real author paired with a title they never wrote is the classic failure and the most instructive.
- Do not gloat. The tone here decides whether the viewer learns a habit or learns contempt. State the mechanism from §6.2 as the explanation.
- Re-run with the source supplied. Same question, with the actual document pasted in. Show the difference, and show the answer pointing back to a specific passage.
- Then break the grounded version too. Ask something the supplied document does not cover and see whether the model says so or fills the gap. This is the most valuable thirty seconds in the lesson.
Key terms introduced here
- Hallucination
- Plausible-sounding but factually incorrect output. A predictable consequence of the training and evaluation regime, not a random defect.
- Grounding
- Supplying the model with source material rather than relying on recall, so that answers can be traced back and checked.
- Retrieval-augmented generation (RAG)
- Retrieving relevant documents and supplying them as context at generation time. Introduced in 2020; built in Lesson 15.
- Calibration
- The match between a system's confidence and its accuracy. Standard benchmark scoring actively discourages it, which is the mechanism this lesson describes.
Evidence at a glance
| Claim | Specifics | Source |
|---|---|---|
| Hallucinations are statistical, not mysterious | Arise from pretraining pressures; incorrect statements are not distinguishable from facts in the training signal | Note 2 |
| Evaluation rewards guessing over abstaining | Models optimised as test-takers; an “epidemic of penalising uncertain responses” | Note 2 |
| Proposed remedy is socio-technical | Rescore existing dominant benchmarks rather than add hallucination-specific ones | Note 2 |
| Rates worsen with poor training data | Sparse, contradictory, or low-quality source material | Note 1 |
| Grounding has a documented method | Retrieval-augmented generation, NeurIPS 2020 | Note 3 |
The misconception to kill
“Newer models don't hallucinate.”
Scale and better training reduce the rate. Neither removes the mechanism.12 As long as the training signal cannot separate true statements from plausible ones, and as long as the dominant benchmarks score a confident wrong answer the same as an honest “I don't know,” the incentive to bluff survives every model release.
Why believing otherwise is expensive: a lower rate with the same mechanism is more dangerous to a careless user, not less. Errors that appear once in fifty outputs get caught. Errors that appear once in five hundred arrive after you have stopped checking.
Self-check
Answer before revealing.4
Give both halves of the explanation for hallucination — the training cause and the evaluation cause.
Training: the model learns from text in which incorrect statements are not distinguishable from facts, so it acquires plausibility rather than truth. Evaluation: benchmarks score a wrong answer and an abstention identically at zero, so guessing dominates abstaining and the model is optimised to bluff.
What single question decides whether a task is a safe one to delegate?
Can I verify the output faster than I could have produced it? Rewriting passes. Recalling a figure I would have to look up anyway fails — because a wrong number is indistinguishable from a right one by inspection.
Why is “give the model the source” more powerful than “tell the model not to make things up”?
Because the instruction does not change the incentive, and the model has no reliable internal signal for which of its outputs are fabricated. Supplying the source converts the task from recall — where plausibility is all it has — into comprehension, where the answer can be pointed back at a passage and checked.
Name the three conditions that together make hallucination most dangerous.
Verification is expensive; the material is thin in training data; and being wrong is costly. Professional specialist work frequently satisfies all three at once, which is why the answer is grounding plus verification rather than abstinence.
A model gives you a citation with a real author and a real journal. What is your next move, and why that one?
Look up the specific title and confirm that author actually wrote it. Real-author, real-journal, invented-title is the classic failure, precisely because every checkable surface detail is correct. Plausibility is what the system optimises for; verifying the surface is not verifying the claim.
Verification checklist
Keep this beside you. Fill the last two fields from real work this week.
Before delegating — can I verify the output faster than I could produce it? If no, what would make that true?
Rule 1 — numbers, citations, and proper names checked? List what you checked and what failed:
Rule 2 — did you ask for reasoning separately from the answer? What did the reasoning reveal?
Rule 3 — did you supply the source, or ask for recall? Rerun with the source and note the difference:
Endnotes
Source tiers: Primary peer-reviewed or archival; Institutional research body or government; Documentation first-party technical documentation; Journalism established newsroom; Trade vendor or aggregator — verify figures independently.
- Institutional Duke University Libraries, “It's 2026. Why Are LLMs Still Hallucinating?”, 5 January 2026, for the working definition of a hallucination as plausible-sounding but factually incorrect output, and for the observation that rates worsen where training data is sparse, contradictory, or low quality. blogs.library.duke.edu
- Primary A. T. Kalai, O. Nachum, S. S. Vempala, E. Zhang, “Why Language Models Hallucinate,” arXiv:2509.04664, submitted 4 September 2025. The paper argues that hallucinations arise from natural statistical pressures in pretraining — where “incorrect statements cannot be distinguished from facts” — compounded by evaluation practices that optimise models as test-takers and penalise expressions of uncertainty, which the authors characterise as an “epidemic of penalizing uncertain responses.” Their proposed mitigation is socio-technical: modify the scoring of existing dominant benchmarks rather than introduce further hallucination-specific evaluations. See also the accompanying research summary, OpenAI, “Why language models hallucinate.” arxiv.org/abs/2509.04664 · openai.com
- Primary P. Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” Advances in Neural Information Processing Systems 33 (2020): 9459–9474; arXiv:2005.11401. arxiv.org/abs/2005.11401
- Primary H. L. Roediger III and J. D. Karpicke, “Test-Enhanced Learning: Taking Memory Tests Improves Long-Term Retention,” Psychological Science 17, no. 3 (2006): 249–255. doi:10.1111/j.1467-9280.2006.01693.x
Bibliography
Primary research
- Kalai, Adam Tauman, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang. “Why Language Models Hallucinate.” arXiv:2509.04664 (2025). arxiv.org
- Lewis, Patrick, et al. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” NeurIPS 33 (2020): 9459–9474. arxiv.org
- Roediger, Henry L., III, and Jeffrey D. Karpicke. “Test-Enhanced Learning: Taking Memory Tests Improves Long-Term Retention.” Psychological Science 17, no. 3 (2006): 249–255.
Institutional and first-party
- Duke University Libraries. “It's 2026. Why Are LLMs Still Hallucinating?” 5 January 2026. blogs.library.duke.edu
- OpenAI. “Why language models hallucinate.” openai.com