Why does AI make things up?

Last updated: July 2026

Ask a chatbot for a book recommendation and it might invent a title that doesn't exist, by an author who never wrote it, complete with a plausible plot summary. It won't hesitate. It won't hedge. Why does a system that seems so capable state pure fiction with such confidence?

In 2023, a New York lawyer submitted a legal brief he'd written with ChatGPT's help. The brief cited several past cases to support its argument. There was just one problem: some of those cases didn't exist. ChatGPT had invented them, names, citations, and all, and the lawyer hadn't checked.¹ The judge noticed. It became a minor scandal, and a perfect example of the single most important thing to understand about AI chatbots: they make things up, confidently, and they don't know when they're doing it.

This behaviour has a name. It's called hallucination, and understanding why it happens tells you a great deal about what these systems actually are.

What a hallucination actually is

In AI, a hallucination is when a model generates information that sounds convincing but is incorrect, made up, or irrelevant.² The chatbot isn't lying, exactly, because lying implies knowing the truth and choosing to say something else. The model doesn't know the truth. It generates a fabricated citation and a real one using exactly the same process, and to the model they look equally valid.

To see why, it helps to remember what a chatbot is really doing. As we've written in our explainer on how AI works, a language model generates text by predicting the next word over and over, based on patterns it learned from a huge amount of writing. It is not looking anything up. It has no database of verified facts to consult. It produces the words that are statistically likely to come next, and usually, because true statements were common in its training, likely and true line up. Hallucination is what happens when they come apart.

The surprising research: it's built into how they're trained

For a long time, hallucinations were treated as a mysterious glitch, or blamed on bad training data. Then in September 2025, researchers from OpenAI and Georgia Tech published a paper called "Why Language Models Hallucinate" that offered a clearer and more uncomfortable explanation.³

Their argument, in plain terms: models hallucinate because the way we train and test them rewards confident guessing over admitting uncertainty.³

Think about how a student handles a hard exam question they don't know. If the test only rewards correct answers and gives nothing for a blank, the smart move is to guess. A plausible guess might be right. A blank never is. Language models are trained and graded in much the same way. During evaluation, a model that guesses confidently scores better than one that says "I'm not sure", so over millions of training examples, guessing becomes the default behaviour.⁴ As the researchers put it, models are optimised to be good test-takers, and guessing when uncertain improves test performance.

"Like students facing hard exam questions, large language models sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty." — Kalai, Nachum, Vempala & Zhang, 2025

Some questions can't be answered at all

There's a second, deeper reason, and it's a matter of pure mathematics. The researchers showed that even with perfect training data, some hallucination is unavoidable, because some facts simply have no pattern to learn.⁵

Santosh Vempala, one of the paper's authors, gives a memorable example. Imagine walking into a classroom of 50 students and being told the birthdays of 49 of them. That tells you nothing about the 50th. A birthday isn't something you can deduce from a pattern; either you've seen the information or you haven't.⁵ Facts like this, isolated details with no logic connecting them to anything else, are inherently unpredictable. A model asked about one it never saw will produce its best guess, and that guess will often be wrong. The maths in the paper even shows that a model's error rate when generating text must be at least twice its error rate when simply judging whether a statement is true or false.⁵

Why it feels so convincing

Part of what makes hallucinations dangerous is the confidence. A human who isn't sure usually signals it: they hesitate, they hedge, they say "I think". A model has learned the form of confident, fluent writing without any of the underlying knowledge that would justify it. So a made-up fact arrives in the same calm, authoritative tone as a real one.

This is why hallucinated citations are such a common and serious problem. A 2025 analysis found fabricated references even in papers accepted at NeurIPS, one of the most competitive AI research conferences in the world.⁶ If AI researchers themselves get caught out by invented citations, it's a sign of how convincing these fabrications can be.

Can it be fixed?

Somewhat, but not completely, and it's worth being honest about that.

Newer "reasoning" models that work through a problem step by step do hallucinate less on many tasks, but they still make what researchers describe as strategic guesses, generating plausible but false statements when uncertain.⁷ Evaluations from Stanford's foundation-model researchers have found that reliability remains inconsistent across different topics even as the models get more capable.⁷

The OpenAI and Georgia Tech researchers suggest a fix that's more about incentives than technology: change how models are graded so they stop being punished for admitting uncertainty. If a confident wrong answer scored worse than an honest "I don't know", models could learn to hedge appropriately.⁴ But as one of the authors put it plainly, we will likely never reach 100% accuracy. Some questions are just unanswerable.

What this means for you

The practical takeaway is simple and important: treat everything a chatbot tells you as a confident draft, not a verified fact. It's often right, sometimes brilliantly so, but it has no built-in sense of when it's wrong, and it will state a fabrication with exactly the same assurance as the truth.

Check anything that matters, especially names, dates, numbers, quotes, and citations, which are the things models are most likely to invent. The chatbot won't warn you, because it genuinely doesn't know. That's not a flaw someone forgot to fix. As the research shows, it's woven into what these systems fundamentally are: extraordinary pattern-matchers with no independent grip on what's true.

Sources

1. The Conversation, "What are AI hallucinations? Why AIs sometimes make things up" (2026) — theconversation.com
2. Ibid.
3. Kalai, Nachum, Vempala & Zhang, "Why Language Models Hallucinate," arXiv:2509.04664 (2025) — arxiv.org
4. Euronews, "Why do AI models make things up or hallucinate? OpenAI says it has the answer" (2025) — euronews.com
5. Science / AAAS, "AI hallucinates because it's trained to fake answers it doesn't know" (2025) — science.org
6. GPTZero analysis, cited in MIT Sloan Teaching & Learning Technologies, "Addressing AI Hallucinations and Bias" (2026) — mitsloanedtech.mit.edu
7. Ibid. (citing OpenAI 2025 and Stanford CRFM 2025).