Article · 8 min read

What is an AI hallucination?

Published September 2026

You ask an AI chatbot a simple question. It answers confidently, with detail, in fluent sentences. The only problem: the answer is completely wrong. The study it cited doesn't exist. The court case it referenced was invented. The historical quote it gave you was never said by anyone. That is an AI hallucination, and it happens far more often than most people realise.

The term sounds dramatic, even a little sci-fi. But the definition is pretty straightforward. An AI hallucination is when an AI system generates information or responses that are incorrect, misleading, or entirely fabricated, but presented as factual. There is no moment of doubt, no disclaimer, no lowered confidence. The model just says the wrong thing as though it were obviously true.

What does "hallucination" actually mean here?

The word is borrowed from human experience, and it is a bit of a stretch. When a person hallucinates, their brain produces a perception with no external cause. When an AI "hallucinates", it produces a plausible-sounding output that has no grounding in fact. The mechanism is different, but the result rhymes: something that feels real, but isn't.

The model isn't perceiving anything. It's predicting the next likely word, and sometimes that prediction is a plausible-sounding falsehood. That is the core of what's going on. Large language models, the kind that power ChatGPT, Claude, Gemini and the rest, generate text by working out which word or phrase is most likely to follow from what came before. They are pattern-matching engines of remarkable sophistication, but they are not looking things up in a verified database. They are, in a sense, always improvising.

So when you ask about a medical study, or a legal precedent, or a historical statistic, the model does not fetch the actual record. It produces text that looks and sounds like what an answer to that question would look like. Usually, that text is broadly accurate, because its training data contained a lot of accurate material. But sometimes it isn't, and the model has no reliable way to know the difference.

There's more where this came from. New articles most weeks.Browse all articles →

Why does it happen?

Hallucinations aren't bugs. They come from how generative AI models are built and trained. A few things make them almost inevitable, at least some of the time.

First, there is the prediction problem. Models generate text by guessing the most likely next word based on patterns they've learned, not by looking up facts. So they'll confidently fill in an answer even when they don't actually have the right information.

Second, there is what researchers sometimes call source amnesia. While generating content based on the training data, the model can produce content by disconnecting from the source of that training data. It absorbed millions of texts during training, but it cannot reliably trace a specific claim back to a specific source. When it can't find the signal it needs, it sometimes fills the gap with something that fits the pattern.

Third, a lack of proper grounding plays a role. An AI model may struggle to accurately understand real-world knowledge, physical properties, or factual information, and this can cause it to generate outputs that, while seemingly plausible, are actually factually incorrect, irrelevant, or nonsensical. This can extend, as Google notes, to fabricating links to web pages that never existed.

What does a hallucination look like in practice?

The most vivid examples tend to come from professional contexts, where the consequences of a wrong answer are hard to ignore.

The legal world has been particularly badly hit. Two New York lawyers cited six nonexistent decisions generated by ChatGPT in a personal injury suit against Avianca Airlines, and the Southern District of New York fined them $5,000. That case, in 2023, was widely treated as a curiosity. It turned out to be a sign of things to come.

As of June 9, 2026, the public AI Hallucination Cases database has identified 1,598 court cases involving AI-fabricated citations or content, up from roughly 200 a year ago. The researcher behind that database, Damien Charlotin, tracks only cases where a court has explicitly found that someone relied on hallucinated content. The real number is almost certainly higher.

The pattern is remarkably consistent across countries and legal systems. Citations bear real case numbers and the initials of real judges. All of those judges exist. The cases do not. The AI fills in every plausible detail, which is precisely what makes the error so hard to catch without going to verify each source.

It has also caught out people who really should know better. In one case that challenged a statute prohibiting AI-generated deepfakes, an expert on the dangers of AI used a generative AI tool to draft a statement, which cited non-existent academic articles. The expert plausibly explained how the mistakes happened, but his credibility with the court had been shattered.

Outside the courtroom, the same pattern plays out everywhere from academic papers to news summaries to medical queries. In one study evaluating AI-generated research proposals, out of 178 references generated by ChatGPT, 69 had no valid digital identifiers and 28 did not appear in any search or have any existing record.

Are hallucinations getting better or worse?

This is where the picture gets awkward. You might expect that as AI models grow more sophisticated, hallucinations would reduce. The evidence for that is mixed, and in some cases the opposite has happened.

AI hallucinations occur when generative AI models produce outputs that sound plausible but are factually wrong, fabricated, or unsupported by their training data. The problem is getting worse, not better: newer reasoning models from OpenAI and others have shown higher hallucination rates in some benchmarks.

Part of the explanation is that more capable models are being asked harder questions, and the failure modes are just more elaborate. A model that can reason through multi-step problems can also construct a more convincing wrong answer. The confidence does not diminish as the question gets harder. If anything, it increases.

Some approaches are helping. Retrieval-augmented generation (often shortened to RAG) connects a model to a verified external database before it answers, so it can ground its responses in actual sources rather than pure pattern-matching. Web search integration works similarly. These do reduce hallucinations in practice, but they do not eliminate them entirely, because the model still has to interpret what it retrieves, and that interpretation can go wrong.

Where does it matter most?

Hallucinations are frustrating anywhere, but they are a built-in property of how these models work, and they create real risk for anyone deploying AI, from legal liability and regulatory compliance to customer trust.

Healthcare is an obvious concern. When clinicians use AI tools to assist with diagnosis, treatment planning, or drug interaction checks, a hallucinated output can directly affect patient safety. A model that fabricates a drug dosage recommendation or invents a contraindication that does not exist creates risk that is difficult to catch in fast-paced clinical environments.

The law, as described above, is another. And education is quietly becoming a third: students submitting essays with AI-generated references, researchers building on citations that do not exist, all trusting that the model would not invent something so specific and checkable.

How to protect yourself

The single most practical thing to understand about AI hallucinations is that the model's tone gives you no information about whether its content is accurate. Confident, well-structured, fluently written sentences are what these systems produce regardless of whether they are correct. You cannot tell from the writing itself.

Some habits that genuinely help:

Verify anything specific. Dates, statistics, names, case citations, study results: check them against an original source before using them for anything important.
Ask the model for sources, then check those sources actually exist. Do not assume a convincing-sounding citation is real.
Use AI for drafting and structure, not primary research. Where a model shines is synthesis and prose. Where it struggles is precise factual recall.
Be more careful in high-stakes domains. A hallucination in a casual email matters a lot less than one in a legal brief or a medical query.

Organisations can reduce these risks through stronger guardrails, adversarial testing, continuous monitoring, source verification, and human review for high-stakes outputs. For an individual, though, the simpler version of that same principle is: treat AI output like a smart first draft from someone who occasionally makes things up, and check the bits that matter.

That is not a reason to stop using these tools. They are genuinely useful. But it is a reason to understand what they are, which is not a knowledge base you can trust blindly. It is a very fluent, very confident text generator that is, at any moment, one unlikely word prediction away from something that sounds authoritative and is entirely wrong.

From Telltale

Keep reading

If this one was useful, there's plenty more on the site. Pieces on how AI works, plus coverage of AI news, the downsides included. All free to read, no account needed.

See all articles →

References

Published September 2026 · telltale-ai.com
All articles · Privacy · Terms