Have you ever wondered why AI Large Language Models (LLMs) sometimes make things up? That is what we call a hallucination. According to new research from OpenAI, hallucinations are not quirks of transformer models or messy data. They are an expected outcome of how these systems are trained and evaluated.
Published in September 2025, the report titled “Why Language Models Hallucinate” explains that LLMs produce confident but incorrect information because their training rewards guessing rather than admitting uncertainty.
Let us break down what the researchers found, why it matters, and what they think we should do next.
Why Models Guess
OpenAI uses a simple analogy to explain LLM behaviour. Imagine a student in an exam. If a question is hard and the exam rewards attempts but not blank answers, most students will guess. A guess might earn some marks, while leaving it blank guarantees zero.
LLMs behave in the same way.
Their training and evaluation processes reward them for providing an answer, not for saying “I do not know.” In many benchmarks, refusing to answer counts as failure. So from the model’s point of view, guessing is the rational move. Even a low probability answer might increase its score.
The key conclusion is that we built models to be excellent test takers, not careful truth tellers.
It Is Not Just the Data
OpenAI is not saying that bad data does not matter. It does. If the internet contains incorrect facts, models will absorb some of them.
However, the paper makes a stronger point: even with perfect data, hallucinations would still occur.
This happens because the model generates text one token at a time, choosing each word based on probability. That statistical process inevitably makes mistakes, especially when questions are:
- rare
- ambiguous
- poorly represented in the training data
The researchers explain this using learning theory. Internally, the model is constantly trying to decide whether the next token fits the answer. When it cannot reliably distinguish between true and false continuations, errors naturally slip through. These errors are hallucinations.
Rareness and Risk
One important finding is that the rarer a fact is in the training data, the more likely the model is to hallucinate.
If something appears only once or a handful of times, the model has weak statistical grounding. It fills the gap with whatever seems plausible. This is why LLMs perform well on common knowledge but struggle with:
- obscure names
- local policy details
- niche research
- new events
OpenAI illustrates this with an example. When asked about an author’s PhD dissertation title, multiple models produced confident and detailed answers that were all incorrect. They were not being mischievous. They were responding the way they were trained to respond.
What OpenAI Recommends
This is where the paper becomes practical.
OpenAI argues that the biggest lever for reducing hallucinations is not a new model architecture. It is changing how we evaluate LLMs.
Current benchmarks use binary scoring: right or wrong. There is no credit for caution or uncertainty. The entire field has been incentivised to optimise for answers, not accuracy.
They make three recommendations.
1. Reward honest uncertainty
Benchmarks should give credit for careful refusals. Saying “I am not sure” should score better than offering a confident falsehood.
2. Train models to express uncertainty
Models should be encouraged to say “I do not know” when appropriate. OpenAI notes that users may sometimes find this less helpful, but the trade off is necessary for reliability.
3. Recognise that some questions cannot be answered
This is especially relevant for marketers, communications teams, and policy professionals. Some questions simply do not have a clear answer in the available data. In those cases, the responsible behaviour is to signal uncertainty.
What This Means for You
1. Treat AI output as a draft, not a source
This is especially true for high-stakes or niche topics. Models are structurally incentivised to guess.
2. Be sceptical of confident answers to obscure questions
Certainty is not evidence. It is a style produced by training incentives.
3. Expect future models to be more cautious
If the industry shifts toward rewarding honesty, LLMs may feel more reserved. This might seem less smooth, but it will be better for trust.
The Bottom Line
Hallucinations are not a glitch. They are what happens when you teach a system that guessing is better than admitting uncertainty.
If we want more reliable AI, we must stop rewarding models for bluffing.
About the Podcast Host
Joanne Sweeney is a specialist in public-sector marketing and digital communications. She’s the author of Public Sector Marketing Pro and the creator of the Professional Diploma in AI for Public Sector Marketing. With years of experience in training, consulting, and advocating for digital transformation, Joanne provides insight and guidance for government agencies navigating the digital landscape. She is also the host of the award-winning Public Sector Marketing Show podcast.
