Large language models (LLMs) like ChatGPT, Claude, and Gemini have made impressive advancements since their release within just the past two years. But the widespread adoption of artificial intelligence (AI) still faces a significant challenge when it comes to dealing with hallucinations.
An AI hallucination is an error mode that differs from the human experience of a hallucination. In AI, a hallucination occurs when a language model generates false, misleading, or entirely fabricated information.
These errors can range from subtle inaccuracies to entirely invented references. That creates significant problems for users incorporating LLMs into research, professional work, or decision-making.
Why do LLMs hallucinate, and what can developers do to minimize the extent to which they happen?
- AI hallucinations occur when LLMs generate false or fabricated information with confidence.
- LLMs hallucinate as a result of how they are trained and evaluated, according to an OpenAI research paper.
- A common example of an AI hallucination is a made-up research citation or statistic.
- Hallucinations in AI can cause real-world problems in high-stakes fields like law, healthcare, and finance if outputs are used without verification.
- Reducing the AI hallucination problem requires introducing confidence thresholds and maintaining human oversight.
Why Do LLMs Hallucinate?
The hallucination problem lies in how large language models are trained. Instead of retrieving facts from a structured database, LLMs predict the most likely sequence of words based on patterns they identify in the massive amounts of training data they are fed.
In a recent research paper, ChatGPT developer OpenAI noted that hallucinations can be caused by both pretraining and post-training processes, “because the training and evaluation procedures reward guessing over acknowledging uncertainty.”
It likens LLM training to standardized human exams in which students may guess the answers to multiple-choice questions and even bluff plausible responses to written questions if they are uncertain.
In both scenarios, binary scoring that awards one point for a correct answer penalizes blank or “I don’t know” responses. But while humans learn the value of expressing uncertainty outside of school, language models are always in “test-taking” mode, the paper notes.
A Confident Wrong Answer Scores Higher Than ‘I Don’t Know’
LLM bluffs can be specific and overly confident, such as responding with “September 30” rather than stating “sometime in autumn” when given a prompt that asks a question about a date.
David Brudenell, Executive Director at Decidr, told Techopedia:
“Hallucinations are not a sign of AI gone rogue. They are the natural byproduct of how models are trained and evaluated. During pretraining, models learn patterns of language, not truths. Even if the training data were flawless, the maths guarantees some level of error. On rare facts the model simply has to guess.
“Post-training makes this worse. Most benchmarks reward bravado and penalize uncertainty. Like students winging it on an exam, models learn that a confident wrong answer scores higher than saying ‘I don’t know’.”
Hallucinations in pretraining can occur because of a combination of multiple factors.
AI systems have been found to hallucinate in response to computationally hard problems, as “no algorithm run on a classical computer, even an AI with superhuman capabilities, can violate the laws of computational complexity theory,” the paper states.
LLMs can return arbitrary facts when the correct information is missing from the training data. If it has not been fed enough high-quality data on a certain topic, it may fill in the gaps with a response that is plausible but incorrect.
Errors are even more likely to arise when the quality of the underlying model is poor. Training and test data distributions often diverge, and incorrect answers from LLMs can often stem from prompts that differ substantially from the training distribution.
An ambiguous prompt from the user can also push the model to generate overly confident, but inaccurate, answers.
And ultimately, the concept of garbage in, garbage out (GIGO) applies, as training data that contains numerous factual errors is likely to be replicated by the LLM in its responses.
The post-training stages refine the base model, with reducing hallucinations as a goal, but their binary approach often reinforces the tendency for the model to hallucinate rather than admit it does not have the correct information to respond.
Brudenell said:
“And here lies the paradox. This is an extraordinary piece of technology, promoted as near magical, yet it cannot afford to show weakness. If it says, ‘I don’t know,’ the obvious question follows. What does it know? Worse, there is no quantitative data on what it does not know. The OpenAI study makes clear that the benchmarks we (the AI R&D community) have built do not capture this gap. We have created a rod for our own back.”
Why Do Hallucinations in AI Matter?
While they can be harmless in casual chat with individuals, LLM hallucinations can have serious consequences in professional settings.
A common hallucination would be asking an LLM for a reference to an analyst report or scientific paper, and rather than returning the response “I don’t know,” the model might generate a realistic-looking citation, complete with author names, journal titles, and page numbers, all for a paper that doesn’t exist. This highlights the risks of trusting outputs without verifying them.
There have been several examples of media publications that broke large language model news for the wrong reasons – having cited references that do not exist or published entire articles fabricated by chatbots, even down to fictional writers. The damage to these publications’ reputation can have a substantial impact on their business.
Brudenell said:
“The risk is not only factual mistakes, it’s misplaced trust. Authoritative hallucinations can mislead decision-making in high-stakes contexts such as finance, medicine, or governance. They also create a cultural problem. When people discover the technology invents answers, the hype backfires.
“Trust, once lost, is hard to win back. Yet the industry has no choice but to keep users engaged. They need people to form habits and to return daily. What happens if people stop using it?”
Many people remain skeptical about using AI-based applications such as chatbots, and prominent examples of hallucinations make it difficult for the industry to encourage widespread adoption.
Solutions for Minimizing Hallucinations
For this reason, researchers and developers are working to minimize the AI hallucination problem through better training methods, retrieval-augmented generation (RAG), and transparency tools.
As LLMs continue to become more sophisticated, the level of hallucinations will improve, but not automatically. Brudenell explained:
“Hallucinations are not inevitable, but progress depends on changing incentives. Current benchmarks reward confident guessing over calibrated honesty. If we redesigned evaluations to give credit for “I don’t know” where appropriate, models would learn to calibrate rather than bluff.
“Techniques such as retrieval augmented generation, reinforcement learning from human feedback, and risk-informed prompting help, but they will not solve the paradox while the dominant leaderboards punish humility and prize confidence.”
OpenAI’s researchers propose that post-training should shift from training models with an autocomplete approach to one that does not return false responses (except when appropriate, such as if a prompt asks the model to produce fiction).
A socio-technical mitigation that specifies confidence thresholds and incorporates confidence targets into established evaluations already in use, by modifying the scoring of existing benchmarks, could reduce hallucinations by allowing LLMs to express uncertainty rather than make incorrect statements.
“This change may steer the field toward more trustworthy AI systems,” the paper concludes. “Simple modifications of mainstream evaluations can realign incentives, rewarding appropriate expressions of uncertainty rather than penalizing them. This can remove barriers to the suppression of hallucinations, and open the door to future work on nuanced language models.”
LLMs are powerful computational systems, but they are not infallible. By design, they generate text based on the probability that the response is correct, not guaranteed to be true. Rather than just a flaw, hallucinations reflect the incentive structures built around the technology, Brudenell said.
“The brilliance of the system is clear. But its weakness, that it cannot admit weakness, remains unresolved. We should expect models to get better at caution, calibration, and factual grounding. To eliminate hallucinations entirely is unlikely,” he admitted. “The real work is socio-technical. Creating systems, incentives, and organizations that can absorb uncertainty rather than punish it.”
As the use of large language models grows, developers and users must remain alert to the risk of prompts returning hallucinations rather than facts. By combining LLMs with fact-checking systems and human oversight, they can harness their potential while reducing the dangers of misinformation.
What does it mean when an AI hallucinates?
An AI hallucination happens when an LLM generates false or misleading information that sounds plausible but is incorrect.
Why do LLMs like ChatGPT hallucinate?
LLMs hallucinate because they predict text based on patterns in training data, not verified facts, often guessing when uncertain.
How can AI hallucinations be reduced?
Developers use better training methods, fact-checking systems, and human oversight to minimize hallucinations and improve AI reliability.