An AI hallucination is an answer that presents invented, distorted or unsupported information as fact. To reduce hallucinations, find the step where the answer first went wrong, fix that step, and test the fix on the failed case and on cases that should still work.
A chatbot can promise a refund the policy does not allow. A support bot can invent a rule to explain a bug. A research tool can cite a court decision that does not exist. A coding assistant can install a package nobody published. Each answer sounds well supported, which is what makes it expensive.
This guide explains why models guess, traces four public failures to the step where each one broke, then groups 14 methods by that step. Pick the methods that match the failure you see: each one adds maintenance, so the list is a menu, not a checklist.
Answer in 60 seconds
- Keep one wrong answer with its question, the documents supplied and any tool results, and trace the disputed claim back through that record.
- Wrong or stale source: decide which document governs and keep the searchable copy in sync.
- Evidence present but distorted: extract the evidence first, then check each claim against it.
- Live status or numbers: read them with a tool or compute them with a program, never from the model’s memory.
- Missing information: give the assistant a legitimate way out (a question, a partial answer, a hand-over), then replay the failed case after every change.
In this article
- Why AI models make things up
- Four public failures and where they first broke
- Trace a wrong answer to its first error
- Fix what the model receives
- Reduce guessing while the answer is written
- Check the answer before it leaves
- Code: invented packages and functions
- Keep fixed errors from coming back
- Advice we would argue with
- When an assistant should answer, ask or hand over
1. Why AI models make things up
A language model writes the most plausible continuation of its input. When the input does not contain the answer, a plausible guess and a true answer can look identical.
Training and testing make this worse. In a September 2025 paper, OpenAI researchers argue that “language models hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty” (Kalai et al., 2025). A test that only counts correct answers gives a guess some chance of scoring and “I don’t know” none. OpenAI’s accompanying article shows the effect on the SimpleQA benchmark:
| Model | Abstained | Correct | Wrong |
|---|---|---|---|
| gpt-5-thinking-mini | 52% | 22% | 26% |
| o4-mini | 1% | 24% | 75% |
On accuracy alone, the second model looks slightly better. It is also wrong almost three times as often. So a model will not reliably say “I don’t know” unless your system lets it and treats that as an acceptable outcome, and a leaderboard score says little about how often it invents answers in your context.
Not every wrong answer is an invention: an assistant can faithfully repeat an outdated instruction. That needs a different fix, which is why diagnosis comes first.
2. Four public failures and where they first broke
Each case is documented in a court decision, a public correction or the company’s own statement. None was resolved by a better model, and each broke at a different step.
Air Canada: the chatbot contradicted the policy page
A passenger asked Air Canada’s website chatbot about bereavement fares. It said that travellers who “have already travelled” could claim the reduced rate “within 90 days of the date your ticket was issued”. The airline’s policy page said the opposite, and Air Canada refused the refund. Before British Columbia’s Civil Resolution Tribunal, the airline argued it could not be held liable for information from its agents, “including a chatbot”. The tribunal called this “a remarkable submission”: “It makes no difference whether the information comes from a static page or a chatbot.” It ordered CAD 812.02 in damages, interest and fees (Moffatt v. Air Canada, 2024 BCCRT 149).
Where it first broke: a policy question was answered without being tied to the page that governs it, although that page was on the same website.
Cursor: the support bot invented a policy to explain a bug
In April 2025, a user of the Cursor code editor was logged out whenever they switched machines. The email support reply said: “Cursor is designed to work with one device per subscription as a core security feature.” No such policy existed; the reply came from an AI support bot, and users announced cancellations. A co-founder called it “an incorrect response from a front-line AI support bot” and later traced the logouts to “a race condition that appears on very slow internet connections”. The company refunded the user, fixed the bug and began labelling AI replies (Hacker News, Ars Technica).
Where it first broke: the bot lacked the one fact that mattered, that a bug existed, and filled the gap with a plausible rule.
Mata v. Avianca: invented court cases, confirmed by the same tool
In 2023, lawyers in a New York federal case filed a brief citing decisions generated by ChatGPT; six did not exist. The court record includes the exchange in which ChatGPT, asked whether a case was real, answered: “Upon double-checking, I found that the case […] does indeed exist and can be found on legal research databases such as Westlaw and LexisNexis.” The judge fined the two lawyers and their firm $5,000, noting that “existing rules impose a gatekeeping role on attorneys to ensure the accuracy of their filings” (Opinion and Order on Sanctions, 22 June 2023).
Where it first broke: nobody checked the citations in the databases the model itself named. Asking the model to confirm its own answer produced a second, equally confident invention.
Deloitte: unverifiable references in a government report
In 2025, Deloitte delivered an assurance review of the system Australia’s Department of Employment and Workplace Relations uses to apply welfare penalties. A University of Sydney researcher reported up to 20 errors, including a cited book that, according to AP, does not exist, and a quotation attributed to a Federal Court judge that the corrected version removed. The corrected report disclosed “a generative AI large language model (Azure OpenAI GPT-4o) based tool chain”. Deloitte said the findings were unaffected, and repaid the final instalment, AUD 97,587.11, of a AUD 439,142 contract (DEWR; Australian government contract register, CN4118426).
Where it first broke: references reached the client without anyone checking that each source exists and says what the report claims.
The four cases side by side
| Case | What the user received | First broken step | Methods |
|---|---|---|---|
| Air Canada | A refund rule the policy did not allow | Answer not tied to the governing document | 1, 8, 9 |
| Cursor | A policy that did not exist | Missing fact filled with a guess | 3, 7, 8 |
| Mata v. Avianca | Six court decisions that did not exist | Claims not checked against the source | 9, 11 |
| Deloitte | References and a quotation that could not be traced | Citations not verified before delivery | 9 |
Two lessons run through all four. The organisation owns what its AI says, whether the words come from a web page, a bot or a report. And the fix sits in the system around the model: the source it answers from, the exit it is allowed to take and the check before the answer leaves.
3. Trace a wrong answer to its first error
Keep the failed answer with the question, the documents supplied and any tool results, then follow the disputed claim backwards to the first step where it went wrong.
| What you observe | Inspect first | Methods |
|---|---|---|
| The answer repeats an outdated instruction | Which version governs, and whether the searchable copy was updated | 1, 2 |
| It uses a similar document for the wrong product or error code | Search scope, exact matching and ranking | 2 |
| It drops a condition or an exception | The passages supplied and the claims produced | 2, 6, 9 |
| It invents a status or gives a wrong total | Tool arguments, the returned record, the calculation | 3, 4, 10 |
| It answers fully despite missing information | Clarification and hand-over rules | 5, 7 |
| It states a policy, price or commitment nobody approved | Whether fixed policies have approved answers | 8 |
| It recommends a package or function that does not exist | Package and documentation checks | 12 |
| A checking model approves a wrong answer | The checker’s criteria and labelled examples | 11 |
| An old error returns after an update | Saved test cases and the change history | 13, 14 |
When an assistant answers from search results (retrieval-augmented generation, or RAG), retrieval and writing are separate steps. A relevant document in the results does not prove the model received or used the passage that mattered.
4. Fix what the model receives
Method 1. Decide which source governs the answer
For each kind of fact, name the source allowed to establish it: the current policy page for refunds, the documentation for the installed version of a library, the signed edition of a contract. Air Canada’s correct policy was on the same website as its chatbot; more documents do not help when the system cannot tell which one governs.
Check the fix: ask a question whose answer differs between versions and inspect which document the answer follows.
Method 2. Make sure the right passage reaches the model
- Stale copies. Editing the original does not update the search index. Synchronise on change, test deletions as well as additions, and make failed updates visible.
- Wrong scope. Separate material by product, version or purpose wherever that changes the answer.
- Lost identifiers. Combine semantic search with exact matching so that codes such as
SYNC-42find their own instructions. - Lost context. Keep the heading, product name or exception attached to each fragment.
Check the fix: inspect the text actually passed to the model before judging the prose. We go deeper in indexing documents without stale answers and testing freshness and retrieval.
Method 3. Read changing facts with a tool
Account status, order state, test results and anything else that changes should come from a tool call that reads the current record, not from documents or the model’s memory. The Cursor bot needed one fact, that a login bug existed, and had no way to read it.
Check the fix: test a missing record and a failed call. A failed call must leave the status unknown, not turn into a confident answer.
Method 4. Let a program do the calculation
Have the model choose a function or write a query, and let the database compute the result. If finished experiments are marked status = 'completed' and the query filters on complete, it runs without error and returns nothing, and “no experiments were completed” becomes a false conclusion.
Check the fix: compare the query and its result with a case whose answer you already know.
5. Reduce guessing while the answer is written
Method 5. Ask for the missing detail
List the details a task depends on (version, operating system, period, account). If one is missing, ask a specific question instead of silently picking a default.
Method 6. Extract the evidence before writing
First pull out the passages or fields the answer needs, then write only from them and keep gaps visible. If a report never measures time saved, its summary should not gain a percentage.
Method 7. Give the assistant a legitimate way out
Because models are rewarded for guessing (section 1), “don’t make things up” is not enough on its own. Design the alternatives: answer the supported part, say what is missing, ask a question or pass the case to a named person. Our guide to when company AI should answer, draft, request approval or execute covers where those lines sit.
Check the fix: measure unsupported answers and unnecessary refusals separately. Fixing one often worsens the other.
Method 8. Use approved answers for fixed policies
Refund rules, cancellation terms and security policies should come from maintained, approved text, with variable fields filled from verified data. The assistant identifies the situation; it does not compose policy. Both Air Canada and Cursor ended up with a bot stating a policy the company did not have.
Check the fix: test a matching request and a similar one that should take another route.
6. Check the answer before it leaves
Method 9. Check each claim against its source
Break the draft into factual statements and compare each with its passage, record or cited source, including conditions and quantities. For citations, confirm in the primary database that the source exists and says what is claimed. This step was missing in the Avianca filing and the Deloitte report, and it must go to the source, not back to the model.
Check the fix: plant a fake reference in a test answer and see whether the check catches it.
Method 10. Check rules and structure separately
When a requirement can be written as a rule (a date range, a required field, a maximum amount), check it in code, outside the generated text. A response that matches a schema proves its shape, not its facts.
Method 11. Test the checker
If another model judges answers, compare its verdicts with examples people have reviewed against explicit criteria, including convincing wrong answers. Record missed errors and false alarms separately. Two models agreeing does not prove that either is right.
7. Code: invented packages and functions
A study presented at USENIX Security 2025 generated 576,000 code samples with 16 models. Of 2.23 million package references, 19.7% pointed to packages that did not exist: at least 5.2% for commercial models and 21.7% for open-source ones, with 205,474 unique invented names. 43% of those names came back in all ten repeats of the same prompt (Spracklen et al., 2025).
Names that repeat are predictable, so an attacker can register them and wait. The practice is called slopsquatting, a term coined by Seth Larson and first published by Andrew Nesbitt in April 2025.
Method 12. Verify every package and API before it runs
- Check that a suggested package exists, is the one intended and has a credible publisher before installing it. Lock versions and review new dependencies.
- Give the assistant documentation for the installed version instead of relying on what the model remembers.
- Require evidence for “done”: test output, the file on disk, the API response. An agent saying the tests pass is a claim, not a result.
Hardening AI-written code and its dependencies is part of our AI Security & Code Hardening service.
8. Keep fixed errors from coming back
Method 13. Replay saved hard cases after every change
Keep a stable set of ordinary questions, exceptions, ambiguous requests and questions with no available answer, in every language you support. Run it whenever the model, prompt, retrieval settings or key documents change, so a regression shows up next to the improvement.
Method 14. Record enough to trace a failure
Log the selected passages, the context sent to the model, tool arguments and results, and versions. If the right document was never retrieved, look at search. If its exception was dropped, look at context assembly. If the exception reached the model and vanished from the answer, look at generation and checking. Turn every real failure into a saved case.
9. Advice we would argue with
“Connect your documents and it will stop making things up.” Retrieval changes what the model sees, not whether it guesses. Air Canada’s correct policy was on its own website. Documents help only when the system knows which one governs and checks the answer against it.
“A second model will check it.” A checker is useful for finding likely errors, not for accepting answers. Asked to double-check, ChatGPT confirmed the invented cases in Mata v. Avianca. Test any checker against human-reviewed examples before trusting its verdict.
“Just use a bigger model.” Newer models may guess less, but accuracy scores hide guessing: in OpenAI’s own example, the model with the higher accuracy was wrong 75% of the time. A model swap does not fix a stale source, a missing tool or an unchecked citation.
“Set the temperature to zero.” A lower temperature makes output more repeatable, not more correct. A wrong answer produced deterministically is wrong every time.
“Tell it not to hallucinate.” An instruction helps only if the system offers a real alternative (ask, answer partially, hand over) and you measure how often the assistant takes it.
10. When an assistant should answer, ask or hand over
The costly errors in all four cases were commitments: a refund, a policy, a legal citation, a published reference. We draw the line by consequence, not by how confident the answer sounds.
| Request | Default | Condition |
|---|---|---|
| Internal draft for a person to review | Answer | The reviewer sees which sources were used |
| Fact from a governed document | Answer with its source | Passage found and claim checked (methods 1, 9) |
| Detail missing from the request | Ask | One targeted question (method 5) |
| Live status of an order, account or job | Answer from a tool | Current record read; a failed read is reported as unknown (method 3) |
| Policy, refund or security rule | Approved answer only | Otherwise hand over (method 8) |
| Price, deadline or contractual commitment | Hand over | A person confirms |
| Change to a customer’s account | Act through a tool, with confirmation | Report the state the tool returned, not the intended one |
Start with one failure you can reproduce
Pick one recurring wrong answer and keep the inputs that produced it. Fix the first step where it went wrong, then replay it alongside cases that must keep working. Track unsupported answers, unnecessary refusals and correction time separately to decide whether the fix stays.
Bring us one wrong answer
Bring one answer your assistant got wrong, the question behind it and what the system had access to. We trace it to the first broken step and show which controls would have caught it, before you add documents, switch models or add a checker.
Designing and operating those controls is the scope of our AI Agents & LLM Products service.
Frequently asked questions
An answer that presents invented, distorted or unsupported information as fact, such as a court case that does not exist or a policy the company never had. An answer can also faithfully repeat an outdated source, which needs a different fix.
A model writes the most plausible continuation of its input, and plausible is not the same as true. OpenAI researchers also argue that training and evaluation reward guessing over admitting uncertainty.
No method guarantees correct answers to arbitrary questions. For a defined task, you can measure whether a change reduces unsupported answers without causing too many refusals.
No. Retrieved material can be outdated, incomplete or misread, and the model can still go beyond it. Check that the right passage was found and that the answer keeps its meaning and limits.
It can flag likely errors, but it can also miss errors and reject correct answers. Test it against human-reviewed examples, and use direct lookups, rules and calculations wherever they can settle the question.
In the Air Canada case, the tribunal held the company responsible for its chatbot’s answers like any other page on its website. It is one Canadian small-claims decision, not a universal rule, but a sound working assumption.