What to Do When AI Makes Things Up: Engineering Guardrails Against Hallucination

Every team that adopts LLMs eventually hits the moment: the model, in perfectly fluent and supremely confident prose, describes something that does not exist. It looks up an order that isn't there, cites a policy clause you never wrote, invents a plan you've never sold. This is hallucination, and most people's first reaction — "switch to a smarter model" — is wrong. A smarter model just makes things up more convincingly.

The good news: hallucination isn't black magic, it's an engineering problem. Our own product lines put AI output in front of real users every day, and what keeps that safe is four layers of defense. Let's go through them one at a time.

Layer one: retrieval — make the model answer from the documents, not from memory

The root of hallucination is that a model will generate plausible-looking content even when it has no basis for it. So the first defense attacks the root: for any answer that requires facts, first retrieve the relevant material from your knowledge base, feed it to the model together with the question, and explicitly instruct in the prompt: "answer only from the provided material; if it's not in the material, say you don't know." This is the core logic of RAG (retrieval-augmented generation).

Two practical points. First, retrieval quality determines everything — if the knowledge base is stale or self-contradictory, the model can only pick something from bad data. When AI says the wrong thing, the disease is usually in the documents, not the model (this is also why we made knowledge-base maturity a hard gate in our AI customer service evaluation framework). Second, require the model to cite which passage it used in each answer — that gives the next layer something to verify.

Layer two: output validation — don't trust, check

Before a model's output reaches the user, it passes through programmatic checks:

• Format validation: If you asked for JSON, validate the schema; missing fields or wrong types trigger a retry, not a pass.
• Fact-anchor validation: Any order number, product name, amount, or link appearing in the answer gets checked against the database — if it doesn't exist, it gets blocked. One of our iron rules is "no link may 404," and it applies equally to AI-generated content: models love inventing URLs that look extremely real.
• Rule scanning: Domain-specific prohibited terms and prohibited meanings get caught here (we run a health-food e-commerce business, so for us this is a regulatory-grade necessity).

The essence of this layer is demoting "what the AI said" to "a draft pending verification," using cheap conventional code to guard against expensive mistakes.

Layer three: confidence thresholds — teach the model to say "I'm not sure"

More dangerous than being wrong is never admitting ignorance. Our approach: have the model assess its own confidence, and branch the flow accordingly. Low retrieval relevance, low model self-assessed confidence, or a question outside the knowledge base's scope — in those three cases, don't force an answer; route to "let me hand you to a human" or "we don't have enough information on that."

Note that the model's self-score can't be fully trusted either (it can confidently score itself high), so in practice we weight objective signals like retrieval relevance above the model's self-assessment. There's no standard answer for where to set the threshold; the principle is: better to hand a few extra cases to a human than to let one wrong answer through — especially early in adoption, before trust has been built.

The goal of hallucination defense isn't an AI that never errs — it's errors intercepted before they reach the user, or at least a seat kept for a human.

Layer four: the human escalation path — the last line of defense, and part of the design

The first three layers are machines catching machines; the fourth admits machines can't catch everything. High-risk operations (refunds, order changes, regulation-adjacent replies) are always "AI drafts, human hits send." When a user expresses clear frustration or keeps re-asking, escalate to a human proactively. Keep records of all AI conversations, review samples regularly, and feed discovered error cases back into the knowledge base and validation rules — that feedback loop is what makes the system improve over time.

Four layers sounds like a lot of engineering, but they can be built incrementally: retrieval plus human review is enough to launch, and validation and confidence routing can grow with your traffic. What you really shouldn't skimp on is the mindset: treat the LLM as a very talented intern who needs an editor, not an all-knowing oracle. If you'd like a hallucination risk check-up for your AI application, or want the defenses designed right from the start, take a look at our AI development services or just get in touch.