Hallucination Detection Pipeline: Catch LLM Lies Before Your Users Do

Your LLM just told a customer that your product has a feature it does not have. Or it cited a research paper that does not exist. Or it confidently stated a legal regulation that was repealed three years ago. These are hallucinations, and they are the single biggest reason production AI systems lose trust.

In this tutorial, we build a complete detection pipeline — from cheap string-level checks to LLM-powered claim verification — so you catch the lies before your users do. If you have completed our Your First AI App tutorial and are comfortable with the OpenAI API basics, you have everything you need.

An LLM hallucination is any generated output that looks plausible but is factually wrong, unsupported by the source material, or entirely fabricated. The model does not "know" it is lying — it is simply predicting the most likely next token, and sometimes the most likely continuation is wrong.

I have seen hallucinations range from subtle (wrong version numbers in documentation) to catastrophic (fabricated legal citations in a compliance tool). The tricky part is that hallucinated text reads exactly like correct text — same confidence, same formatting, same authoritative tone. You cannot eyeball it at scale.

Three broad categories of hallucination matter for detection. Intrinsic hallucinations contradict the provided source, extrinsic ones add claims no source mentions, and factual ones state things that are objectively wrong about the world. The code below defines each with an example and a risk rating.

Our detection pipeline will tackle all three. We start with the cheapest, fastest checks and escalate to more expensive LLM-powered verification only when needed.

Every code block in this tutorial runs directly in your browser. We will use the OpenAI API for the LLM-powered verification steps. The pure Python checks (string matching, claim extraction logic) need no API key at all.

Before burning API credits on fancy verification, catch the obvious problems with plain Python. These checks cost zero tokens and run in microseconds. I always run these first in production — they catch more problems than you would expect.

The simplest hallucination signal: the LLM contradicts itself within the same response. The find_number_contradictions function below uses a regex pattern to extract every phrase matching "Entity is/was/has Number" (e.g., "Revenue was 5.2 million"). It groups extracted numbers by entity name, then checks whether any entity appears with two different values. If "Acme Corp" is paired with both 2015 and 2013, that is an immediate red flag. The function returns a list of dictionaries, each with the entity, its conflicting values, and a severity rating.

The detector catches two contradictions: "Acme Corp" has conflicting founding years (2015 and 2013), and "Revenue" has conflicting values (5.2 and 8.1 million). Simple regex, zero API cost, immediate signal that this response needs re-generation.

LLMs sometimes hedge when they are uncertain — phrases like "I believe," "approximately," or "it is possible that." These phrases are not proof of hallucination, but they correlate with lower factual accuracy. When an LLM is confident and correct, it rarely hedges.

Quick check: What hedge score would you expect for the text "The answer is definitely 42. I am sure of it."? Answer: 0.0 — "definitely" and "I am sure" signal certainty, not uncertainty. Our detector only flags hedging phrases like "I believe" or "possibly." Confident-sounding text gets a clean bill of health, which is exactly what we want.

A paragraph of LLM output might contain five distinct factual claims. To verify each one, we first need to extract them. This is where most people skip a step: they try to verify the entire paragraph as one blob, which misses individual false claims buried in otherwise correct text.

The extract_factual_claims function below scans text for three distinct regex patterns. First, it finds date claims — years that appear after context words like "in" or "since." Second, it catches numerical claims — amounts followed by units like "million" or "%." Third, it extracts entity claims using "X is/was Y" patterns. A helper function, get_containing_sentence, grabs the full sentence around each match so we have context for downstream verification.

Notice that our test text contains a deliberate error: TensorFlow was released by Google, not Microsoft. The claim extractor does not judge accuracy — it just identifies the claims. Verification comes next.

Pattern-based extraction misses claims that do not follow neat "X is Y" patterns — comparison claims, causal statements, and implicit assertions all slip through. The function below sends the text to gpt-4o-mini with a system prompt that asks for a JSON array of objects, each containing the claim text, its type (date, number, attribution, definition, or comparison), and whether it is objectively verifiable. We strip any markdown code fences the model might wrap around its JSON before parsing.

The LLM identifies claims that regex would miss — like "surpassing R in 2018" (a comparison claim) and the attribution of TensorFlow to Microsoft (which is wrong — it should be Google). Each claim now becomes an independent verification target.

This is one of my favourite detection techniques because the intuition is dead simple: if you ask the same question five times and get five different answers, the model is probably guessing. If all five agree, the answer is more likely correct.

The self_consistency_check function below sends the same question to gpt-4o-mini N times at temperature 0.7, collects all responses, counts unique answers, and computes an agreement ratio between 0.0 (total disagreement) and 1.0 (perfect agreement). When the ratio drops below 0.5, we flag the answer as a likely hallucination.

The agreement ratio alone is useful, but we can sharpen it. The function below goes deeper: it runs self-consistency, then extracts individual claims from each response with our LLM extractor, and compares those claims across responses. Claims that appear in some responses but not others are flagged as potentially hallucinated — a much more granular signal than comparing whole answers.

Self-consistency tells you whether the model is sure of itself. Source-grounded verification tells you whether the model is correct. The idea: given a reference document (your knowledge base, a retrieved passage, a product spec), check whether each claim in the LLM output is actually supported by that source.

This is the core technique behind RAG evaluation and is essential if your LLM answers questions about specific documents. Without it, the model can confidently fabricate details that sound like they came from the document but did not.

Before reading the results, try this: look at the six claims above and the source document. Which ones would you flag? The verification should catch three problems: the widget count is 50, not 100 (CONTRADICTED); there is no Python notebook feature (NOT_FOUND — the LLM fabricated it); and data retention is 12 months on Standard, not 24 (CONTRADICTED). The other three claims check out.

We have built the individual pieces. The class below wires them into a single reusable pipeline that runs checks in strict cost order. Three methods handle the three levels: check_contradictions and check_hedging are free string checks (Level 1), extract_claims sends text to the LLM for claim decomposition (Level 2), and verify_claims fact-checks each claim against a source document (Level 3). The run_pipeline method orchestrates all three, accumulating a risk score as issues surface.

Time to see it in action. The test below feeds the detector a product description that contains several fabrications: Oracle is not listed as a supported data source, the node limit is 50 not 100, the user limit is 10 not 25, and "ML model training" and "$50M valuation" are entirely invented. Watch how the pipeline surfaces each one.

The pipeline catches multiple issues: Oracle is not a supported data source (contradicted), 100 nodes is wrong — it is 50 (contradicted), 25 users is wrong — it is 10 (contradicted), and the ML training capabilities and $50M valuation are entirely fabricated (not found in source). This is exactly the kind of output that would damage user trust if served unchecked.

In production, you rarely run the full pipeline on every response. Here are the patterns I have seen work well across different use cases.

The gate-and-escalate function below runs free string checks on every response. If those checks pass cleanly (no contradictions, low hedging), it returns immediately without spending any API tokens. Only when a red flag appears — or when strict mode is enabled for high-stakes outputs — does it invoke the full LLM-powered pipeline. Two test cases demonstrate both paths.

When your LLM generates answers based on retrieved documents, you need to verify the answer against the retrieved chunks. The verify_rag_response function below concatenates all retrieved chunks into a single source document, runs the full detection pipeline against it, and computes a groundedness score — the fraction of claims that the source material actually supports. If you are building a RAG pipeline from scratch or using LangChain for RAG, this verification step slots in right after generation.

Another mistake I see frequently: running self-consistency checks at temperature 0. At zero temperature, you get nearly identical responses every time, so the agreement ratio is always high. That tells you nothing about whether the answer is actually correct. Use temperature 0.7 or higher for self-consistency.

A third trap: treating NOT_FOUND as acceptable. When a claim is not found in the source, that often means the LLM fabricated it. In a customer-facing product, fabricated claims are just as damaging as contradicted ones. My rule: in strict mode, NOT_FOUND should be treated the same as CONTRADICTED.

Hallucination detection adds latency and cost to every LLM response. The table below compares the three pipeline levels so you can make informed trade-offs for your specific use case.

Yes, and in many cases you should. Using the same model to verify its own output is like asking a student to grade their own exam. A different model (or even a smaller, cheaper model focused on fact-checking) can catch errors the original model is blind to. In practice, gpt-4o-mini works well as a verifier even for responses generated by larger models.

You have two options. First, buffer the complete response and verify before displaying — this adds latency but catches everything. Second, stream the response to the user and run verification in parallel, then display a warning badge if issues are found after the fact. The second approach is better for UX but requires a mechanism to update or annotate the response retroactively.

It depends on the stakes. For a casual chatbot, a groundedness score of 0.7 (70% of claims supported by source) is often acceptable. For medical, legal, or financial applications, target 0.95 or higher. Below that threshold, either re-generate the response with more explicit grounding instructions or fall back to a canned response.

NLI (Natural Language Inference) uses a smaller, specialized model like DeBERTa to classify whether a claim is entailed, contradicted, or neutral relative to a source passage. It is fast and free to run locally, but less flexible than an LLM-as-judge for complex or nuanced claims. LLM-as-judge (what we built in this tutorial) handles more varied claim types but costs API tokens and is slower. I prefer NLI for high-volume pre-filtering and LLM-as-judge for the final verdict on flagged claims.

The pipeline we built covers the most practical techniques, but the research landscape is broader. I want to mention the major alternatives so you know what exists when you need to scale beyond string checks and LLM-as-judge verification.

You now have a working hallucination detection pipeline that scales from free regex checks to LLM-powered claim verification. Here is where to go from here on PythonBook.