Catching LLM Hallucinations After Generation: Knowledge Graphs, Cheap NLP Cascades and Production Design
Catching LLM Hallucinations After Generation
A hands-on tutorial on post-processing, knowledge-graph verification, cost-efficient NLP cascades, agentic explainability and production deployment, without retraining the model.
Research date: 27 October 2026 | Level: intermediate | Reading time: about 45 minutes
Large language models still produce fluent statements that are wrong, unsupported or misleading, even after instruction tuning and reinforcement learning. Improving the model is one remedy. A complementary remedy is to put a verification and correction layer after the model: split the answer into atomic claims, collect evidence from documents and knowledge graphs, check support and contradiction, validate citations, and then publish, correct, qualify or abstain. The generator's weights are never touched, so the layer works around proprietary APIs and open-weight models alike.
- Why post-processing, and what it is not
- The verification vocabulary: verdicts, abstention, independence
- The big picture: five architectures and the one to build
- Building blocks: claims, detectors, provenance and where to get them
- Knowledge graphs as a verification tool
- Making it affordable: NLP-first, LLM-last
- Going to production on Google Cloud
- Explainability for agentic systems
- Evaluating the whole thing
- Final recommendation and checklist
- References
1. Why post-processing, and what it is not
1.1 A fluent sentence is not a verified sentence
Imagine a model returns this sentence:
"Researcher A published a paper on knowledge graphs in 2024, demonstrating a 35% improvement in factual accuracy."
It looks like one statement, but it hides five independently checkable claims. Any of them could be true, partly true or invented, and the fluent wording gives no hint which.
Unless the application adds a check, the sentence travels straight from the model to the user. The question this tutorial answers is therefore:
How can we detect, explain and correct unsupported claims after an LLM has produced an answer, without changing the underlying model?
This matters most for enterprise assistants, research tools, financial and healthcare information systems and agentic workflows, where a plausible but unsupported statement has real consequences.
1.2 The problem is not solved at the frontier
Google's FACTS Benchmark Suite (December 2025) evaluates grounding, parametric factuality, search-based factuality and multimodal factuality separately. The best overall FACTS Score reported was 68.8% (Gemini 3 Pro), and every evaluated model stayed below 70%. Those numbers are benchmark-specific and not a universal accuracy estimate, but they show why an independent verification layer is still worth building.
1.3 Post-training versus post-processing
The two terms are easy to confuse. The difference is when they act and whether the weights change.
A tiny example makes it concrete. Suppose a deployed model answers The company was founded in 2018. A post-generation layer can extract that claim, fetch the official company record, notice the contradiction, replace the date with the supported one and validate the corrected answer before publishing it. The model itself stays exactly as it was.
1.4 Post-processing is not a magic fix
The new layer can fail too:
- A verifier can reject a true statement.
- A second LLM can agree with the first model's mistake.
- A knowledge graph can hold a wrong or outdated relationship.
- A citation can exist and still not support the claim it is attached to.
So the goal is not "add another model after the generator". The goal is an independent, evidence-based decision layer that finds errors, explains why a claim is unsupported and chooses an appropriate action.
2. The verification vocabulary: verdicts, abstention, independence
2.1 Three verdicts, three default actions
A production verifier must make its decision semantics explicit. Every claim ends in exactly one of three states.
| Verdict | Formal meaning | Required evidence | Default action |
|---|---|---|---|
| Supported | At least one authoritative evidence item entails the claim | Claim-level entailment, exact structured match or authoritative record | Publish with citation |
| Contradicted | Authoritative evidence entails the negation or an incompatible value | Direct contradiction, preferably from a higher-authority source | Correct, qualify or reject |
| Unresolved | No sufficient evidence either way | No relevant passage, absent graph edge, failed retrieval or unresolved conflict | Retrieve again, qualify or abstain |
A very common production bug is to treat a missing knowledge-graph edge as a contradiction. If the graph has no link between Researcher A and Paper B, the paper may simply not be indexed yet, the relation may not have been extracted, the schema may not model it, or the query may have failed. Return unresolved, never contradicted.
2.2 Abstention is part of the design
OpenAI's September 2025 article Why Language Models Hallucinate argues that standard training and evaluation can reward guessing over admitting uncertainty. A peer-reviewed formalisation appeared in Nature (2026), arguing that accuracy-only evaluation rewards guessing and proposing abstention-aware scoring. The practical consequence for us: the verifier must be allowed to say no, and abstaining is an application-level decision that needs no change to the generator.
2.3 Verifier independence: avoiding circular checking
The generator's output is untrusted input. If the generator, the retriever, the graph constructor and the verifier all rely on the same corpus or the same extraction model, they share the same blind spots. An incorrect authorship relation extracted from one document and stored in the graph will be confidently "confirmed" by a graph verifier built on that graph.
To keep the verifier independent:
- Keep source provenance and the original passages next to every structured fact.
- Use independently maintained, authoritative records for high-risk facts.
- Audit graph edges against source records on a schedule.
- Avoid using the same model, prompt and corpus for generation, graph construction and final verification.
- Log the model, prompt, retrieval configuration and KG version of every verification run.
2.4 Whole-answer versus claim-level checking
Early evaluation asked "is this answer correct?". That is fine for short question answering, but a long answer with ten claims, nine right and one invented, either gets rejected wholesale or waved through. Claim-level verification avoids both outcomes: it points at the exact statement to fix and leaves a clean audit trail.
RAGTruth gave this direction an empirical base: nearly 18,000 naturally generated RAG responses from open and closed LLMs, annotated at case and word level, with hallucination-intensity labels that separate minor unsupported spans from severe errors.
Use three verdicts. Treat "unresolved" as legitimate. Never turn a missing edge into a contradiction. Keep the verifier independent of the generator. Check claims, not whole answers.
2.5 A one-page glossary
These terms are used all through the tutorial. Skim them now and come back when one appears in a later section.
| Term | Meaning in this tutorial |
|---|---|
| Claim | One atomic, checkable statement extracted from the answer (section 4.1 shows how) |
| Evidence | Text passages, graph facts or records that a claim is checked against |
| Grounding vs factuality | Grounded: supported by the evidence you supplied. Factual: true in the world. A claim can be grounded in a wrong document, so you need both ideas |
| NLI / entailment | Natural language inference: a model labels a claim as entailed, contradicted or neutral given the evidence |
| Knowledge graph (KG), triple | A graph of entities and relations; a triple is one edge, for example (Paper B, published_in, 2024) |
| Provenance | The recorded path from a claim back to the source passages that support it |
| Abstention | The system declines to publish a claim or answer because support is insufficient |
| RAG | Retrieval-augmented generation: the generator is given retrieved passages before it answers |
| Cascade | Checks ordered from cheap to expensive; each claim stops at the first decisive one (section 6) |
| Golden set | A labelled test set you own and re-run before every release (section 9) |
3. The big picture: five architectures and the one to build
Before the details, here is the whole map. Terms you have not met yet (Claimify, GraphRAG, VeriTrail and so on) are explained in the sections that follow; this section only shows where each one plugs in.
3.1 Five architectures, from the lightest to the most complete
Start simple and add layers only when your error analysis justifies them. Here are five architectures, ordered from the lightest to the most complete.
Architecture 1: generator plus specialised verifier
Purpose: detect unsupported statements with minimal change. The candidate answer and its evidence go to a verifier (Lynx, HHEM or similar); a supported answer is published, a contradicted one is corrected, an unresolved one is abstained on.
- Pros: simple, model-agnostic, easy to benchmark.
- Cons: depends on evidence quality, weaker on relational or temporal errors, adds latency.
- Best for: customer-support assistants, internal document assistants, existing RAG systems.
Architecture 2: atomic claims plus independent verification
Purpose: find the exact unsupported statement. Take the answer "Company A introduced Product B in 2023, and the product reduced processing costs by 40%." The extractor should return two claims, and the second should carry its qualifiers and what evidence it needs:
[
{ "id": "C1", "claim": "Company A introduced Product B in 2023." },
{ "id": "C2", "claim": "Product B reduced processing costs by 40%.",
"qualifiers": ["baseline system", "reported by vendor"],
"needs_evidence": ["metric definition", "baseline", "evaluation period"] }
]
A useful verdict for C2 would be:
{
"claim_id": "C2",
"status": "unresolved",
"evidence_ids": ["E12", "E13"],
"contradiction_ids": [],
"reason": "The sources describe the product but do not establish the reported cost reduction.",
"required_evidence": ["cost metric", "baseline system", "measurement period", "independent evaluation"]
}
Foundation: Claimify, RAGTruth, VeriTrail, Lynx, HHEM. Main risk: extraction is itself a model task and can distort meaning, so keep the original sentence and check that the claims together cover it.
Architecture 3: KG-based post-generation verification
Purpose: catch wrong entities, relations, dates, quantities and multi-hop conclusions. Pipeline: claim extraction, entity linking, graph queries, structured comparison, NLI or semantic verification, decision. Handle three cases separately:
- Direct support: graph and authoritative source records agree.
- Direct contradiction: authoritative records conflict with the claim.
- Missing information: the relation is absent, so retrieve more evidence rather than rejecting.
Not every statement maps to a triple: qualified interpretations still need NLI over source passages, while the graph checks the underlying entities and relations (the hybrid GraphEval approach). Main risk: graph and verifier may inherit the same upstream error.
Architecture 4: citation validation and provenance
Purpose: make sure every factual statement has appropriate evidence, using three escalating levels:
- Existence: does the cited source exist?
- Relevance: does it address the claim?
- Entailment: does it support the claim as written?
For research papers read methods, results and limitations, not only the abstract, and for numbers establish metric, dataset, baseline and setting. Record the result in a provenance graph:
GeneratedAnswer ── CONTAINS_CLAIM ──> Claim C1 Claim C1 ── SUPPORTED_BY ──> Evidence E1 ── FROM ──> Source S1 Claim C1 ── VERIFIED_BY ──> VerificationRun V1
Main risk: a citation can be authentic and relevant yet not support the full claim.
Architecture 5: detect, retrieve, correct, re-verify
Purpose: turn detection into mitigation. Choose the response by error type:
| Error | Recommended response |
|---|---|
| Incorrect date | Replace with the supported date |
| Incorrect entity | Resolve the entity, regenerate the affected statement |
| Unsupported quantitative claim | Remove the number or retrieve evidence |
| Contradictory sources | Explain the disagreement or prefer the authoritative source |
| Missing graph relationship | Search more evidence; do not treat absence as contradiction |
| Unverifiable citation | Replace with a verified source or remove it |
| Unsupported causal inference | Weaken the claim or obtain causal evidence |
| Persistent uncertainty | Abstain or qualify explicitly |
Self-RAG and Corrective RAG are not purely post-generation, but their corrective-retrieval idea adapts well. Main risk: a correction model can introduce new errors, so corrected output goes through the same verification again.
3.2 The recommended combined architecture
The five architectures are not rivals. The recommended system combines them in one cascade. Read the figure from top to bottom: the generator proposes, everything below decides.
Each box in the figure maps to a technique explained later. Use this table as the index:
| Piece | Its job in the cascade | Explained in |
|---|---|---|
| Claimify-style extraction | High-quality atomic claims | section 4.1 |
| HHEM, MiniCheck, NLI, Lynx | Cheap, specialised factual-consistency screening | section 4.2 |
| NeMo Guardrails (or custom code) | Configurable enforcement of the policy | section 4.2.5 |
| VeriTrail-inspired provenance | Locating the stage that introduced an error in multi-step workflows | section 4.3 |
| Graph evidence store | Entities, relations, source passages, validity dates, verification results, model and prompt versions, correction history | sections 5.1 to 5.4 |
| GraphEval / GraphCorrect-inspired repair | Triple-level localisation and local repair | section 5.5 |
| GraphRAG-style indexing, GRIL-style retrieval | Corpus-level and multi-hop evidence selection | section 5.6 |
| NLP-first cascades, caching, batching, risk routing | Keeping the cost per decision low | section 6 |
Separate the generator from the decision to publish. The generator proposes an answer; the verification layer determines whether it is adequately supported.
4. Building blocks: claims, detectors, provenance and where to get them
No single tool does everything. The practical industrial pattern is layered verification: each layer does the job it is best at.
4.1 Claim extraction: Claimify, step by step
Everything downstream, retrieval, graph lookups, NLI, citations, depends on the claims you feed in. If extraction drops a statement, it is never checked (a coverage failure). If it keeps a statement but loses a qualifier or a referent, it checks the wrong thing (a decontextualization failure). Microsoft's Claimify (ACL 2025) was designed around exactly these two failures. Rather than asking one prompt to "list the claims", it runs four small, focused stages and refuses to guess: if a sentence cannot be made unambiguous from its context, it is dropped instead of being turned into a plausible-looking claim.
Reported performance (as summarised from the paper)
The paper evaluates claims on three criteria: entailment (are the claims supported by the source text?), coverage (are all verifiable elements captured?) and decontextualization (can each claim be checked on its own?). As summarised by a secondary source, Claimify matched the best baseline on entailment (about 99%) and led on coverage and decontextualization. Treat the exact numbers as indicative and read the paper before quoting them.
| Method | Entailment | Sentence-level coverage F1 | Element-level coverage F1 |
|---|---|---|---|
| Claimify | 99% | 93.5% | 91.9% |
| VeriScore | 99% | 90.2% | 85.4% |
| DnD | 97% | 88.5% | 86.3% |
| SAFE | 94% | 87.0% | 84.0% |
Is there code? An honest answer
The Microsoft Research pages describe the method and VeriTrail uses it internally, but I could not find an official public code release or model weights. VeriTrail itself is described as research-only and not commercially available. A third-party Claimify-style MCP server exists, but it is unofficial. The practical route is to re-implement the four stages from the paper with an LLM of your choice, which is what the code below does. Validate it with the checks that follow before trusting it.
A working re-implementation (Gemini on Vertex AI)
This sketch uses the google-genai SDK with structured JSON output, so each stage returns a typed object instead of free text. It is unofficial and simplified: the prompts are short paraphrases of the paper's stage descriptions, and you should tune them on your own examples.
# claimify_lite.py (UNOFFICIAL: re-implements the four published stages)
# pip install google-genai pydantic nltk ; python -c "import nltk; nltk.download('punkt_tab')"
import re
from typing import Literal
from pydantic import BaseModel
from nltk.tokenize import sent_tokenize
from google import genai
from google.genai import types
client = genai.Client(vertexai=True, project="PROJECT", location="LOCATION") # pick a region where the model is offered
MODEL = "gemini-2.5-flash"
def ask(prompt: str, schema: type[BaseModel]):
r = client.models.generate_content(
model=MODEL, contents=prompt,
config=types.GenerateContentConfig(
temperature=0, response_mime_type="application/json", response_schema=schema))
return r.parsed
class Selection(BaseModel):
label: Literal["non_verifiable", "partially_verifiable", "fully_verifiable"]
rewritten: str # only the checkable, fact-focused content
class Disambiguation(BaseModel):
resolved_sentence: str
unresolved: bool # True if an ambiguity could not be resolved from the context
notes: str
class Decomposition(BaseModel):
claims: list[str] # standalone claims; context added from outside goes in [brackets]
def stage2_select(context: str, sentence: str) -> Selection:
return ask(f"""Context (earlier text):
{context}
Sentence: {sentence}
Does the sentence contain verifiable factual content? Label it non_verifiable,
partially_verifiable or fully_verifiable. If partially verifiable, rewrite it keeping
ONLY the checkable factual content and every qualifier (e.g. "reportedly", "about").""", Selection)
def stage3_disambiguate(context: str, sentence: str) -> Disambiguation:
return ask(f"""Context (earlier text):
{context}
Sentence: {sentence}
Find ambiguity: pronouns, vague times ("last year"), unclear scope. Resolve each one
using ONLY the context. If any ambiguity cannot be resolved from the context, set
unresolved=true. Never guess.""", Disambiguation)
def stage4_decompose(sentence: str) -> Decomposition:
return ask(f"""Split this sentence into the smallest standalone factual claims.
Each claim must be understandable alone and keep all qualifiers. Put any information
that you add from outside the sentence in [square brackets].
Sentence: {sentence}""", Decomposition)
def extract_claims(text: str, p: int = 1, f: int = 0, drop_unresolved: bool = True):
sents = sent_tokenize(text) # stage 1: split
out = []
for i, s in enumerate(sents):
context = " ".join(sents[max(0, i - p): i] + sents[i + 1: i + 1 + f]) # stage 1: context window
sel = stage2_select(context, s) # stage 2
if sel.label == "non_verifiable":
continue
dis = stage3_disambiguate(context, sel.rewritten) # stage 3
if dis.unresolved and drop_unresolved:
out.append({"sentence": i, "claims": [], "dropped": dis.notes}) # log it, do not guess
continue
dec = stage4_decompose(dis.resolved_sentence) # stage 4
out.append({"sentence": i, "claims": dec.claims, "source": sel.rewritten})
return out
Validating the extractor before you trust it
Two cheap automatic checks catch most extraction bugs. A decontextualization lint flags claims that still contain loose pronouns or relative dates. A coverage check asks an entailment model whether the extracted claims, taken together, still support the verifiable part of the original sentence; a low score means something was lost.
import re
PRONOUNS = re.compile(r"\b(it|its|they|them|their|this|that|these|those|he|she|his|her)\b", re.I)
RELATIVE_TIME = re.compile(r"\b(last year|next year|recently|currently|yesterday|today|soon|previously)\b", re.I)
def lint_claim(claim: str) -> list[str]:
bare = re.sub(r"\[[^\]]*\]", "", claim) # ignore text added in [brackets]
issues = []
if PRONOUNS.search(bare): issues.append("loose pronoun")
if RELATIVE_TIME.search(bare): issues.append("relative time, needs a date")
return issues
def coverage_gap(hhem, rewritten_sentence: str, claims: list[str], threshold: float = 0.5) -> bool:
"""True if the claims no longer support the sentence they came from."""
score = hhem.predict([(" ".join(claims), rewritten_sentence)])[0].item()
return score < threshold
Tuning and cost levers
| Knob | Effect | Advice |
|---|---|---|
Context window p, f | More context resolves more ambiguity, costs more tokens | Start with one or two preceding sentences and no following ones; raise only if many sentences are dropped as unresolved |
drop_unresolved | Strict: never guess, may lose claims. Lenient: keep with a "needs evidence" flag | Strict for high-risk domains; lenient plus a flag when recall matters more |
| Model size per stage | Four LLM calls per sentence add up | A small model often suffices for selection; use a stronger one for disambiguation |
| Skip non-claims early | Greetings, instructions and opinions need no LLM call | Cheap regex or a tiny classifier before stage 2 |
| Batching and caching | Repeated sentences recur across answers | Cache by hash of sentence plus context; send a whole paragraph's sentences in one request where possible |
| Temperature | Randomness breaks reproducibility | Keep at 0 and log the prompt and model version with every run |
What can still go wrong
| Failure | Example | Mitigation |
|---|---|---|
| Over-dropping | Many sentences removed as "unresolved" | Log drops; widen the context window; review a sample weekly |
| Qualifier loss | "reportedly cut costs" becomes "cut costs" | Prompt must name qualifiers; add a test set with hedges, numbers and conditions |
| Over-splitting | One causal statement becomes two unrelated claims | Keep the original sentence next to its claims so a reviewer can compare |
| Hallucinated context | A bracketed addition that is not in the context | Check bracketed text against the context; reject claims whose brackets introduce new facts |
4.2 Specialised detectors: what they take, what they return, and how to call them
Dedicated evaluators separate generating from judging. They are not interchangeable: they take different inputs, return different outputs and have different licences. The practical way to live with that is an adapter layer: wrap every detector behind one small interface that returns a normalised verdict, so the cascade of section 6 can mix and swap them freely.
HHEM, MiniCheck and groundedness services answer "is this supported by the evidence?". A low score means not supported, which covers both "the evidence says the opposite" and "the evidence says nothing". Following the verdict rules of section 2, map a low score to unresolved by default, and to contradicted only when a 3-class NLI model or a judge reports an actual contradiction.
4.2.1 Vectara HHEM-2.1-Open: the fast first screen
HHEM (Hughes Hallucination Evaluation Model) takes a (premise, hypothesis) pair, where the premise is your evidence and the hypothesis is the claim, and returns a factual-consistency score between 0 (not supported at all) and 1 (fully supported). The open version is released on Hugging Face under Apache-2.0, is fine-tuned from google/flan-t5-base and has no fixed context-length limit. Vectara reports it runs on a consumer GPU or a modern CPU, at roughly half a second per judgment on a good GPU. A commercial version is available only through Vectara's own platform.
# pip install transformers torch
from transformers import AutoModelForSequenceClassification
hhem = AutoModelForSequenceClassification.from_pretrained(
"vectara/hallucination_evaluation_model",
trust_remote_code=True, # runs code from the repo: pin a revision (see 3.5)
)
pairs = [ # (premise = evidence, hypothesis = claim)
("The Eiffel Tower is located in Paris, France.", "The Eiffel Tower is located in Berlin."),
("The Eiffel Tower is located in Paris, France.", "The Eiffel Tower is in France."),
("The Eiffel Tower is located in Paris, France.", "The Eiffel Tower is a major tourist attraction."),
]
scores = hhem.predict(pairs) # use predict(), not model(pairs); returns scores in [0, 1]
print([round(float(s), 3) for s in scores])
# expect: low for Berlin, high for "in France", low for the tourist-attraction claim,
# which is plausible but NOT established by the source
The model card does not give a recommended threshold, and the right one depends on your data. Choose it on a labelled development set by fixing the precision you need:
import numpy as np
from sklearn.metrics import precision_recall_curve
def pick_threshold(scores, is_supported, min_precision=0.95):
"""Smallest threshold whose precision for 'supported' is at least min_precision."""
prec, rec, thr = precision_recall_curve(is_supported, scores)
ok = np.where(prec[:-1] >= min_precision)[0]
return float(thr[ok[0]]) if len(ok) else None # None: this model cannot reach that precision
# thr = pick_threshold(dev_scores, dev_labels) # then: score >= thr -> supported, else -> unresolved
4.2.2 MiniCheck: small, fast, built for claim-level checking
MiniCheck is a family of compact fact-checking models trained for checking a claim against a (possibly long) document. Its GitHub repository (Liyan06/MiniCheck) states an MIT licence and offers RoBERTa-Large, DeBERTa-v3-Large and Flan-T5-Large checkpoints, plus larger variants published by Bespoke Labs. It returns a binary label and a probability.
# git clone https://github.com/Liyan06/MiniCheck.git && pip install -r requirements.txt from minicheck.minicheck import MiniCheck doc = "A group of students gather in the school library to study for their upcoming final exams." claims = ["The students are preparing for an examination.", "The students are on vacation."] scorer = MiniCheck(model_name="flan-t5-large", device="cuda:0", cache_dir="./ckpts") # model_name: 'roberta-large' | 'deberta-v3-large' | 'flan-t5-large' labels, probs, _, _ = scorer.score(docs=[doc] * len(claims), claims=claims) print(labels, probs) # 1 = supported, 0 = unsupported; e.g. [1, 0] with probabilities
4.2.3 A 3-class NLI model: the only cheap way to see "contradicted"
A natural-language-inference model classifies a pair as entailment, neutral or contradiction. That third class is what lets you separate "the evidence says otherwise" from "the evidence says nothing". Widely used checkpoints are fine-tuned DeBERTa models; the one below is a popular example, but check its model card for licence and intended use.
from transformers import pipeline
nli = pipeline("text-classification",
model="MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli",
top_k=None) # return all three class scores
def nli_verdict(evidence: str, claim: str):
scores = {d["label"]: d["score"] for d in nli({"text": evidence, "text_pair": claim})[0]}
return scores # {'entailment': .., 'neutral': .., 'contradiction': ..}
print(nli_verdict("Paper B was published in 2024.", "Paper B was published in 2025."))
# 'contradiction' should dominate: this is the case that deserves the label "contradicted"
4.2.4 Patronus Lynx: an LLM judge that explains itself
Lynx is a fine-tuned LLM (an 8B and a 70B variant are referenced in public documentation) that receives a question, a document and an answer and outputs a verdict with its reasoning. It checks three things: the answer's information is contained in the document, the answer adds nothing absent from it and the answer does not contradict it. Because it reasons, it handles harder cases than a score-only model, at higher cost. Check the licence: the Hugging Face card for the 8B model lists cc-by-nc-4.0, which restricts commercial use.
# Serve it (OpenAI-compatible server, port 8000 by default):
# pip install vllm
# vllm serve PatronusAI/Llama-3-Patronus-Lynx-8B-Instruct
import json
from openai import OpenAI
lynx = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
MODEL = "PatronusAI/Llama-3-Patronus-Lynx-8B-Instruct"
# PARAPHRASED template. For best accuracy copy the exact wording from the model card
# or from the Lynx paper (arXiv 2407.08488); the model was trained on that format.
TEMPLATE = """Given the following QUESTION, DOCUMENT and ANSWER you must analyze the provided answer
and determine whether it is faithful to the contents of the DOCUMENT. The ANSWER must not offer
new information beyond the context provided in the DOCUMENT and must not contradict it.
Output "PASS" if the answer is faithful, "FAIL" otherwise. Show your reasoning.
QUESTION: {question}
DOCUMENT: {document}
ANSWER: {answer}
Your output should be JSON with the keys "REASONING" and "SCORE"."""
def lynx_check(question: str, document: str, answer: str) -> dict:
r = lynx.chat.completions.create(
model=MODEL, temperature=0, max_tokens=256,
messages=[{"role": "user", "content": TEMPLATE.format(
question=question, document=document, answer=answer)}])
try:
out = json.loads(r.choices[0].message.content)
return {"faithful": str(out["SCORE"]).upper() == "PASS", "reasoning": out.get("REASONING")}
except (ValueError, KeyError):
return {"faithful": None, "reasoning": "unparseable"} # unresolved, NEVER approved
4.2.5 NVIDIA NeMo Guardrails: running Lynx as an output rail
NeMo Guardrails documents a ready-made Patronus Lynx output rail. You serve Lynx behind an OpenAI-compatible endpoint, declare it as a model, and enable the rail. If Lynx says the answer is hallucinated, the bot's reply is replaced with an "answer unknown" response. By NVIDIA's documentation the rail fails safe: if Lynx's output cannot be parsed, it treats the answer as a hallucination. The rail needs the retrieved chunks to be available as relevant_chunks and runs on the LLMRails engine.
# config.yml (from the NeMo Guardrails Lynx integration page)
models:
- type: patronus_lynx
engine: openai
model: PatronusAI/Patronus-Lynx-70B-Instruct # or the 8B model
parameters:
base_url: "http://localhost:5000/v1" # your vLLM server
api_key: EMPTY
rails:
output:
flows:
- patronus lynx check output hallucination
The rail also needs a prompt for the task patronus_lynx_check_output_hallucination in prompts.yml; start from the one shipped with NeMo Guardrails instead of writing your own. Install the framework with pip install nemoguardrails.
4.2.6 Managed services: Azure groundedness and Google check grounding
If you prefer not to host a model, two cloud providers offer groundedness as an API. Both answer "is this text supported by these sources?", not "is this text true". Both have size limits, so claim-level calls with small source sets are the natural fit.
Azure AI Content Safety, groundedness detection. A REST call takes the text to check and the grounding sources, and can optionally ask for a reason for each ungrounded span (which uses an Azure OpenAI deployment you supply). The documented limits are 7,500 characters of text and 55,000 characters of sources per request. The documentation also flags that the semantics of ungroundedPercentage look inconsistent, so rely on ungroundedDetected and the span list and test the field on known examples.
import requests
ENDPOINT = "https://<resource>.cognitiveservices.azure.com"
URL = f"{ENDPOINT}/contentsafety/text:detectGroundedness?api-version=2024-02-15-preview" # check the current api-version
def azure_grounded(text: str, sources: list[str], key: str) -> dict:
body = {"domain": "Generic", "task": "Summarization",
"text": text, "groundingSources": sources, "reasoning": False}
r = requests.post(URL, json=body, headers={"Ocp-Apim-Subscription-Key": key}, timeout=10)
r.raise_for_status()
j = r.json()
return {"ungrounded": j["ungroundedDetected"],
"spans": [d["text"] for d in j.get("ungroundedDetails", [])]}
Google Cloud, check grounding (Agent Search / Discovery Engine API). You send an answerCandidate (up to 4,096 tokens) and up to 200 facts, and receive a support score between 0 and 1 plus claim-level citations to the facts that support each claim. A citationThreshold (default 0.6) controls how strict the citations are. This fits a GCP deployment naturally, with IAM-based authentication and no model to host.
# pip install google-cloud-discoveryengine
from google.cloud import discoveryengine_v1 as de
client = de.GroundedGenerationServiceClient()
config = client.grounding_config_path(project="PROJECT", location="global",
grounding_config="default_grounding_config")
req = de.CheckGroundingRequest(
grounding_config=config,
answer_candidate="Titanic was directed by James Cameron.",
facts=[de.GroundingFact(fact_text="Titanic is a 1997 film directed by James Cameron.",
attributes={"author": "Example Source"})],
grounding_spec=de.CheckGroundingSpec(citation_threshold=0.6),
)
resp = client.check_grounding(request=req)
print(resp.support_score) # 0..1; per-claim citations are in resp.claims
4.2.7 Putting them behind one interface
Here is the adapter layer of Figure 8 in code. Each detector returns the same Verdict, and a small policy function applies the verdict rules from section 2.
from dataclasses import dataclass
from typing import Literal, Optional, Protocol
Label = Literal["supported", "contradicted", "unresolved"]
@dataclass
class Verdict:
label: Label
score: Optional[float]
rationale: str
detector: str
class Detector(Protocol):
def check(self, claim: str, evidence: str) -> Verdict: ...
class HHEMDetector:
def __init__(self, model, thr: float): self.m, self.thr = model, thr
def check(self, claim, evidence):
s = float(self.m.predict([(evidence, claim)])[0])
# a low score means "not supported", never "contradicted"
return Verdict("supported" if s >= self.thr else "unresolved", s, f"HHEM score {s:.2f}", "hhem")
class NLIDetector:
def check(self, claim, evidence):
p = nli_verdict(evidence, claim)
if p["entailment"] >= 0.9: return Verdict("supported", p["entailment"], "entailed", "nli")
if p["contradiction"] >= 0.9: return Verdict("contradicted", p["contradiction"], "contradicts evidence", "nli")
return Verdict("unresolved", max(p.values()), "neutral or low confidence", "nli")
def cascade(claim: str, evidence: str, detectors: list[Detector]) -> Verdict:
"""Cheapest detector first; stop at the first decisive verdict (section 6)."""
last = Verdict("unresolved", None, "no detector was decisive", "none")
for d in detectors:
try:
v = d.check(claim, evidence)
except Exception as e: # a timeout or outage must never become approval
last = Verdict("unresolved", None, f"{type(d).__name__} failed: {e}", "error")
continue
if v.label != "unresolved":
return v
last = v
return last
4.2.8 Which one to start with
| Need | Start with | Why |
|---|---|---|
| Fast, cheap first screen, commercial use | HHEM-2.1-Open (Apache-2.0) or MiniCheck (MIT) | Small, CPU or one modest GPU, permissive licences |
| Need to see contradictions | 3-class NLI model | Only it separates "contradicted" from "neutral" cheaply |
| Hard cases, need explanations | An LLM judge (Lynx, or Gemini through your own prompt) | Reasoning handles multi-sentence and implicit cases; check Lynx's non-commercial licence |
| No hosting capacity | Google check grounding or Azure groundedness | Managed, IAM-integrated, but limits and per-call cost apply |
| Policy and enforcement around the checks | NeMo Guardrails | Orchestrates rails; it is a control plane, not a detector |
Whatever you pick, calibrate on your own labelled claims. Published scores on public benchmarks tell you which models are worth testing, not what threshold to ship.
4.3 Provenance: Microsoft VeriTrail, step by step
Agentic and GraphRAG workflows have many steps, and an error can enter at an intermediate summary, a tool call, a retrieval step or a graph-construction step, not only at the last generation call. VeriTrail treats the whole workflow as a directed acyclic graph (DAG) in which every node is a piece of text: source documents, intermediate outputs and the final answer. It extracts claims from the final output (with Claimify) and verifies each claim by walking the graph backwards, from the final output toward the sources. At each step a language model first selects evidence sentences from the nodes just upstream and then gives a verdict: Fully Supported, Not Fully Supported or Inconclusive. The walk stops when it reaches source text, or when a user-set number of consecutive "Not Fully Supported" verdicts is reached.
Microsoft describes VeriTrail as used for research purposes and not available commercially, and I could not find a public code release. So treat it as a design pattern to re-implement, not a package to install. The core is small, as the sketch shows.
# veritrail_lite.py (illustrative, simplified from the published description)
from dataclasses import dataclass, field
@dataclass
class Node:
id: str
text: str
parents: list[str] = field(default_factory=list) # upstream nodes this text was produced from
is_source: bool = False
def verify_claim(claim: str, start: str, dag: dict[str, Node], llm, k_stop: int = 2):
"""Walk backwards from the node holding the claim; return verdict + evidence trail."""
trail, current, bad_in_a_row = [], [start], 0
while current:
upstream = [dag[p] for n in current for p in dag[n].parents]
if not upstream: # reached the top of the DAG
break
evidence = llm.select_sentences(claim, upstream) # LLM picks supporting sentences (with node IDs)
verdict = llm.judge(claim, evidence) # 'fully' | 'not_fully' | 'inconclusive'
trail.append({"nodes": [n.id for n in upstream], "evidence": evidence, "verdict": verdict})
if verdict == "inconclusive":
return {"verdict": "inconclusive", "trail": trail}
bad_in_a_row = bad_in_a_row + 1 if verdict == "not_fully" else 0
if bad_in_a_row >= k_stop: # the error entered near here
return {"verdict": "not_fully_supported", "trail": trail,
"localised_at": [n.id for n in upstream]}
if all(n.is_source for n in upstream):
break
current = [n.id for n in upstream]
final = "not_fully_supported" if bad_in_a_row else "fully_supported"
return {"verdict": final, "trail": trail}
The returned trail is exactly the evidence trail that section 8 turns into an explanation, and localised_at is what tells you whether to fix retrieval, the summariser or the final generation.
Five failures VeriTrail can tell apart
Walking backwards through the workflow lets a verifier distinguish five different failures, and each needs a different fix. Figure 10 shows where they sit.
4.4 Where to get the models and code
Not everything in this tutorial is a downloadable model. Some pieces are open weights, some are managed APIs and some are research methods with no public code. The table is deliberately honest about which is which, with the licence as listed on each source at the research date. Licences change and model cards are updated, so re-check before you ship.
| Piece | Where | Terms as listed | How you run it |
|---|---|---|---|
| HHEM-2.1-Open consistency scorer | Hugging Face vectara/hallucination_evaluation_model; also Kaggle | Apache-2.0 | transformers with trust_remote_code=True; CPU or consumer GPU. The commercial HHEM is only inside Vectara's platform |
| MiniCheck claim checker | GitHub Liyan06/MiniCheck; Hugging Face lytang/MiniCheck-RoBERTa-Large, -DeBERTa-v3-Large, -Flan-T5-Large; larger bespokelabs/Bespoke-MiniCheck-7B | MIT for the repository (check each checkpoint and the 7B model's own card) | Python package from the repo; GPU recommended |
| 3-class NLI | Hugging Face, for example MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli | Check the model card | transformers text-classification pipeline |
| Patronus Lynx LLM judge | Hugging Face PatronusAI/Llama-3-Patronus-Lynx-8B-Instruct; the 70B model PatronusAI/Patronus-Lynx-70B-Instruct is referenced in NVIDIA's docs; quantised GGUF builds also exist | cc-by-nc-4.0 on the 8B card: non-commercial. Check the others | vLLM, SGLang, Ollama or Docker Model Runner behind an OpenAI-compatible API |
| NeMo Guardrails control plane | pip install nemoguardrails; docs at docs.nvidia.com/nemo/guardrails | Open-source toolkit, check the repository licence | Library inside your service; config in YAML plus Colang |
| Azure groundedness | Azure AI Content Safety (managed API) | Commercial service, per-call pricing | REST call; request size limits apply |
| Google check grounding | Discovery Engine (Agent Search) API on Google Cloud | Commercial service, per-call pricing | REST or client library, IAM authentication |
| Claimify | ACL 2025 paper and Microsoft Research page | No official code found; a third-party MCP implementation exists, unofficial | Re-implement from the paper (section 4.1) |
| VeriTrail | arXiv 2505.21786 and the Microsoft Research blog | Described as research use, not commercially available | Re-implement the pattern (section 4.3) |
| HalluGuard (4B verifier) | arXiv 2510.00880 | Check the paper for released weights and licence | Evaluate only if weights are published |
Extract atomic claims first. Use specialised detectors (Lynx, HHEM, managed groundedness) as components, not as oracles. Use a guardrail framework to orchestrate and enforce. Use provenance tracing for multi-step workflows. Make abstention a normal outcome.
5. Knowledge graphs as a verification tool
5.1 Why a graph helps after generation
A knowledge graph (KG) stores entities, relations and attributes. That makes some claims much easier to check than free text. The sentence "Researcher A authored Paper B in 2025" becomes two small facts the verifier can inspect separately:
Researcher A ── AUTHORED ──> Paper B Paper B ── PUBLICATION_YEAR ──> 2025
The catch: the graph must be linked to reliable source records. If the authorship edge was extracted wrongly, the graph simply reproduces the mistake.
5.2 Six kinds of graph-based checks
| Check | Example question | Method |
|---|---|---|
| Entity | Does this paper exist? | Entity resolution and authoritative lookup |
| Relation | Did Researcher A author Paper B? | Graph query plus source check |
| Attribute | Was it published in 2025? | Structured attribute comparison |
| Temporal | Was Person A CEO in 2022? | Time-aware graph query |
| Multi-hop | Did a researcher work for an organisation that acquired Company B? | Graph traversal and inference validation |
| Provenance | Which source supports this edge? | Source-linked graph and evidence inspection |
5.3 Store evidence, not just facts
Each fact in the graph should carry its origin and status, and should separate when the fact was valid from when it was recorded. Keep the original source passage reachable so a verifier can re-read it when the structured form is ambiguous.
{
"subject": "Paper B",
"predicate": "publication_year",
"object": 2025,
"source_id": "S17",
"source_type": "publisher_metadata",
"source_location": "publication_record",
"observed_at": "2026-10-27",
"valid_from": "2025-01-01",
"verification_status": "verified"
}
5.4 Two verification patterns you can implement today
Two more patterns round out the toolbox:
- Pattern C, community or subgraph grounding: for broad analytical questions, retrieve the relevant GraphRAG communities or subgraphs into an evidence pack, then generate or verify against it.
- Pattern D, claim-to-graph reconstruction and correction: convert the generated answer into its own small graph, verify triple by triple, repair only the bad triple, then re-verify. This follows GraphEval and GraphCorrect and is shown next.
5.5 Localising the error: the GraphEval / GraphCorrect idea
Amazon Science's GraphEval represents a response as a knowledge graph, flags the triples most likely to be hallucinated and combines this with NLI models. The authors report better balanced accuracy than raw NLI alone. GraphCorrect then uses the graph structure to attempt repair, and the authors report that a majority of hallucinations can be rectified in their evaluated setting. The key contribution is localisation: the system names the faulty entity, relation or triple instead of condemning the whole answer.
Here is a worked repair on a factual example:
Response : "Marie Curie won the Nobel Prize in Physics in 1911
and the Nobel Prize in Chemistry in 1903."
Triples : T1 (Marie Curie, WON, Nobel Prize in Physics) year=1911
T2 (Marie Curie, WON, Nobel Prize in Chemistry) year=1903
Evidence : "Curie shared the 1903 Nobel Prize in Physics and won
the 1911 Nobel Prize in Chemistry."
Verdicts : T1 year=1911 -> CONTRADICTED (evidence: Physics = 1903)
T2 year=1903 -> CONTRADICTED (evidence: Chemistry = 1911)
Repair : swap the two years, then RE-VERIFY both triples
Output : "Marie Curie won the Nobel Prize in Physics in 1903
and the Nobel Prize in Chemistry in 1911."
Notice what happened: the model had the right two prizes and the right two years but crossed them. A whole-answer "not factual" label would say nothing about this; triple-level localisation points at the exact attributes and enables a two-word repair that is then verified again.
5.6 State of the art: what the big players actually built
Industrial knowledge-graph systems are easier to understand once you separate five jobs that are often blurred together: constructing the graph from text, retrieving from it, grounding generation in what was retrieved, verifying the finished answer and tracing where an error entered. Almost no single product does all five. The map below shows who contributes what; this section walks through the graph-side contributions with a scheme and a worked example: GraphRAG, dynamic community selection, GRIL and Google's encoding-versus-recall work. GraphEval and GraphCorrect are in section 5.5, VeriTrail in section 4.3, NeMo Guardrails in section 4.2.5 and the benchmarks in section 9.4.
5.6.1 Microsoft GraphRAG: turning a corpus into an index you can query globally
Ordinary vector RAG finds passages that look like the question. That works for "what did the audit memo say about Beta?" and fails for "what are the main risks across all our documents?", because no single passage contains the answer. GraphRAG attacks this second kind of question, called global sensemaking. An LLM first extracts entities and relationships from every document and builds a graph. The graph is then split into hierarchical communities of tightly connected nodes, and a short report is written for each community. At query time each community report produces a partial answer and the partial answers are combined, a map-reduce pattern. The original evaluation reported better comprehensiveness and diversity than vector RAG on global questions.
5.6.2 Dynamic community selection: retrieve the smallest sufficient evidence
Asking every community report to contribute is expensive. Dynamic community selection starts at the root of the hierarchy and asks an LLM whether each community report is relevant to the question. A relevant community is kept and its children are examined; an irrelevant one is pruned together with its whole subtree. Only the surviving reports feed the answer.
5.6.3 Amazon GRIL: let the question steer the graph traversal
Many GraphRAG systems retrieve first and reason afterwards, so the retriever never learns what the reasoner needs. GRIL trains a graph retriever jointly with the LLM. An attention-based grow-and-prune mechanism walks multi-hop neighbourhoods and filters noise. The chosen subgraph reaches the LLM in two forms, soft graph tokens (a compact learned encoding) and a verbalised graph (readable text). The LLM's own logits flow back as an implicit training signal for the retriever.
GRIL is a generation-time method, but it teaches a rule you can reuse after generation: retrieve conditioned on the claim being checked.
| Claim type | Example | Retrieval strategy |
|---|---|---|
| Direct relation | "A authored B" | One-hop lookup of the edge plus its source passage |
| Temporal | "A was CEO in 2022" | Time-aware path: filter edges by validity interval |
| Causal | "X caused the delay" | Constrain relation types (CAUSES, DELAYS); require text evidence, not just co-occurrence |
| Multi-hop | "A acquired a subsidiary of B" | Expand the path hop by hop until the chain is complete or declared incomplete |
| Aggregate | "Most suppliers are in Asia" | Community or subgraph retrieval, then count |
5.6.4 Google: separating "never learned" from "cannot recall"
Google contributes two ideas about measurement. The FACTS Benchmark Suite scores four skills separately (grounding in a supplied document, parametric facts from memory, search-assisted facts and multimodal facts) so one factuality number cannot hide qualitatively different failures. Knowledge profiling goes one level deeper and separates encoding, whether the model ever acquired a fact, from recall, whether it can bring the fact back when asked.
5.7 When a KG helps, and when it does not
| KG is strong for structural failures | KG is weak for linguistic or interpretive failures |
|---|---|
| Wrong entity linking Incorrect relation Wrong temporal attribute Invalid multi-hop inference Missing provenance Contradicting structured records | Misreading a qualification Ignoring a negation Turning correlation into causation Misrepresenting uncertainty Fabricating a citation on the right topic |
That is why the strongest design is hybrid: structured claims go to the graph, textual claims to retrieval plus NLI, cited claims to the citation checker, and multi-step claims to provenance tracing (exactly the routing shown in Figure 6).
More research worth knowing: Can Knowledge Graphs Reduce Hallucinations in LLMs? A Survey (NAACL 2024); Lie to Me, which converts responses into entity-relation graphs for self-detection and reports up to 16% relative accuracy and 20% F1 gains over standard self-detection and SelfCheckGPT with GPT-4o and Gemini-2.5-Flash (ICPRAM 2026); and KGHaluBench (EACL 2026 Findings), a graph-built benchmark whose automated verification agrees with human judgement 79.19% (conceptual) and 87.74% (correctness) of the time.
A graph is evidence only when it carries provenance and validity dates. Use it for relations, dates and multi-hop paths. Never read a missing edge as falsehood. Localise errors at triple level, and retrieve the smallest sufficient subgraph first.
6. Making it affordable: NLP-first, LLM-last
If a large LLM judges every claim, verification can cost several times more than generation. Classical and lightweight NLP components can do most of the work, so the principle is progressive filtering: run the cheapest reliable test first and escalate only when needed.
6.1 The NLP toolbox
| Component | Role in verification | Cost benefit |
|---|---|---|
| Sentence segmentation | Split answers into units | Enables selective verification |
| Claim detection | Separate factual from non-factual sentences | Skips opinions, instructions, greetings |
| Named-entity recognition | Find people, organisations, dates, places | Routes entity claims to the KG |
| Entity linking | Map surface forms to canonical IDs | Avoids false graph misses and duplicate retrieval |
| Relation extraction | Turn a claim into a candidate triple | Enables cheap structured matching |
| Semantic similarity | Find likely relevant passages | Fewer passages reach NLI or the LLM |
| Natural-language inference | Entailment, contradiction or neutral | Resolves easy claims without an LLM |
| Semantic deduplication | Merge equivalent claims | Verify each unique claim once |
| Query expansion | Alternative evidence queries | Better recall without regenerating |
| Citation parsing | Normalise references | Deterministic citation checks first |
6.2 The five-tier cascade
Evidence that this works: a deep-research verification study reports that an NLI-then-LLM cascade resolved about 40% of claims directly with a specialised NLI model on both FEVER and SciFact, cutting expensive LLM inference by roughly 40% without hurting overall accuracy.
The cascade in words:
- Deterministic checks first. Validate dates, identifiers, URLs, citation format, numeric ranges and exact KG triples with no model call.
- Embedding retrieval second. Keep only the top-k relevant passages.
- Small NLI model third. Classify each claim-evidence pair.
- LLM verification fourth. Only for neutral, conflicting, multi-hop, temporal, causal or high-risk claims.
- Correction last. Regenerate only the affected claim or sentence.
6.3 Seven cost levers
1. Verify unique claims, not whole answers
Extract atomic claims, deduplicate paraphrases, verify each unique claim once and attach the verdict to every occurrence.
2. Cache with the right keys
| Cache key | Example | Benefit |
|---|---|---|
| Claim hash | Normalised claim text | No re-verification of repeats |
| Claim + evidence hash | Claim plus passage IDs | No repeated NLI or LLM run |
| Entity + relation + value | Paper B / publication_year / 2025 | Reuses structured KG checks |
| Citation hash | DOI, URL or bibliographic ID | Reuses existence and metadata checks |
| Source version | Document ID plus content hash | Invalidates when sources change |
Always include source-version and verifier-version metadata in cache entries. A cached verdict must not outlive a change to the evidence, KG, prompt or verifier model.
3. Filter with embeddings before expensive reasoning
Use a small k for low-risk claims and expand it only when the verifier returns "unresolved". E-Verify moves part of the verification into embedding space and reports a 22× speedup over an NLI-DeBERTa baseline, 46× over MiniCheck and a 4.84× lower per-document cost in its setting. Those figures are implementation- and benchmark-specific, but they show the potential.
4. Prefer small specialised verifiers
HalluGuard is a 4B-parameter model that classifies document-claim pairs as grounded or hallucinated with evidence-based justifications, reporting 84.0% average balanced accuracy on RAGTruth, comparable to MiniCheck-7B and above Granite Guardian 3.3 at roughly half the parameters. In FEVER 2025 a fine-tuned 270M-parameter XLM-R model reached 57.7% macro-F1 against 16.9% for the best 7B-12B decoder LLM tested on an in-domain task. Small models do not always win, but for narrow, well-defined verification they are a serious option.
5. Batch
Batch several claims against one passage, or several passages against one claim, ask for structured JSON output, and use asynchronous processing for non-interactive workflows.
6. Route by risk
| Claim type | Examples | Policy |
|---|---|---|
| Low risk | Definitions, navigation, formatting | Claim detection; skip deep verification |
| Medium risk | Product facts, history, document summaries | Embedding filter + NLI |
| High risk | Medical, legal, financial, safety, security | KG validation + NLI + LLM verification |
| Multi-hop | "A acquired a subsidiary of B" | Path validation + NLI + optional LLM |
| Citation-dependent | Research claims, quantitative results | Citation parsing + entailment + authority check |
| Temporal | "As of 2026…" | Time-aware KG query + freshness check |
Putting the signals together. The routing policy should combine five signals: claim risk (medical, legal, financial, safety and security claims get deeper checks), claim structure (multi-hop and temporal claims get path validation), evidence availability (unresolved claims get more retrieval before LLM escalation), citation dependence (cited claims get existence, relevance and entailment checks) and business impact (high-impact claims may always get the strong verifier). Calibrate every threshold on representative production data.
7. Let the graph prune retrieval, and cap correction loops
Entity linking narrows the search to a candidate subgraph: an exact triple is checked against its source passage, a path candidate fetches only path-supporting documents, and no candidate means broaden retrieval or return unresolved. For correction, fix only contradicted claims, allow one or two attempts, re-verify only the corrected claim and its context, prefer qualification over regeneration when evidence is incomplete, and abstain after repeated failure.
Two further ideas: mixed strong/weak model consortia can improve the cost-performance frontier over spending the whole sampling budget on one strong model, and a strong verifier can be distilled into a compact one (run it offline, store labels and rationales, fine-tune a small model, evaluate on held-out data, route disagreements back to the strong model). A cost-focused QA-agent study found self-training and knowledge distillation give comparable hallucination reduction, with self-training much cheaper.
6.4 Measure cost per decision
Track the total, and the cost per verified claim, not just API calls:
Cost per answer = C_generation + C_retrieval + C_KG + C_NLI
+ C_LLM_verifier + C_correction + C_reverification
Cost per verified claim = (C_NLI + C_LLM_verifier + C_retrieval + C_KG)
/ number of unique claims verified
Also log the share of claims resolved by rules, by NLI, escalated to an LLM, needing extra retrieval, corrected or abstained; evidence passages and KG queries per claim; LLM calls per answer; cost per prevented hallucination; and cost per published answer.
Deduplicate, cache, filter with embeddings, run NLI first, escalate by risk, and cap correction loops. The expensive LLM judge should see only the hard residue.
7. Going to production on Google Cloud
This section turns the combined architecture of section 3.2 into something you can deploy and operate on Google Cloud Platform (GCP). It is written as a build guide: first the shape of the system, then each component and why it was chosen, then how a request flows, how the knowledge graph is fed, how you ship changes safely, and how you keep it secure, observable and affordable. Product names are current as of the research date; check quotas, regions and prices in the official documentation before committing.
7.1 Design principles that drive every choice
- Verification is a separate service, not a library inside the generator. It has its own scaling, its own failure modes and its own deployment cycle.
- Stateless services, stateful stores. Every Cloud Run service can be killed and restarted at any time; state lives in the cache, graph, index and BigQuery.
- Idempotent by claim hash. The same claim against the same evidence and verifier version always returns the same verdict, so retries and caches are safe.
- Fail safe, never fail open. A timeout or a missing dependency produces "unresolved", never "approved".
- Everything leaves a trace. Every verdict carries evidence IDs, verifier and prompt versions and the policy that fired.
- Keep data where the law says. Choose one region (for example
europe-west9in Paris oreurope-west1in Belgium) for services and stores, and make sure model endpoints respect the same residency.
7.2 The architecture at a glance
Read it left to right. A caller reaches the edge, which authenticates and rate-limits. The request enters verify-api, the orchestrator that owns the policy. It calls the small stateless services in the dashed box, and those call the managed data and model services on the right. Everything slow or optional happens on the async plane. Security and observability wrap the whole thing.
7.3 Component by component: what to use, why and which knobs matter
| Component | GCP service | Why this choice | Knobs that matter |
|---|---|---|---|
| Edge | External Application Load Balancer + Cloud Armor; API Gateway or IAP | WAF and DDoS protection, per-client quotas, one public door while services stay private | Rate-limit rules, API keys or OIDC audiences, Cloud Run ingress set to internal and load balancing only |
| Orchestrator, extractor, retriever, corrector | Cloud Run services | Stateless HTTP, autoscaling, revisions with traffic splitting, pay per use | Concurrency, min instances (avoid cold starts), request timeout, startup CPU boost, CPU always allocated for latency-sensitive paths |
| NLI and specialised verifiers (HHEM, Lynx) | Cloud Run with NVIDIA L4 GPU, or Vertex AI endpoints | Cloud Run GPUs run model containers without cluster management; Vertex endpoints add model registry, versioning and autoscaling | Batch size, max sequence length, min instances (GPU cold starts are slow), model warm-up on startup |
| Strong judge and claim model | Vertex AI (Gemini, or a hosted model) | Managed, no GPU operations, same region and IAM as everything else | Quota, retry with exponential backoff and jitter, per-request timeout, structured JSON output |
| Claim and citation cache | Memorystore for Redis | Sub-millisecond lookups; the claim hash is the natural key (see section 6) | TTL, eviction policy, key includes source-version and verifier-version |
| Vector index | Vertex AI Vector Search (or AlloyDB with pgvector for one database) | Low-latency nearest-neighbour search for evidence passages | Embedding model version, top-k, metadata filters (source, date, access group) |
| Knowledge graph | Spanner Graph, Neo4j or Apache Jena (table below) | Entity, relation and provenance queries | Schema with validity times and source IDs, indexes on entity ID and relation type |
| Audit and evaluation records | BigQuery | Cheap analytical queries over every verdict; drives dashboards and the golden set | Partition by day, cluster by status and claim type, retention policy |
| Source documents and snapshots | Cloud Storage | Durable, versioned objects | Object versioning, lifecycle rules, CMEK |
| Async work | Pub/Sub, Cloud Tasks, Cloud Run jobs, Eventarc | Decouple slow, retryable or scheduled work from the request path | Ack deadlines, dead-letter topics, max retries, concurrency per queue |
Choosing the knowledge-graph store
| Option | Best when | Trade-offs |
|---|---|---|
| Spanner Graph | You want a fully managed graph on GCP with strong consistency and horizontal scale, queried with GQL; it became generally available in 2025 | GQL rather than classic Cypher (Google documents how the two compare); you design the property graph over Spanner tables |
| Neo4j (managed service or self-run on Compute Engine or GKE) | Your team already knows Cypher, or you need its mature graph algorithms and tooling | More operations if self-run; check your cloud region and networking for the managed offering |
| Apache Jena | Your sources are already RDF or you need SPARQL and ontology reasoning | You run and scale it yourself; fewer managed conveniences |
For a first version, pick the store your team can operate confidently. The verification logic only needs three queries: find entity, get relation with its source and validity and find path up to k hops. Hide them behind a small kg-client interface so the store can be swapped later.
7.4 Three ways to run verification: pick per use case
7.5 Anatomy of one request, with a latency budget
Here is where the time goes in an inline-gate request. The numbers are illustrative targets to design against, not measurements: replace them with your own after load testing.
Three design consequences follow. Parallelise stage 4 because text retrieval and the graph query are independent. Batch stage 6 so the GPU sees one large batch instead of many tiny ones. And do not block on the audit write: publish it to Pub/Sub and let a worker write BigQuery. A skeleton of the orchestrator shows the shape:
# verify_api.py (illustrative skeleton)
import asyncio, hashlib
from fastapi import FastAPI
from pydantic import BaseModel
app = FastAPI()
class VerifyRequest(BaseModel):
answer: str
evidence: list[dict] = []
risk: str = "medium" # low | medium | high
def claim_key(claim: str, src_ver: str, ver_ver: str) -> str:
return hashlib.sha256(f"{claim}|{src_ver}|{ver_ver}".encode()).hexdigest()
@app.post("/verify")
async def verify(req: VerifyRequest):
claims = await extractor.extract(req.answer) # stage 2
claims = dedupe(claims)
verdicts, todo = {}, []
for c in claims: # stage 3
hit = await cache.get(claim_key(c.text, SRC_VER, VERIFIER_VER))
(verdicts.__setitem__(c.id, hit) if hit else todo.append(c))
if todo:
text_ev, kg_ev = await asyncio.gather( # stage 4
retriever.search(todo), kg_client.lookup(todo))
for c in todo:
v = rules.check(c, kg_ev) or None # stage 5
if v: verdicts[c.id] = v
rest = [c for c in todo if c.id not in verdicts]
for c, v in zip(rest, await nli.batch(rest, text_ev)): # stage 6
verdicts[c.id] = v
hard = [c for c in rest if needs_escalation(verdicts[c.id], req.risk)]
for c, v in zip(hard, await judge.batch(hard, text_ev, kg_ev)): # stage 7
verdicts[c.id] = v
await cache.set_many(verdicts)
decision = policy.decide(verdicts, req.risk) # publish, correct, qualify, abstain
publisher.fire_and_forget("audit", build_record(req, verdicts, decision)) # stage 9
return decision.to_response()
An API contract you can start from
Request:
{
"answer": "Researcher A published Paper B in 2025.",
"evidence": [ { "id": "E1", "text": "Paper B was published in 2024." } ],
"verification_mode": "claim_level"
}
Response:
{
"status": "contradicted",
"claims": [
{
"claim_id": "C1",
"text": "Researcher A published Paper B in 2025.",
"status": "contradicted",
"evidence_ids": ["E1"],
"reason": "The supplied evidence states that Paper B was published in 2024.",
"action": "retrieve_and_correct"
}
],
"final_action": "retrieve_and_correct"
}
Failure handling: never fail open
Verification failure must never silently become approval.
Plan explicit behaviour for: missing source passages, graph-query failures, invalid or unavailable citations, verifier timeouts, conflicting evidence, malformed claim extraction, repeated correction failures, incomplete source updates and ambiguous entity links. The safe default in each case is "unresolved", followed by retry, qualification or abstention.
7.6 Feeding the graph: the ingestion pipeline
The verifier is only as good as the evidence behind it, so the knowledge graph and the vector index need their own production pipeline. It runs asynchronously whenever a source document appears or changes.
Three habits prevent most graph-quality incidents. First, audit by sampling: every batch, compare a random sample of extracted triples with the source text before the batch goes live. Second, version the evidence: the cache key from section 6 includes the source version, so a graph update invalidates old verdicts automatically. Third, keep verifier independence (section 2.3): where you can, use a different model and prompt for extraction than for the final verification.
7.7 Bringing a model into your platform safely
Downloading weights straight from the internet at container start-up is convenient and risky: a repository can change, disappear or ship code you have not reviewed (HHEM needs trust_remote_code=True, which executes code from the repository). The safer pattern is a one-time, reviewed import into storage you control.
In commands, steps 3 to 6 look like this (illustrative; replace names, check flags and quotas in the current documentation):
# 3-4. download a pinned revision and mirror it to your own versioned path pip install -U "huggingface_hub[cli]" huggingface-cli download vectara/hallucination_evaluation_model \ --revision <commit-sha> --local-dir ./hhem gcloud storage cp -r ./hhem gs://YOUR_MODEL_BUCKET/hhem/<commit-sha>/ # 6. serve on Cloud Run with a GPU, mounting the bucket read-only (no runtime download) gcloud run deploy nli-gpu \ --image europe-west9-docker.pkg.dev/PROJECT/verif/nli-gpu:GIT_SHA \ --region LOCATION --gpu 1 --gpu-type nvidia-l4 \ --cpu 4 --memory 16Gi --no-cpu-throttling \ --min-instances 1 --max-instances 5 \ --add-volume name=models,type=cloud-storage,bucket=YOUR_MODEL_BUCKET,readonly=true \ --add-volume-mount volume=models,mount-path=/models \ --set-env-vars HHEM_PATH=/models/hhem/<commit-sha> \ --no-allow-unauthenticated
Inside the container, load from the mounted path instead of the hub name, for example from_pretrained("/models/hhem/<commit-sha>", trust_remote_code=True), and keep HF_HUB_OFFLINE=1 set so nothing is fetched at start-up.
7.8 Deploying a service: the settings that matter
Each Cloud Run service is a container built from the same repository. The command below shows the settings that most affect latency and cost for the orchestrator; the comments say why. Treat it as a starting point and check the current flags and limits in the Cloud Run documentation.
gcloud run deploy verify-api \ --image europe-west9-docker.pkg.dev/PROJECT/verif/verify-api:GIT_SHA \ --region europe-west9 \ --service-account verify-api@PROJECT.iam.gserviceaccount.com \ --ingress internal-and-cloud-load-balancing \ --no-allow-unauthenticated \ --cpu 2 --memory 2Gi \ --concurrency 40 \ --min-instances 2 --max-instances 50 \ --cpu-boost --no-cpu-throttling \ --timeout 20 \ --set-secrets JUDGE_CONFIG=judge-config:latest \ --set-env-vars SRC_VER=2026-10-27,VERIFIER_VER=v3
| Setting | What it does and why |
|---|---|
--ingress internal-and-cloud-load-balancing | No direct public access; traffic must come through the load balancer or the private network |
--no-allow-unauthenticated | Callers must present a valid identity token |
--concurrency 40 | The service mostly waits on other services (I/O bound), so one instance can serve many requests at once |
--min-instances 2 --max-instances 50 | A warm floor that avoids cold starts, and a ceiling that caps cost during spikes |
--cpu-boost --no-cpu-throttling | Faster startup, and CPU stays available for background work after a response |
--timeout 20 | Fail fast and return "unresolved" instead of hanging the caller |
--set-secrets / --set-env-vars | Secrets come from Secret Manager; SRC_VER and VERIFIER_VER become part of every cache key |
The GPU service for NLI is similar but runs on a GPU-enabled configuration (an NVIDIA L4 on Cloud Run, which requires more CPU and memory than a CPU-only service) and usually sets a minimum of one warm instance, because loading a model on a cold GPU instance can take many seconds. Infrastructure itself should live in Terraform so environments are reproducible:
# main.tf (illustrative excerpt)
resource "google_cloud_run_v2_service" "verify_api" {
name = "verify-api"
location = "europe-west9"
ingress = "INGRESS_TRAFFIC_INTERNAL_LOAD_BALANCER"
template {
service_account = google_service_account.verify_api.email
max_instance_request_concurrency = 40
timeout = "20s"
scaling { min_instance_count = 2 max_instance_count = 50 }
containers {
image = var.image
resources { limits = { cpu = "2", memory = "2Gi" } cpu_idle = false startup_cpu_boost = true }
env { name = "VERIFIER_VER" value = var.verifier_version }
}
}
}
7.9 Shipping changes safely: CI/CD and environments
Verifier changes are risky in a particular way: a new threshold, prompt or model can silently shift what gets published. So the pipeline gates on measured quality, not only on unit tests, and it releases gradually.
The evaluation gate is the most valuable step. It runs the golden set (section 9) against the candidate revision and fails the build if claim-level recall drops, false rejections rise or the cost per answer jumps beyond limits you set. A trimmed workflow:
# .github/workflows/deploy.yml (illustrative excerpt)
permissions: { id-token: write, contents: read } # needed for Workload Identity Federation
jobs:
deploy:
steps:
- uses: actions/checkout@v4
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.WIF_PROVIDER }}
service_account: ${{ vars.DEPLOY_SA }}
- run: python eval/run_golden.py --min-recall 0.90 --max-false-reject 0.05 # the gate
- run: gcloud run deploy verify-api --image $IMAGE --no-traffic --tag canary --region europe-west9
- run: python eval/shadow_compare.py --url $CANARY_URL --minutes 30
- run: gcloud run services update-traffic verify-api --to-tags canary=5 --region europe-west9
The thresholds above are placeholders; set them from your own baseline. Cloud Run keeps every revision, so the rollback is one command that shifts traffic back to the previous revision.
7.10 Security: identities, networks and data
Give each service its own service account with the smallest set of roles. The table shows an illustrative least-privilege layout; adapt role names to your exact resources.
| Identity | May do | May not do |
|---|---|---|
verify-api | Invoke the other Cloud Run services, call Vertex AI, read the cache, publish to the audit topic | Write to the graph or the source bucket, read other projects' data |
evidence-retriever | Read the vector index and the graph (read-only role), read the source bucket | Write anywhere, call model endpoints |
audit-writer | Subscribe to the audit topic, insert into the BigQuery audit dataset only | Read production data stores |
ingest-job | Read sources, call extraction models, write to the graph, index and provenance tables | Be invoked by the online path |
deployer (CI) | Deploy revisions, push images, apply Terraform in its own environment | Read production data, act in other environments |
- Network: keep Cloud Run on internal ingress; reach the data tier over private connectivity; wrap the project in VPC Service Controls to limit data exfiltration.
- Secrets and keys: configuration secrets in Secret Manager, mounted at deploy time; customer-managed encryption keys (CMEK) for buckets, BigQuery and the graph where policy requires.
- Per-user access: pass the end user's identity or access group down to retrieval and filter evidence by it, so the verifier cannot "confirm" a claim from a document the user may not see.
- Prompt-injection hygiene: retrieved text and graph fields are evidence, never instructions. Strip or neutralise instructions inside evidence before it reaches an LLM judge, validate any model-generated query arguments, and never let evidence change the policy.
- Audit data is sensitive: it contains user questions and source passages. Apply retention limits and redact where required.
7.11 Observability and service-level objectives
Instrument with OpenTelemetry so each request produces one trace with a span per stage (the same trace feeds the explainability layer in section 8). Send traces, logs and metrics to Cloud Trace, Cloud Logging and Cloud Monitoring. Define a few indicators that match what users and the business care about:
| Indicator | Example objective (set your own) | Alert when |
|---|---|---|
| Availability of the verify endpoint | 99.5% of requests succeed | Burn rate of the error budget is high |
| Latency, inline mode | p95 below your budget (Figure 21) | p95 above budget for 10 minutes |
| Share of claims resolved by cache and rules | Stable around your baseline | Sudden drop (a cache or graph problem) or sudden rise (a stale cache) |
| Share escalated to the LLM judge | About the designed fraction | Above design for an hour (cost and latency risk) |
| Unresolved rate | Within a band you have reviewed | Spike (retrieval or graph outage) or collapse (verifier failing open) |
| Abstention and correction rates | Tracked per claim type | Large change after a release |
| Cost per answer and per verified claim | Below your ceiling | Above ceiling for a day |
| Offline audit: unsupported-claim rate in published answers | Falling or flat | Rising (drift in the generator, sources or verifier) |
Because every verdict lands in BigQuery, analysis is plain SQL. For example, to see where claims are being resolved this week:
SELECT resolved_by, status, COUNT(*) AS claims FROM `PROJECT.verification.claims` WHERE DATE(ts) >= DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY) GROUP BY resolved_by, status ORDER BY claims DESC;
7.12 Capacity and cost: a worked example
The cascade from section 6 is what makes this affordable. Take 1,000 extracted claims. With the illustrative split below, only about one claim in ten reaches the expensive judge. These percentages are assumptions to plan with; measure your own.
For capacity planning, multiply your peak answers per second by the average unique claims per answer to get claims per second, then size each tier from the funnel: the NLI GPU service needs to absorb about 45% of that rate in batches, and the judge quota about 10%. Use the cost formula from section 6.4 with real unit prices from your billing data, and review it monthly. Two cheap levers: raise the cache hit rate (longer TTL where sources are stable) and loosen the escalation rule for low-risk claim types.
7.13 Rolling it out without surprises
Cloud Run services for stateless logic, Cloud Run GPU or Vertex AI for models, a managed graph and vector store for evidence, Pub/Sub for everything slow, BigQuery for the audit trail, Terraform and Workload Identity Federation for repeatable, keyless delivery, and an evaluation gate plus canary so that a change to the verifier can never quietly change what you publish.
8. Explainability for agentic systems
A verifier that only says "not supported" is hard to act on. In an agentic system the final answer is the end of a long chain: query planning, retrieval, tool calls, memory access, sub-agent delegation, summarisation, synthesis. A final-output check alone cannot tell you where an unsupported statement entered the workflow. The goal is therefore a system that can answer six questions:
- Which evidence supported this claim?
- Which retrieval step produced that evidence?
- Which tool call or intermediate summary introduced the error?
- Why did the verifier accept, reject or abstain?
- Which policy or threshold produced the final action?
8.1 Eight explanation layers
An explanation written by an LLM can be plausible and wrong. A production explanation must be grounded in stored structure: evidence IDs, tool-call spans, KG paths, verifier decisions and policy outcomes. Test explanation faithfulness against the stored trace, never against how convincing it sounds.
8.2 The explainable agent architecture
A single decision record then looks like this, and it lets an auditor rebuild the decision without trusting the generator's own account:
{
"claim_id": "C7",
"claim": "Company A acquired a subsidiary of Company B in 2024.",
"status": "supported",
"evidence_ids": ["E31", "E42"],
"kg_path": [
"Company A --ACQUIRED--> Company C",
"Company C --SUBSIDIARY_OF--> Company B"
],
"tool_calls": ["kg_query_17", "document_search_08"],
"verifier": "nli-v3",
"verifier_score": 0.93,
"policy": "medium_risk_claim",
"final_action": "publish_with_citations"
}
8.3 Where did the error enter? VeriTrail
Section 4.3 explained the backward walk. What matters here is what it hands to the explanation layer: per claim, a verdict, the evidence trail and, for unsupported claims, the node where support was lost. That gives two kinds of explanation: provenance for supported or inconclusive claims, and error localisation for unsupported ones.
This is why the service should return a traceable result, not a bare label:
{
"claim_id": "C12",
"status": "not_fully_supported",
"error_localisation": {
"likely_introduced_at": "intermediate_summary_node_4",
"source_node": "retrieved_document_18",
"final_node": "final_answer"
},
"evidence_trail": [
{ "node_id": "retrieved_document_18",
"selected_sentence": "The pilot reduced processing time by 18%." },
{ "node_id": "intermediate_summary_node_4",
"generated_summary": "The system reduced costs by 40%." },
{ "node_id": "final_answer",
"generated_claim": "The system reduced costs by 40%." }
],
"recommended_action": "correct_or_abstain"
}
8.4 Operational explainability: traces and online evaluation
Amazon Bedrock AgentCore Observability is a useful reference for the operational side: it emits OpenTelemetry-compatible traces covering model invocations, tool calls, reasoning steps, memory operations, token use, latency and guardrail interventions. AWS recommends instrumenting from the first test query and propagating W3C trace context across agent, tool and service boundaries. Its evaluation layer can score behaviour at session, trace and tool-call level and can sample a configurable share of live traces.
Google documents that Vertex Explainable AI is deprecated as of 16 March 2026 and unavailable after 16 March 2027. The durable approach is application-level provenance: retain prompts, model versions, retrieval IDs and passages, tool inputs and outputs, KG paths, verifier scores and rationales, and policy decisions, and emit them through OpenTelemetry-compatible traces.
8.5 Three explanation records worth copying
Supported claim:
{
"claim": "Paper B was published in 2024.",
"status": "supported",
"evidence": [{ "source_id": "S17", "passage": "Paper B was published in 2024.",
"source_type": "publisher_metadata" }],
"kg_path": ["Paper B --PUBLICATION_YEAR--> 2024"],
"verifier": "nli-v3",
"decision": "publish",
"explanation": "The publisher metadata and the retrieved passage both state 2024."
}
Unresolved claim. The explanation says what is missing and what would resolve it:
{
"claim": "Paper B reported a 35% improvement in factual accuracy.",
"status": "unresolved",
"evidence": [],
"kg_path": [],
"verifier": "nli-v3",
"decision": "abstain",
"explanation": "The sources describe Paper B but do not report a 35% improvement. The claim requires the metric, baseline, dataset and experimental setting."
}
Contradicted claim:
{
"claim": "Paper B was published in 2025.",
"status": "contradicted",
"evidence": [{ "source_id": "S17", "passage": "Paper B was published in 2024.",
"source_type": "publisher_metadata" }],
"verifier": "nli-v3",
"decision": "correct",
"explanation": "Authoritative publisher metadata states 2024, contradicting the generated year 2025."
}
8.6 Human review for high-risk claims
Explanations should support a human reviewer, not replace one. The review screen shows the original request, the final answer, each extracted claim, evidence passages and citations, the KG path (or its absence), the verifier score and rationale, the policy decision and the correction history, with one-click approve, edit, reject or escalate. Route a claim to a person when the verdict is unresolved, sources conflict, a correction was generated, the claim is medical, legal, financial, safety or security related, a tool returned an unusual result, or the agent took an unusual path.
8.7 Explainability metrics
| Metric | Definition |
|---|---|
| Evidence-trail coverage | Published claims with at least one linked evidence item |
| Citation faithfulness | Explanations whose cited evidence entails the claim |
| Tool-trace completeness | Agent runs with complete tool-call spans |
| KG-path completeness | KG-supported claims with a complete path and provenance |
| Explanation faithfulness | Explanations consistent with the stored trace |
| Error-localisation accuracy | Unsupported claims where the correct failing step is identified |
| Review acceptance rate | Explanations accepted by human reviewers |
| Explanation latency and cost | Extra latency, tokens, retrieval and storage per explanation |
8.8 Eight production rules for explainability
- Instrument every agent step: reasoning, retrieval, KG queries, tools, memory, model calls, guardrails, final output.
- Use OpenTelemetry as the interoperability layer and propagate trace context across boundaries.
- Store evidence, not only explanations: source IDs, passage text or hashes, KG paths, tool outputs, scores, decisions.
- Separate explanation from verification. The verifier decides; the explanation layer communicates.
- Evaluate explanation faithfulness against the trace.
- Support human review for high-risk claims.
- Make abstention explainable: say what evidence was missing and what would resolve it.
- Avoid explanation-only trust. A fluent rationale is not proof.
9. Evaluating the whole thing
Treat unsupported claims as the positive class and compute the standard measures:
Precision = TP / (TP + FP) Recall = TP / (TP + FN) F1 = 2 · Precision · Recall / (Precision + Recall)
The metric that matters most is the quality of the final answer, not the number of errors the detector finds in isolation.
9.1 What to measure
- Claim-level precision, recall and F1, with contradiction and unsupported-claim detection reported separately.
- Final-answer factuality after correction, plus correction success and newly introduced errors.
- Citation precision: citations that exist and entail the claim; also citation coverage.
- Entity-linking precision and recall, relation-extraction accuracy, temporal consistency, provenance completeness, source authority and freshness.
- False rejection caused by missing graph edges.
- Correct and false abstention rates.
- Latency, tokens, retrieval calls, KG queries and verification iterations, per answer and per verified claim.
- Results stratified by claim type, with confidence intervals or paired statistical tests.
9.2 Reading a detector's numbers: a worked example
Suppose you run a claim verifier on 200 claims. 60 are truly unsupported (the positive class). The verifier flags 50 claims, of which 40 are truly unsupported. Then:
TP = 40 FP = 10 FN = 20 (flagged 50, missed 20 of the 60) Precision = TP / (TP + FP) = 40 / 50 = 0.80 Recall = TP / (TP + FN) = 40 / 60 = 0.667 F1 = 2 · 0.80 · 0.667 / (0.80 + 0.667) = 0.727
Now split the 10 false positives by cause. If 6 of them were true claims rejected only because the graph had no edge, your false-rejection rate from missing edges is 6 out of the 140 supported claims, about 4.3%, and the fix is "return unresolved and retrieve more", not "tune the threshold". This is the kind of diagnosis a single F1 hides.
9.3 The ablation ladder
Build the system one rung at a time and measure what each rung adds. This is also how you avoid paying for components that do not help.
The question that decides whether the KG is worth its upkeep:
Does knowledge-graph-based post-processing improve detection and correction of relational, temporal and multi-hop hallucinations beyond conventional RAG plus an independent claim verifier, at acceptable latency and cost?
9.4 Public benchmarks: what each one can and cannot tell you
No benchmark covers every production failure. Choose benchmarks by the capability you need to measure, then add a domain test set for what they miss. The matrix below is a simplified guide.
| Benchmark | What one sample looks like (illustrative) | Use it to |
|---|---|---|
| RAGTruth | Retrieved context + the model's response + word-level labels marking unsupported spans and their severity, e.g. the span "40%" tagged as baseless information | Train and test span-level detectors in RAG; measure localisation |
| HaluBench | (context, question, answer) triplet labelled faithful or hallucinated; 15,000 samples across several domains, introduced with Lynx | Compare detectors such as Lynx, HHEM and an LLM judge on the same triplets |
| FACTS Grounding and Suite | A long document + a user request; the response is judged on whether it is fully grounded in the document (suite adds parametric, search and multimodal) | Track frontier-model grounding and separate it from memory-based factuality |
| GraphEval | A response converted to triples; each triple checked against the context, the flagged triple corrected by GraphCorrect | Test triple-level localisation and KG-guided repair |
| KGHaluBench | A question generated from a knowledge-graph path, with difficulty controlled by path depth; answers verified automatically for conceptual and fact-level correctness | Probe parametric breadth and multi-hop depth; reported agreement with human judgement is 79.19% (conceptual) and 87.74% (correctness) |
Where to get them
| Dataset | Where | Use |
|---|---|---|
| RAGTruth | GitHub ParticleMedia/RAGTruth; paper ACL 2024 | Response- and word-level hallucination labels in RAG outputs |
| HaluBench | Introduced with Lynx (arXiv 2407.08488); published on Hugging Face by Patronus AI | 15,000 context-question-answer samples for comparing detectors |
| FACTS Grounding and Suite | Google DeepMind blog posts; hosted by Google (Kaggle) | Grounding, parametric, search and multimodal factuality |
| GraphEval | arXiv 2407.10793; Amazon Science page | KG-based evaluation and correction method |
| KGHaluBench | arXiv 2602.19643 (EACL 2026 Findings) | Graph-generated multi-hop questions; check the paper for released artefacts |
Which benchmark fits which rung
| Ablation rung (section 9.3) | Benchmark that fits |
|---|---|
| E1-E2: verifier, claim-level checking | HaluBench, RAGTruth |
| E3: add external evidence (RAG) | RAGTruth, FACTS Grounding |
| E4: add the knowledge graph | GraphEval, KGHaluBench, plus your own relational and temporal claims |
| E5-E6: citations, correction | Your own domain set (no public benchmark covers it fully) |
9.5 Your own test set, and a prototype comparison
Compare, on the same test set: (1) HHEM as a screening model, (2) Lynx as a specialised evaluator, (3) a general LLM judge as baseline, (4) a graph-plus-NLI verifier for relational claims, (5) a Claimify-style extractor with VeriTrail-style provenance for multi-step workflows, and (6) a fine-tuned compact verifier for high-volume, narrow domains. The test set should include direct contradictions, plausible but unsupported statements, wrong entity relations, temporal errors, multi-hop claims, incomplete evidence, claims that require abstention, and citations that exist but do not entail the claim.
10. Final recommendation and checklist
For an existing production LLM, implement this pipeline and keep the generator unchanged:
10.1 Implementation checklist
- ☐ Three verdicts implemented; a missing graph edge returns unresolved.
- ☐ Claims are atomic, keep their qualifiers and are deduplicated.
- ☐ Graph facts carry source, validity time and verification status.
- ☐ Cascade: rules, then NLI, then small verifier, then LLM judge for the residue.
- ☐ Caches keyed on claim, evidence, source version and verifier version.
- ☐ Risk routing defined and thresholds calibrated on your own data.
- ☐ Citations checked for existence, relevance and entailment.
- ☐ Correction is local, capped at one or two attempts and always re-verified.
- ☐ Verification failure fails safe (unresolved, never approved).
- ☐ Retrieved text treated as evidence, never as instructions.
- ☐ OpenTelemetry traces, audit records and human review for high-risk claims.
- ☐ Ablation E0 to E7 run on a domain test set, with final-answer factuality as the headline metric.
10.2 Two rules worth printing
Never publish a corrected answer merely because a second generation attempt sounds convincing. Publish it only after the relevant claims pass the verification policy, or state explicitly that the evidence is insufficient.
An agentic system should not merely generate an explanation. It should preserve a traceable evidence trail that makes the explanation verifiable, auditable and correctable.
10.3 The three experiments to run first
- Does adding a KG improve verification over a strong RAG-plus-verifier baseline?
- Does an NLI-first cascade preserve accuracy while reducing LLM cost?
- Does provenance-aware explanation improve human review quality and error localisation?
Open research challenges: generalising specialised verifiers across domains and languages, preserving meaning during claim decomposition, avoiding false acceptance when the graph is wrong and false rejection when it is incomplete, and evaluating correction, retrieval and abstention as separate actions.
11. References
Post-generation hallucination detection
- Ravi, Mielczarek, Kannappan, Kiela & Qian (2024). Lynx: An Open Source Hallucination Evaluation Model. arXiv:2407.08488. arxiv.org/abs/2407.08488
- Niu et al. (2024). RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models. ACL 2024. aclanthology.org/2024.acl-long.585
- Sansford, Richardson, Petric Maretic & Nait Saada (2024). GraphEval: A Knowledge-Graph Based LLM Hallucination Evaluation Framework. arXiv:2407.10793. arxiv.org/abs/2407.10793
- Paudel, Lyzhov, Joshi & Anand (2025). HalluciNot: Hallucination Detection Through Context and Common Knowledge Verification. arXiv:2504.07069. arxiv.org/abs/2504.07069
Claim extraction and provenance
- Metropolitansky & Larson (2025). Towards Effective Extraction and Evaluation of Factual Claims (Claimify). ACL 2025. Microsoft Research
- Metropolitansky et al. (2025). VeriTrail: Closed-Domain Hallucination Detection with Traceability. arXiv:2505.21786. arxiv.org/html/2505.21786
- Microsoft Research (5 August 2025). VeriTrail: Detecting hallucination and tracing provenance in multi-step AI workflows. Blog post
Knowledge graphs and hallucination mitigation
- Agrawal, Kumarage, Alghamdi & Liu (2024). Can Knowledge Graphs Reduce Hallucinations in LLMs? A Survey. NAACL 2024. aclanthology.org/2024.naacl-long.219
- Wagner, Kitzelmann & Boersch (2025). Mitigating Hallucination by Integrating Knowledge Graphs into LLM Inference: A Systematic Literature Review. ACL 2025 SRW. aclanthology.org/2025.acl-srw.53
- Edge et al. (2024). From Local to Global: A GraphRAG Approach to Query-Focused Summarization. arXiv:2404.16130. arxiv.org/abs/2404.16130
- Microsoft Research (2024). GraphRAG: Improving global search via dynamic community selection. Blog post
- Kale & Alfeo (2025). Lie to Me: Knowledge Graphs for Robust Hallucination Self-Detection in LLMs. arXiv:2512.23547, ICPRAM 2026. arxiv.org/abs/2512.23547
- Robertson et al. (2026). KGHaluBench: A Knowledge Graph-Based Hallucination Benchmark. EACL 2026 Findings, arXiv:2602.19643. arxiv.org/html/2602.19643
Major technology-company research
- Google DeepMind (2024). FACTS Grounding. Blog post
- Google DeepMind (2025). FACTS Benchmark Suite. Blog post
- OpenAI (2025). Why Language Models Hallucinate. openai.com
- Kalai, Nachum, Vempala & Zhang (2026). Evaluating large language models for accuracy incentivizes hallucinations. Nature 653(8116), 1047-1051. doi:10.1038/s41586-026-10549-w
- Amazon Science. GraphEval. amazon.science
- Amazon Science. GRIL: Knowledge Graph Retrieval-Integrated Learning with Large Language Models. amazon.science
- Google Research. Empty Shelves or Lost Keys? Recall Is the Bottleneck for Parametric Factuality. research.google
Industrial models and deployment tools
- Vectara, Hughes Hallucination Evaluation Model (HHEM). Hugging Face
- NVIDIA, NeMo Guardrails. docs.nvidia.com/nemo/guardrails
- Microsoft Azure AI Content Safety, groundedness detection. Microsoft Learn
- Google Cloud, grounding and evidence-based generation. Grounding with Google Search · Check grounding
NLP-efficient verification and cost reduction
- E-Verify: A Paradigm Shift to Scalable Embedding-Based Factuality Verification. Findings of EMNLP 2025. PDF
- HalluGuard: Evidence-Grounded Small Reasoning Models to Mitigate Hallucinations in RAG. arXiv:2510.00880. arxiv.org
- Proceedings of the Eighth Fact Extraction and Verification Workshop (FEVER 2025). ACL Anthology
- Why Your Deep Research Agent Fails? On Hallucination Detection and Mitigation. arXiv:2601.22984. arxiv.org
- Teaming LLMs to Detect and Mitigate Hallucinations. arXiv:2510.19507. arxiv.org
- Winning Big with Small Models: Knowledge Distillation vs. Self-Training for Reducing Hallucination in QA Agents. GEM Workshop 2025. PDF
Explainability and agentic systems
- AWS (5 October 2026). Evaluating multi-agent systems for explainability and helpfulness with Amazon Bedrock AgentCore. AWS blog
- AWS (31 March 2026). Build reliable AI agents with Amazon Bedrock AgentCore Evaluations. AWS blog
- AWS (29 June 2026). Debugging production agents with Amazon Bedrock AgentCore Observability. AWS blog
- AWS Well-Architected (2026). AGENTOPS05-BP01: Establish end-to-end tracing and observability. Agentic AI Lens
- Google Cloud (2026). Introduction to Vertex Explainable AI. Documentation
Corrective retrieval
- Asai et al. (2024). Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. ICLR 2024. ICLR proceedings
- Yan, Gu, Zhu & Ling (2024). Corrective Retrieval Augmented Generation. arXiv:2401.15884. arxiv.org/abs/2401.15884
This tutorial restructures and illustrates a research report dated 27 October 2026. Reported figures are benchmark- and setting-specific and should be re-measured on your own data before you rely on them.
Comments
Post a Comment