{"schemaVersion":1,"id":"https://lucioduran.com/projects/citation-grounding-checker","url":"https://lucioduran.com/projects/citation-grounding-checker","title":"Citation Grounding Checker","description":"Citation support checking with rules, FLAN-T5 and DeBERTa NLI. Predictions, abstentions and precision linked to each claim’s exact source passage.","language":"en","datePublished":"2026-10-21","dateModified":"2026-10-21","contentFormat":"text/markdown","content":"[Projects]() / Citation Grounding Checker\n\n# Citation Grounding Checker\n\nCitation support checking with rules, FLAN-T5 and DeBERTa NLI. Predictions, abstentions and precision linked to each claim’s exact source passage.\n\nAuthor: [Lucio Durán]() · Python 3.11.13 · License: [MIT]()\n\nPage published: 2026-10-21 · Page updated: 2026-10-21\n\n[Repository]() · [Original evidence]() · [Release v0.1.0]()\n\n## Problem and relevance to agents\n\nRetrieving a relevant source does not establish that it supports a claim. A changed number, negation or omitted context can alter what a citation actually supports.\n\nThe project exposes the claim, passage, reference label and verdict together. Grounding errors can be inspected separately from search failures: the passage is already provided here.\n\n## Architecture\n\n1. **Evidence contract.** Validates exact quotes, offsets, files and development/evaluation isolation. Each verdict links to a passage at an immutable revision.\n2. **Three checkers.** Conservative rules, FLAN-T5-small and DeBERTa v3 xsmall NLI receive only passage and claim. Reference labels, categories and rationales stay outside model inputs.\n3. **Explicit evaluation.** Separates support, contradiction, insufficient evidence and abstention. Retains raw outputs, hashes, confusion matrices and per-category metrics.\n\n- [Design and methodology]()\n\n## What this demonstrates\n\n- Traceability from a claim to its exact textual evidence.\n- Comparative checker evaluation with visible coverage and abstentions.\n- Local execution of pinned models with labels isolated from inputs.\n\n## Published results\n\nOriginal corpus of 48 synthetic pairs across 16 passages: 12 development cases and 36 evaluation cases. Models ran on Intel Xeon W-3223 with two CPU threads.\n\nEach of the three reference classes has 12 evaluation cases. Precision is measured among predictions emitted for a class; correct counts use all 36 cases. Abstention is distinct from an insufficient-evidence classification.\n\n### Verdicts on 36 evaluation cases\n\n| Method | Correct | Abstentions | Support precision | Contradiction precision | Insufficient precision |\n| --- | --- | --- | --- | --- | --- |\n| Rules | 3 / 36 | 33 / 36 | 1.000 | 1.000 | No predictions |\n| FLAN-T5-small | 21 / 36 | 0 / 36 | 0.500 | 0.583 | 0.833 |\n| DeBERTa NLI | 31 / 36 | 1 / 36 | 0.923 | 0.800 | 1.000 |\n\nDeBERTa classified 31 of 36 cases correctly and abstained on one. Rules classified three correctly and abstained on 33. Precision on the rules’ few answers must be read alongside their coverage.\n\nLabels are synthetic author-supplied references. There is no independent human annotation or measured reviewer disagreement. The analysis retains numeric, paraphrase and context errors. [Results and analysis]().\n\n## Reproduce\n\nValidation, rules and replay use Python 3.11 or later without model dependencies. Full inference requires downloading the pinned weights and installing model packages.\n\nEnvironment commands use a POSIX shell. Measurement requires a clean tracked Git tree and a new output directory. The lock targets Intel macOS; each new run retains its own metadata.\n\n### Replay saved evidence\n\n```\ngit clone https://github.com/lucioduran/citation-grounding-checker.git\ncd citation-grounding-checker\ngit checkout 1bb2d88a9a434e3193949f8e92f1aff3e118199c\npython -m grounding validate\npython -m grounding verify --report results/eval/report.json\npython -m grounding check --passage \"The gate is open.\" --claim \"The gate is not open.\"\n```\n\n### Repeat the local measurement\n\n```\npython -m venv .venv\n. .venv/bin/activate\npython -m pip install -r requirements-local.lock\npython -m grounding download\nHF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=2 MKL_NUM_THREADS=2 python -m grounding run --split dev --output build/dev\nHF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=2 MKL_NUM_THREADS=2 python -m grounding run --split eval --output build/eval\npython -m grounding verify --report build/eval/report.json\n```\n\n## Scope and limitations\n\n- Small synthetic corpus known to the author; several cases share passages and templates.\n- Fixed uncalibrated NLI threshold of 0.6. No web retrieval, human validation or production security evaluation.\n- Timings are individual observations. Replay recomputes saved results; it neither repeats inference nor establishes its authenticity.\n\n- [RAG architecture: retrieval, citations and evaluation]()\n\n## Sources and revision\n\nValues on this page come from the linked report. They are not live measurements. The report records the executed revision; the reviewed revision includes the subsequent publication of that report.\n\n- Reviewed revision: [` 1bb2d88a9a434e3193949f8e92f1aff3e118199c `]()\n- Revision recorded by the experiment: [` 7fb3a7ca724d31fa4db75a1057d20659a8d33a45 `]()\n- Report timestamp (UTC): 2026-10-21T21:57:14.496757+00:00\n- Release published: [v0.1.0]() · 2026-10-21T21:59:32Z\n- Corpus SHA-256: ` 9cce1c138bbefbb52c360d9cd2a7eb7839ef5664348d0465b6409314f538e00d `\n\n- [README and usage]()\n- [Original evidence]()\n- [Development evidence]()\n- [Results and analysis]()\n- [Design and methodology]()\n- [Corpus and provenance]()\n- [Models, revisions and licenses]()\n- [Measured environment dependencies]()\n- [MIT license]()","headings":[{"level":1,"text":"Citation Grounding Checker","url":"https://lucioduran.com/projects/citation-grounding-checker#title"},{"level":2,"text":"Problem and relevance to agents","url":"https://lucioduran.com/projects/citation-grounding-checker#problem"},{"level":2,"text":"Architecture","url":"https://lucioduran.com/projects/citation-grounding-checker#architecture"},{"level":2,"text":"What this demonstrates","url":"https://lucioduran.com/projects/citation-grounding-checker#demonstrates"},{"level":2,"text":"Published results","url":"https://lucioduran.com/projects/citation-grounding-checker#results"},{"level":3,"text":"Verdicts on 36 evaluation cases","url":"https://lucioduran.com/projects/citation-grounding-checker#results-judgments"},{"level":2,"text":"Reproduce","url":"https://lucioduran.com/projects/citation-grounding-checker#reproduce"},{"level":3,"text":"Replay saved evidence","url":"https://lucioduran.com/projects/citation-grounding-checker#reproduce-replay"},{"level":3,"text":"Repeat the local measurement","url":"https://lucioduran.com/projects/citation-grounding-checker#reproduce-inference"},{"level":2,"text":"Scope and limitations","url":"https://lucioduran.com/projects/citation-grounding-checker#scope"},{"level":2,"text":"Sources and revision","url":"https://lucioduran.com/projects/citation-grounding-checker#sources"}],"sections":[{"heading":"Citation Grounding Checker","level":1,"url":"https://lucioduran.com/projects/citation-grounding-checker","content":"[Projects]() / Citation Grounding Checker"},{"heading":"Citation Grounding Checker","level":1,"url":"https://lucioduran.com/projects/citation-grounding-checker#title","content":"Citation support checking with rules, FLAN-T5 and DeBERTa NLI. Predictions, abstentions and precision linked to each claim’s exact source passage.\n\nAuthor: [Lucio Durán]() · Python 3.11.13 · License: [MIT]()\n\nPage published: 2026-10-21 · Page updated: 2026-10-21\n\n[Repository]() · [Original evidence]() · [Release v0.1.0]()"},{"heading":"Problem and relevance to agents","level":2,"url":"https://lucioduran.com/projects/citation-grounding-checker#problem","content":"Retrieving a relevant source does not establish that it supports a claim. A changed number, negation or omitted context can alter what a citation actually supports.\n\nThe project exposes the claim, passage, reference label and verdict together. Grounding errors can be inspected separately from search failures: the passage is already provided here."},{"heading":"Architecture","level":2,"url":"https://lucioduran.com/projects/citation-grounding-checker#architecture","content":"1. **Evidence contract.** Validates exact quotes, offsets, files and development/evaluation isolation. Each verdict links to a passage at an immutable revision.\n2. **Three checkers.** Conservative rules, FLAN-T5-small and DeBERTa v3 xsmall NLI receive only passage and claim. Reference labels, categories and rationales stay outside model inputs.\n3. **Explicit evaluation.** Separates support, contradiction, insufficient evidence and abstention. Retains raw outputs, hashes, confusion matrices and per-category metrics.\n\n- [Design and methodology]()"},{"heading":"What this demonstrates","level":2,"url":"https://lucioduran.com/projects/citation-grounding-checker#demonstrates","content":"- Traceability from a claim to its exact textual evidence.\n- Comparative checker evaluation with visible coverage and abstentions.\n- Local execution of pinned models with labels isolated from inputs."},{"heading":"Published results","level":2,"url":"https://lucioduran.com/projects/citation-grounding-checker#results","content":"Original corpus of 48 synthetic pairs across 16 passages: 12 development cases and 36 evaluation cases. Models ran on Intel Xeon W-3223 with two CPU threads.\n\nEach of the three reference classes has 12 evaluation cases. Precision is measured among predictions emitted for a class; correct counts use all 36 cases. Abstention is distinct from an insufficient-evidence classification."},{"heading":"Verdicts on 36 evaluation cases","level":3,"url":"https://lucioduran.com/projects/citation-grounding-checker#results-judgments","content":"| Method | Correct | Abstentions | Support precision | Contradiction precision | Insufficient precision |\n| --- | --- | --- | --- | --- | --- |\n| Rules | 3 / 36 | 33 / 36 | 1.000 | 1.000 | No predictions |\n| FLAN-T5-small | 21 / 36 | 0 / 36 | 0.500 | 0.583 | 0.833 |\n| DeBERTa NLI | 31 / 36 | 1 / 36 | 0.923 | 0.800 | 1.000 |\n\nDeBERTa classified 31 of 36 cases correctly and abstained on one. Rules classified three correctly and abstained on 33. Precision on the rules’ few answers must be read alongside their coverage.\n\nLabels are synthetic author-supplied references. There is no independent human annotation or measured reviewer disagreement. The analysis retains numeric, paraphrase and context errors. [Results and analysis]()."},{"heading":"Reproduce","level":2,"url":"https://lucioduran.com/projects/citation-grounding-checker#reproduce","content":"Validation, rules and replay use Python 3.11 or later without model dependencies. Full inference requires downloading the pinned weights and installing model packages.\n\nEnvironment commands use a POSIX shell. Measurement requires a clean tracked Git tree and a new output directory. The lock targets Intel macOS; each new run retains its own metadata."},{"heading":"Replay saved evidence","level":3,"url":"https://lucioduran.com/projects/citation-grounding-checker#reproduce-replay","content":"```\ngit clone https://github.com/lucioduran/citation-grounding-checker.git\ncd citation-grounding-checker\ngit checkout 1bb2d88a9a434e3193949f8e92f1aff3e118199c\npython -m grounding validate\npython -m grounding verify --report results/eval/report.json\npython -m grounding check --passage \"The gate is open.\" --claim \"The gate is not open.\"\n```"},{"heading":"Repeat the local measurement","level":3,"url":"https://lucioduran.com/projects/citation-grounding-checker#reproduce-inference","content":"```\npython -m venv .venv\n. .venv/bin/activate\npython -m pip install -r requirements-local.lock\npython -m grounding download\nHF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=2 MKL_NUM_THREADS=2 python -m grounding run --split dev --output build/dev\nHF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=2 MKL_NUM_THREADS=2 python -m grounding run --split eval --output build/eval\npython -m grounding verify --report build/eval/report.json\n```"},{"heading":"Scope and limitations","level":2,"url":"https://lucioduran.com/projects/citation-grounding-checker#scope","content":"- Small synthetic corpus known to the author; several cases share passages and templates.\n- Fixed uncalibrated NLI threshold of 0.6. No web retrieval, human validation or production security evaluation.\n- Timings are individual observations. Replay recomputes saved results; it neither repeats inference nor establishes its authenticity.\n\n- [RAG architecture: retrieval, citations and evaluation]()"},{"heading":"Sources and revision","level":2,"url":"https://lucioduran.com/projects/citation-grounding-checker#sources","content":"Values on this page come from the linked report. They are not live measurements. The report records the executed revision; the reviewed revision includes the subsequent publication of that report.\n\n- Reviewed revision: [` 1bb2d88a9a434e3193949f8e92f1aff3e118199c `]()\n- Revision recorded by the experiment: [` 7fb3a7ca724d31fa4db75a1057d20659a8d33a45 `]()\n- Report timestamp (UTC): 2026-10-21T21:57:14.496757+00:00\n- Release published: [v0.1.0]() · 2026-10-21T21:59:32Z\n- Corpus SHA-256: ` 9cce1c138bbefbb52c360d9cd2a7eb7839ef5664348d0465b6409314f538e00d `\n\n- [README and usage]()\n- [Original evidence]()\n- [Development evidence]()\n- [Results and analysis]()\n- [Design and methodology]()\n- [Corpus and provenance]()\n- [Models, revisions and licenses]()\n- [Measured environment dependencies]()\n- [MIT license]()"}],"links":[{"text":"Projects","url":"https://lucioduran.com/projects"},{"text":"Lucio Durán","url":"https://lucioduran.com/"},{"text":"MIT","url":"https://github.com/lucioduran/citation-grounding-checker/blob/1bb2d88a9a434e3193949f8e92f1aff3e118199c/LICENSE"},{"text":"Repository","url":"https://github.com/lucioduran/citation-grounding-checker"},{"text":"Original evidence","url":"https://raw.githubusercontent.com/lucioduran/citation-grounding-checker/1bb2d88a9a434e3193949f8e92f1aff3e118199c/results/eval/report.json"},{"text":"Release v0.1.0","url":"https://github.com/lucioduran/citation-grounding-checker/releases/tag/v0.1.0"},{"text":"Design and methodology","url":"https://github.com/lucioduran/citation-grounding-checker/blob/1bb2d88a9a434e3193949f8e92f1aff3e118199c/docs/methodology.md"},{"text":"Results and analysis","url":"https://github.com/lucioduran/citation-grounding-checker/blob/1bb2d88a9a434e3193949f8e92f1aff3e118199c/docs/results.md"},{"text":"RAG architecture: retrieval, citations and evaluation","url":"https://lucioduran.com/blog/rag-pipeline-architecture-chunking-reranking-evaluation"},{"text":"1bb2d88a9a434e3193949f8e92f1aff3e118199c","url":"https://github.com/lucioduran/citation-grounding-checker/commit/1bb2d88a9a434e3193949f8e92f1aff3e118199c"},{"text":"7fb3a7ca724d31fa4db75a1057d20659a8d33a45","url":"https://github.com/lucioduran/citation-grounding-checker/commit/7fb3a7ca724d31fa4db75a1057d20659a8d33a45"},{"text":"v0.1.0","url":"https://github.com/lucioduran/citation-grounding-checker/releases/tag/v0.1.0"},{"text":"README and usage","url":"https://github.com/lucioduran/citation-grounding-checker/blob/1bb2d88a9a434e3193949f8e92f1aff3e118199c/README.md"},{"text":"Development evidence","url":"https://raw.githubusercontent.com/lucioduran/citation-grounding-checker/1bb2d88a9a434e3193949f8e92f1aff3e118199c/results/dev/report.json"},{"text":"Corpus and provenance","url":"https://github.com/lucioduran/citation-grounding-checker/blob/1bb2d88a9a434e3193949f8e92f1aff3e118199c/data/README.md"},{"text":"Models, revisions and licenses","url":"https://github.com/lucioduran/citation-grounding-checker/blob/1bb2d88a9a434e3193949f8e92f1aff3e118199c/docs/model-notices.md"},{"text":"Measured environment dependencies","url":"https://github.com/lucioduran/citation-grounding-checker/blob/1bb2d88a9a434e3193949f8e92f1aff3e118199c/requirements-local.lock"},{"text":"MIT license","url":"https://github.com/lucioduran/citation-grounding-checker/blob/1bb2d88a9a434e3193949f8e92f1aff3e118199c/LICENSE"}],"structuredData":[{"@context":"https://schema.org","@graph":[{"@type":"WebPage","@id":"https://lucioduran.com/projects/citation-grounding-checker#page","url":"https://lucioduran.com/projects/citation-grounding-checker","name":"Citation Grounding Checker","description":"Citation support checking with rules, FLAN-T5 and DeBERTa NLI. Predictions, abstentions and precision linked to each claim’s exact source passage.","inLanguage":"en","datePublished":"2026-10-21","dateModified":"2026-10-21","author":{"@id":"https://lucioduran.com/#person"},"mainEntity":{"@id":"https://lucioduran.com/projects/citation-grounding-checker#software"}},{"@type":"SoftwareSourceCode","@id":"https://lucioduran.com/projects/citation-grounding-checker#software","name":"Citation Grounding Checker","description":"Citation support checking with rules, FLAN-T5 and DeBERTa NLI. Predictions, abstentions and precision linked to each claim’s exact source passage.","url":"https://lucioduran.com/projects/citation-grounding-checker","codeRepository":"https://github.com/lucioduran/citation-grounding-checker","programmingLanguage":"Python","runtimePlatform":"Python 3.11.13","license":"https://github.com/lucioduran/citation-grounding-checker/blob/1bb2d88a9a434e3193949f8e92f1aff3e118199c/LICENSE","version":"v0.1.0","identifier":{"@type":"PropertyValue","propertyID":"git-commit","value":"1bb2d88a9a434e3193949f8e92f1aff3e118199c"},"author":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"isAccessibleForFree":true,"citation":["https://github.com/lucioduran/citation-grounding-checker/blob/1bb2d88a9a434e3193949f8e92f1aff3e118199c/README.md","https://raw.githubusercontent.com/lucioduran/citation-grounding-checker/1bb2d88a9a434e3193949f8e92f1aff3e118199c/results/eval/report.json","https://raw.githubusercontent.com/lucioduran/citation-grounding-checker/1bb2d88a9a434e3193949f8e92f1aff3e118199c/results/dev/report.json","https://github.com/lucioduran/citation-grounding-checker/blob/1bb2d88a9a434e3193949f8e92f1aff3e118199c/docs/results.md","https://github.com/lucioduran/citation-grounding-checker/blob/1bb2d88a9a434e3193949f8e92f1aff3e118199c/docs/methodology.md","https://github.com/lucioduran/citation-grounding-checker/blob/1bb2d88a9a434e3193949f8e92f1aff3e118199c/data/README.md","https://github.com/lucioduran/citation-grounding-checker/blob/1bb2d88a9a434e3193949f8e92f1aff3e118199c/docs/model-notices.md","https://github.com/lucioduran/citation-grounding-checker/blob/1bb2d88a9a434e3193949f8e92f1aff3e118199c/requirements-local.lock","https://github.com/lucioduran/citation-grounding-checker/releases/tag/v0.1.0"]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Projects","item":"https://lucioduran.com/projects"},{"@type":"ListItem","position":2,"name":"Citation Grounding Checker","item":"https://lucioduran.com/projects/citation-grounding-checker"}]}]}],"contentHash":"5a3585222e4d11eaf8d8beb487e7f58d85d9509710bf118eec0faed1b76f7351","provenance":{"method":"Deterministic serialization of the server-rendered main element","source":"https://lucioduran.com/projects/citation-grounding-checker","hashAlgorithm":"sha256","hashScope":"JSON document fields before contentHash and provenance","sourceHTMLBytes":45171,"contentBytes":7496}}