[Projects]() / Citation Grounding Checker # Citation Grounding Checker Citation support checking with rules, FLAN-T5 and DeBERTa NLI. Predictions, abstentions and precision linked to each claim’s exact source passage. Author: [Lucio Durán]() · Python 3.11.13 · License: [MIT]() Page published: 2026-10-21 · Page updated: 2026-10-21 [Repository]() · [Original evidence]() · [Release v0.1.0]() ## Problem and relevance to agents Retrieving a relevant source does not establish that it supports a claim. A changed number, negation or omitted context can alter what a citation actually supports. The project exposes the claim, passage, reference label and verdict together. Grounding errors can be inspected separately from search failures: the passage is already provided here. ## Architecture 1. **Evidence contract.** Validates exact quotes, offsets, files and development/evaluation isolation. Each verdict links to a passage at an immutable revision. 2. **Three checkers.** Conservative rules, FLAN-T5-small and DeBERTa v3 xsmall NLI receive only passage and claim. Reference labels, categories and rationales stay outside model inputs. 3. **Explicit evaluation.** Separates support, contradiction, insufficient evidence and abstention. Retains raw outputs, hashes, confusion matrices and per-category metrics. - [Design and methodology]() ## What this demonstrates - Traceability from a claim to its exact textual evidence. - Comparative checker evaluation with visible coverage and abstentions. - Local execution of pinned models with labels isolated from inputs. ## Published results Original corpus of 48 synthetic pairs across 16 passages: 12 development cases and 36 evaluation cases. Models ran on Intel Xeon W-3223 with two CPU threads. Each of the three reference classes has 12 evaluation cases. Precision is measured among predictions emitted for a class; correct counts use all 36 cases. Abstention is distinct from an insufficient-evidence classification. ### Verdicts on 36 evaluation cases | Method | Correct | Abstentions | Support precision | Contradiction precision | Insufficient precision | | --- | --- | --- | --- | --- | --- | | Rules | 3 / 36 | 33 / 36 | 1.000 | 1.000 | No predictions | | FLAN-T5-small | 21 / 36 | 0 / 36 | 0.500 | 0.583 | 0.833 | | DeBERTa NLI | 31 / 36 | 1 / 36 | 0.923 | 0.800 | 1.000 | DeBERTa classified 31 of 36 cases correctly and abstained on one. Rules classified three correctly and abstained on 33. Precision on the rules’ few answers must be read alongside their coverage. Labels are synthetic author-supplied references. There is no independent human annotation or measured reviewer disagreement. The analysis retains numeric, paraphrase and context errors. [Results and analysis](). ## Reproduce Validation, rules and replay use Python 3.11 or later without model dependencies. Full inference requires downloading the pinned weights and installing model packages. Environment commands use a POSIX shell. Measurement requires a clean tracked Git tree and a new output directory. The lock targets Intel macOS; each new run retains its own metadata. ### Replay saved evidence ``` git clone https://github.com/lucioduran/citation-grounding-checker.git cd citation-grounding-checker git checkout 1bb2d88a9a434e3193949f8e92f1aff3e118199c python -m grounding validate python -m grounding verify --report results/eval/report.json python -m grounding check --passage "The gate is open." --claim "The gate is not open." ``` ### Repeat the local measurement ``` python -m venv .venv . .venv/bin/activate python -m pip install -r requirements-local.lock python -m grounding download HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=2 MKL_NUM_THREADS=2 python -m grounding run --split dev --output build/dev HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=2 MKL_NUM_THREADS=2 python -m grounding run --split eval --output build/eval python -m grounding verify --report build/eval/report.json ``` ## Scope and limitations - Small synthetic corpus known to the author; several cases share passages and templates. - Fixed uncalibrated NLI threshold of 0.6. No web retrieval, human validation or production security evaluation. - Timings are individual observations. Replay recomputes saved results; it neither repeats inference nor establishes its authenticity. - [RAG architecture: retrieval, citations and evaluation]() ## Sources and revision Values on this page come from the linked report. They are not live measurements. The report records the executed revision; the reviewed revision includes the subsequent publication of that report. - Reviewed revision: [` 1bb2d88a9a434e3193949f8e92f1aff3e118199c `]() - Revision recorded by the experiment: [` 7fb3a7ca724d31fa4db75a1057d20659a8d33a45 `]() - Report timestamp (UTC): 2026-10-21T21:57:14.496757+00:00 - Release published: [v0.1.0]() · 2026-10-21T21:59:32Z - Corpus SHA-256: ` 9cce1c138bbefbb52c360d9cd2a7eb7839ef5664348d0465b6409314f538e00d ` - [README and usage]() - [Original evidence]() - [Development evidence]() - [Results and analysis]() - [Design and methodology]() - [Corpus and provenance]() - [Models, revisions and licenses]() - [Measured environment dependencies]() - [MIT license]()