Citation Grounding Checker

Problem and relevance to agents

Retrieving a relevant source does not establish that it supports a claim. A changed number, negation or omitted context can alter what a citation actually supports.

The project exposes the claim, passage, reference label and verdict together. Grounding errors can be inspected separately from search failures: the passage is already provided here.

Architecture

1. Evidence contract. Validates exact quotes, offsets, files and development/evaluation isolation. Each verdict links to a passage at an immutable revision.
2. Three checkers. Conservative rules, FLAN-T5-small and DeBERTa v3 xsmall NLI receive only passage and claim. Reference labels, categories and rationales stay outside model inputs.
3. Explicit evaluation. Separates support, contradiction, insufficient evidence and abstention. Retains raw outputs, hashes, confusion matrices and per-category metrics.

• Design and methodology

What this demonstrates

• Traceability from a claim to its exact textual evidence.
• Comparative checker evaluation with visible coverage and abstentions.
• Local execution of pinned models with labels isolated from inputs.

Published results

Original corpus of 48 synthetic pairs across 16 passages: 12 development cases and 36 evaluation cases. Models ran on Intel Xeon W-3223 with two CPU threads.

Each of the three reference classes has 12 evaluation cases. Precision is measured among predictions emitted for a class; correct counts use all 36 cases. Abstention is distinct from an insufficient-evidence classification.

DeBERTa classified 31 of 36 cases correctly and abstained on one. Rules classified three correctly and abstained on 33. Precision on the rules’ few answers must be read alongside their coverage.

Labels are synthetic author-supplied references. There is no independent human annotation or measured reviewer disagreement. The analysis retains numeric, paraphrase and context errors. Results and analysis.

Reproduce

Validation, rules and replay use Python 3.11 or later without model dependencies. Full inference requires downloading the pinned weights and installing model packages.

Environment commands use a POSIX shell. Measurement requires a clean tracked Git tree and a new output directory. The lock targets Intel macOS; each new run retains its own metadata.

git clone https://github.com/lucioduran/citation-grounding-checker.git
cd citation-grounding-checker
git checkout 1bb2d88a9a434e3193949f8e92f1aff3e118199c
python -m grounding validate
python -m grounding verify --report results/eval/report.json
python -m grounding check --passage "The gate is open." --claim "The gate is not open."

python -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements-local.lock
python -m grounding download
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=2 MKL_NUM_THREADS=2 python -m grounding run --split dev --output build/dev
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=2 MKL_NUM_THREADS=2 python -m grounding run --split eval --output build/eval
python -m grounding verify --report build/eval/report.json

Scope and limitations

• Small synthetic corpus known to the author; several cases share passages and templates.
• Fixed uncalibrated NLI threshold of 0.6. No web retrieval, human validation or production security evaluation.
• Timings are individual observations. Replay recomputes saved results; it neither repeats inference nor establishes its authenticity.

• RAG architecture: retrieval, citations and evaluation

Sources and revision

Values on this page come from the linked report. They are not live measurements. The report records the executed revision; the reviewed revision includes the subsequent publication of that report.

• Reviewed revision: 1bb2d88a9a434e3193949f8e92f1aff3e118199c
• Revision recorded by the experiment: 7fb3a7ca724d31fa4db75a1057d20659a8d33a45
• Report timestamp (UTC): 2026-10-21T21:57:14.496757+00:00
• Release published: v0.1.0 · 2026-10-21T21:59:32Z
• Corpus SHA-256: 9cce1c138bbefbb52c360d9cd2a7eb7839ef5664348d0465b6409314f538e00d

• README and usage
• Original evidence
• Development evidence
• Results and analysis
• Design and methodology
• Corpus and provenance
• Models, revisions and licenses
• Measured environment dependencies
• MIT license