Chunking Ablation Suite

Problem and relevance to agents

Finding the right document is insufficient when a chunk loses a condition, table cell or reference target. This suite measures whether retrieved text retains all required evidence.

It supports corpus-specific segmentation choices and failure analysis. Decisions can be inspected through saved offsets, chunks, vectors and rankings.

Architecture

1. Corpus with localized evidence. Eighteen original fictional documents contain code, tables and cross references. Queries declare required spans and use document-disjoint development and evaluation partitions.
2. Eight frozen configurations. Compares fixed windows, structural block packing and MiniLM semantic boundaries. Contrasts change method, word budget, overlap or threshold while keeping embeddings and top-3 retrieval fixed.
3. Retrieval replay. Rebuilds boundaries from saved block embeddings, rankings from chunk and query vectors, evidence coverage and index costs.

• Design and methodology
• Data contract

What this demonstrates

• Ablation design with one variable per contrast and results by document type.
• Separation of document retrieval from complete-evidence retrieval.
• Publication of null results and segmentation failures with inspectable data.

Published results

Local MiniLM measurement on Intel Xeon W-3223 with two CPU threads. There are 18 development queries and 36 evaluation queries; every index contains the same 18 documents.

Complete@3 requires the union of the first three chunks to cover every required evidence span. It does not measure generated answers. Sizes 64 and 96 are word budgets, not model-token budgets.

Structure-96 retrieved complete evidence for 33 of 36 queries. Semantic configurations did so for 12. At 64 words, changing the threshold changes no boundaries because no adjacent block pair fits; this ablation is uninformative about threshold sensitivity.

The analysis breaks down span recall, completion, text bytes, vector payload and construction time for code, tables and cross references. Timings are sequential observations on one host. Results and analysis.

Reproduce

Replay uses Python 3.11 and the standard library. Repeating inference requires the pinned MiniLM model and published environment packages.

Commands use a POSIX shell. Replace MODEL_PATH with the snapshot path printed by download and choose new output directories. Run development before evaluation and avoid concurrent compute-intensive work.

git clone https://github.com/lucioduran/chunking-ablation-suite.git
cd chunking-ablation-suite
git checkout 12193ba09bb1081a05d3ea7a043748089aa2330b
python -m chunking_suite verify --report results/dev/report.json
python -m chunking_suite verify --report results/eval/report.json

python -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements-local.lock
python -m chunking_suite download
# Replace MODEL_PATH with the snapshot path printed by download.
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 OMP_NUM_THREADS=2 MKL_NUM_THREADS=2 OPENBLAS_NUM_THREADS=2 python -m chunking_suite run --model-path MODEL_PATH --split dev --output build/reproduced-dev
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 OMP_NUM_THREADS=2 MKL_NUM_THREADS=2 OPENBLAS_NUM_THREADS=2 python -m chunking_suite run --model-path MODEL_PATH --split eval --output build/reproduced-eval
python -m chunking_suite verify --report build/reproduced-eval/report.json

Scope and limitations

• Synthetic corpus known to the author, with shared templates across partitions and no independent human review.
• Atomic-block preservation differs between methods. Method comparisons alone do not isolate the causal contribution of semantic embeddings.
• No answer generation, link following or unanswerable-query evaluation. Payload size does not represent total vector database memory.

• RAG architecture: chunking and evaluation

Sources and revision

Values on this page come from the linked report. They are not live measurements. The report records the executed revision; the reviewed revision includes the subsequent publication of that report.

• Reviewed revision: 12193ba09bb1081a05d3ea7a043748089aa2330b
• Revision recorded by the experiment: 5e1f27c2aba7624f181e4b1a172873b5180b8f53
• Report timestamp (UTC): 2026-10-21T21:57:38.943953+00:00
• Release published: v0.1.0 · 2026-10-21T22:01:14Z

• README and usage
• Original evidence
• Development evidence
• Evaluation summary JSON
• Results and analysis
• Design and methodology
• Data contract
• Models, revisions and licenses
• Measured environment dependencies
• MIT license