[Projects]() / Incremental RAG Index # Incremental RAG Index Versioned SQLite RAG indexes with content-hash embedding reuse and atomic publication. Compares full and incremental rebuilds and injects process exits. Author: [Lucio Durán]() · Python 3.11.13 · License: [MIT]() Page published: 2026-10-22 · Page updated: 2026-10-22 [Repository]() · [Original evidence]() · [Release v0.1.0]() ## Problem and relevance to agents An update can fail after extraction or embedding. Readers need a complete version, and retries after a lost acknowledgement must not duplicate versions or restore deleted documents. The index preserves versions and a change journal. Its evidence shows what changed, interrupts publication and compares recovery with a complete rebuild. ## Architecture 1. **Content-addressed reuse.** Accepts the complete corpus as an ID-to-text map. Reuses identical-content embeddings, invalidates reuse when the adapter changes and removes missing IDs from the current version. 2. **Transactional publication.** SQLite commits a complete version and journal atomically. Writers based on stale versions are rejected. 3. **Observable recovery.** Process exits before and after commit record version, integrity and retry outcomes. Reports compare documents, vectors, roots and rankings. - [Design and methodology]() ## What this demonstrates - Consistency between incremental updates and full rebuilds. - Atomic publication, writer coordination and retries after lost acknowledgement. - A clear distinction between index equivalence and retrieval quality. ## Published results Two published experiments use a deterministic adapter and local MiniLM. Each covers five scenarios and four exit checkpoints. The table shows MiniLM on six synthetic documents, Python 3.11.13, SQLite 3.50.4 and two CPU threads. Documents, vectors, roots and rankings match exactly across full and incremental builds in all five scenarios for both adapters. Timings exclude model loading and query embedding. ### Computed embeddings and update time | Scenario | Incremental embeddings | Full embeddings | Incremental (ms) | Full (ms) | | --- | --- | --- | --- | --- | | Initial load | 6 | 6 | 88.6 | 98.0 | | Edit, add and delete | 2 | 6 | 40.2 | 92.8 | | Rename one document | 0 | 6 | 9.8 | 96.2 | | Repeat identical corpus | 0 | 6 | 7.8 | 86.0 | | Empty corpus | 0 | 0 | 7.0 | 3.3 | Three pre-commit exits preserved the old version; the post-commit exit preserved the new one. Retrying after commit created no additional version. All four recovered databases per adapter passed SQLite integrity checks. Deleted IDs disappear from the current version and its searches. Historical versions retain those documents; deletion is not physical erasure. [Design and methodology](). ## Reproduce The library, CLI and replay require Python 3.11 or later without model dependencies. Neural reproduction downloads MiniLM and uses the measured environment dependencies. Use new output paths and run experiments serially. Commands use a POSIX shell; the dependency lock targets Intel macOS. The default adapter is deterministic hashing for storage verification, not a semantic encoder. ### Replay saved evidence ``` git clone https://github.com/lucioduran/incremental-rag-index.git cd incremental-rag-index git checkout 9de502f73b0992f79fafda8c079e4363fa9d8fa9 python -m experiments.verify results/deterministic.json python -m experiments.verify results/minilm.json ``` ### Repeat the local measurement ``` python3.11 -m venv .venv . .venv/bin/activate python -m pip install -r requirements-local.lock python -m experiments.download --output build/minilm OMP_NUM_THREADS=2 MKL_NUM_THREADS=2 TOKENIZERS_PARALLELISM=false HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python -m experiments.reproduce --model build/minilm --output build/minilm-report.json python -m experiments.verify build/minilm-report.json python -m experiments.reproduce --output build/deterministic-report.json python -m experiments.verify build/deterministic-report.json ``` ## Scope and limitations - Whole-corpus hashing and full snapshot writes remain linear. No compaction, ANN index or incremental crawler is provided. - Six documents and one timing observation per scenario do not establish production speedups. Power loss, disk-full conditions and production concurrent load were not evaluated. - Guarantees depend on SQLite and the local filesystem. Existing readers and historical versions retain earlier data. - MiniLM limits documents to 256 tokens. Hashes detect inconsistency but do not authenticate data against an attacker with write access. - [RAG architecture: index consistency and updates]() ## Sources and revision Values on this page come from the linked report. They are not live measurements. The report records the executed revision; the reviewed revision includes the subsequent publication of that report. - Reviewed revision: [` 9de502f73b0992f79fafda8c079e4363fa9d8fa9 `]() - Revision recorded by the experiment: [` 37559ac9cfd8c039dbbf884c8814f2e4018e311b `]() - Report timestamp (UTC): 2026-10-21T22:54:50.852898+00:00 - Release published: [v0.1.0]() · 2026-10-21T22:59:53Z - [README and usage]() - [Original evidence]() - [Deterministic adapter evidence]() - [Design and methodology]() - [Data and provenance]() - [Models, revisions and licenses]() - [Measured environment dependencies]() - [MIT license]()