LLM Semantic Cache

Problem and relevance to agents

Similar questions can require different answers when users, permissions, entities, dates or sources change. Similarity does not authorize answer reuse, and embedding computation has a cost.

The project makes reuse boundaries explicit and measures whether net savings survive semantic-cache overhead. An unsafe diagnostic demonstrates what happens when those boundaries are ignored.

Architecture

1. Scope before similarity. Filter by user, permissions, version, entity, date and embedding-model namespace before comparison. Attempt exact lookup before computing a query vector.
2. Transactional invalidation. SQLite coordinates writes. A generation ticket captured before source access fences writes that predate an invalidation. Expiration, replacement and logical deletion are explicit.
3. Local model evaluation. MiniLM embeddings and FLAN-T5-small generation process 48 fictional requests per policy. Saved answers, decisions, vectors and timings support replay without inference.

• Design and methodology

What this demonstrates

• Reuse boundaries and rejection of stale in-flight writes after invalidation.
• Separate measurement of correct hits, incorrect reuse and answer accuracy.
• Net savings accounting including embedding, lookup, generation, writes and invalidation.

Published results

One fixed-order local run with 48 requests per policy on Intel Xeon W-3223 using four threads. MiniLM and FLAN-T5-small are revision-pinned; the similarity threshold is 0.90.

Hit rates use all requests as their denominator. Correct reuse requires identical scope, intended fact and reference value. Net savings include request-time work but exclude downloads, model loading and warmup.

Semantic caching reused four more answers than exact caching but saved less time: 15.0% versus 25.1%. Query embeddings consumed 0.308 s. Generation variation also contributes; one run does not establish a stable performance ranking.

The scope-blind diagnostic produced 32 incorrect reuses, 80% of its hits. No cache, scoped exact and scoped semantic reached 100% answer exact match on simple numeric questions; this does not establish general safety. Results summary.

Reproduce

Library and replay use Python 3.11 or later and SQLite without model dependencies. Neural reproduction downloads two public models and uses the measured environment lock.

Use a fresh output directory and one measurement at a time. Commands use POSIX and the lock targets Intel macOS. Callers must derive permissions and scope from trusted sources on every request.

git clone https://github.com/lucioduran/llm-semantic-cache.git
cd llm-semantic-cache
git checkout 78efb52757165a2f1dc5cc2f76197317b853643a
python -m scripts.verify results/reference/report.json

python3.11 -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements-models.lock
python -m scripts.evaluate download
OMP_NUM_THREADS=4 OPENBLAS_NUM_THREADS=4 MKL_NUM_THREADS=4 python -m scripts.evaluate run --output results/my-run
python -m scripts.verify results/my-run/report.json

Scope and limitations

• The library does not authenticate users or infer permissions. A version omitting prompt, model, content or policy dependencies can allow stale reuse.
• Overlapping reads may already hold an answer when invalidation occurs; there is no distributed revocation barrier. Global ticket invalidation may reject unrelated in-flight writes.
• Linear semantic search per scope, without a vector index, single-flight generation, size limit or automatic pruning. Deletion is logical.
• Small author-written corpus, one run and no timing intervals. This is not a security audit, scale test or general answer-quality evaluation.

• AI orchestration, memory and answer reuse

Sources and revision

Results come from the report and traces included in v0.1.0. The report identifies the implementation commit but contains no execution timestamp. The release date is shown separately.

• Reviewed revision: 78efb52757165a2f1dc5cc2f76197317b853643a
• Revision recorded by the experiment: c9c450254b5e7875c04611b5e3e4d22addca6db6
• Release published: v0.1.0 · 2026-10-21T23:13:38Z

• README and usage
• Original evidence
• Results summary
• Original responses and traces
• Design and methodology
• Data and provenance
• Frozen configuration
• Error analysis
• Models, revisions and licenses
• Measured environment dependencies
• MIT license