<!-- Canonical: https://lucioduran.com/projects/llm-semantic-cache
Language: en
Content-SHA256: 0e614edca40d4c051f2dbf61cd7bf51f70382d733b8c547545b64ca3c63026ee
-->

[Projects](<https://lucioduran.com/projects>) / LLM Semantic Cache

# LLM Semantic Cache

SQLite exact and semantic answer caching with user, permission and version isolation, transactional invalidation and local evaluation of correct and incorrect reuse.

Author: [Lucio Durán](<https://lucioduran.com/>) · Python 3.11.13 · License: [MIT](<https://github.com/lucioduran/llm-semantic-cache/blob/78efb52757165a2f1dc5cc2f76197317b853643a/LICENSE>)

Page published: 2026-10-22 · Page updated: 2026-10-22

[Repository](<https://github.com/lucioduran/llm-semantic-cache>) · [Original evidence](<https://raw.githubusercontent.com/lucioduran/llm-semantic-cache/78efb52757165a2f1dc5cc2f76197317b853643a/results/reference/report.json>) · [Release v0.1.0](<https://github.com/lucioduran/llm-semantic-cache/releases/tag/v0.1.0>)

## Problem and relevance to agents

Similar questions can require different answers when users, permissions, entities, dates or sources change. Similarity does not authorize answer reuse, and embedding computation has a cost.

The project makes reuse boundaries explicit and measures whether net savings survive semantic-cache overhead. An unsafe diagnostic demonstrates what happens when those boundaries are ignored.

## Architecture

1. **Scope before similarity.** Filter by user, permissions, version, entity, date and embedding-model namespace before comparison. Attempt exact lookup before computing a query vector.
2. **Transactional invalidation.** SQLite coordinates writes. A generation ticket captured before source access fences writes that predate an invalidation. Expiration, replacement and logical deletion are explicit.
3. **Local model evaluation.** MiniLM embeddings and FLAN-T5-small generation process 48 fictional requests per policy. Saved answers, decisions, vectors and timings support replay without inference.

- [Design and methodology](<https://github.com/lucioduran/llm-semantic-cache/blob/78efb52757165a2f1dc5cc2f76197317b853643a/docs/design.md>)

## What this demonstrates

- Reuse boundaries and rejection of stale in-flight writes after invalidation.
- Separate measurement of correct hits, incorrect reuse and answer accuracy.
- Net savings accounting including embedding, lookup, generation, writes and invalidation.

## Published results

One fixed-order local run with 48 requests per policy on Intel Xeon W-3223 using four threads. MiniLM and FLAN-T5-small are revision-pinned; the similarity threshold is 0.90.

Hit rates use all requests as their denominator. Correct reuse requires identical scope, intended fact and reference value. Net savings include request-time work but exclude downloads, model loading and warmup.

### Reuse and net time

| Policy | Correct hits | Incorrect hits | Total time (s) | Time saved |
| --- | --- | --- | --- | --- |
| No cache | 0/48 | 0/48 | 3.893 | 0% |
| Scoped exact | 12/48 | 0/48 | 2.915 | 25.1% |
| Scoped semantic | 16/48 | 0/48 | 3.310 | 15.0% |
| Scope-blind diagnostic | 8/48 | 32/48 | 0.717 | Invalid benefit |

Semantic caching reused four more answers than exact caching but saved less time: 15.0% versus 25.1%. Query embeddings consumed 0.308 s. Generation variation also contributes; one run does not establish a stable performance ranking.

The scope-blind diagnostic produced 32 incorrect reuses, 80% of its hits. No cache, scoped exact and scoped semantic reached 100% answer exact match on simple numeric questions; this does not establish general safety. [Results summary](<https://raw.githubusercontent.com/lucioduran/llm-semantic-cache/78efb52757165a2f1dc5cc2f76197317b853643a/results/reference/summary.json>).

## Reproduce

Library and replay use Python 3.11 or later and SQLite without model dependencies. Neural reproduction downloads two public models and uses the measured environment lock.

Use a fresh output directory and one measurement at a time. Commands use POSIX and the lock targets Intel macOS. Callers must derive permissions and scope from trusted sources on every request.

### Replay saved evidence

```
git clone https://github.com/lucioduran/llm-semantic-cache.git
cd llm-semantic-cache
git checkout 78efb52757165a2f1dc5cc2f76197317b853643a
python -m scripts.verify results/reference/report.json
```

### Repeat the local measurement

```
python3.11 -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements-models.lock
python -m scripts.evaluate download
OMP_NUM_THREADS=4 OPENBLAS_NUM_THREADS=4 MKL_NUM_THREADS=4 python -m scripts.evaluate run --output results/my-run
python -m scripts.verify results/my-run/report.json
```

## Scope and limitations

- The library does not authenticate users or infer permissions. A version omitting prompt, model, content or policy dependencies can allow stale reuse.
- Overlapping reads may already hold an answer when invalidation occurs; there is no distributed revocation barrier. Global ticket invalidation may reject unrelated in-flight writes.
- Linear semantic search per scope, without a vector index, single-flight generation, size limit or automatic pruning. Deletion is logical.
- Small author-written corpus, one run and no timing intervals. This is not a security audit, scale test or general answer-quality evaluation.

- [AI orchestration, memory and answer reuse](<https://lucioduran.com/blog/semantic-kernel-langchain-ai-orchestration>)

## Sources and revision

Results come from the report and traces included in v0.1.0. The report identifies the implementation commit but contains no execution timestamp. The release date is shown separately.

- Reviewed revision: [` 78efb52757165a2f1dc5cc2f76197317b853643a `](<https://github.com/lucioduran/llm-semantic-cache/commit/78efb52757165a2f1dc5cc2f76197317b853643a>)
- Revision recorded by the experiment: [` c9c450254b5e7875c04611b5e3e4d22addca6db6 `](<https://github.com/lucioduran/llm-semantic-cache/commit/c9c450254b5e7875c04611b5e3e4d22addca6db6>)
- Release published: [v0.1.0](<https://github.com/lucioduran/llm-semantic-cache/releases/tag/v0.1.0>) · 2026-10-21T23:13:38Z

- [README and usage](<https://github.com/lucioduran/llm-semantic-cache/blob/78efb52757165a2f1dc5cc2f76197317b853643a/README.md>)
- [Original evidence](<https://raw.githubusercontent.com/lucioduran/llm-semantic-cache/78efb52757165a2f1dc5cc2f76197317b853643a/results/reference/report.json>)
- [Results summary](<https://raw.githubusercontent.com/lucioduran/llm-semantic-cache/78efb52757165a2f1dc5cc2f76197317b853643a/results/reference/summary.json>)
- [Original responses and traces](<https://raw.githubusercontent.com/lucioduran/llm-semantic-cache/78efb52757165a2f1dc5cc2f76197317b853643a/results/reference/trace.jsonl>)
- [Design and methodology](<https://github.com/lucioduran/llm-semantic-cache/blob/78efb52757165a2f1dc5cc2f76197317b853643a/docs/design.md>)
- [Data and provenance](<https://raw.githubusercontent.com/lucioduran/llm-semantic-cache/78efb52757165a2f1dc5cc2f76197317b853643a/data/requests.json>)
- [Frozen configuration](<https://raw.githubusercontent.com/lucioduran/llm-semantic-cache/78efb52757165a2f1dc5cc2f76197317b853643a/data/config.json>)
- [Error analysis](<https://github.com/lucioduran/llm-semantic-cache/blob/78efb52757165a2f1dc5cc2f76197317b853643a/docs/methodology.md>)
- [Models, revisions and licenses](<https://github.com/lucioduran/llm-semantic-cache/blob/78efb52757165a2f1dc5cc2f76197317b853643a/docs/model-notices.md>)
- [Measured environment dependencies](<https://github.com/lucioduran/llm-semantic-cache/blob/78efb52757165a2f1dc5cc2f76197317b853643a/requirements-models.lock>)
- [MIT license](<https://github.com/lucioduran/llm-semantic-cache/blob/78efb52757165a2f1dc5cc2f76197317b853643a/LICENSE>)