[Projects]() / RAG Evaluation Workbench # RAG Evaluation Workbench Reproducible RAG evaluation with BM25, dense and hybrid retrieval, local CPU inference, and separate retrieval and answer metrics. Author: [Lucio Durán]() · Python 3.11.13 · License: [MIT]() Page published: 2026-10-21 · Page updated: 2026-10-21 [Repository]() · [Raw evaluation report]() ## Problem and relevance to agents Retrieving the right document does not guarantee a correct answer. RAG Evaluation Workbench compares three retrieval methods while keeping the answer model, prompt, context budgets and decoding fixed. It records rankings, included context, predictions and errors to evaluate each stage separately. An agent that retrieves evidence can still confuse facts or invent an answer when it should abstain. This project makes those failures inspectable without crediting retrieval with a success the final answer did not achieve. ## Architecture 1. **Corpus and split.** 24 original fictional documents under MIT, 12 development questions and 26 evaluation questions. Development and evaluation use disjoint relevant-document groups while searching the same corpus. Two evaluation questions are unanswerable. 2. **Comparable retrieval.** BM25, MiniLM cosine similarity and reciprocal rank fusion with constant 60. All three retrieve up to three documents. Each complete document is a retrieval unit; this experiment does not compare chunking strategies or rerankers. 3. **Frozen generation.** Local CPU FLAN-T5-small, four threads, seed 1729 and greedy decoding. Budgets are 256 embedding input tokens, 384 generation input tokens and 32 output tokens. Whole documents are packed in ranking order until the budget is exhausted. Labels and reference answers never enter the prompt. 4. **Evidence and verification.** Every run preserves input and model hashes, revisions, environment details, rankings, prompts, predictions and timings. A standard-library verifier reconstructs rankings and prompts and recalculates metrics and intervals from saved data without rerunning the models. - [Methodology and measurement boundaries]() - [Frozen configuration]() ## What this demonstrates - Controlled comparisons that separate retrieval quality, answer quality and abstention. - Recall@3, NDCG@3, exact match and token F1 implementation with grouped bootstrap and paired method differences. - Traceability from aggregate results to individual queries, rankings, prompts and predictions, including unfavorable results and scoring limitations. ## Published results Author-published results from a local Intel Xeon W-3223 run with Python 3.11.13 and four CPU threads. Retrieval metrics use 24 answerable queries; exact match and token F1 use all 26 evaluation queries, including the two unanswerable questions. The 95% intervals use 2,000 bootstrap resamples of source groups. Both questions about a document stay together. Method differences use paired observations with aligned group resampling. There are 12 answerable groups and two unanswerable groups. Tables round values; the reports retain full precision. ### Retrieval: 24 answerable queries Recall@3 and NDCG@3 over 24 answerable questions in 12 source groups. Unanswerable questions are excluded from these metrics. The 95% intervals use grouped bootstrap. | Method | Recall@3 | 95% interval | NDCG@3 | 95% interval | | --- | --- | --- | --- | --- | | BM25 | 1.000 | \[1.000, 1.000\] | 0.969 | \[0.923, 1.000\] | | Dense | 1.000 | \[1.000, 1.000\] | 1.000 | \[1.000, 1.000\] | | Hybrid | 1.000 | \[1.000, 1.000\] | 1.000 | \[1.000, 1.000\] | ### Answers: 26 evaluation queries Exact match and token F1 over all 26 questions in 14 groups, including two unanswerable questions. These are surface-form metrics, not factuality measures. The 95% intervals use grouped bootstrap. | Method | Exact match | 95% interval | Token F1 | 95% interval | | --- | --- | --- | --- | --- | | BM25 | 0.731 | \[0.560, 0.885\] | 0.756 | \[0.583, 0.889\] | | Dense | 0.769 | \[0.600, 0.893\] | 0.795 | \[0.627, 0.926\] | | Hybrid | 0.731 | \[0.560, 0.885\] | 0.767 | \[0.605, 0.899\] | ### Paired exact-match differences Exact-match differences over the same 26 questions with paired resampling of 14 groups. The dense-minus-BM25 interval includes zero; this sample does not establish a general improvement. | Comparison | Mean difference | 95% interval | | --- | --- | --- | | Dense minus BM25 | 0.038 | \[0.000, 0.120\] | | Hybrid minus BM25 | 0.000 | \[0.000, 0.000\] | ### Request latency and paid API cost Milliseconds per request in the recorded Intel Xeon W-3223 run with four threads. One observation per query and method, 26 queries per method. Model loading, warmup and indexing are excluded. USD 0 covers paid APIs only; electricity and hardware costs were not measured. | Method | P50 (ms) | P95 (ms) | API (USD) | | --- | --- | --- | --- | | BM25 | 93.7 | 133.4 | 0 | | Dense | 105.9 | 138.8 | 0 | | Hybrid | 105.9 | 144.4 | 0 | All three methods reach Recall@3 = 1 and retain relevant context in the prompt, but none answers every query correctly. Dense gets one more exact-match success than BM25: 20/26 versus 19/26. Its paired difference is 0.038 and the interval \[0.000, 0.120\] includes zero. This result does not establish a general winner. All three methods fail both abstention questions: 0/2 per method. Across 78 predictions there are 20 exact-match failures: 14 with relevant context present and six on unanswerable questions. [Error analysis](). Some failures are scoring limitations: “90 requests per minute” is correct but excluded by short aliases frozen before evaluation. Other failures are substantive, such as confusing a door access code with a network enrollment code. Aliases were not expanded after inspecting predictions. [Per-query errors](). Request latency includes retrieval, context packing and generation. Model loading, warmup and index construction are measured separately. There is one observation per query and method. Paid API cost is USD 0; electricity and hardware costs are unmeasured. [Methodology and measurement boundaries](). ## Reproduce Report verification uses only the Python standard library. Reproducing inference requires installing dependencies and explicitly downloading two public models. No account, paid API or rented GPU is required. With Python 3.11.13, the first block checks recorded evidence consistency. The second runs the models from the same repository. The lock targets the measured Intel macOS environment; other platforms may need compatible PyTorch packages. Use new output directories. Downloading requires internet and several hundred MB; inference then runs offline. A new run records its own environment, timestamp and timings. ### Verify the recorded report without running models ``` git clone https://github.com/lucioduran/rag-evaluation-workbench.git cd rag-evaluation-workbench git checkout 60246831be5cf68768710e83d7b41614e49a8341 python -m rag_workbench verify --report results/eval/report.json ``` ### Repeat local inference in the reference environment ``` python -m venv .venv . .venv/bin/activate python -m pip install -r requirements-local.lock python -m rag_workbench download HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 TOKENIZERS_PARALLELISM=false python -m rag_workbench run --split dev --output build/reproduced-dev HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 TOKENIZERS_PARALLELISM=false python -m rag_workbench run --split eval --output build/reproduced-eval python -m rag_workbench verify --report build/reproduced-eval/report.json ``` ## Scope and limitations - The small synthetic corpus is easy for retrieval. The split is reserved against parameter tuning but known to its author. This is not an externally blind evaluation or evidence of production generalization. - Exact match and token F1 compare surface forms, not factuality. Frozen aliases can penalize valid paraphrases. Manual analysis does not change the scores. - The small answer model fails abstention on this sample. Perfect recall does not mean correct end-to-end answers. - Intervals are based on few groups. A degenerate interval does not prove certainty beyond the corpus. Latencies describe one local run, not a load test or SLA. - Offline verification checks internal consistency. It does not authenticate measurements, rerun neural inference or independently recount tokenizer tokens. Hardware and backend changes can alter both timing and predictions. - The CLI expects trusted local files and loads inputs and results into memory. It is not an upload service. Code, corpus and results use MIT; downloaded models use Apache-2.0 and are not redistributed in the repository. - [Article: RAG pipeline architecture, retrieval and evaluation]() ## Sources and revision Values on this page come from the linked report. They are not live measurements. The report records the executed revision; the reviewed revision includes the subsequent publication of that report. - Reviewed revision: [` 60246831be5cf68768710e83d7b41614e49a8341 `]() - Revision recorded by the experiment: [` 1087f9b0b7985ce719204595c7a90b45e1e186b2 `]() - Report timestamp (UTC): 2026-10-21T21:08:41.652766+00:00 - Corpus SHA-256: ` ecf0e3d23dbee38f61aad545e67d7ad3e81195e39659a4619b7035babb5938e6 ` - [README]() - [Methodology and measurement boundaries]() - [Frozen configuration]() - [Dataset provenance and split policy]() - [Original corpus]() - [Questions and labels]() - [Raw evaluation report]() - [Metric summary]() - [Per-query errors]() - [Error analysis]() - [Raw development report]() - [Reference environment dependencies]() - [Model licenses]() - [MIT license]()