[Projects]() / Reranker Budget Router # Reranker Budget Router Reranking router with hard pair budgets, development-fitted policies and a local MiniLM cross-encoder. Publishes quality, latency, decisions and routing errors. Author: [Lucio Durán]() · Python 3.11.13 · License: [MIT]() Page published: 2026-10-21 · Page updated: 2026-10-21 [Repository]() · [Original evidence]() · [Release v0.1.0]() ## Problem and relevance to agents Reranking every candidate spends compute even when BM25 already resolves the query. Skipping the reranker can also miss a gain. The decision needs measured comparisons and visible failure cases. The project exposes how much compute a policy assigns to each query and which quality gains it misses. It separates pair budgets from dollars, energy and latency guarantees. ## Architecture 1. **Retrieval and signals.** BM25 produces candidates. Score gaps and term coverage feed a router choosing depth 0, 4 or 12 under a hard pair budget. 2. **Frozen policies.** Six budget/penalty combinations fit thresholds using development data only. A pinned MiniLM cross-encoder reorders the chosen prefix and preserves the tail. 3. **Auditable decisions.** Per-query traces, counterfactual scores, policies and hashes support metric replay, avoidable-compute analysis and localization of quality losses. - [Design and methodology]() ## What this demonstrates - Explicit inference-budget allocation with tuning separated from evaluation. - Joint quality, latency and scored-pair measurement against two baselines. - Negative-result analysis: identifying when a router adds cost and misses a gain. ## Published results Corpus of 24 fictional documents, 12 development queries and 26 evaluation queries, including two unanswerable questions. MiniLM ran on Intel Xeon W-3223 with two threads and three serial repetitions per policy. Warm latency includes retrieval, signals, routing, tokenization and inference. Unanswerable questions receive zero quality. Each policy has 26 quality units; repeated timing does not create 78 independent queries. ### Evaluation quality and cost | Policy | nDCG@3 | Mean pairs | Mean (ms) | p95 (ms) | | --- | --- | --- | --- | --- | | always-12 | 0.9089 | 12.00 | 101.321 | 126.578 | | never | 0.8947 | 0.00 | 0.108 | 0.152 | | router-b12-p0 | 0.8947 | 0.00 | 0.114 | 0.176 | | router-b12-p0.01 | 0.8947 | 0.00 | 0.112 | 0.154 | | router-b12-p0.04 | 0.8947 | 0.00 | 0.109 | 0.139 | | router-b4-p0 | 0.8947 | 0.00 | 0.160 | 0.201 | | router-b4-p0.01 | 0.8947 | 0.00 | 0.127 | 0.194 | | router-b4-p0.04 | 0.8947 | 0.00 | 0.116 | 0.152 | All six learned policies skipped reranking. They matched never’s quality and added overhead. The always-12 baseline improved eval-iris-1 and reached 0.9089 nDCG@3 versus 0.8947. This evaluation does not demonstrate a router advantage. The six quality-loss errors are one query counted across six policies. The observed frontier contains always-12 and never. Tiny timing differences between policies with identical decisions are noise. [Results and analysis](). ## Reproduce Replay and route require Python 3.11 or later without model dependencies. route displays signals and candidates without inference. Full reproduction downloads a public cross-encoder. Environment commands use a POSIX shell. reproduce.py acquires an exclusive lock, runs both partitions and verifies evidence. Output must be new; dev and eval remain sibling folders so policy provenance can be checked. ### Replay saved evidence ``` git clone https://github.com/lucioduran/reranker-budget-router.git cd reranker-budget-router git checkout b34802875068f89c8fb3550f53680d60afdae513 python -m budget_router verify --output results/dev python -m budget_router verify --output results/eval python -m budget_router route "Where can I restore a deleted notebook?" --budget 4 ``` ### Repeat the local measurement ``` python3.11 -m venv .venv . .venv/bin/activate python -m pip install -r requirements-models.txt -r requirements-dev.txt python -m budget_router download python scripts/reproduce.py --output build/reproduction ``` ## Scope and limitations - Small synthetic dataset known to the author. Query groups are correlated and relevance has no independent human validation. - The budget limits pairs, not maximum elapsed time. The model cannot recover documents outside its prefix and truncates inputs to 256 tokens. - Three warm repetitions on one CPU do not characterize concurrency, energy, monetary cost or stable production tail percentiles. - [RAG architecture: reranking and cost evaluation]() ## Sources and revision Values come from the summary, traces and analysis included in v0.1.0. Metadata identifies the executed revision and environment but contains no execution timestamp. The release date is shown separately. - Reviewed revision: [` b34802875068f89c8fb3550f53680d60afdae513 `]() - Revision recorded by the experiment: [` 09ac402ba70b1456e4e49dfa3d3fc30de18d56d3 `]() - Release published: [v0.1.0]() · 2026-10-21T22:16:27Z - Corpus SHA-256: ` ecf0e3d23dbee38f61aad545e67d7ad3e81195e39659a4619b7035babb5938e6 ` - [README and usage]() - [Original evidence]() - [Results and analysis]() - [Design and methodology]() - [Corpus and provenance]() - [Runtime environment]() - [Frozen policies]() - [Routing errors]() - [Execution traces]() - [Development evidence]() - [Models, revisions and licenses]() - [Measured environment dependencies]() - [MIT license]()