[Projects]() / AI Model Routing Policy # AI Model Routing Policy Routing between FLAN-T5-small and base with development-only fitting, held-out evaluation, measured router cost and injected local outages. Includes its negative result. Author: [Lucio Durán]() · Python 3.11.13 · License: [MIT]() Page published: 2026-10-22 · Page updated: 2026-10-22 [Repository]() · [Original evidence]() · [Release v0.1.0]() ## Problem and relevance to agents Adding a router does not automatically make inference more efficient than one model. Its overhead, failed attempts and task-level quality losses must be counted. The comparison holds tasks fixed and exposes decisions, fallback and unavailable outcomes. It supports deciding whether routing complexity is justified by measured results. ## Architecture 1. **Development-fitted policy.** 16 tasks fit a per-type lookup table using accuracy minus 0.02 times mean inference seconds. Held-out evaluation has 24 tasks; gold labels never reach the router. 2. **Four strategies.** Compare always-small, always-base, a fixed 120-character selector and the fitted policy. Both revision-pinned models remain resident. 3. **Fallback and accounting.** Only a typed unavailable error triggers the second adapter. Overhead, failed attempts, tokens, loading, warmup and policy fitting are recorded separately. - [Design and methodology]() ## What this demonstrates - Policy fitting separated from held-out evaluation with explicit baselines. - Routing cost and task-level quality degradation accounting. - Controlled fallback, explicit both-unavailable failure and analysis of a negative result. ## Published results 16 development and 24 evaluation tasks across sentiment, extraction, arithmetic and logic. Local FLAN-T5-small and base ran on Intel Xeon W-3223 with two threads. Request time includes routing, dispatch, failed attempts and inference. Loading, warmup and fitting are separately reported outside that table. Outages are immediate exceptions injected into owned local adapters. ### Comparison without injected failures | Strategy | Correct | Total time (s) | Router total (ms) | Generated tokens | | --- | --- | --- | --- | --- | | Always small | 12/24 | 4.006823 | 0.021426 | 229 | | Always base | 12/24 | 8.376350 | 0.020290 | 214 | | Length selector | 12/24 | 4.359402 | 0.035640 | 229 | | Fitted policy | 12/24 | 4.401570 | 0.044034 | 229 | ### Fallback with fitted policy | Injected unavailability | Correct | Unavailable | Total time (s) | | --- | --- | --- | --- | | Small down | 12/24 | 0 | 9.149960 | | Base down | 12/24 | 0 | 3.981469 | | Both down | 0/24 | 24 | 0.000176 | All four strategies reached 12/24. The fitted router was slower than always-small with no quality gain. All failed six extraction and six arithmetic tasks: zero relative degradation can conceal poor absolute quality. Development inference took 3.600025 s and policy fitting took 0.124500 ms. Full evaluation wall time was 41.797864 s, including setup, four strategies, outage scenarios and evidence recording. [Original evidence](). ## Reproduce Replay uses Python 3.11 and the standard library without model downloads. Neural reproduction requires pinned dependencies and public snapshots of both models. POSIX commands omit the author-specific coordinator. Run one measurement at a time, keeping development and evaluation in fresh directories connected through --dev. ### Replay saved evidence ``` git clone https://github.com/lucioduran/ai-model-routing-policy.git cd ai-model-routing-policy git checkout 1e891db0bfc7c6ad77a1baae3b93f5a76a558435 python -m routing verify --split dev --output results/dev python -m routing verify --split eval --dev results/dev --output results/eval ``` ### Repeat the local measurement ``` python3.11 -m venv .venv . .venv/bin/activate python -m pip install -r requirements-local.lock python -m routing download OMP_NUM_THREADS=2 MKL_NUM_THREADS=2 python -m routing run --split dev --output build/dev OMP_NUM_THREADS=2 MKL_NUM_THREADS=2 python -m routing run --split eval --dev build/dev --output build/eval python -m routing verify --split eval --dev build/dev --output build/eval ``` ## Scope and limitations - Forty synthetic tasks and one run do not establish production savings. Task families overlap across splits and there was no independent human review. - The caller supplies task types; no intent-classification cost is measured. The policy penalizes latency without enforcing a hard quality guarantee. - Immediate exceptions do not model network failures, timeouts or remote retry costs. Two resident models also do not represent memory-pressure deployments. - Paid API cost: USD 0. Measured CPU time is neither energy nor a bill. Request-level timings exclude loading and warmup. - [AI orchestration and model selection]() ## Sources and revision Values on this page come from the linked report. They are not live measurements. The report records the executed revision; the reviewed revision includes the subsequent publication of that report. - Reviewed revision: [` 1e891db0bfc7c6ad77a1baae3b93f5a76a558435 `]() - Revision recorded by the experiment: [` ae4081c9097561b40652b5c2f2680cba37660af2 `]() - Report timestamp (UTC): 2026-10-21T23:08:03.603602+00:00 - Release published: [v0.1.0]() · 2026-10-21T23:11:21Z - [README and usage]() - [Original evidence]() - [Original responses and traces]() - [Development evidence]() - [Fitted policy]() - [Design and methodology]() - [Data and provenance]() - [Models, revisions and licenses]() - [Measured environment dependencies]() - [MIT license]()