Agent Context Budget
Compare full-window input, truncation, selection and extractive compression while measuring evidence loss separately from answer quality.
Problem and relevance to agents
A compressed record can keep a citation identifier but lose the sentence that supports its answer. Token savings alone cannot establish useful context.
Frozen tasks with late and contradictory facts expose which policies lose decisive evidence and whether a small local model uses what remains.
Architecture
- Freeze the experiment. Twelve original fictional tasks and the evaluation protocol were committed before inference.
- Apply exact token budgets. A pinned tokenizer accounts for prompt overhead across chronological input, prefix truncation, record selection and sentence extraction.
- Preserve raw decisions. Prompts, fragments, source IDs, responses, token traces and resource observations support deterministic replay.
What this demonstrates
- Separate scoring for retained evidence, retained citation identifiers, generated citations and exact answers.
- Explicit recency and first-sentence ablations.
- Actual pinned FLAN-T5-small CPU inference with recorded failures.
Published results
One local Xeon CPU run made 132 model calls across eleven settings using two PyTorch CPU threads.
Each setting evaluates the same twelve tasks. Original prompts contain 142 to 418 tokens; none exceeds the model's 512-token input limit. Selected evidence and generated answers are scored independently.
Context policy evaluation
| Policy and token budget | Exact answers | Evidence retained | Citation IDs retained | Mean input tokens |
|---|---|---|---|---|
| full:512 | 10/12 | 1.000 | 1.000 | 280.4 |
| head:128 | 0/12 | 0.000 | 0.000 | 128.0 |
| head:256 | 5/12 | 0.500 | 0.500 | 205.2 |
| select:128 | 6/12 | 1.000 | 1.000 | 111.4 |
| select:256 | 3/12 | 1.000 | 1.000 | 195.9 |
| compress:128 | 1/12 | 1.000 | 1.000 | 106.4 |
| compress:256 | 2/12 | 1.000 | 1.000 | 141.4 |
| select_no_recency:128 | 6/12 | 1.000 | 1.000 | 112.4 |
| select_no_recency:256 | 4/12 | 1.000 | 1.000 | 196.4 |
| compress_first:128 | 2/12 | 0.667 | 1.000 | 111.7 |
| compress_first:256 | 3/12 | 0.667 | 1.000 | 157.2 |
Every setting produced zero correct output citations and zero grounded exact matches. First-sentence compression lost four corrections while retaining their citation IDs. Retaining evidence did not guarantee correct model use.
Reproduce
Portable replay requires Python 3.11 or newer and no runtime model packages. Inference uses the pinned Intel macOS Python 3.11.13 environment with PyTorch 2.2.2 and Transformers 4.46.3.
Run measurements in a new destination and serialize heavy workloads. Within the original project series, use the shared benchmark_lock.py wrapper. Commands below use POSIX shell syntax.
Verify saved evidence
git clone https://github.com/lucioduran/agent-context-budget.git
cd agent-context-budget
git checkout 72d150669a9fa4ab85496b4b62440ce95a9be7fe
python -m context_budget verify results/evaluation.jsonRepeat the local experiment
python3.11 -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements-local.lock
python scripts/download_model.py
python -m context_budget evaluate --model-path .cache/model --output build/new-run.json
python -m context_budget verify build/new-run.jsonScope and limitations
- Small author-known synthetic tasks, a single model and one observation per cell do not establish production quality or stable latency rankings.
- Compression is extractive, not learned summarization. Selection uses lexical overlap and can introduce chronology ambiguity after ranking.
- The implementation tests overflow handling, but the recorded run does not measure prompts longer than the declared model window.
- Replay trusts recorded tokenization and checks consistency, not inference authenticity. Energy and hardware costs are unmeasured; no paid API was used.
Sources and revision
The raw report identifies its implementation revision, timestamp, source hashes, task hashes and model hashes. Release documentation is separate from the recorded execution.
- Reviewed revision:
72d150669a9fa4ab85496b4b62440ce95a9be7fe - Revision recorded by the experiment:
4b38a6150b152758f99e4a12002d9b7e03a5d057 - Report timestamp (UTC):
- Release published: v0.1.1 ·