<!-- Canonical: https://lucioduran.com/topics/llm-evaluation
Language: en
Content-SHA256: c10f780e08649a64d0c781950fe4208cd316303b3322d7d559654b59bd06f37a
-->

# LLM and AI agent evaluation

Evaluation connects an engineering decision to observable evidence. These projects separate dimensions that are often combined: an answer can be valid JSON yet incorrect, include an unsupported citation, or complete a task while violating an execution boundary.

By [Lucio Durán](<https://lucioduran.com/>) · Uruguay

## Which decision are you making?

### When deciding a model change

Compare both candidates on versioned tasks and inspect segments that regress. Model Regression Gate retains paired results and distinguishes acceptance, rejection and review.

### When evaluating answers

Measure content as well as format. Structured Output Contracts contrasts validity with accuracy; Citation Grounding Checker links each claim to the passage that might support it.

### When evaluating agents

Record observable effects and failures as well as the final answer. Agent Injection Evals separates capability, attack attempts and realized effects; Agent Context Budget studies evidence loss and answer quality.

## Projects and evidence

Each project page provides code, environment details, published results and limitations. The experiments describe those configurations; they do not establish results in client systems.

- ### [Model Regression Gate](<https://lucioduran.com/projects/model-regression-gate>)

  A reproducible model promotion gate with versioned tasks, paired accuracy comparisons, slice tolerances and a separate review-required outcome.

  Python · License: MIT

  [Read the project](<https://lucioduran.com/projects/model-regression-gate>) · [Repository](<https://github.com/lucioduran/model-regression-gate>) · [Release v0.1.0](<https://github.com/lucioduran/model-regression-gate/releases/tag/v0.1.0>)
- ### [Agent Injection Evals](<https://lucioduran.com/projects/agent-injection-evals>)

  Reproducible prompt-injection evaluation with a pinned local model, simulated tools and separate metrics for task capability, attack attempts and realized effects.

  Python · License: MIT

  [Read the project](<https://lucioduran.com/projects/agent-injection-evals>) · [Repository](<https://github.com/lucioduran/agent-injection-evals>)
- ### [Structured Output Contracts](<https://lucioduran.com/projects/structured-output-contracts>)

  Bounded JSON contracts with strict validation, explicit refusals and limited retries. Separates validity from correctness using two local Qwen models and preserved raw responses.

  Python · License: MIT

  [Read the project](<https://lucioduran.com/projects/structured-output-contracts>) · [Repository](<https://github.com/lucioduran/structured-output-contracts>) · [Release v0.1.0](<https://github.com/lucioduran/structured-output-contracts/releases/tag/v0.1.0>)
- ### [Citation Grounding Checker](<https://lucioduran.com/projects/citation-grounding-checker>)

  Citation support checking with rules, FLAN-T5 and DeBERTa NLI. Predictions, abstentions and precision linked to each claim’s exact source passage.

  Python · License: MIT

  [Read the project](<https://lucioduran.com/projects/citation-grounding-checker>) · [Repository](<https://github.com/lucioduran/citation-grounding-checker>) · [Release v0.1.0](<https://github.com/lucioduran/citation-grounding-checker/releases/tag/v0.1.0>)
- ### [AI Model Routing Policy](<https://lucioduran.com/projects/ai-model-routing-policy>)

  Routing between FLAN-T5-small and base with development-only fitting, held-out evaluation, measured router cost and injected local outages. Includes its negative result.

  Python · License: MIT

  [Read the project](<https://lucioduran.com/projects/ai-model-routing-policy>) · [Repository](<https://github.com/lucioduran/ai-model-routing-policy>) · [Release v0.1.0](<https://github.com/lucioduran/ai-model-routing-policy/releases/tag/v0.1.0>)
- ### [Agent Context Budget](<https://lucioduran.com/projects/agent-context-budget>)

  Compare full-window input, truncation, selection and extractive compression while measuring evidence loss separately from answer quality.

  Python · License: MIT

  [Read the project](<https://lucioduran.com/projects/agent-context-budget>) · [Repository](<https://github.com/lucioduran/agent-context-budget>) · [Release v0.1.1](<https://github.com/lucioduran/agent-context-budget/releases/tag/v0.1.1>)

## Architecture and methodology reading

- ### [Evaluating agents without self-deception: harnesses, contamination, and cost per success](<https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success>)

  How to evaluate long-running agents through verifiable state, procedural checks, repeated trials, comparable infrastructure, and costs that include failures.