{"schemaVersion":1,"id":"https://lucioduran.com/topics/llm-evaluation","url":"https://lucioduran.com/topics/llm-evaluation","title":"LLM and AI agent evaluation","description":"Reproducible model and agent evaluation: regressions, citations, structured output and prompt injection. Source code and results by Lucio Durán.","language":"en","datePublished":"2026-11-10","dateModified":"2026-11-10","contentFormat":"text/markdown","content":"# LLM and AI agent evaluation\n\nEvaluation connects an engineering decision to observable evidence. These projects separate dimensions that are often combined: an answer can be valid JSON yet incorrect, include an unsupported citation, or complete a task while violating an execution boundary.\n\nBy [Lucio Durán](<https://lucioduran.com/>) · Uruguay\n\n## Which decision are you making?\n\n### When deciding a model change\n\nCompare both candidates on versioned tasks and inspect segments that regress. Model Regression Gate retains paired results and distinguishes acceptance, rejection and review.\n\n### When evaluating answers\n\nMeasure content as well as format. Structured Output Contracts contrasts validity with accuracy; Citation Grounding Checker links each claim to the passage that might support it.\n\n### When evaluating agents\n\nRecord observable effects and failures as well as the final answer. Agent Injection Evals separates capability, attack attempts and realized effects; Agent Context Budget studies evidence loss and answer quality.\n\n## Projects and evidence\n\nEach project page provides code, environment details, published results and limitations. The experiments describe those configurations; they do not establish results in client systems.\n\n- ### [Model Regression Gate](<https://lucioduran.com/projects/model-regression-gate>)\n\n  A reproducible model promotion gate with versioned tasks, paired accuracy comparisons, slice tolerances and a separate review-required outcome.\n\n  Python · License: MIT\n\n  [Read the project](<https://lucioduran.com/projects/model-regression-gate>) · [Repository](<https://github.com/lucioduran/model-regression-gate>) · [Release v0.1.0](<https://github.com/lucioduran/model-regression-gate/releases/tag/v0.1.0>)\n- ### [Agent Injection Evals](<https://lucioduran.com/projects/agent-injection-evals>)\n\n  Reproducible prompt-injection evaluation with a pinned local model, simulated tools and separate metrics for task capability, attack attempts and realized effects.\n\n  Python · License: MIT\n\n  [Read the project](<https://lucioduran.com/projects/agent-injection-evals>) · [Repository](<https://github.com/lucioduran/agent-injection-evals>)\n- ### [Structured Output Contracts](<https://lucioduran.com/projects/structured-output-contracts>)\n\n  Bounded JSON contracts with strict validation, explicit refusals and limited retries. Separates validity from correctness using two local Qwen models and preserved raw responses.\n\n  Python · License: MIT\n\n  [Read the project](<https://lucioduran.com/projects/structured-output-contracts>) · [Repository](<https://github.com/lucioduran/structured-output-contracts>) · [Release v0.1.0](<https://github.com/lucioduran/structured-output-contracts/releases/tag/v0.1.0>)\n- ### [Citation Grounding Checker](<https://lucioduran.com/projects/citation-grounding-checker>)\n\n  Citation support checking with rules, FLAN-T5 and DeBERTa NLI. Predictions, abstentions and precision linked to each claim’s exact source passage.\n\n  Python · License: MIT\n\n  [Read the project](<https://lucioduran.com/projects/citation-grounding-checker>) · [Repository](<https://github.com/lucioduran/citation-grounding-checker>) · [Release v0.1.0](<https://github.com/lucioduran/citation-grounding-checker/releases/tag/v0.1.0>)\n- ### [AI Model Routing Policy](<https://lucioduran.com/projects/ai-model-routing-policy>)\n\n  Routing between FLAN-T5-small and base with development-only fitting, held-out evaluation, measured router cost and injected local outages. Includes its negative result.\n\n  Python · License: MIT\n\n  [Read the project](<https://lucioduran.com/projects/ai-model-routing-policy>) · [Repository](<https://github.com/lucioduran/ai-model-routing-policy>) · [Release v0.1.0](<https://github.com/lucioduran/ai-model-routing-policy/releases/tag/v0.1.0>)\n- ### [Agent Context Budget](<https://lucioduran.com/projects/agent-context-budget>)\n\n  Compare full-window input, truncation, selection and extractive compression while measuring evidence loss separately from answer quality.\n\n  Python · License: MIT\n\n  [Read the project](<https://lucioduran.com/projects/agent-context-budget>) · [Repository](<https://github.com/lucioduran/agent-context-budget>) · [Release v0.1.1](<https://github.com/lucioduran/agent-context-budget/releases/tag/v0.1.1>)\n\n## Architecture and methodology reading\n\n- ### [Evaluating agents without self-deception: harnesses, contamination, and cost per success](<https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success>)\n\n  How to evaluate long-running agents through verifiable state, procedural checks, repeated trials, comparable infrastructure, and costs that include failures.","headings":[{"level":1,"text":"LLM and AI agent evaluation","url":"https://lucioduran.com/topics/llm-evaluation#title"},{"level":2,"text":"Which decision are you making?","url":"https://lucioduran.com/topics/llm-evaluation#decisions"},{"level":3,"text":"When deciding a model change","url":"https://lucioduran.com/topics/llm-evaluation#decision-1"},{"level":3,"text":"When evaluating answers","url":"https://lucioduran.com/topics/llm-evaluation#decision-2"},{"level":3,"text":"When evaluating agents","url":"https://lucioduran.com/topics/llm-evaluation#decision-3"},{"level":2,"text":"Projects and evidence","url":"https://lucioduran.com/topics/llm-evaluation#projects"},{"level":3,"text":"Model Regression Gate","url":"https://lucioduran.com/topics/llm-evaluation#project-model-regression-gate"},{"level":3,"text":"Agent Injection Evals","url":"https://lucioduran.com/topics/llm-evaluation#project-agent-injection-evals"},{"level":3,"text":"Structured Output Contracts","url":"https://lucioduran.com/topics/llm-evaluation#project-structured-output-contracts"},{"level":3,"text":"Citation Grounding Checker","url":"https://lucioduran.com/topics/llm-evaluation#project-citation-grounding-checker"},{"level":3,"text":"AI Model Routing Policy","url":"https://lucioduran.com/topics/llm-evaluation#project-ai-model-routing-policy"},{"level":3,"text":"Agent Context Budget","url":"https://lucioduran.com/topics/llm-evaluation#project-agent-context-budget"},{"level":2,"text":"Architecture and methodology reading","url":"https://lucioduran.com/topics/llm-evaluation#articles"},{"level":3,"text":"Evaluating agents without self-deception: harnesses, contamination, and cost per success","url":"https://lucioduran.com/topics/llm-evaluation#article-agent-evaluation-harness-contamination-cost-per-success"}],"sections":[{"heading":"LLM and AI agent evaluation","level":1,"url":"https://lucioduran.com/topics/llm-evaluation","content":""},{"heading":"LLM and AI agent evaluation","level":1,"url":"https://lucioduran.com/topics/llm-evaluation#title","content":"Evaluation connects an engineering decision to observable evidence. These projects separate dimensions that are often combined: an answer can be valid JSON yet incorrect, include an unsupported citation, or complete a task while violating an execution boundary.\n\nBy [Lucio Durán](<https://lucioduran.com/>) · Uruguay"},{"heading":"Which decision are you making?","level":2,"url":"https://lucioduran.com/topics/llm-evaluation#decisions","content":""},{"heading":"When deciding a model change","level":3,"url":"https://lucioduran.com/topics/llm-evaluation#decision-1","content":"Compare both candidates on versioned tasks and inspect segments that regress. Model Regression Gate retains paired results and distinguishes acceptance, rejection and review."},{"heading":"When evaluating answers","level":3,"url":"https://lucioduran.com/topics/llm-evaluation#decision-2","content":"Measure content as well as format. Structured Output Contracts contrasts validity with accuracy; Citation Grounding Checker links each claim to the passage that might support it."},{"heading":"When evaluating agents","level":3,"url":"https://lucioduran.com/topics/llm-evaluation#decision-3","content":"Record observable effects and failures as well as the final answer. Agent Injection Evals separates capability, attack attempts and realized effects; Agent Context Budget studies evidence loss and answer quality."},{"heading":"Projects and evidence","level":2,"url":"https://lucioduran.com/topics/llm-evaluation#projects","content":"Each project page provides code, environment details, published results and limitations. The experiments describe those configurations; they do not establish results in client systems."},{"heading":"Model Regression Gate","level":3,"url":"https://lucioduran.com/topics/llm-evaluation#project-model-regression-gate","content":"A reproducible model promotion gate with versioned tasks, paired accuracy comparisons, slice tolerances and a separate review-required outcome.\n\nPython · License: MIT\n\n[Read the project](<https://lucioduran.com/projects/model-regression-gate>) · [Repository](<https://github.com/lucioduran/model-regression-gate>) · [Release v0.1.0](<https://github.com/lucioduran/model-regression-gate/releases/tag/v0.1.0>)"},{"heading":"Agent Injection Evals","level":3,"url":"https://lucioduran.com/topics/llm-evaluation#project-agent-injection-evals","content":"Reproducible prompt-injection evaluation with a pinned local model, simulated tools and separate metrics for task capability, attack attempts and realized effects.\n\nPython · License: MIT\n\n[Read the project](<https://lucioduran.com/projects/agent-injection-evals>) · [Repository](<https://github.com/lucioduran/agent-injection-evals>)"},{"heading":"Structured Output Contracts","level":3,"url":"https://lucioduran.com/topics/llm-evaluation#project-structured-output-contracts","content":"Bounded JSON contracts with strict validation, explicit refusals and limited retries. Separates validity from correctness using two local Qwen models and preserved raw responses.\n\nPython · License: MIT\n\n[Read the project](<https://lucioduran.com/projects/structured-output-contracts>) · [Repository](<https://github.com/lucioduran/structured-output-contracts>) · [Release v0.1.0](<https://github.com/lucioduran/structured-output-contracts/releases/tag/v0.1.0>)"},{"heading":"Citation Grounding Checker","level":3,"url":"https://lucioduran.com/topics/llm-evaluation#project-citation-grounding-checker","content":"Citation support checking with rules, FLAN-T5 and DeBERTa NLI. Predictions, abstentions and precision linked to each claim’s exact source passage.\n\nPython · License: MIT\n\n[Read the project](<https://lucioduran.com/projects/citation-grounding-checker>) · [Repository](<https://github.com/lucioduran/citation-grounding-checker>) · [Release v0.1.0](<https://github.com/lucioduran/citation-grounding-checker/releases/tag/v0.1.0>)"},{"heading":"AI Model Routing Policy","level":3,"url":"https://lucioduran.com/topics/llm-evaluation#project-ai-model-routing-policy","content":"Routing between FLAN-T5-small and base with development-only fitting, held-out evaluation, measured router cost and injected local outages. Includes its negative result.\n\nPython · License: MIT\n\n[Read the project](<https://lucioduran.com/projects/ai-model-routing-policy>) · [Repository](<https://github.com/lucioduran/ai-model-routing-policy>) · [Release v0.1.0](<https://github.com/lucioduran/ai-model-routing-policy/releases/tag/v0.1.0>)"},{"heading":"Agent Context Budget","level":3,"url":"https://lucioduran.com/topics/llm-evaluation#project-agent-context-budget","content":"Compare full-window input, truncation, selection and extractive compression while measuring evidence loss separately from answer quality.\n\nPython · License: MIT\n\n[Read the project](<https://lucioduran.com/projects/agent-context-budget>) · [Repository](<https://github.com/lucioduran/agent-context-budget>) · [Release v0.1.1](<https://github.com/lucioduran/agent-context-budget/releases/tag/v0.1.1>)"},{"heading":"Architecture and methodology reading","level":2,"url":"https://lucioduran.com/topics/llm-evaluation#articles","content":""},{"heading":"Evaluating agents without self-deception: harnesses, contamination, and cost per success","level":3,"url":"https://lucioduran.com/topics/llm-evaluation#article-agent-evaluation-harness-contamination-cost-per-success","content":"How to evaluate long-running agents through verifiable state, procedural checks, repeated trials, comparable infrastructure, and costs that include failures."}],"links":[{"text":"Lucio Durán","url":"https://lucioduran.com/"},{"text":"Model Regression Gate","url":"https://lucioduran.com/projects/model-regression-gate"},{"text":"Read the project","url":"https://lucioduran.com/projects/model-regression-gate"},{"text":"Repository","url":"https://github.com/lucioduran/model-regression-gate"},{"text":"Release v0.1.0","url":"https://github.com/lucioduran/model-regression-gate/releases/tag/v0.1.0"},{"text":"Agent Injection Evals","url":"https://lucioduran.com/projects/agent-injection-evals"},{"text":"Read the project","url":"https://lucioduran.com/projects/agent-injection-evals"},{"text":"Repository","url":"https://github.com/lucioduran/agent-injection-evals"},{"text":"Structured Output Contracts","url":"https://lucioduran.com/projects/structured-output-contracts"},{"text":"Read the project","url":"https://lucioduran.com/projects/structured-output-contracts"},{"text":"Repository","url":"https://github.com/lucioduran/structured-output-contracts"},{"text":"Release v0.1.0","url":"https://github.com/lucioduran/structured-output-contracts/releases/tag/v0.1.0"},{"text":"Citation Grounding Checker","url":"https://lucioduran.com/projects/citation-grounding-checker"},{"text":"Read the project","url":"https://lucioduran.com/projects/citation-grounding-checker"},{"text":"Repository","url":"https://github.com/lucioduran/citation-grounding-checker"},{"text":"Release v0.1.0","url":"https://github.com/lucioduran/citation-grounding-checker/releases/tag/v0.1.0"},{"text":"AI Model Routing Policy","url":"https://lucioduran.com/projects/ai-model-routing-policy"},{"text":"Read the project","url":"https://lucioduran.com/projects/ai-model-routing-policy"},{"text":"Repository","url":"https://github.com/lucioduran/ai-model-routing-policy"},{"text":"Release v0.1.0","url":"https://github.com/lucioduran/ai-model-routing-policy/releases/tag/v0.1.0"},{"text":"Agent Context Budget","url":"https://lucioduran.com/projects/agent-context-budget"},{"text":"Read the project","url":"https://lucioduran.com/projects/agent-context-budget"},{"text":"Repository","url":"https://github.com/lucioduran/agent-context-budget"},{"text":"Release v0.1.1","url":"https://github.com/lucioduran/agent-context-budget/releases/tag/v0.1.1"},{"text":"Evaluating agents without self-deception: harnesses, contamination, and cost per success","url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success"}],"structuredData":[{"@context":"https://schema.org","@graph":[{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com/"},{"@type":"WebSite","@id":"https://lucioduran.com/#website","url":"https://lucioduran.com/","name":"Lucio Durán","inLanguage":["en","es"],"publisher":{"@id":"https://lucioduran.com/#person"}},{"@type":"CollectionPage","@id":"https://lucioduran.com/topics/llm-evaluation#page","url":"https://lucioduran.com/topics/llm-evaluation","name":"LLM and AI agent evaluation","description":"Reproducible model and agent evaluation: regressions, citations, structured output and prompt injection. Source code and results by Lucio Durán.","inLanguage":"en","author":{"@id":"https://lucioduran.com/#person"},"publisher":{"@id":"https://lucioduran.com/#person"},"isPartOf":{"@id":"https://lucioduran.com/#website"},"isAccessibleForFree":true,"datePublished":"2026-11-10","dateModified":"2026-11-10","mainEntity":{"@type":"ItemList","numberOfItems":6,"itemListElement":[{"@type":"ListItem","position":1,"name":"Model Regression Gate","url":"https://lucioduran.com/projects/model-regression-gate"},{"@type":"ListItem","position":2,"name":"Agent Injection Evals","url":"https://lucioduran.com/projects/agent-injection-evals"},{"@type":"ListItem","position":3,"name":"Structured Output Contracts","url":"https://lucioduran.com/projects/structured-output-contracts"},{"@type":"ListItem","position":4,"name":"Citation Grounding Checker","url":"https://lucioduran.com/projects/citation-grounding-checker"},{"@type":"ListItem","position":5,"name":"AI Model Routing Policy","url":"https://lucioduran.com/projects/ai-model-routing-policy"},{"@type":"ListItem","position":6,"name":"Agent Context Budget","url":"https://lucioduran.com/projects/agent-context-budget"}]}}]}],"contentHash":"c10f780e08649a64d0c781950fe4208cd316303b3322d7d559654b59bd06f37a","provenance":{"method":"Deterministic serialization of the server-rendered main element","source":"https://lucioduran.com/topics/llm-evaluation","hashAlgorithm":"sha256","hashScope":"JSON document fields before contentHash and provenance","sourceHTMLBytes":41904,"contentBytes":4720}}