LLM and AI agent evaluation
Evaluation connects an engineering decision to observable evidence. These projects separate dimensions that are often combined: an answer can be valid JSON yet incorrect, include an unsupported citation, or complete a task while violating an execution boundary.
Which decision are you making?
When deciding a model change
Compare both candidates on versioned tasks and inspect segments that regress. Model Regression Gate retains paired results and distinguishes acceptance, rejection and review.
When evaluating answers
Measure content as well as format. Structured Output Contracts contrasts validity with accuracy; Citation Grounding Checker links each claim to the passage that might support it.
When evaluating agents
Record observable effects and failures as well as the final answer. Agent Injection Evals separates capability, attack attempts and realized effects; Agent Context Budget studies evidence loss and answer quality.
Projects and evidence
Each project page provides code, environment details, published results and limitations. The experiments describe those configurations; they do not establish results in client systems.
Model Regression Gate
A reproducible model promotion gate with versioned tasks, paired accuracy comparisons, slice tolerances and a separate review-required outcome.
Agent Injection Evals
Reproducible prompt-injection evaluation with a pinned local model, simulated tools and separate metrics for task capability, attack attempts and realized effects.
Structured Output Contracts
Bounded JSON contracts with strict validation, explicit refusals and limited retries. Separates validity from correctness using two local Qwen models and preserved raw responses.
Citation Grounding Checker
Citation support checking with rules, FLAN-T5 and DeBERTa NLI. Predictions, abstentions and precision linked to each claim’s exact source passage.
AI Model Routing Policy
Routing between FLAN-T5-small and base with development-only fitting, held-out evaluation, measured router cost and injected local outages. Includes its negative result.
Agent Context Budget
Compare full-window input, truncation, selection and extractive compression while measuring evidence loss separately from answer quality.
Architecture and methodology reading
Evaluating agents without self-deception: harnesses, contamination, and cost per success
How to evaluate long-running agents through verifiable state, procedural checks, repeated trials, comparable infrastructure, and costs that include failures.