Agentic AI # Evaluating agents without self-deception: harnesses, contamination, and cost per success 2026-10-217 min read Lucio Durán Engineering Manager & AI Solutions Architect Also available in: [Español]() ## The demo has a statistical distribution of one round of applause An agent completed a complex task. Excellent. Now repeat it from the same initial state, change a reasonable detail, interrupt a tool, and measure recovery cost. That is where useful information for an operational decision begins. The video has already done its commercial job. **Evaluating an agent means measuring a system acting on an environment under a defined success contract and budget.** The model is one variable. Tools, memory, permissions, infrastructure, and completion logic are variables too. I propose a protocol for an agent preparing repository changes. The instructional numbers and formats below are illustrative, not measurements I have performed. ## Define success before reading the response For a task such as “fix a defect and prepare a reviewable change,” I would accept a run only when observable conditions hold: 1. The requested behavior is implemented according to an independent evaluator. 2. Selected regression checks still pass. 3. Changes stay within authorized scope. 4. No prohibited external effects or secret exposure occur. 5. Execution respects the specified budget and deadline. The evaluator inspects final state and tool events. “Everything works” is a claim to check. Using that claim to approve the run hires the candidate to grade their own exam. The March 2026 preprint [Beyond Task Completion]() studies apparently successful outcomes that violate procedural requirements. This provides a concrete reason to record how a result was obtained as well as its final state. In a custom rubric, a critical criterion acts as an acceptance gate. Good documentation does not offset a data leak. Partial scores remain useful diagnostically even when a run is rejected. ## Version the whole experiment This conceptual manifest shows information I would require to reproduce a comparison. Replace ` REQUIRED ` values with real identifiers: ```yaml experiment: dataset_revision: REQUIRED task_split: held_out model_snapshot: REQUIRED harness_commit: REQUIRED system_prompt_sha256: REQUIRED tool_contract_revision: REQUIRED grader_commit: REQUIRED environment: image_digest: REQUIRED initial_repository_commit: REQUIRED cpu_limit: REQUIRED memory_limit_mb: REQUIRED network_policy: REQUIRED dependency_cache_policy: REQUIRED execution: trials_per_task: REQUIRED deadline_seconds: REQUIRED max_total_tokens: REQUIRED max_tool_calls: REQUIRED max_cost_usd: REQUIRED ``` Each trial starts from clean state. A solution left on the filesystem, memory written by the previous attempt, or a cache containing answers can change the task. External effects are isolated per run or replaced with fixtures whose behavior is specified. If the provider cannot pin an exact model version, document the available identifier and time window. Reproducing the environment reduces ambiguity; it does not make a remote service deterministic. ## The harness is taking the exam too In February 2026, Anthropic reported a six-percentage-point difference on Terminal-Bench 2.0 between infrastructure configurations in its experiment. That result belongs to that study, not to every agent. [Primary source](). To compare designs, I would hold resources constant and block executions by task and time period. Alternating candidate order reduces the effect of always running one during more favorable conditions. I would record saturation, provider latency, and environment errors. There are two valid questions: how well the complete system performs operationally, and what a component contributes under controlled conditions. Publish both with their denominators. Users experience infrastructure failures too, however inconvenient that may be for the model team. Exclusions must be defined before the experiment and reported. Removing the favored candidate's timeouts afterward creates a wonderful metric for a rather selective reality. ## Capability and consistency need different numbers [Tau-bench]() introduced ` pass^k ` to study consistency across trials. It should be distinguished from ` pass@k `, which measures the possibility of obtaining at least one success. With ` n ` exchangeable trials of one task, ` c ` successes, and ` k <= n `, the combinatorial estimators are: ```text pass@k = 1 - C(n - c, k) / C(n, k) pass^k = C(c, k) / C(n, k) C(a, k) = 0 when a < k ``` Estimate per task and then aggregate. Do not pool successes from easy and difficult tasks before applying the formula. As a purely mathematical example, if each independent attempt had success probability ` p = 0.8 `, four attempts would yield ` pass@4 = 0.9984 `, while ` pass^4 = 0.4096 `. The same behavior supports a triumphant slide and a fairly nervous operation. Those population formulas assume independent attempts with constant probability. Real executions can share a tool failure or model limitation. I would report per-task rates and intervals from resampling tasks; dependencies by repository or incident require preserving those groups in the sampling scheme. ` Pass@k ` also assumes the correct solution can be recognized among attempts. In production, that selector has costs and errors of its own. The benchmark oracle is not included in the API. ## Contamination can now happen during execution In March 2026, Anthropic documented BrowseComp cases where an agent identified the benchmark and located its answers. The issue extends beyond training exposure. Tools can retrieve the solved exam too. [Original analysis](). For research with browsing, I would separate public tasks from held-out cases and record sources supporting each answer. A correct answer obtained from a published solution should be marked contaminated for the capability being measured. A task explicitly about finding existing answers would have a different contract. For coding, I would keep held-out tests outside the agent's environment and record attempts to modify the evaluator. A task dependent on public documentation needs access to that documentation; blocking the entire network can prevent leakage while changing the problem being evaluated. The report should explain that tradeoff. Labeling a dataset private does not replace investigating what the system could actually see. ## Judges with bounded responsibilities I would use code for invariants and verifiable outcomes, a semantic rubric for open-ended questions, and human review to calibrate disagreements. [Anthropic's agent evaluation guide]() likewise distinguishes code-based, model-based, and human graders. The semantic judge receives evidence and explicit criteria, with an option to answer “undetermined.” I would measure false positives and false negatives against reviewed cases, including long, persuasive, incorrect responses. Agent evidence is untrusted judge input. Instructions inside a file or response must not change the rubric. Versioning the judging prompt and model makes measurement changes visible even when the agent itself is unchanged. ## Cost per success includes the defeats I define an accepted execution as one satisfying the complete contract. For a fixed workload: ```text cost_per_accepted_execution = total_cost_of_all_executions / accepted_executions ``` The numerator includes models, tools, infrastructure, and retries. Human work is reported in minutes and, if monetized, at a declared rate. With zero successes, the ratio offers no useful finite cost. I would publish this metric alongside acceptance rate, p50/p95 latency, time to abandonment, and interventions. Results should also be broken down by difficulty: accepting only easy tasks can improve aggregate cost while reducing product utility. Abstention is correct when the contract requires it for insufficient evidence or a prohibited action. When the task should have been solved, abstention is an unresolved task. Coverage must be reported so this distinction does not become an accounting escape hatch. ## An adoption criterion that can fail Before running anything, I would specify maximum acceptable cost, minimum success, and critical conditions blocking release. I would compare against the current system, retain raw results, and explain which improvements persist when memory, delegation, or additional retrieval is removed. The result may be that the new design does not justify its complexity. That is what the experiment is for. If every evaluation validates the architecture chosen in advance, you may be measuring the team's narrative ability. **An agent is ready for more autonomy when its performance, limits, and costs withstand repeatable evaluation on relevant tasks.** The demo shows a possibility. An operational decision needs a distribution. Apply this evaluation to the [recursive agent design]() and the [tool execution protocol](). ## Frequently asked questions ### What is the difference between pass@k and pass^k? Pass@k asks whether at least one of k attempts succeeds. Pass^k asks whether all of them succeed. They measure capability with multiple attempts and consistency, respectively, and are not interchangeable. ### Is another LLM sufficient as a judge? Not for every criterion. State changes and checkable constraints should be evaluated with code; semantic judgment needs a calibrated rubric and disagreement review. ### Which economic metric is useful for an agent? Total cost of the evaluated workload divided by accepted executions, accompanied by acceptance rate, latency, and interventions. Include retries, tools, infrastructure, and failures. ## Tips 1. Define success, budget, and exclusion rules before running. After seeing the score, methodology becomes suspiciously creative. 2. Retain final state, tool events, and grader version. The agent's final response is not an execution receipt. 3. Include failed attempts in the cost. Counting only the winners manufactures an efficiency nobody will actually pay for. - agentic-ai - agent-evaluation - evals - agent-harness - benchmark-contamination - reliability - cost-per-success [Back to Blog]()