{"schemaVersion":1,"id":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success","url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success","title":"Evaluating agents without self-deception: harnesses, contamination, and cost per success","description":"How to evaluate long-running agents through verifiable state, procedural checks, repeated trials, comparable infrastructure, and costs that include failures.","language":"en","datePublished":"2026-10-21","dateModified":null,"contentFormat":"text/markdown","content":"Agentic AI\n\n# Evaluating agents without self-deception: harnesses, contamination, and cost per success\n\n2026-10-217 min read\n\nLucio Durán\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/agent-evaluation-harness-contamination-cost-per-success>)\n\n## The demo has a statistical distribution of one round of applause\n\nAn agent completed a complex task. Excellent. Now repeat it from the same initial state, change a reasonable detail, interrupt a tool, and measure recovery cost.\n\nThat is where useful information for an operational decision begins. The video has already done its commercial job.\n\n**Evaluating an agent means measuring a system acting on an environment under a defined success contract and budget.** The model is one variable. Tools, memory, permissions, infrastructure, and completion logic are variables too.\n\nI propose a protocol for an agent preparing repository changes. The instructional numbers and formats below are illustrative, not measurements I have performed.\n\n## Define success before reading the response\n\nFor a task such as “fix a defect and prepare a reviewable change,” I would accept a run only when observable conditions hold:\n\n1. The requested behavior is implemented according to an independent evaluator.\n2. Selected regression checks still pass.\n3. Changes stay within authorized scope.\n4. No prohibited external effects or secret exposure occur.\n5. Execution respects the specified budget and deadline.\n\nThe evaluator inspects final state and tool events. “Everything works” is a claim to check. Using that claim to approve the run hires the candidate to grade their own exam.\n\nThe March 2026 preprint [Beyond Task Completion](<https://arxiv.org/abs/2603.03116v1>) studies apparently successful outcomes that violate procedural requirements. This provides a concrete reason to record how a result was obtained as well as its final state.\n\nIn a custom rubric, a critical criterion acts as an acceptance gate. Good documentation does not offset a data leak. Partial scores remain useful diagnostically even when a run is rejected.\n\n## Version the whole experiment\n\nThis conceptual manifest shows information I would require to reproduce a comparison. Replace ` REQUIRED ` values with real identifiers:\n\n```yaml\nexperiment:\n  dataset_revision: REQUIRED\n  task_split: held_out\n  model_snapshot: REQUIRED\n  harness_commit: REQUIRED\n  system_prompt_sha256: REQUIRED\n  tool_contract_revision: REQUIRED\n  grader_commit: REQUIRED\nenvironment:\n  image_digest: REQUIRED\n  initial_repository_commit: REQUIRED\n  cpu_limit: REQUIRED\n  memory_limit_mb: REQUIRED\n  network_policy: REQUIRED\n  dependency_cache_policy: REQUIRED\nexecution:\n  trials_per_task: REQUIRED\n  deadline_seconds: REQUIRED\n  max_total_tokens: REQUIRED\n  max_tool_calls: REQUIRED\n  max_cost_usd: REQUIRED\n```\n\nEach trial starts from clean state. A solution left on the filesystem, memory written by the previous attempt, or a cache containing answers can change the task. External effects are isolated per run or replaced with fixtures whose behavior is specified.\n\nIf the provider cannot pin an exact model version, document the available identifier and time window. Reproducing the environment reduces ambiguity; it does not make a remote service deterministic.\n\n## The harness is taking the exam too\n\nIn February 2026, Anthropic reported a six-percentage-point difference on Terminal-Bench 2.0 between infrastructure configurations in its experiment. That result belongs to that study, not to every agent. [Primary source](<https://www.anthropic.com/research/infrastructure-noise>).\n\nTo compare designs, I would hold resources constant and block executions by task and time period. Alternating candidate order reduces the effect of always running one during more favorable conditions. I would record saturation, provider latency, and environment errors.\n\nThere are two valid questions: how well the complete system performs operationally, and what a component contributes under controlled conditions. Publish both with their denominators. Users experience infrastructure failures too, however inconvenient that may be for the model team.\n\nExclusions must be defined before the experiment and reported. Removing the favored candidate's timeouts afterward creates a wonderful metric for a rather selective reality.\n\n## Capability and consistency need different numbers\n\n[Tau-bench](<https://arxiv.org/abs/2406.12045v1>) introduced ` pass^k ` to study consistency across trials. It should be distinguished from ` pass@k `, which measures the possibility of obtaining at least one success.\n\nWith ` n ` exchangeable trials of one task, ` c ` successes, and ` k <= n `, the combinatorial estimators are:\n\n```text\npass@k = 1 - C(n - c, k) / C(n, k)\npass^k =     C(c, k)     / C(n, k)\n\nC(a, k) = 0 when a < k\n```\n\nEstimate per task and then aggregate. Do not pool successes from easy and difficult tasks before applying the formula.\n\nAs a purely mathematical example, if each independent attempt had success probability ` p = 0.8 `, four attempts would yield ` pass@4 = 0.9984 `, while ` pass^4 = 0.4096 `. The same behavior supports a triumphant slide and a fairly nervous operation.\n\nThose population formulas assume independent attempts with constant probability. Real executions can share a tool failure or model limitation. I would report per-task rates and intervals from resampling tasks; dependencies by repository or incident require preserving those groups in the sampling scheme.\n\n` Pass@k ` also assumes the correct solution can be recognized among attempts. In production, that selector has costs and errors of its own. The benchmark oracle is not included in the API.\n\n## Contamination can now happen during execution\n\nIn March 2026, Anthropic documented BrowseComp cases where an agent identified the benchmark and located its answers. The issue extends beyond training exposure. Tools can retrieve the solved exam too. [Original analysis](<https://www.anthropic.com/research/eval-awareness-browsecomp>).\n\nFor research with browsing, I would separate public tasks from held-out cases and record sources supporting each answer. A correct answer obtained from a published solution should be marked contaminated for the capability being measured. A task explicitly about finding existing answers would have a different contract.\n\nFor coding, I would keep held-out tests outside the agent's environment and record attempts to modify the evaluator. A task dependent on public documentation needs access to that documentation; blocking the entire network can prevent leakage while changing the problem being evaluated.\n\nThe report should explain that tradeoff. Labeling a dataset private does not replace investigating what the system could actually see.\n\n## Judges with bounded responsibilities\n\nI would use code for invariants and verifiable outcomes, a semantic rubric for open-ended questions, and human review to calibrate disagreements. [Anthropic's agent evaluation guide](<https://www.anthropic.com/research/demystifying-evals-for-ai-agents>) likewise distinguishes code-based, model-based, and human graders.\n\nThe semantic judge receives evidence and explicit criteria, with an option to answer “undetermined.” I would measure false positives and false negatives against reviewed cases, including long, persuasive, incorrect responses.\n\nAgent evidence is untrusted judge input. Instructions inside a file or response must not change the rubric. Versioning the judging prompt and model makes measurement changes visible even when the agent itself is unchanged.\n\n## Cost per success includes the defeats\n\nI define an accepted execution as one satisfying the complete contract. For a fixed workload:\n\n```text\ncost_per_accepted_execution =\n  total_cost_of_all_executions / accepted_executions\n```\n\nThe numerator includes models, tools, infrastructure, and retries. Human work is reported in minutes and, if monetized, at a declared rate. With zero successes, the ratio offers no useful finite cost.\n\nI would publish this metric alongside acceptance rate, p50/p95 latency, time to abandonment, and interventions. Results should also be broken down by difficulty: accepting only easy tasks can improve aggregate cost while reducing product utility.\n\nAbstention is correct when the contract requires it for insufficient evidence or a prohibited action. When the task should have been solved, abstention is an unresolved task. Coverage must be reported so this distinction does not become an accounting escape hatch.\n\n## An adoption criterion that can fail\n\nBefore running anything, I would specify maximum acceptable cost, minimum success, and critical conditions blocking release. I would compare against the current system, retain raw results, and explain which improvements persist when memory, delegation, or additional retrieval is removed.\n\nThe result may be that the new design does not justify its complexity. That is what the experiment is for. If every evaluation validates the architecture chosen in advance, you may be measuring the team's narrative ability.\n\n**An agent is ready for more autonomy when its performance, limits, and costs withstand repeatable evaluation on relevant tasks.** The demo shows a possibility. An operational decision needs a distribution.\n\nApply this evaluation to the [recursive agent design](<https://lucioduran.com/blog/recursive-agent-harnesses-context-budgets-provenance>) and the [tool execution protocol](<https://lucioduran.com/blog/agent-tool-execution-authorization-idempotency-reconciliation>).\n\n## Frequently asked questions\n\n### What is the difference between pass@k and pass^k?\n\nPass@k asks whether at least one of k attempts succeeds. Pass^k asks whether all of them succeed. They measure capability with multiple attempts and consistency, respectively, and are not interchangeable.\n\n### Is another LLM sufficient as a judge?\n\nNot for every criterion. State changes and checkable constraints should be evaluated with code; semantic judgment needs a calibrated rubric and disagreement review.\n\n### Which economic metric is useful for an agent?\n\nTotal cost of the evaluated workload divided by accepted executions, accompanied by acceptance rate, latency, and interventions. Include retries, tools, infrastructure, and failures.\n\n## Tips\n\n1. Define success, budget, and exclusion rules before running. After seeing the score, methodology becomes suspiciously creative.\n2. Retain final state, tool events, and grader version. The agent's final response is not an execution receipt.\n3. Include failed attempts in the cost. Counting only the winners manufactures an efficiency nobody will actually pay for.\n\n- agentic-ai\n- agent-evaluation\n- evals\n- agent-harness\n- benchmark-contamination\n- reliability\n- cost-per-success\n\n[Back to Blog](<https://lucioduran.com/blog>)","headings":[{"level":1,"text":"Evaluating agents without self-deception: harnesses, contamination, and cost per success","url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#title"},{"level":2,"text":"The demo has a statistical distribution of one round of applause","url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#the-demo-has-a-statistical-distribution-of-one-round-of-applause"},{"level":2,"text":"Define success before reading the response","url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#define-success-before-reading-the-response"},{"level":2,"text":"Version the whole experiment","url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#version-the-whole-experiment"},{"level":2,"text":"The harness is taking the exam too","url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#the-harness-is-taking-the-exam-too"},{"level":2,"text":"Capability and consistency need different numbers","url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#capability-and-consistency-need-different-numbers"},{"level":2,"text":"Contamination can now happen during execution","url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#contamination-can-now-happen-during-execution"},{"level":2,"text":"Judges with bounded responsibilities","url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#judges-with-bounded-responsibilities"},{"level":2,"text":"Cost per success includes the defeats","url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#cost-per-success-includes-the-defeats"},{"level":2,"text":"An adoption criterion that can fail","url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#an-adoption-criterion-that-can-fail"},{"level":2,"text":"Frequently asked questions","url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#questions"},{"level":3,"text":"What is the difference between pass@k and pass^k?","url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#question-1"},{"level":3,"text":"Is another LLM sufficient as a judge?","url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#question-2"},{"level":3,"text":"Which economic metric is useful for an agent?","url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#question-3"},{"level":2,"text":"Tips","url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#article-tips"}],"sections":[{"heading":"Evaluating agents without self-deception: harnesses, contamination, and cost per success","level":1,"url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success","content":"Agentic AI"},{"heading":"Evaluating agents without self-deception: harnesses, contamination, and cost per success","level":1,"url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#title","content":"2026-10-217 min read\n\nLucio Durán\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/agent-evaluation-harness-contamination-cost-per-success>)"},{"heading":"The demo has a statistical distribution of one round of applause","level":2,"url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#the-demo-has-a-statistical-distribution-of-one-round-of-applause","content":"An agent completed a complex task. Excellent. Now repeat it from the same initial state, change a reasonable detail, interrupt a tool, and measure recovery cost.\n\nThat is where useful information for an operational decision begins. The video has already done its commercial job.\n\n**Evaluating an agent means measuring a system acting on an environment under a defined success contract and budget.** The model is one variable. Tools, memory, permissions, infrastructure, and completion logic are variables too.\n\nI propose a protocol for an agent preparing repository changes. The instructional numbers and formats below are illustrative, not measurements I have performed."},{"heading":"Define success before reading the response","level":2,"url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#define-success-before-reading-the-response","content":"For a task such as “fix a defect and prepare a reviewable change,” I would accept a run only when observable conditions hold:\n\n1. The requested behavior is implemented according to an independent evaluator.\n2. Selected regression checks still pass.\n3. Changes stay within authorized scope.\n4. No prohibited external effects or secret exposure occur.\n5. Execution respects the specified budget and deadline.\n\nThe evaluator inspects final state and tool events. “Everything works” is a claim to check. Using that claim to approve the run hires the candidate to grade their own exam.\n\nThe March 2026 preprint [Beyond Task Completion](<https://arxiv.org/abs/2603.03116v1>) studies apparently successful outcomes that violate procedural requirements. This provides a concrete reason to record how a result was obtained as well as its final state.\n\nIn a custom rubric, a critical criterion acts as an acceptance gate. Good documentation does not offset a data leak. Partial scores remain useful diagnostically even when a run is rejected."},{"heading":"Version the whole experiment","level":2,"url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#version-the-whole-experiment","content":"This conceptual manifest shows information I would require to reproduce a comparison. Replace ` REQUIRED ` values with real identifiers:\n\n```yaml\nexperiment:\n  dataset_revision: REQUIRED\n  task_split: held_out\n  model_snapshot: REQUIRED\n  harness_commit: REQUIRED\n  system_prompt_sha256: REQUIRED\n  tool_contract_revision: REQUIRED\n  grader_commit: REQUIRED\nenvironment:\n  image_digest: REQUIRED\n  initial_repository_commit: REQUIRED\n  cpu_limit: REQUIRED\n  memory_limit_mb: REQUIRED\n  network_policy: REQUIRED\n  dependency_cache_policy: REQUIRED\nexecution:\n  trials_per_task: REQUIRED\n  deadline_seconds: REQUIRED\n  max_total_tokens: REQUIRED\n  max_tool_calls: REQUIRED\n  max_cost_usd: REQUIRED\n```\n\nEach trial starts from clean state. A solution left on the filesystem, memory written by the previous attempt, or a cache containing answers can change the task. External effects are isolated per run or replaced with fixtures whose behavior is specified.\n\nIf the provider cannot pin an exact model version, document the available identifier and time window. Reproducing the environment reduces ambiguity; it does not make a remote service deterministic."},{"heading":"The harness is taking the exam too","level":2,"url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#the-harness-is-taking-the-exam-too","content":"In February 2026, Anthropic reported a six-percentage-point difference on Terminal-Bench 2.0 between infrastructure configurations in its experiment. That result belongs to that study, not to every agent. [Primary source](<https://www.anthropic.com/research/infrastructure-noise>).\n\nTo compare designs, I would hold resources constant and block executions by task and time period. Alternating candidate order reduces the effect of always running one during more favorable conditions. I would record saturation, provider latency, and environment errors.\n\nThere are two valid questions: how well the complete system performs operationally, and what a component contributes under controlled conditions. Publish both with their denominators. Users experience infrastructure failures too, however inconvenient that may be for the model team.\n\nExclusions must be defined before the experiment and reported. Removing the favored candidate's timeouts afterward creates a wonderful metric for a rather selective reality."},{"heading":"Capability and consistency need different numbers","level":2,"url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#capability-and-consistency-need-different-numbers","content":"[Tau-bench](<https://arxiv.org/abs/2406.12045v1>) introduced ` pass^k ` to study consistency across trials. It should be distinguished from ` pass@k `, which measures the possibility of obtaining at least one success.\n\nWith ` n ` exchangeable trials of one task, ` c ` successes, and ` k <= n `, the combinatorial estimators are:\n\n```text\npass@k = 1 - C(n - c, k) / C(n, k)\npass^k =     C(c, k)     / C(n, k)\n\nC(a, k) = 0 when a < k\n```\n\nEstimate per task and then aggregate. Do not pool successes from easy and difficult tasks before applying the formula.\n\nAs a purely mathematical example, if each independent attempt had success probability ` p = 0.8 `, four attempts would yield ` pass@4 = 0.9984 `, while ` pass^4 = 0.4096 `. The same behavior supports a triumphant slide and a fairly nervous operation.\n\nThose population formulas assume independent attempts with constant probability. Real executions can share a tool failure or model limitation. I would report per-task rates and intervals from resampling tasks; dependencies by repository or incident require preserving those groups in the sampling scheme.\n\n` Pass@k ` also assumes the correct solution can be recognized among attempts. In production, that selector has costs and errors of its own. The benchmark oracle is not included in the API."},{"heading":"Contamination can now happen during execution","level":2,"url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#contamination-can-now-happen-during-execution","content":"In March 2026, Anthropic documented BrowseComp cases where an agent identified the benchmark and located its answers. The issue extends beyond training exposure. Tools can retrieve the solved exam too. [Original analysis](<https://www.anthropic.com/research/eval-awareness-browsecomp>).\n\nFor research with browsing, I would separate public tasks from held-out cases and record sources supporting each answer. A correct answer obtained from a published solution should be marked contaminated for the capability being measured. A task explicitly about finding existing answers would have a different contract.\n\nFor coding, I would keep held-out tests outside the agent's environment and record attempts to modify the evaluator. A task dependent on public documentation needs access to that documentation; blocking the entire network can prevent leakage while changing the problem being evaluated.\n\nThe report should explain that tradeoff. Labeling a dataset private does not replace investigating what the system could actually see."},{"heading":"Judges with bounded responsibilities","level":2,"url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#judges-with-bounded-responsibilities","content":"I would use code for invariants and verifiable outcomes, a semantic rubric for open-ended questions, and human review to calibrate disagreements. [Anthropic's agent evaluation guide](<https://www.anthropic.com/research/demystifying-evals-for-ai-agents>) likewise distinguishes code-based, model-based, and human graders.\n\nThe semantic judge receives evidence and explicit criteria, with an option to answer “undetermined.” I would measure false positives and false negatives against reviewed cases, including long, persuasive, incorrect responses.\n\nAgent evidence is untrusted judge input. Instructions inside a file or response must not change the rubric. Versioning the judging prompt and model makes measurement changes visible even when the agent itself is unchanged."},{"heading":"Cost per success includes the defeats","level":2,"url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#cost-per-success-includes-the-defeats","content":"I define an accepted execution as one satisfying the complete contract. For a fixed workload:\n\n```text\ncost_per_accepted_execution =\n  total_cost_of_all_executions / accepted_executions\n```\n\nThe numerator includes models, tools, infrastructure, and retries. Human work is reported in minutes and, if monetized, at a declared rate. With zero successes, the ratio offers no useful finite cost.\n\nI would publish this metric alongside acceptance rate, p50/p95 latency, time to abandonment, and interventions. Results should also be broken down by difficulty: accepting only easy tasks can improve aggregate cost while reducing product utility.\n\nAbstention is correct when the contract requires it for insufficient evidence or a prohibited action. When the task should have been solved, abstention is an unresolved task. Coverage must be reported so this distinction does not become an accounting escape hatch."},{"heading":"An adoption criterion that can fail","level":2,"url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#an-adoption-criterion-that-can-fail","content":"Before running anything, I would specify maximum acceptable cost, minimum success, and critical conditions blocking release. I would compare against the current system, retain raw results, and explain which improvements persist when memory, delegation, or additional retrieval is removed.\n\nThe result may be that the new design does not justify its complexity. That is what the experiment is for. If every evaluation validates the architecture chosen in advance, you may be measuring the team's narrative ability.\n\n**An agent is ready for more autonomy when its performance, limits, and costs withstand repeatable evaluation on relevant tasks.** The demo shows a possibility. An operational decision needs a distribution.\n\nApply this evaluation to the [recursive agent design](<https://lucioduran.com/blog/recursive-agent-harnesses-context-budgets-provenance>) and the [tool execution protocol](<https://lucioduran.com/blog/agent-tool-execution-authorization-idempotency-reconciliation>)."},{"heading":"Frequently asked questions","level":2,"url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#questions","content":""},{"heading":"What is the difference between pass@k and pass^k?","level":3,"url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#question-1","content":"Pass@k asks whether at least one of k attempts succeeds. Pass^k asks whether all of them succeed. They measure capability with multiple attempts and consistency, respectively, and are not interchangeable."},{"heading":"Is another LLM sufficient as a judge?","level":3,"url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#question-2","content":"Not for every criterion. State changes and checkable constraints should be evaluated with code; semantic judgment needs a calibrated rubric and disagreement review."},{"heading":"Which economic metric is useful for an agent?","level":3,"url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#question-3","content":"Total cost of the evaluated workload divided by accepted executions, accompanied by acceptance rate, latency, and interventions. Include retries, tools, infrastructure, and failures."},{"heading":"Tips","level":2,"url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#article-tips","content":"1. Define success, budget, and exclusion rules before running. After seeing the score, methodology becomes suspiciously creative.\n2. Retain final state, tool events, and grader version. The agent's final response is not an execution receipt.\n3. Include failed attempts in the cost. Counting only the winners manufactures an efficiency nobody will actually pay for.\n\n- agentic-ai\n- agent-evaluation\n- evals\n- agent-harness\n- benchmark-contamination\n- reliability\n- cost-per-success\n\n[Back to Blog](<https://lucioduran.com/blog>)"}],"links":[{"text":"Español","url":"https://lucioduran.com/es/blog/agent-evaluation-harness-contamination-cost-per-success"},{"text":"Beyond Task Completion","url":"https://arxiv.org/abs/2603.03116v1"},{"text":"Primary source","url":"https://www.anthropic.com/research/infrastructure-noise"},{"text":"Tau-bench","url":"https://arxiv.org/abs/2406.12045v1"},{"text":"Original analysis","url":"https://www.anthropic.com/research/eval-awareness-browsecomp"},{"text":"Anthropic's agent evaluation guide","url":"https://www.anthropic.com/research/demystifying-evals-for-ai-agents"},{"text":"recursive agent design","url":"https://lucioduran.com/blog/recursive-agent-harnesses-context-budgets-provenance"},{"text":"tool execution protocol","url":"https://lucioduran.com/blog/agent-tool-execution-authorization-idempotency-reconciliation"},{"text":"Back to Blog","url":"https://lucioduran.com/blog"}],"structuredData":[{"@context":"https://schema.org","@graph":[{"@type":"BlogPosting","@id":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success#article","mainEntityOfPage":{"@type":"WebPage","@id":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success"},"headline":"Evaluating agents without self-deception: harnesses, contamination, and cost per success","description":"How to evaluate long-running agents through verifiable state, procedural checks, repeated trials, comparable infrastructure, and costs that include failures.","datePublished":"2026-10-21","author":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"publisher":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"image":"https://statics.lucioduran.com/images/lucioduran-social-card.jpg","url":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success","inLanguage":"en","isPartOf":{"@id":"https://lucioduran.com/blog#blog"},"articleSection":"Agentic AI","keywords":"agentic-ai, agent-evaluation, evals, agent-harness, benchmark-contamination, reliability, cost-per-success","wordCount":1247,"timeRequired":"PT7M","workTranslation":[{"@type":"BlogPosting","inLanguage":"es","url":"https://lucioduran.com/es/blog/agent-evaluation-harness-contamination-cost-per-success"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://lucioduran.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://lucioduran.com/blog"},{"@type":"ListItem","position":3,"name":"Evaluating agents without self-deception: harnesses, contamination, and cost per success","item":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success"}]},{"@type":"FAQPage","mainEntity":[{"@type":"Question","name":"What is the difference between pass@k and pass^k?","acceptedAnswer":{"@type":"Answer","text":"Pass@k asks whether at least one of k attempts succeeds. Pass^k asks whether all of them succeed. They measure capability with multiple attempts and consistency, respectively, and are not interchangeable."}},{"@type":"Question","name":"Is another LLM sufficient as a judge?","acceptedAnswer":{"@type":"Answer","text":"Not for every criterion. State changes and checkable constraints should be evaluated with code; semantic judgment needs a calibrated rubric and disagreement review."}},{"@type":"Question","name":"Which economic metric is useful for an agent?","acceptedAnswer":{"@type":"Answer","text":"Total cost of the evaluated workload divided by accepted executions, accompanied by acceptance rate, latency, and interventions. Include retries, tools, infrastructure, and failures."}}]}]}],"contentHash":"02b8755469e811bee9ea52b91332b27dc2aa27c8f4a43e7b7d8902c13a2e85ab","provenance":{"method":"Deterministic serialization of the server-rendered main element","source":"https://lucioduran.com/blog/agent-evaluation-harness-contamination-cost-per-success","hashAlgorithm":"sha256","hashScope":"JSON document fields before contentHash and provenance","sourceHTMLBytes":60381,"contentBytes":10890}}