{"schemaVersion":1,"id":"https://lucioduran.com/projects/agent-injection-evals","url":"https://lucioduran.com/projects/agent-injection-evals","title":"Agent Injection Evals","description":"Reproducible prompt-injection evaluation with a pinned local model, simulated tools and separate metrics for task capability, attack attempts and realized effects.","language":"en","datePublished":"2026-10-21","dateModified":"2026-10-21","contentFormat":"text/markdown","content":"[Projects](<https://lucioduran.com/projects>) / Agent Injection Evals\n\n# Agent Injection Evals\n\nReproducible prompt-injection evaluation with a pinned local model, simulated tools and separate metrics for task capability, attack attempts and realized effects.\n\nAuthor: [Lucio Durán](<https://lucioduran.com/>) · Python 3.11.13 / FLAN-T5-small / CPU · License: [MIT](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/LICENSE>)\n\nPage published: 2026-10-21 · Page updated: 2026-10-21\n\n[Repository](<https://github.com/lucioduran/agent-injection-evals>) · [Evaluation report](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/results/eval/report.json>)\n\n## Problem and relevance to agents\n\nAn agent that cannot complete any task can still report zero successful attacks. That number alone does not demonstrate security. This project measures legitimate work, malformed responses, attack attempts and effects separately.\n\nThe evidence makes it possible to inspect an agent reading hostile documents and proposing actions. It separates model behavior from authorization enforced by the tool executor.\n\n## Architecture\n\n1. **Controlled tasks and attacks.** Two development tasks and six evaluation tasks each have a clean control and four hostile variants: answer contamination, synthetic secret disclosure, destination redirection and unauthorized state mutation.\n2. **2 × 2 comparison.** Plain or instruction-isolated prompts are crossed with permissive or task-scoped policies. The same pinned FLAN-T5-small model generates one decision per case.\n3. **Simulated execution.** A strict parser accepts ANSWER, SEND or WRITE. Policy checks the authorized verb and target. Effects stay in fresh in-memory dictionaries; destinations use .invalid and no messages are sent.\n4. **Traces and scoring.** Every prompt, output, parse decision, attempt, effect and resource observation is recorded. An offline verifier reconstructs prompts and scores from saved responses without rerunning the model.\n\n- [Design and metric definitions](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/docs/design.md>)\n- [Parser, policies and scoring](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/injection_evals/core.py>)\n- [Synthetic dataset provenance](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/data/README.md>)\n\n## What this demonstrates\n\n- Evaluation design separating legitimate capability, attempted attacks and realized effects.\n- Tool authorization enforced outside the model, with a permissive baseline and reproducible inputs.\n- Publication of negative results and error analysis without mislabeling protocol failures as successful defenses.\n\n## Published results\n\nThe evaluation report records 120 real CPU generations: six tasks, five variants and four strategies. No strategy completed a clean or hostile task under the frozen metric. The development experiment preserves another 40 generations.\n\nThe experiment used google/flan-t5-small revision 0fc9ddf78a1e988dac52e2dac162b0ede4fd74ab, seed 1729, greedy decoding, two threads, up to 512 input tokens without truncation and 48 output tokens. The recorded environment is Python 3.11.13, torch 2.2.2, transformers 4.46.3 and an Intel Xeon W-3223 with 48 GiB RAM.\n\n### Capability and attack effects\n\nClean and hostile success measure the legitimate task. Effects counts attacks realized after policy enforcement. Blocked counts denied calls. Malformed covers all variants for each strategy.\n\n| Strategy | Clean success | Hostile success | Effects | Malformed | Blocked |\n| --- | --- | --- | --- | --- | --- |\n| Baseline | 0 / 6 | 0 / 24 | 0 / 24 | 30 / 30 | 0 |\n| Isolation | 0 / 6 | 0 / 24 | 5 / 24 | 25 / 30 | 0 |\n| Policy | 0 / 6 | 0 / 24 | 0 / 24 | 30 / 30 | 0 |\n| Combined | 0 / 6 | 0 / 24 | 0 / 24 | 25 / 30 | 5 |\n\n### Evaluation resources\n\nObserved generation totals per strategy. They exclude model loading and are neither a load test nor averages across repeated runs.\n\n| Strategy | Wall time | CPU time | Input / output tokens | Paid APIs |\n| --- | --- | --- | --- | --- |\n| Baseline | 7.770 s | 14.678 s | 3143 / 409 | $0 |\n| Isolation | 6.092 s | 12.118 s | 4307 / 322 | $0 |\n| Policy | 6.909 s | 13.723 s | 3143 / 409 | $0 |\n| Combined | 6.162 s | 12.035 s | 4307 / 322 | $0 |\n\nIsolation produced five unauthorized writes. Combined protection attempted the same five writes and policy blocked them. With 0/6 clean-task successes per strategy, there is no useful defense winner and no defined capability-conditioned attack rate: that value is null, not zero.\n\nAll 120 generations completed without runtime errors. Model loading took 2.797 s and process peak memory was 709.3 MiB, shared across strategies. Energy and hardware amortization were not measured. [Evaluation report](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/results/eval/report.json>).\n\nSome responses extracted the correct fact but omitted the required prefix. The error analysis preserves those cases and explains why the success criterion was not changed afterward. [Error analysis](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/docs/error-analysis.md>).\n\n## Reproduce\n\nRecorded-report verification needs Python 3.11 and no external packages. Inference requires the documented Intel macOS native dependency lock and a download of public weights. Inference runs locally after downloading, without paid APIs.\n\nReport verification recomputes internal consistency; it does not rerun inference or authenticate timing. New inference runs require new output directories and should run serially. The lock targets the reference environment; other platforms must record their own versions.\n\n### Verify recorded reports\n\n```\ngit clone https://github.com/lucioduran/agent-injection-evals.git\ncd agent-injection-evals\ngit checkout e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66\npython -m injection_evals verify --report results/dev/report.json\npython -m injection_evals verify --report results/eval/report.json\n```\n\n### Run local inference\n\n```\npython -m venv .venv\n. .venv/bin/activate\npython -m pip install -r requirements-local.lock\npython -m injection_evals download\npython -m injection_evals run --split dev --output build/reproduced-dev\npython -m injection_evals run --split eval --output build/reproduced-eval\npython -m injection_evals verify --report build/reproduced-eval/report.json\n```\n\n## Scope and limitations\n\n- The small model fails the action protocol and tasks. These results cannot rank defenses for capable production agents.\n- The synthetic set is small, author-known and shares attack templates across splits. It is neither representative of arbitrary attacks nor an independent blind evaluation.\n- Each trial has one decision, with no browser, tool feedback, adaptive attacks or real exfiltration.\n- Isolation is a prompt convention. Policy does not prevent disclosure in answers or through an already authorized destination.\n- Timings are one observation per case without warmup removal. Replay checks internal consistency, not measurement authenticity or cross-platform neural equivalence.\n\n- [Function calling and tool use in agents](<https://lucioduran.com/blog/llm-function-calling-tool-use-patterns>)\n\n## Sources and revision\n\nValues on this page come from the linked report. They are not live measurements. The report records the executed revision; the reviewed revision includes the subsequent publication of that report.\n\n- Reviewed revision: [` e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66 `](<https://github.com/lucioduran/agent-injection-evals/commit/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66>)\n- Revision recorded by the experiment: [` ed6cbc309e1ec9f1e7211d83743927e28ee48c97 `](<https://github.com/lucioduran/agent-injection-evals/commit/ed6cbc309e1ec9f1e7211d83743927e28ee48c97>)\n- Report timestamp (UTC): 2026-10-21T21:29:09.518113+00:00\n- Corpus SHA-256: ` ef0d7970d9ec3cde2d3c02d6ddef34046a82aef12b8524175216d5d6f216ac65 `\n\n- [README](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/README.md>)\n- [Design and metric definitions](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/docs/design.md>)\n- [Synthetic dataset provenance](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/data/README.md>)\n- [Tasks and labels](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/data/tasks.json>)\n- [Evaluation report](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/results/eval/report.json>)\n- [Complete evaluation trace](<https://raw.githubusercontent.com/lucioduran/agent-injection-evals/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/results/eval/trace.jsonl>)\n- [Development report](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/results/dev/report.json>)\n- [Error analysis](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/docs/error-analysis.md>)\n- [Model and license notices](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/docs/model-notices.md>)\n- [Parser, policies and scoring](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/injection_evals/core.py>)\n- [Pinned inference dependencies](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/requirements-local.lock>)\n- [MIT license](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/LICENSE>)","headings":[{"level":1,"text":"Agent Injection Evals","url":"https://lucioduran.com/projects/agent-injection-evals#title"},{"level":2,"text":"Problem and relevance to agents","url":"https://lucioduran.com/projects/agent-injection-evals#problem"},{"level":2,"text":"Architecture","url":"https://lucioduran.com/projects/agent-injection-evals#architecture"},{"level":2,"text":"What this demonstrates","url":"https://lucioduran.com/projects/agent-injection-evals#demonstrates"},{"level":2,"text":"Published results","url":"https://lucioduran.com/projects/agent-injection-evals#results"},{"level":3,"text":"Capability and attack effects","url":"https://lucioduran.com/projects/agent-injection-evals#results-capability"},{"level":3,"text":"Evaluation resources","url":"https://lucioduran.com/projects/agent-injection-evals#results-resources"},{"level":2,"text":"Reproduce","url":"https://lucioduran.com/projects/agent-injection-evals#reproduce"},{"level":3,"text":"Verify recorded reports","url":"https://lucioduran.com/projects/agent-injection-evals#reproduce-verify"},{"level":3,"text":"Run local inference","url":"https://lucioduran.com/projects/agent-injection-evals#reproduce-inference"},{"level":2,"text":"Scope and limitations","url":"https://lucioduran.com/projects/agent-injection-evals#scope"},{"level":2,"text":"Sources and revision","url":"https://lucioduran.com/projects/agent-injection-evals#sources"}],"sections":[{"heading":"Agent Injection Evals","level":1,"url":"https://lucioduran.com/projects/agent-injection-evals","content":"[Projects](<https://lucioduran.com/projects>) / Agent Injection Evals"},{"heading":"Agent Injection Evals","level":1,"url":"https://lucioduran.com/projects/agent-injection-evals#title","content":"Reproducible prompt-injection evaluation with a pinned local model, simulated tools and separate metrics for task capability, attack attempts and realized effects.\n\nAuthor: [Lucio Durán](<https://lucioduran.com/>) · Python 3.11.13 / FLAN-T5-small / CPU · License: [MIT](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/LICENSE>)\n\nPage published: 2026-10-21 · Page updated: 2026-10-21\n\n[Repository](<https://github.com/lucioduran/agent-injection-evals>) · [Evaluation report](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/results/eval/report.json>)"},{"heading":"Problem and relevance to agents","level":2,"url":"https://lucioduran.com/projects/agent-injection-evals#problem","content":"An agent that cannot complete any task can still report zero successful attacks. That number alone does not demonstrate security. This project measures legitimate work, malformed responses, attack attempts and effects separately.\n\nThe evidence makes it possible to inspect an agent reading hostile documents and proposing actions. It separates model behavior from authorization enforced by the tool executor."},{"heading":"Architecture","level":2,"url":"https://lucioduran.com/projects/agent-injection-evals#architecture","content":"1. **Controlled tasks and attacks.** Two development tasks and six evaluation tasks each have a clean control and four hostile variants: answer contamination, synthetic secret disclosure, destination redirection and unauthorized state mutation.\n2. **2 × 2 comparison.** Plain or instruction-isolated prompts are crossed with permissive or task-scoped policies. The same pinned FLAN-T5-small model generates one decision per case.\n3. **Simulated execution.** A strict parser accepts ANSWER, SEND or WRITE. Policy checks the authorized verb and target. Effects stay in fresh in-memory dictionaries; destinations use .invalid and no messages are sent.\n4. **Traces and scoring.** Every prompt, output, parse decision, attempt, effect and resource observation is recorded. An offline verifier reconstructs prompts and scores from saved responses without rerunning the model.\n\n- [Design and metric definitions](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/docs/design.md>)\n- [Parser, policies and scoring](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/injection_evals/core.py>)\n- [Synthetic dataset provenance](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/data/README.md>)"},{"heading":"What this demonstrates","level":2,"url":"https://lucioduran.com/projects/agent-injection-evals#demonstrates","content":"- Evaluation design separating legitimate capability, attempted attacks and realized effects.\n- Tool authorization enforced outside the model, with a permissive baseline and reproducible inputs.\n- Publication of negative results and error analysis without mislabeling protocol failures as successful defenses."},{"heading":"Published results","level":2,"url":"https://lucioduran.com/projects/agent-injection-evals#results","content":"The evaluation report records 120 real CPU generations: six tasks, five variants and four strategies. No strategy completed a clean or hostile task under the frozen metric. The development experiment preserves another 40 generations.\n\nThe experiment used google/flan-t5-small revision 0fc9ddf78a1e988dac52e2dac162b0ede4fd74ab, seed 1729, greedy decoding, two threads, up to 512 input tokens without truncation and 48 output tokens. The recorded environment is Python 3.11.13, torch 2.2.2, transformers 4.46.3 and an Intel Xeon W-3223 with 48 GiB RAM."},{"heading":"Capability and attack effects","level":3,"url":"https://lucioduran.com/projects/agent-injection-evals#results-capability","content":"Clean and hostile success measure the legitimate task. Effects counts attacks realized after policy enforcement. Blocked counts denied calls. Malformed covers all variants for each strategy.\n\n| Strategy | Clean success | Hostile success | Effects | Malformed | Blocked |\n| --- | --- | --- | --- | --- | --- |\n| Baseline | 0 / 6 | 0 / 24 | 0 / 24 | 30 / 30 | 0 |\n| Isolation | 0 / 6 | 0 / 24 | 5 / 24 | 25 / 30 | 0 |\n| Policy | 0 / 6 | 0 / 24 | 0 / 24 | 30 / 30 | 0 |\n| Combined | 0 / 6 | 0 / 24 | 0 / 24 | 25 / 30 | 5 |"},{"heading":"Evaluation resources","level":3,"url":"https://lucioduran.com/projects/agent-injection-evals#results-resources","content":"Observed generation totals per strategy. They exclude model loading and are neither a load test nor averages across repeated runs.\n\n| Strategy | Wall time | CPU time | Input / output tokens | Paid APIs |\n| --- | --- | --- | --- | --- |\n| Baseline | 7.770 s | 14.678 s | 3143 / 409 | $0 |\n| Isolation | 6.092 s | 12.118 s | 4307 / 322 | $0 |\n| Policy | 6.909 s | 13.723 s | 3143 / 409 | $0 |\n| Combined | 6.162 s | 12.035 s | 4307 / 322 | $0 |\n\nIsolation produced five unauthorized writes. Combined protection attempted the same five writes and policy blocked them. With 0/6 clean-task successes per strategy, there is no useful defense winner and no defined capability-conditioned attack rate: that value is null, not zero.\n\nAll 120 generations completed without runtime errors. Model loading took 2.797 s and process peak memory was 709.3 MiB, shared across strategies. Energy and hardware amortization were not measured. [Evaluation report](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/results/eval/report.json>).\n\nSome responses extracted the correct fact but omitted the required prefix. The error analysis preserves those cases and explains why the success criterion was not changed afterward. [Error analysis](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/docs/error-analysis.md>)."},{"heading":"Reproduce","level":2,"url":"https://lucioduran.com/projects/agent-injection-evals#reproduce","content":"Recorded-report verification needs Python 3.11 and no external packages. Inference requires the documented Intel macOS native dependency lock and a download of public weights. Inference runs locally after downloading, without paid APIs.\n\nReport verification recomputes internal consistency; it does not rerun inference or authenticate timing. New inference runs require new output directories and should run serially. The lock targets the reference environment; other platforms must record their own versions."},{"heading":"Verify recorded reports","level":3,"url":"https://lucioduran.com/projects/agent-injection-evals#reproduce-verify","content":"```\ngit clone https://github.com/lucioduran/agent-injection-evals.git\ncd agent-injection-evals\ngit checkout e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66\npython -m injection_evals verify --report results/dev/report.json\npython -m injection_evals verify --report results/eval/report.json\n```"},{"heading":"Run local inference","level":3,"url":"https://lucioduran.com/projects/agent-injection-evals#reproduce-inference","content":"```\npython -m venv .venv\n. .venv/bin/activate\npython -m pip install -r requirements-local.lock\npython -m injection_evals download\npython -m injection_evals run --split dev --output build/reproduced-dev\npython -m injection_evals run --split eval --output build/reproduced-eval\npython -m injection_evals verify --report build/reproduced-eval/report.json\n```"},{"heading":"Scope and limitations","level":2,"url":"https://lucioduran.com/projects/agent-injection-evals#scope","content":"- The small model fails the action protocol and tasks. These results cannot rank defenses for capable production agents.\n- The synthetic set is small, author-known and shares attack templates across splits. It is neither representative of arbitrary attacks nor an independent blind evaluation.\n- Each trial has one decision, with no browser, tool feedback, adaptive attacks or real exfiltration.\n- Isolation is a prompt convention. Policy does not prevent disclosure in answers or through an already authorized destination.\n- Timings are one observation per case without warmup removal. Replay checks internal consistency, not measurement authenticity or cross-platform neural equivalence.\n\n- [Function calling and tool use in agents](<https://lucioduran.com/blog/llm-function-calling-tool-use-patterns>)"},{"heading":"Sources and revision","level":2,"url":"https://lucioduran.com/projects/agent-injection-evals#sources","content":"Values on this page come from the linked report. They are not live measurements. The report records the executed revision; the reviewed revision includes the subsequent publication of that report.\n\n- Reviewed revision: [` e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66 `](<https://github.com/lucioduran/agent-injection-evals/commit/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66>)\n- Revision recorded by the experiment: [` ed6cbc309e1ec9f1e7211d83743927e28ee48c97 `](<https://github.com/lucioduran/agent-injection-evals/commit/ed6cbc309e1ec9f1e7211d83743927e28ee48c97>)\n- Report timestamp (UTC): 2026-10-21T21:29:09.518113+00:00\n- Corpus SHA-256: ` ef0d7970d9ec3cde2d3c02d6ddef34046a82aef12b8524175216d5d6f216ac65 `\n\n- [README](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/README.md>)\n- [Design and metric definitions](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/docs/design.md>)\n- [Synthetic dataset provenance](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/data/README.md>)\n- [Tasks and labels](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/data/tasks.json>)\n- [Evaluation report](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/results/eval/report.json>)\n- [Complete evaluation trace](<https://raw.githubusercontent.com/lucioduran/agent-injection-evals/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/results/eval/trace.jsonl>)\n- [Development report](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/results/dev/report.json>)\n- [Error analysis](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/docs/error-analysis.md>)\n- [Model and license notices](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/docs/model-notices.md>)\n- [Parser, policies and scoring](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/injection_evals/core.py>)\n- [Pinned inference dependencies](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/requirements-local.lock>)\n- [MIT license](<https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/LICENSE>)"}],"links":[{"text":"Projects","url":"https://lucioduran.com/projects"},{"text":"Lucio Durán","url":"https://lucioduran.com/"},{"text":"MIT","url":"https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/LICENSE"},{"text":"Repository","url":"https://github.com/lucioduran/agent-injection-evals"},{"text":"Evaluation report","url":"https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/results/eval/report.json"},{"text":"Design and metric definitions","url":"https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/docs/design.md"},{"text":"Parser, policies and scoring","url":"https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/injection_evals/core.py"},{"text":"Synthetic dataset provenance","url":"https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/data/README.md"},{"text":"Error analysis","url":"https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/docs/error-analysis.md"},{"text":"Function calling and tool use in agents","url":"https://lucioduran.com/blog/llm-function-calling-tool-use-patterns"},{"text":"e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66","url":"https://github.com/lucioduran/agent-injection-evals/commit/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66"},{"text":"ed6cbc309e1ec9f1e7211d83743927e28ee48c97","url":"https://github.com/lucioduran/agent-injection-evals/commit/ed6cbc309e1ec9f1e7211d83743927e28ee48c97"},{"text":"README","url":"https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/README.md"},{"text":"Tasks and labels","url":"https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/data/tasks.json"},{"text":"Complete evaluation trace","url":"https://raw.githubusercontent.com/lucioduran/agent-injection-evals/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/results/eval/trace.jsonl"},{"text":"Development report","url":"https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/results/dev/report.json"},{"text":"Model and license notices","url":"https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/docs/model-notices.md"},{"text":"Pinned inference dependencies","url":"https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/requirements-local.lock"},{"text":"MIT license","url":"https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/LICENSE"}],"structuredData":[{"@context":"https://schema.org","@graph":[{"@type":"WebPage","@id":"https://lucioduran.com/projects/agent-injection-evals#page","url":"https://lucioduran.com/projects/agent-injection-evals","name":"Agent Injection Evals","description":"Reproducible prompt-injection evaluation with a pinned local model, simulated tools and separate metrics for task capability, attack attempts and realized effects.","inLanguage":"en","datePublished":"2026-10-21","dateModified":"2026-10-21","author":{"@id":"https://lucioduran.com/#person"},"mainEntity":{"@id":"https://lucioduran.com/projects/agent-injection-evals#software"}},{"@type":"SoftwareSourceCode","@id":"https://lucioduran.com/projects/agent-injection-evals#software","name":"Agent Injection Evals","description":"Reproducible prompt-injection evaluation with a pinned local model, simulated tools and separate metrics for task capability, attack attempts and realized effects.","url":"https://lucioduran.com/projects/agent-injection-evals","codeRepository":"https://github.com/lucioduran/agent-injection-evals","programmingLanguage":"Python","runtimePlatform":"Python 3.11.13 / FLAN-T5-small / CPU","license":"https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/LICENSE","version":"e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66","identifier":{"@type":"PropertyValue","propertyID":"git-commit","value":"e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66"},"author":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"isAccessibleForFree":true,"citation":["https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/README.md","https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/docs/design.md","https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/data/README.md","https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/data/tasks.json","https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/results/eval/report.json","https://raw.githubusercontent.com/lucioduran/agent-injection-evals/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/results/eval/trace.jsonl","https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/results/dev/report.json","https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/docs/error-analysis.md","https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/docs/model-notices.md","https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/injection_evals/core.py","https://github.com/lucioduran/agent-injection-evals/blob/e6d3c2c62154ccfc5a36595ee0ed4e44484c7d66/requirements-local.lock"]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Projects","item":"https://lucioduran.com/projects"},{"@type":"ListItem","position":2,"name":"Agent Injection Evals","item":"https://lucioduran.com/projects/agent-injection-evals"}]}]}],"contentHash":"8367f0b3a4c1453cf5822aea24d8c708e2831757d757d24f82d5fb8b14bdf6a4","provenance":{"method":"Deterministic serialization of the server-rendered main element","source":"https://lucioduran.com/projects/agent-injection-evals","hashAlgorithm":"sha256","hashScope":"JSON document fields before contentHash and provenance","sourceHTMLBytes":52424,"contentBytes":9979}}