{"schemaVersion":1,"id":"https://lucioduran.com/projects/agent-trace-observability","url":"https://lucioduran.com/projects/agent-trace-observability","title":"Agent Trace Observability","description":"OpenTelemetry instrumentation for an owned local agent pipeline with bounded, allowlisted exports and tested exclusion of sensitive payloads.","language":"en","datePublished":"2026-11-09","dateModified":"2026-11-09","contentFormat":"text/markdown","content":"[Projects](<https://lucioduran.com/projects>) / Agent Trace Observability\n\n# Agent Trace Observability\n\nOpenTelemetry instrumentation for an owned local agent pipeline with bounded, allowlisted exports and tested exclusion of sensitive payloads.\n\nAuthor: [Lucio Durán](<https://lucioduran.com/>) · Python 3.11.13 and OpenTelemetry SDK 1.39.1 · License: [MIT](<https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/LICENSE>)\n\nPage published: 2026-11-09 · Page updated: 2026-11-09\n\n[Repository](<https://github.com/lucioduran/agent-trace-observability>) · [Original evidence](<https://raw.githubusercontent.com/lucioduran/agent-trace-observability/b5d9d3465507d050a82839b7655cc6a0bb67f345/results/local/report.json>) · [Release v0.1.0](<https://github.com/lucioduran/agent-trace-observability/releases/tag/v0.1.0>)\n\n## Problem and relevance to agents\n\nAgent traces can leak prompts, credentials and documents through attributes, events, exceptions and status descriptions. Removing a prompt field alone is insufficient.\n\nA narrow export contract preserves operational relationships while excluding arbitrary payloads before they reach the local sink.\n\n## Architecture\n\n1. **Instrument an owned pipeline.** Token-overlap retrieval, a trained naive Bayes classifier and a catalog tool emit real OpenTelemetry spans.\n2. **Project approved fields.** Fresh records retain fixed component names, versions, latency, error categories and correlation IDs. Arbitrary payload and exception text is excluded.\n3. **Bound exports and failures.** A dedicated synchronous JSONL exporter enforces lifetime byte/record limits and permanently stops after a sink failure without retries.\n\n- [Design and trust boundaries](<https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/docs/design.md>)\n\n## What this demonstrates\n\n- Async parentage, concurrent context isolation and explicit W3C thread handoff.\n- Tests of attributes, events, exceptions, nested payloads and sink errors.\n- Version, failure-cause and latency linkage across 400 real local executions.\n\n## Published results\n\nA short Xeon CPU run executed 400 requests at concurrency four with OpenTelemetry SDK 1.39.1.\n\nThe run cycles evenly through success, retrieval failure, model failure and tool timeout. Failures are injected; execution and timings are real. Latency includes synchronous export but excludes waiting for the request semaphore.\n\n### Local trace evidence\n\n| Measurement | Observed result |\n| --- | --- |\n| Requests / concurrency | 400 / 4 |\n| Successes | 100 |\n| Injected retrieval / model / tool failures | 100 / 100 / 100 |\n| Exported spans | 1,300 |\n| Dropped spans / export failures | 0 / 0 |\n| Exported bytes | 463,011 |\n| Median / maximum request latency | 1.388 / 4.380 ms |\n| Cost / energy | Not measured / not measured |\n\nThe run demonstrates trace linkage and failure attribution in this owned pipeline. It does not measure tracing overhead, production throughput, LLM quality or remote collector interoperability.\n\n## Reproduce\n\nPython 3.11 and pinned OpenTelemetry SDK 1.39.1. Installation accesses PyPI; evaluation needs no credentials, model-weight download or external service.\n\nRun measurements in a new destination and serialize heavy workloads. Within the original project series, use the shared benchmark\\_lock.py wrapper. Commands below use POSIX shell syntax.\n\n### Verify saved evidence\n\n```\ngit clone https://github.com/lucioduran/agent-trace-observability.git\ncd agent-trace-observability\ngit checkout b5d9d3465507d050a82839b7655cc6a0bb67f345\npython -m venv .venv\n. .venv/bin/activate\npython -m pip install -r requirements-dev.txt\npython -m agent_trace verify --output results/local\n```\n\n### Repeat the local experiment\n\n```\nOMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 python -m agent_trace run --output build/new-run --runs 400 --concurrency 4\npython -m agent_trace verify --output build/new-run\n```\n\n## Scope and limitations\n\n- The export boundary does not protect process memory, application logs, other exporters or malicious application code.\n- Trace IDs are correlation hints, not authenticated identity. Omitting payloads deliberately sacrifices content debugging.\n- A slow sink can block; no hard I/O timeout, remote-delivery guarantee, retries, rotation or disk durability is provided. Partial writes can leave an incomplete final line.\n- Only a local classifier and JSONL sink were evaluated. Cost and energy were not measured; there was no independent security or human review.\n\n- [Related architecture article](<https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices>)\n\n## Sources and revision\n\nThe release includes the raw report and traces with fixture and trace digests and runtime versions. The report does not record an execution commit or timestamp; the release revision and date identify publication, not the run time.\n\n- Reviewed revision: [` b5d9d3465507d050a82839b7655cc6a0bb67f345 `](<https://github.com/lucioduran/agent-trace-observability/commit/b5d9d3465507d050a82839b7655cc6a0bb67f345>)\n- Release published: [v0.1.0](<https://github.com/lucioduran/agent-trace-observability/releases/tag/v0.1.0>) · 2026-11-09T18:46:18Z\n- [Original span records](<https://raw.githubusercontent.com/lucioduran/agent-trace-observability/b5d9d3465507d050a82839b7655cc6a0bb67f345/results/local/traces.jsonl>) SHA-256: ` 060760b9d196dd12375aab0d8b7baf9c25c39e251fdc919f394030a0a3ef7a00 `\n\n- [README and usage](<https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/README.md>)\n- [Original evidence](<https://raw.githubusercontent.com/lucioduran/agent-trace-observability/b5d9d3465507d050a82839b7655cc6a0bb67f345/results/local/report.json>)\n- [MIT license](<https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/LICENSE>)\n- [Design and trust boundaries](<https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/docs/design.md>)\n- [Measurement methodology](<https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/docs/reproduction.md>)\n- [Original span records](<https://raw.githubusercontent.com/lucioduran/agent-trace-observability/b5d9d3465507d050a82839b7655cc6a0bb67f345/results/local/traces.jsonl>)","headings":[{"level":1,"text":"Agent Trace Observability","url":"https://lucioduran.com/projects/agent-trace-observability#title"},{"level":2,"text":"Problem and relevance to agents","url":"https://lucioduran.com/projects/agent-trace-observability#problem"},{"level":2,"text":"Architecture","url":"https://lucioduran.com/projects/agent-trace-observability#architecture"},{"level":2,"text":"What this demonstrates","url":"https://lucioduran.com/projects/agent-trace-observability#demonstrates"},{"level":2,"text":"Published results","url":"https://lucioduran.com/projects/agent-trace-observability#results"},{"level":3,"text":"Local trace evidence","url":"https://lucioduran.com/projects/agent-trace-observability#results-Local%20trace%20evidence"},{"level":2,"text":"Reproduce","url":"https://lucioduran.com/projects/agent-trace-observability#reproduce"},{"level":3,"text":"Verify saved evidence","url":"https://lucioduran.com/projects/agent-trace-observability#reproduce-replay"},{"level":3,"text":"Repeat the local experiment","url":"https://lucioduran.com/projects/agent-trace-observability#reproduce-experiment"},{"level":2,"text":"Scope and limitations","url":"https://lucioduran.com/projects/agent-trace-observability#scope"},{"level":2,"text":"Sources and revision","url":"https://lucioduran.com/projects/agent-trace-observability#sources"}],"sections":[{"heading":"Agent Trace Observability","level":1,"url":"https://lucioduran.com/projects/agent-trace-observability","content":"[Projects](<https://lucioduran.com/projects>) / Agent Trace Observability"},{"heading":"Agent Trace Observability","level":1,"url":"https://lucioduran.com/projects/agent-trace-observability#title","content":"OpenTelemetry instrumentation for an owned local agent pipeline with bounded, allowlisted exports and tested exclusion of sensitive payloads.\n\nAuthor: [Lucio Durán](<https://lucioduran.com/>) · Python 3.11.13 and OpenTelemetry SDK 1.39.1 · License: [MIT](<https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/LICENSE>)\n\nPage published: 2026-11-09 · Page updated: 2026-11-09\n\n[Repository](<https://github.com/lucioduran/agent-trace-observability>) · [Original evidence](<https://raw.githubusercontent.com/lucioduran/agent-trace-observability/b5d9d3465507d050a82839b7655cc6a0bb67f345/results/local/report.json>) · [Release v0.1.0](<https://github.com/lucioduran/agent-trace-observability/releases/tag/v0.1.0>)"},{"heading":"Problem and relevance to agents","level":2,"url":"https://lucioduran.com/projects/agent-trace-observability#problem","content":"Agent traces can leak prompts, credentials and documents through attributes, events, exceptions and status descriptions. Removing a prompt field alone is insufficient.\n\nA narrow export contract preserves operational relationships while excluding arbitrary payloads before they reach the local sink."},{"heading":"Architecture","level":2,"url":"https://lucioduran.com/projects/agent-trace-observability#architecture","content":"1. **Instrument an owned pipeline.** Token-overlap retrieval, a trained naive Bayes classifier and a catalog tool emit real OpenTelemetry spans.\n2. **Project approved fields.** Fresh records retain fixed component names, versions, latency, error categories and correlation IDs. Arbitrary payload and exception text is excluded.\n3. **Bound exports and failures.** A dedicated synchronous JSONL exporter enforces lifetime byte/record limits and permanently stops after a sink failure without retries.\n\n- [Design and trust boundaries](<https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/docs/design.md>)"},{"heading":"What this demonstrates","level":2,"url":"https://lucioduran.com/projects/agent-trace-observability#demonstrates","content":"- Async parentage, concurrent context isolation and explicit W3C thread handoff.\n- Tests of attributes, events, exceptions, nested payloads and sink errors.\n- Version, failure-cause and latency linkage across 400 real local executions."},{"heading":"Published results","level":2,"url":"https://lucioduran.com/projects/agent-trace-observability#results","content":"A short Xeon CPU run executed 400 requests at concurrency four with OpenTelemetry SDK 1.39.1.\n\nThe run cycles evenly through success, retrieval failure, model failure and tool timeout. Failures are injected; execution and timings are real. Latency includes synchronous export but excludes waiting for the request semaphore."},{"heading":"Local trace evidence","level":3,"url":"https://lucioduran.com/projects/agent-trace-observability#results-Local%20trace%20evidence","content":"| Measurement | Observed result |\n| --- | --- |\n| Requests / concurrency | 400 / 4 |\n| Successes | 100 |\n| Injected retrieval / model / tool failures | 100 / 100 / 100 |\n| Exported spans | 1,300 |\n| Dropped spans / export failures | 0 / 0 |\n| Exported bytes | 463,011 |\n| Median / maximum request latency | 1.388 / 4.380 ms |\n| Cost / energy | Not measured / not measured |\n\nThe run demonstrates trace linkage and failure attribution in this owned pipeline. It does not measure tracing overhead, production throughput, LLM quality or remote collector interoperability."},{"heading":"Reproduce","level":2,"url":"https://lucioduran.com/projects/agent-trace-observability#reproduce","content":"Python 3.11 and pinned OpenTelemetry SDK 1.39.1. Installation accesses PyPI; evaluation needs no credentials, model-weight download or external service.\n\nRun measurements in a new destination and serialize heavy workloads. Within the original project series, use the shared benchmark\\_lock.py wrapper. Commands below use POSIX shell syntax."},{"heading":"Verify saved evidence","level":3,"url":"https://lucioduran.com/projects/agent-trace-observability#reproduce-replay","content":"```\ngit clone https://github.com/lucioduran/agent-trace-observability.git\ncd agent-trace-observability\ngit checkout b5d9d3465507d050a82839b7655cc6a0bb67f345\npython -m venv .venv\n. .venv/bin/activate\npython -m pip install -r requirements-dev.txt\npython -m agent_trace verify --output results/local\n```"},{"heading":"Repeat the local experiment","level":3,"url":"https://lucioduran.com/projects/agent-trace-observability#reproduce-experiment","content":"```\nOMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 python -m agent_trace run --output build/new-run --runs 400 --concurrency 4\npython -m agent_trace verify --output build/new-run\n```"},{"heading":"Scope and limitations","level":2,"url":"https://lucioduran.com/projects/agent-trace-observability#scope","content":"- The export boundary does not protect process memory, application logs, other exporters or malicious application code.\n- Trace IDs are correlation hints, not authenticated identity. Omitting payloads deliberately sacrifices content debugging.\n- A slow sink can block; no hard I/O timeout, remote-delivery guarantee, retries, rotation or disk durability is provided. Partial writes can leave an incomplete final line.\n- Only a local classifier and JSONL sink were evaluated. Cost and energy were not measured; there was no independent security or human review.\n\n- [Related architecture article](<https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices>)"},{"heading":"Sources and revision","level":2,"url":"https://lucioduran.com/projects/agent-trace-observability#sources","content":"The release includes the raw report and traces with fixture and trace digests and runtime versions. The report does not record an execution commit or timestamp; the release revision and date identify publication, not the run time.\n\n- Reviewed revision: [` b5d9d3465507d050a82839b7655cc6a0bb67f345 `](<https://github.com/lucioduran/agent-trace-observability/commit/b5d9d3465507d050a82839b7655cc6a0bb67f345>)\n- Release published: [v0.1.0](<https://github.com/lucioduran/agent-trace-observability/releases/tag/v0.1.0>) · 2026-11-09T18:46:18Z\n- [Original span records](<https://raw.githubusercontent.com/lucioduran/agent-trace-observability/b5d9d3465507d050a82839b7655cc6a0bb67f345/results/local/traces.jsonl>) SHA-256: ` 060760b9d196dd12375aab0d8b7baf9c25c39e251fdc919f394030a0a3ef7a00 `\n\n- [README and usage](<https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/README.md>)\n- [Original evidence](<https://raw.githubusercontent.com/lucioduran/agent-trace-observability/b5d9d3465507d050a82839b7655cc6a0bb67f345/results/local/report.json>)\n- [MIT license](<https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/LICENSE>)\n- [Design and trust boundaries](<https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/docs/design.md>)\n- [Measurement methodology](<https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/docs/reproduction.md>)\n- [Original span records](<https://raw.githubusercontent.com/lucioduran/agent-trace-observability/b5d9d3465507d050a82839b7655cc6a0bb67f345/results/local/traces.jsonl>)"}],"links":[{"text":"Projects","url":"https://lucioduran.com/projects"},{"text":"Lucio Durán","url":"https://lucioduran.com/"},{"text":"MIT","url":"https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/LICENSE"},{"text":"Repository","url":"https://github.com/lucioduran/agent-trace-observability"},{"text":"Original evidence","url":"https://raw.githubusercontent.com/lucioduran/agent-trace-observability/b5d9d3465507d050a82839b7655cc6a0bb67f345/results/local/report.json"},{"text":"Release v0.1.0","url":"https://github.com/lucioduran/agent-trace-observability/releases/tag/v0.1.0"},{"text":"Design and trust boundaries","url":"https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/docs/design.md"},{"text":"Related architecture article","url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices"},{"text":"b5d9d3465507d050a82839b7655cc6a0bb67f345","url":"https://github.com/lucioduran/agent-trace-observability/commit/b5d9d3465507d050a82839b7655cc6a0bb67f345"},{"text":"v0.1.0","url":"https://github.com/lucioduran/agent-trace-observability/releases/tag/v0.1.0"},{"text":"Original span records","url":"https://raw.githubusercontent.com/lucioduran/agent-trace-observability/b5d9d3465507d050a82839b7655cc6a0bb67f345/results/local/traces.jsonl"},{"text":"README and usage","url":"https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/README.md"},{"text":"MIT license","url":"https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/LICENSE"},{"text":"Measurement methodology","url":"https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/docs/reproduction.md"}],"structuredData":[{"@context":"https://schema.org","@graph":[{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com/"},{"@type":"WebSite","@id":"https://lucioduran.com/#website","url":"https://lucioduran.com/","name":"Lucio Durán","inLanguage":["en","es"],"publisher":{"@id":"https://lucioduran.com/#person"}},{"@type":"WebPage","@id":"https://lucioduran.com/projects/agent-trace-observability#page","url":"https://lucioduran.com/projects/agent-trace-observability","name":"Agent Trace Observability","description":"OpenTelemetry instrumentation for an owned local agent pipeline with bounded, allowlisted exports and tested exclusion of sensitive payloads.","inLanguage":"en","author":{"@id":"https://lucioduran.com/#person"},"publisher":{"@id":"https://lucioduran.com/#person"},"isPartOf":{"@id":"https://lucioduran.com/#website"},"isAccessibleForFree":true,"mainEntity":{"@id":"https://lucioduran.com/projects/agent-trace-observability#software"},"datePublished":"2026-11-09","dateModified":"2026-11-09","breadcrumb":{"@id":"https://lucioduran.com/projects/agent-trace-observability#breadcrumb"}},{"@type":"SoftwareSourceCode","@id":"https://lucioduran.com/projects/agent-trace-observability#software","name":"Agent Trace Observability","description":"OpenTelemetry instrumentation for an owned local agent pipeline with bounded, allowlisted exports and tested exclusion of sensitive payloads.","inLanguage":"en","url":"https://lucioduran.com/projects/agent-trace-observability","codeRepository":"https://github.com/lucioduran/agent-trace-observability","programmingLanguage":"Python","runtimePlatform":"Python 3.11.13 and OpenTelemetry SDK 1.39.1","license":"https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/LICENSE","version":"v0.1.0","identifier":{"@type":"PropertyValue","propertyID":"git-commit","value":"b5d9d3465507d050a82839b7655cc6a0bb67f345"},"author":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"isAccessibleForFree":true,"citation":["https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/README.md","https://raw.githubusercontent.com/lucioduran/agent-trace-observability/b5d9d3465507d050a82839b7655cc6a0bb67f345/results/local/report.json","https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/docs/design.md","https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/docs/reproduction.md","https://raw.githubusercontent.com/lucioduran/agent-trace-observability/b5d9d3465507d050a82839b7655cc6a0bb67f345/results/local/traces.jsonl","https://github.com/lucioduran/agent-trace-observability/releases/tag/v0.1.0"],"publisher":{"@id":"https://lucioduran.com/#person"},"mainEntityOfPage":{"@id":"https://lucioduran.com/projects/agent-trace-observability#page"}},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Projects","item":"https://lucioduran.com/projects"},{"@type":"ListItem","position":2,"name":"Agent Trace Observability","item":"https://lucioduran.com/projects/agent-trace-observability"}],"@id":"https://lucioduran.com/projects/agent-trace-observability#breadcrumb"}]}],"contentHash":"f88c5d87135931dcb8cfbdda4077aac51d256873ce17e2a47ec0cefacd76bd1a","provenance":{"method":"Deterministic serialization of the server-rendered main element","source":"https://lucioduran.com/projects/agent-trace-observability","hashAlgorithm":"sha256","hashScope":"JSON document fields before contentHash and provenance","sourceHTMLBytes":43525,"contentBytes":6409}}