<!-- Canonical: https://lucioduran.com/projects/agent-trace-observability
Language: en
Content-SHA256: f88c5d87135931dcb8cfbdda4077aac51d256873ce17e2a47ec0cefacd76bd1a
-->

[Projects](<https://lucioduran.com/projects>) / Agent Trace Observability

# Agent Trace Observability

OpenTelemetry instrumentation for an owned local agent pipeline with bounded, allowlisted exports and tested exclusion of sensitive payloads.

Author: [Lucio Durán](<https://lucioduran.com/>) · Python 3.11.13 and OpenTelemetry SDK 1.39.1 · License: [MIT](<https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/LICENSE>)

Page published: 2026-11-09 · Page updated: 2026-11-09

[Repository](<https://github.com/lucioduran/agent-trace-observability>) · [Original evidence](<https://raw.githubusercontent.com/lucioduran/agent-trace-observability/b5d9d3465507d050a82839b7655cc6a0bb67f345/results/local/report.json>) · [Release v0.1.0](<https://github.com/lucioduran/agent-trace-observability/releases/tag/v0.1.0>)

## Problem and relevance to agents

Agent traces can leak prompts, credentials and documents through attributes, events, exceptions and status descriptions. Removing a prompt field alone is insufficient.

A narrow export contract preserves operational relationships while excluding arbitrary payloads before they reach the local sink.

## Architecture

1. **Instrument an owned pipeline.** Token-overlap retrieval, a trained naive Bayes classifier and a catalog tool emit real OpenTelemetry spans.
2. **Project approved fields.** Fresh records retain fixed component names, versions, latency, error categories and correlation IDs. Arbitrary payload and exception text is excluded.
3. **Bound exports and failures.** A dedicated synchronous JSONL exporter enforces lifetime byte/record limits and permanently stops after a sink failure without retries.

- [Design and trust boundaries](<https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/docs/design.md>)

## What this demonstrates

- Async parentage, concurrent context isolation and explicit W3C thread handoff.
- Tests of attributes, events, exceptions, nested payloads and sink errors.
- Version, failure-cause and latency linkage across 400 real local executions.

## Published results

A short Xeon CPU run executed 400 requests at concurrency four with OpenTelemetry SDK 1.39.1.

The run cycles evenly through success, retrieval failure, model failure and tool timeout. Failures are injected; execution and timings are real. Latency includes synchronous export but excludes waiting for the request semaphore.

### Local trace evidence

| Measurement | Observed result |
| --- | --- |
| Requests / concurrency | 400 / 4 |
| Successes | 100 |
| Injected retrieval / model / tool failures | 100 / 100 / 100 |
| Exported spans | 1,300 |
| Dropped spans / export failures | 0 / 0 |
| Exported bytes | 463,011 |
| Median / maximum request latency | 1.388 / 4.380 ms |
| Cost / energy | Not measured / not measured |

The run demonstrates trace linkage and failure attribution in this owned pipeline. It does not measure tracing overhead, production throughput, LLM quality or remote collector interoperability.

## Reproduce

Python 3.11 and pinned OpenTelemetry SDK 1.39.1. Installation accesses PyPI; evaluation needs no credentials, model-weight download or external service.

Run measurements in a new destination and serialize heavy workloads. Within the original project series, use the shared benchmark\_lock.py wrapper. Commands below use POSIX shell syntax.

### Verify saved evidence

```
git clone https://github.com/lucioduran/agent-trace-observability.git
cd agent-trace-observability
git checkout b5d9d3465507d050a82839b7655cc6a0bb67f345
python -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements-dev.txt
python -m agent_trace verify --output results/local
```

### Repeat the local experiment

```
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 python -m agent_trace run --output build/new-run --runs 400 --concurrency 4
python -m agent_trace verify --output build/new-run
```

## Scope and limitations

- The export boundary does not protect process memory, application logs, other exporters or malicious application code.
- Trace IDs are correlation hints, not authenticated identity. Omitting payloads deliberately sacrifices content debugging.
- A slow sink can block; no hard I/O timeout, remote-delivery guarantee, retries, rotation or disk durability is provided. Partial writes can leave an incomplete final line.
- Only a local classifier and JSONL sink were evaluated. Cost and energy were not measured; there was no independent security or human review.

- [Related architecture article](<https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices>)

## Sources and revision

The release includes the raw report and traces with fixture and trace digests and runtime versions. The report does not record an execution commit or timestamp; the release revision and date identify publication, not the run time.

- Reviewed revision: [` b5d9d3465507d050a82839b7655cc6a0bb67f345 `](<https://github.com/lucioduran/agent-trace-observability/commit/b5d9d3465507d050a82839b7655cc6a0bb67f345>)
- Release published: [v0.1.0](<https://github.com/lucioduran/agent-trace-observability/releases/tag/v0.1.0>) · 2026-11-09T18:46:18Z
- [Original span records](<https://raw.githubusercontent.com/lucioduran/agent-trace-observability/b5d9d3465507d050a82839b7655cc6a0bb67f345/results/local/traces.jsonl>) SHA-256: ` 060760b9d196dd12375aab0d8b7baf9c25c39e251fdc919f394030a0a3ef7a00 `

- [README and usage](<https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/README.md>)
- [Original evidence](<https://raw.githubusercontent.com/lucioduran/agent-trace-observability/b5d9d3465507d050a82839b7655cc6a0bb67f345/results/local/report.json>)
- [MIT license](<https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/LICENSE>)
- [Design and trust boundaries](<https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/docs/design.md>)
- [Measurement methodology](<https://github.com/lucioduran/agent-trace-observability/blob/b5d9d3465507d050a82839b7655cc6a0bb67f345/docs/reproduction.md>)
- [Original span records](<https://raw.githubusercontent.com/lucioduran/agent-trace-observability/b5d9d3465507d050a82839b7655cc6a0bb67f345/results/local/traces.jsonl>)