Lucio Durán
lucioduran.com

Projects / Agent Trace Observability

Agent Trace Observability

OpenTelemetry instrumentation for an owned local agent pipeline with bounded, allowlisted exports and tested exclusion of sensitive payloads.

Author: · Python 3.11.13 and OpenTelemetry SDK 1.39.1 · License: MIT

Page published: · Page updated:

Repository · Original evidence · Release v0.1.0

Problem and relevance to agents

Agent traces can leak prompts, credentials and documents through attributes, events, exceptions and status descriptions. Removing a prompt field alone is insufficient.

A narrow export contract preserves operational relationships while excluding arbitrary payloads before they reach the local sink.

Architecture

  1. Instrument an owned pipeline. Token-overlap retrieval, a trained naive Bayes classifier and a catalog tool emit real OpenTelemetry spans.
  2. Project approved fields. Fresh records retain fixed component names, versions, latency, error categories and correlation IDs. Arbitrary payload and exception text is excluded.
  3. Bound exports and failures. A dedicated synchronous JSONL exporter enforces lifetime byte/record limits and permanently stops after a sink failure without retries.

What this demonstrates

  • Async parentage, concurrent context isolation and explicit W3C thread handoff.
  • Tests of attributes, events, exceptions, nested payloads and sink errors.
  • Version, failure-cause and latency linkage across 400 real local executions.

Published results

A short Xeon CPU run executed 400 requests at concurrency four with OpenTelemetry SDK 1.39.1.

The run cycles evenly through success, retrieval failure, model failure and tool timeout. Failures are injected; execution and timings are real. Latency includes synchronous export but excludes waiting for the request semaphore.

Local trace evidence

Original evidence
MeasurementObserved result
Requests / concurrency400 / 4
Successes100
Injected retrieval / model / tool failures100 / 100 / 100
Exported spans1,300
Dropped spans / export failures0 / 0
Exported bytes463,011
Median / maximum request latency1.388 / 4.380 ms
Cost / energyNot measured / not measured

The run demonstrates trace linkage and failure attribution in this owned pipeline. It does not measure tracing overhead, production throughput, LLM quality or remote collector interoperability.

Reproduce

Python 3.11 and pinned OpenTelemetry SDK 1.39.1. Installation accesses PyPI; evaluation needs no credentials, model-weight download or external service.

Run measurements in a new destination and serialize heavy workloads. Within the original project series, use the shared benchmark_lock.py wrapper. Commands below use POSIX shell syntax.

Verify saved evidence

git clone https://github.com/lucioduran/agent-trace-observability.git
cd agent-trace-observability
git checkout b5d9d3465507d050a82839b7655cc6a0bb67f345
python -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements-dev.txt
python -m agent_trace verify --output results/local

Repeat the local experiment

OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 python -m agent_trace run --output build/new-run --runs 400 --concurrency 4
python -m agent_trace verify --output build/new-run

Scope and limitations

  • The export boundary does not protect process memory, application logs, other exporters or malicious application code.
  • Trace IDs are correlation hints, not authenticated identity. Omitting payloads deliberately sacrifices content debugging.
  • A slow sink can block; no hard I/O timeout, remote-delivery guarantee, retries, rotation or disk durability is provided. Partial writes can leave an incomplete final line.
  • Only a local classifier and JSONL sink were evaluated. Cost and energy were not measured; there was no independent security or human review.

Sources and revision

The release includes the raw report and traces with fixture and trace digests and runtime versions. The report does not record an execution commit or timestamp; the release revision and date identify publication, not the run time.