{"schemaVersion":1,"id":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices","url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices","title":"OpenTelemetry in Production: Distributed Tracing Implementation and Cost Control","description":"Auto-instrumentation gotchas, custom spans that actually help, baggage propagation patterns, Jaeger vs Tempo showdown, trace-based testing, and how to not go bankrupt on observability costs.","language":"en","datePublished":"2025-12-11","dateModified":null,"contentFormat":"text/markdown","content":"Observability\n\n# OpenTelemetry in Production: Distributed Tracing Implementation and Cost Control\n\n2025-12-118 min read\n\nLucio Durán\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/opentelemetry-distributed-tracing-microservices>)\n\n## The auto-instrumentation trap\n\nThe typical first step is applying the auto-instrumentation agent without further configuration.\n\nFor Node.js with [OpenTelemetry](<https://opentelemetry.io/>), the initial setup appears simple:\n\n```javascript\n// tracing.js, loaded before anything else via --require\nconst { NodeSDK } = require('@opentelemetry/sdk-node');\nconst { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');\nconst { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');\n\nconst sdk = new NodeSDK({\n traceExporter: new OTLPTraceExporter({\n url: 'http://otel-collector:4317',\n }),\n instrumentations: [getNodeAutoInstrumentations({\n '@opentelemetry/instrumentation-fs': { enabled: false }, // Trust me\n '@opentelemetry/instrumentation-http': {\n ignoreIncomingRequestHook: (req) => {\n return req.url === '/healthz' || req.url === '/readyz';\n },\n },\n })],\n});\n\nsdk.start();\n```\n\nSee that ` instrumentation-fs ` disabled? A common discovery: the collector falls over because the fs instrumentation creates a span for every single file read, including config files, node\\_modules, temp files, everything. Trace volume jumps from 50MB/min to 12GB/min.\n\nFor Go services, the approach is different because Go doesn't have a runtime agent model:\n\n```go\npackage main\n\nimport (\n \"context\"\n \"go.opentelemetry.io/otel\"\n \"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc\"\n \"go.opentelemetry.io/otel/sdk/resource\"\n sdktrace \"go.opentelemetry.io/otel/sdk/trace\"\n semconv \"go.opentelemetry.io/otel/semconv/v1.24.0\"\n)\n\nfunc initTracer() (*sdktrace.TracerProvider, error) {\n exporter, err := otlptracegrpc.New(context.Background(),\n otlptracegrpc.WithEndpoint(\"otel-collector:4317\"),\n otlptracegrpc.WithInsecure(),\n )\n if err != nil {\n return nil, err\n }\n\n tp := sdktrace.NewTracerProvider(\n sdktrace.WithBatcher(exporter,\n sdktrace.WithMaxQueueSize(2048),\n sdktrace.WithMaxExportBatchSize(512),\n sdktrace.WithBatchTimeout(5*time.Second),\n ),\n sdktrace.WithResource(resource.NewWithAttributes(\n semconv.SchemaURL,\n semconv.ServiceNameKey.String(\"checkout-service\"),\n semconv.ServiceVersionKey.String(\"1.4.2\"),\n semconv.DeploymentEnvironmentKey.String(\"production\"),\n )),\n sdktrace.WithSampler(sdktrace.ParentBased(\n sdktrace.TraceIDRatioBased(0.1),\n )),\n )\n\n otel.SetTracerProvider(tp)\n return tp, nil\n}\n```\n\nThe ` WithBatcher ` config is crucial. The defaults are fine for development, but in production with high throughput, you'll hit queue overflows. Services can drop 40% of spans silently because the batch queue was full and the default behavior is to drop, not block.\n\n## Custom spans that actually help\n\nAuto-instrumentation gives you the skeleton. Custom spans give you the meat. But teams sometimes go overboard and create spans for every function call, which just turns your traces into unreadable noise.\n\nA good rule: create custom spans for **business operations** and **external dependencies** that auto-instrumentation doesn't cover.\n\n```go\nfunc (s *CheckoutService) ProcessOrder(ctx context.Context, order *Order) error {\n ctx, span := s.tracer.Start(ctx, \"checkout.process_order\",\n trace.WithAttributes(\n attribute.String(\"order.id\", order.ID),\n attribute.Int(\"order.items_count\", len(order.Items)),\n attribute.String(\"order.payment_method\", order.PaymentMethod),\n attribute.Float64(\"order.total\", order.Total),\n ),\n )\n defer span.End()\n\n // Validate inventory, this talks to another service\n if err := s.validateInventory(ctx, order); err != nil {\n span.SetStatus(codes.Error, \"inventory validation failed\")\n span.RecordError(err, trace.WithAttributes(\n attribute.StringSlice(\"failed_skus\", getFailedSKUs(err)),\n ))\n return fmt.Errorf(\"inventory check: %w\", err)\n }\n\n // Process payment, external gateway\n paymentSpan := trace.SpanFromContext(ctx)\n paymentSpan.AddEvent(\"payment.initiated\", trace.WithAttributes(\n attribute.String(\"gateway\", s.paymentGateway.Name()),\n ))\n\n charge, err := s.processPayment(ctx, order)\n if err != nil {\n span.SetStatus(codes.Error, \"payment processing failed\")\n span.RecordError(err)\n return fmt.Errorf(\"payment: %w\", err)\n }\n\n span.SetAttributes(attribute.String(\"payment.charge_id\", charge.ID))\n return nil\n}\n```\n\nNotice a few things. The ` order.id ` is recorded as an attribute for trace searching by order ID. ` RecordError ` with additional attributes captures which specific SKUs failed. And span events mark significant moments within a single span (payment initiated vs. payment completed).\n\n## Baggage propagation: cross-service context done right\n\nBaggage is the most underused feature of OpenTelemetry. It lets you attach key-value pairs to the context that propagate across service boundaries, without cluttering every span with the same attributes.\n\n```go\n// At the API gateway, inject tenant context\nimport \"go.opentelemetry.io/otel/baggage\"\n\nfunc gatewayMiddleware(next http.Handler) http.Handler {\n return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n tenantID := extractTenantID(r)\n region := resolveRegion(r)\n\n member1, _ := baggage.NewMember(\"tenant.id\", tenantID)\n member2, _ := baggage.NewMember(\"region\", region)\n bag, _ := baggage.New(member1, member2)\n\n ctx := baggage.ContextWithBaggage(r.Context(), bag)\n next.ServeHTTP(w, r.WithContext(ctx))\n })\n}\n\n// In ANY downstream service, read it without knowing how it got there\nfunc (s *InventoryService) CheckStock(ctx context.Context, sku string) (int, error) {\n bag := baggage.FromContext(ctx)\n tenantID := bag.Member(\"tenant.id\").Value()\n region := bag.Member(\"region\").Value()\n\n // Route to tenant-specific inventory\n return s.repo.GetStock(ctx, tenantID, region, sku)\n}\n```\n\nThe collector can also read baggage and add it as span attributes automatically:\n\n```yaml\n# otel-collector config\nprocessors:\n baggage:\n rules:\n - from: baggage\n key: tenant.id\n action: insert\n - from: baggage\n key: region\n action: insert\n```\n\nThis is incredibly powerful for multi-tenant systems. Instead of passing ` tenantID ` through every function signature, it rides the context through the entire request chain.\n\n## Jaeger vs Tempo: the real comparison\n\nHere is an honest breakdown based on production experience.\n\n**Jaeger with Elasticsearch**:\n\n- Setup: Jaeger collector + Elasticsearch cluster (3 nodes, r6g.xlarge)\n- Storage: ~2TB of traces, 7-day retention\n- Monthly cost: ~$2,400 (mostly Elasticsearch)\n- Query performance: Fast for trace ID lookup, slow for tag-based search at scale\n- Operations burden: Elasticsearch index management, JVM tuning, shard rebalancing\n\n**Grafana Tempo with S3**:\n\n- Setup: Tempo (3 replicas in microservices mode) + S3 bucket\n- Storage: ~2TB of traces, 30-day retention (yes, 4x longer)\n- Monthly cost: ~$180 (S3 storage + a bit of compute)\n- Query performance: Trace ID lookup is fast, tag-based search requires TraceQL\n- Operations burden: Minimal, stateless ingesters, object storage does the heavy lifting\n\nThe cost difference is not a typo. Tempo uses object storage (S3, GCS, Azure Blob) instead of a full-text search engine. The trade-off is that you can't do arbitrary field searches, you need to use TraceQL or integrate with Grafana Loki for log-to-trace correlation.\n\n```promql\n# TraceQL, finding slow checkout traces\n{ span.checkout.process_order && duration > 5s }\n\n# With attribute filter\n{ span.checkout.process_order\n && span.order.payment_method = \"credit_card\"\n && status = error }\n```\n\nThe switch from Jaeger to Tempo is often a clear win. The observability bill goes from dreaded to barely noticed.\n\n## Trace-based testing: validation through observability\n\nThis is one of the most promising developments. Trace-based testing uses actual distributed traces to validate system behavior. Instead of mocking everything, you run real requests and assert on the trace structure.\n\nTracetest, which hooks into the OTEL collector, enables this:\n\n```yaml\n# tracetest definition\ntype: Test\nspec:\n name: \"Checkout creates payment and sends confirmation\"\n trigger:\n type: http\n httpRequest:\n url: http://api-gateway/api/v1/checkout\n method: POST\n body: '{\"items\": [{\"sku\": \"WIDGET-1\", \"qty\": 2}], \"payment\": \"credit_card\"}'\n specs:\n - selector: span[name=\"checkout.process_order\"]\n assertions:\n - attr:order.items_count = 2\n - selector: span[name=\"payment.charge\"]\n assertions:\n - attr:tracetest.span.duration < 2000ms\n - attr:payment.status = \"captured\"\n - selector: span[name=\"notification.send_email\"]\n assertions:\n - attr:email.template = \"order_confirmation\"\n - attr:tracetest.span.duration < 500ms\n```\n\nThis test fires a real HTTP request, waits for the trace to be collected, and then asserts on span attributes across multiple services. It's integration testing on steroids, you're validating not just the response, but the entire internal behavior of the system.\n\nRunning these in CI after deploying to a staging environment catches issues that traditional integration tests miss: a service that returns the right response but takes an extra database round-trip, a message queue consumer that processes events but doesn't propagate the trace context, an HTTP client that silently falls back to a default timeout.\n\n## Cost control: the survival guide\n\nObservability costs can spiral out of control faster than you'd expect. Here is a proven three-layer approach:\n\n### Layer 1: Head-based sampling at the SDK\n\n```yaml\n# Environment variable, simple and effective\nOTEL_TRACES_SAMPLER=parentbased_traceidratio\nOTEL_TRACES_SAMPLER_ARG=0.1\n```\n\nThis samples 10% of traces at the source. The ` parentbased ` prefix is critical, it means child spans respect the parent's sampling decision. Without it, you get orphaned spans everywhere.\n\n### Layer 2: Tail-based sampling at the collector\n\n```yaml\n# otel-collector config\nprocessors:\n tail_sampling:\n decision_wait: 30s\n policies:\n - name: errors-always\n type: status_code\n status_code:\n status_codes: [ERROR]\n - name: slow-traces\n type: latency\n latency:\n threshold_ms: 5000\n - name: percentage-sample\n type: probabilistic\n probabilistic:\n sampling_percentage: 5\n```\n\nThis catches 100% of error traces and slow traces, regardless of the head-based sampling decision. The collector needs to buffer spans for the ` decision_wait ` period (30s in this case), which requires memory, budget about 500MB per 10K spans/second.\n\n### Layer 3: Attribute limits\n\n```yaml\nprocessors:\n attributes:\n actions:\n - key: http.request.body\n action: delete # Never store request bodies\n - key: db.statement\n action: hash # Hash SQL queries instead of storing them\n span:\n max_num_attributes: 64\n max_attribute_value_length: 256\n```\n\nThe ` db.statement ` hashing prevents a scenario where full SQL queries including WHERE clause values, which may contain user emails, get logged. Security and cost in one config change.\n\nWith all three layers, tracing costs drop from ~$8,000/month to ~$800/month while actually improving the debugging experience (because 100% error capture means the traces that matter are never missed).\n\n## The collector topology that scales\n\nRunning a single collector is inadvisable. Here is a proven topology:\n\n```\nServices → Agent Collectors (DaemonSet) → Gateway Collectors (Deployment) → Tempo\n (local buffering) (sampling, processing) (storage)\n```\n\nThe agent collectors run as a DaemonSet in Kubernetes. They receive spans over localhost (no network hop), do basic batching, and forward to the gateway. The gateway collectors do the heavy processing: tail-based sampling, attribute manipulation, and export to the backend.\n\n```yaml\n# Agent collector config (lightweight)\nreceivers:\n otlp:\n protocols:\n grpc:\n endpoint: 0.0.0.0:4317\nexporters:\n otlp:\n endpoint: otel-gateway:4317\n compression: zstd\nprocessors:\n batch:\n timeout: 1s\n send_batch_size: 1024\n\n# Gateway collector config (heavy processing)\nreceivers:\n otlp:\n protocols:\n grpc:\n endpoint: 0.0.0.0:4317\nprocessors:\n tail_sampling:\n # ... sampling policies\n attributes:\n # ... attribute processing\nexporters:\n otlp:\n endpoint: tempo:4317\n```\n\nThis separation lets you scale the gateway independently from the agents, and if the gateway goes down, the agents buffer locally without dropping spans from your services.\n\nRunning OpenTelemetry in production isn't hard. Running it *well*, with proper sampling, cost control, meaningful custom spans, and a topology that scales, that's where the craft is. Start with auto-instrumentation, add custom spans for business operations, set up tail-based sampling immediately, and pick a storage backend that won't bankrupt you. Everything else is optimization.\n\n## Frequently asked questions\n\n### Should I use auto-instrumentation or manual instrumentation?\n\nStart with auto-instrumentation to get baseline visibility, then layer manual spans for business-critical paths. Auto-instrumentation covers HTTP, gRPC, and database calls well but misses domain-specific logic.\n\n### Jaeger or Grafana Tempo?\n\nTempo if you're already in the Grafana ecosystem and want to minimize costs, it uses object storage (S3/GCS) instead of Elasticsearch. Jaeger if you need a standalone solution with mature tooling and don't mind running Elasticsearch or Cassandra.\n\n### How do I control OpenTelemetry costs in production?\n\nThree levers: head-based sampling to reduce volume at the source, tail-based sampling at the collector to keep interesting traces, and attribute limits to cap span size. We cut our costs 90% with a 10% head sample rate plus tail-based error sampling.\n\n## Tips\n\n1. **Tip:** Set ` OTEL_TRACES_SAMPLER=parentbased_traceidratio ` with a ratio of 0.1 in production, then use tail-based sampling at the collector to catch 100% of errors regardless.\n2. **Tip:** Never put PII in span attributes. Use ` baggage ` for cross-service context and redact at the collector level with the ` attributes ` processor.\n3. **Tip:** Grafana Tempo with S3 backend costs us ~$180/mo for 2TB of traces. Jaeger with Elasticsearch was $2,400/mo for the same volume. Do the math.\n4. **Tip:** Use ` span.SetStatus(codes.Error, err.Error()) ` religiously, without it, your error traces are invisible in most UIs.\n\n- opentelemetry\n- distributed-tracing\n- observability\n- jaeger\n- tempo\n- microservices\n- otel\n- spans\n- instrumentation\n\n[Back to Blog](<https://lucioduran.com/blog>)","headings":[{"level":1,"text":"OpenTelemetry in Production: Distributed Tracing Implementation and Cost Control","url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#title"},{"level":2,"text":"The auto-instrumentation trap","url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#the-auto-instrumentation-trap"},{"level":2,"text":"Custom spans that actually help","url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#custom-spans-that-actually-help"},{"level":2,"text":"Baggage propagation: cross-service context done right","url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#baggage-propagation-cross-service-context-done-right"},{"level":2,"text":"Jaeger vs Tempo: the real comparison","url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#jaeger-vs-tempo-the-real-comparison"},{"level":2,"text":"Trace-based testing: validation through observability","url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#trace-based-testing-validation-through-observability"},{"level":2,"text":"Cost control: the survival guide","url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#cost-control-the-survival-guide"},{"level":3,"text":"Layer 1: Head-based sampling at the SDK","url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#layer-1-head-based-sampling-at-the-sdk"},{"level":3,"text":"Layer 2: Tail-based sampling at the collector","url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#layer-2-tail-based-sampling-at-the-collector"},{"level":3,"text":"Layer 3: Attribute limits","url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#layer-3-attribute-limits"},{"level":2,"text":"The collector topology that scales","url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#the-collector-topology-that-scales"},{"level":2,"text":"Frequently asked questions","url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#questions"},{"level":3,"text":"Should I use auto-instrumentation or manual instrumentation?","url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#question-1"},{"level":3,"text":"Jaeger or Grafana Tempo?","url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#question-2"},{"level":3,"text":"How do I control OpenTelemetry costs in production?","url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#question-3"},{"level":2,"text":"Tips","url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#article-tips"}],"sections":[{"heading":"OpenTelemetry in Production: Distributed Tracing Implementation and Cost Control","level":1,"url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices","content":"Observability"},{"heading":"OpenTelemetry in Production: Distributed Tracing Implementation and Cost Control","level":1,"url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#title","content":"2025-12-118 min read\n\nLucio Durán\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/opentelemetry-distributed-tracing-microservices>)"},{"heading":"The auto-instrumentation trap","level":2,"url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#the-auto-instrumentation-trap","content":"The typical first step is applying the auto-instrumentation agent without further configuration.\n\nFor Node.js with [OpenTelemetry](<https://opentelemetry.io/>), the initial setup appears simple:\n\n```javascript\n// tracing.js, loaded before anything else via --require\nconst { NodeSDK } = require('@opentelemetry/sdk-node');\nconst { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');\nconst { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');\n\nconst sdk = new NodeSDK({\n traceExporter: new OTLPTraceExporter({\n url: 'http://otel-collector:4317',\n }),\n instrumentations: [getNodeAutoInstrumentations({\n '@opentelemetry/instrumentation-fs': { enabled: false }, // Trust me\n '@opentelemetry/instrumentation-http': {\n ignoreIncomingRequestHook: (req) => {\n return req.url === '/healthz' || req.url === '/readyz';\n },\n },\n })],\n});\n\nsdk.start();\n```\n\nSee that ` instrumentation-fs ` disabled? A common discovery: the collector falls over because the fs instrumentation creates a span for every single file read, including config files, node\\_modules, temp files, everything. Trace volume jumps from 50MB/min to 12GB/min.\n\nFor Go services, the approach is different because Go doesn't have a runtime agent model:\n\n```go\npackage main\n\nimport (\n \"context\"\n \"go.opentelemetry.io/otel\"\n \"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc\"\n \"go.opentelemetry.io/otel/sdk/resource\"\n sdktrace \"go.opentelemetry.io/otel/sdk/trace\"\n semconv \"go.opentelemetry.io/otel/semconv/v1.24.0\"\n)\n\nfunc initTracer() (*sdktrace.TracerProvider, error) {\n exporter, err := otlptracegrpc.New(context.Background(),\n otlptracegrpc.WithEndpoint(\"otel-collector:4317\"),\n otlptracegrpc.WithInsecure(),\n )\n if err != nil {\n return nil, err\n }\n\n tp := sdktrace.NewTracerProvider(\n sdktrace.WithBatcher(exporter,\n sdktrace.WithMaxQueueSize(2048),\n sdktrace.WithMaxExportBatchSize(512),\n sdktrace.WithBatchTimeout(5*time.Second),\n ),\n sdktrace.WithResource(resource.NewWithAttributes(\n semconv.SchemaURL,\n semconv.ServiceNameKey.String(\"checkout-service\"),\n semconv.ServiceVersionKey.String(\"1.4.2\"),\n semconv.DeploymentEnvironmentKey.String(\"production\"),\n )),\n sdktrace.WithSampler(sdktrace.ParentBased(\n sdktrace.TraceIDRatioBased(0.1),\n )),\n )\n\n otel.SetTracerProvider(tp)\n return tp, nil\n}\n```\n\nThe ` WithBatcher ` config is crucial. The defaults are fine for development, but in production with high throughput, you'll hit queue overflows. Services can drop 40% of spans silently because the batch queue was full and the default behavior is to drop, not block."},{"heading":"Custom spans that actually help","level":2,"url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#custom-spans-that-actually-help","content":"Auto-instrumentation gives you the skeleton. Custom spans give you the meat. But teams sometimes go overboard and create spans for every function call, which just turns your traces into unreadable noise.\n\nA good rule: create custom spans for **business operations** and **external dependencies** that auto-instrumentation doesn't cover.\n\n```go\nfunc (s *CheckoutService) ProcessOrder(ctx context.Context, order *Order) error {\n ctx, span := s.tracer.Start(ctx, \"checkout.process_order\",\n trace.WithAttributes(\n attribute.String(\"order.id\", order.ID),\n attribute.Int(\"order.items_count\", len(order.Items)),\n attribute.String(\"order.payment_method\", order.PaymentMethod),\n attribute.Float64(\"order.total\", order.Total),\n ),\n )\n defer span.End()\n\n // Validate inventory, this talks to another service\n if err := s.validateInventory(ctx, order); err != nil {\n span.SetStatus(codes.Error, \"inventory validation failed\")\n span.RecordError(err, trace.WithAttributes(\n attribute.StringSlice(\"failed_skus\", getFailedSKUs(err)),\n ))\n return fmt.Errorf(\"inventory check: %w\", err)\n }\n\n // Process payment, external gateway\n paymentSpan := trace.SpanFromContext(ctx)\n paymentSpan.AddEvent(\"payment.initiated\", trace.WithAttributes(\n attribute.String(\"gateway\", s.paymentGateway.Name()),\n ))\n\n charge, err := s.processPayment(ctx, order)\n if err != nil {\n span.SetStatus(codes.Error, \"payment processing failed\")\n span.RecordError(err)\n return fmt.Errorf(\"payment: %w\", err)\n }\n\n span.SetAttributes(attribute.String(\"payment.charge_id\", charge.ID))\n return nil\n}\n```\n\nNotice a few things. The ` order.id ` is recorded as an attribute for trace searching by order ID. ` RecordError ` with additional attributes captures which specific SKUs failed. And span events mark significant moments within a single span (payment initiated vs. payment completed)."},{"heading":"Baggage propagation: cross-service context done right","level":2,"url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#baggage-propagation-cross-service-context-done-right","content":"Baggage is the most underused feature of OpenTelemetry. It lets you attach key-value pairs to the context that propagate across service boundaries, without cluttering every span with the same attributes.\n\n```go\n// At the API gateway, inject tenant context\nimport \"go.opentelemetry.io/otel/baggage\"\n\nfunc gatewayMiddleware(next http.Handler) http.Handler {\n return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n tenantID := extractTenantID(r)\n region := resolveRegion(r)\n\n member1, _ := baggage.NewMember(\"tenant.id\", tenantID)\n member2, _ := baggage.NewMember(\"region\", region)\n bag, _ := baggage.New(member1, member2)\n\n ctx := baggage.ContextWithBaggage(r.Context(), bag)\n next.ServeHTTP(w, r.WithContext(ctx))\n })\n}\n\n// In ANY downstream service, read it without knowing how it got there\nfunc (s *InventoryService) CheckStock(ctx context.Context, sku string) (int, error) {\n bag := baggage.FromContext(ctx)\n tenantID := bag.Member(\"tenant.id\").Value()\n region := bag.Member(\"region\").Value()\n\n // Route to tenant-specific inventory\n return s.repo.GetStock(ctx, tenantID, region, sku)\n}\n```\n\nThe collector can also read baggage and add it as span attributes automatically:\n\n```yaml\n# otel-collector config\nprocessors:\n baggage:\n rules:\n - from: baggage\n key: tenant.id\n action: insert\n - from: baggage\n key: region\n action: insert\n```\n\nThis is incredibly powerful for multi-tenant systems. Instead of passing ` tenantID ` through every function signature, it rides the context through the entire request chain."},{"heading":"Jaeger vs Tempo: the real comparison","level":2,"url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#jaeger-vs-tempo-the-real-comparison","content":"Here is an honest breakdown based on production experience.\n\n**Jaeger with Elasticsearch**:\n\n- Setup: Jaeger collector + Elasticsearch cluster (3 nodes, r6g.xlarge)\n- Storage: ~2TB of traces, 7-day retention\n- Monthly cost: ~$2,400 (mostly Elasticsearch)\n- Query performance: Fast for trace ID lookup, slow for tag-based search at scale\n- Operations burden: Elasticsearch index management, JVM tuning, shard rebalancing\n\n**Grafana Tempo with S3**:\n\n- Setup: Tempo (3 replicas in microservices mode) + S3 bucket\n- Storage: ~2TB of traces, 30-day retention (yes, 4x longer)\n- Monthly cost: ~$180 (S3 storage + a bit of compute)\n- Query performance: Trace ID lookup is fast, tag-based search requires TraceQL\n- Operations burden: Minimal, stateless ingesters, object storage does the heavy lifting\n\nThe cost difference is not a typo. Tempo uses object storage (S3, GCS, Azure Blob) instead of a full-text search engine. The trade-off is that you can't do arbitrary field searches, you need to use TraceQL or integrate with Grafana Loki for log-to-trace correlation.\n\n```promql\n# TraceQL, finding slow checkout traces\n{ span.checkout.process_order && duration > 5s }\n\n# With attribute filter\n{ span.checkout.process_order\n && span.order.payment_method = \"credit_card\"\n && status = error }\n```\n\nThe switch from Jaeger to Tempo is often a clear win. The observability bill goes from dreaded to barely noticed."},{"heading":"Trace-based testing: validation through observability","level":2,"url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#trace-based-testing-validation-through-observability","content":"This is one of the most promising developments. Trace-based testing uses actual distributed traces to validate system behavior. Instead of mocking everything, you run real requests and assert on the trace structure.\n\nTracetest, which hooks into the OTEL collector, enables this:\n\n```yaml\n# tracetest definition\ntype: Test\nspec:\n name: \"Checkout creates payment and sends confirmation\"\n trigger:\n type: http\n httpRequest:\n url: http://api-gateway/api/v1/checkout\n method: POST\n body: '{\"items\": [{\"sku\": \"WIDGET-1\", \"qty\": 2}], \"payment\": \"credit_card\"}'\n specs:\n - selector: span[name=\"checkout.process_order\"]\n assertions:\n - attr:order.items_count = 2\n - selector: span[name=\"payment.charge\"]\n assertions:\n - attr:tracetest.span.duration < 2000ms\n - attr:payment.status = \"captured\"\n - selector: span[name=\"notification.send_email\"]\n assertions:\n - attr:email.template = \"order_confirmation\"\n - attr:tracetest.span.duration < 500ms\n```\n\nThis test fires a real HTTP request, waits for the trace to be collected, and then asserts on span attributes across multiple services. It's integration testing on steroids, you're validating not just the response, but the entire internal behavior of the system.\n\nRunning these in CI after deploying to a staging environment catches issues that traditional integration tests miss: a service that returns the right response but takes an extra database round-trip, a message queue consumer that processes events but doesn't propagate the trace context, an HTTP client that silently falls back to a default timeout."},{"heading":"Cost control: the survival guide","level":2,"url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#cost-control-the-survival-guide","content":"Observability costs can spiral out of control faster than you'd expect. Here is a proven three-layer approach:"},{"heading":"Layer 1: Head-based sampling at the SDK","level":3,"url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#layer-1-head-based-sampling-at-the-sdk","content":"```yaml\n# Environment variable, simple and effective\nOTEL_TRACES_SAMPLER=parentbased_traceidratio\nOTEL_TRACES_SAMPLER_ARG=0.1\n```\n\nThis samples 10% of traces at the source. The ` parentbased ` prefix is critical, it means child spans respect the parent's sampling decision. Without it, you get orphaned spans everywhere."},{"heading":"Layer 2: Tail-based sampling at the collector","level":3,"url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#layer-2-tail-based-sampling-at-the-collector","content":"```yaml\n# otel-collector config\nprocessors:\n tail_sampling:\n decision_wait: 30s\n policies:\n - name: errors-always\n type: status_code\n status_code:\n status_codes: [ERROR]\n - name: slow-traces\n type: latency\n latency:\n threshold_ms: 5000\n - name: percentage-sample\n type: probabilistic\n probabilistic:\n sampling_percentage: 5\n```\n\nThis catches 100% of error traces and slow traces, regardless of the head-based sampling decision. The collector needs to buffer spans for the ` decision_wait ` period (30s in this case), which requires memory, budget about 500MB per 10K spans/second."},{"heading":"Layer 3: Attribute limits","level":3,"url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#layer-3-attribute-limits","content":"```yaml\nprocessors:\n attributes:\n actions:\n - key: http.request.body\n action: delete # Never store request bodies\n - key: db.statement\n action: hash # Hash SQL queries instead of storing them\n span:\n max_num_attributes: 64\n max_attribute_value_length: 256\n```\n\nThe ` db.statement ` hashing prevents a scenario where full SQL queries including WHERE clause values, which may contain user emails, get logged. Security and cost in one config change.\n\nWith all three layers, tracing costs drop from ~$8,000/month to ~$800/month while actually improving the debugging experience (because 100% error capture means the traces that matter are never missed)."},{"heading":"The collector topology that scales","level":2,"url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#the-collector-topology-that-scales","content":"Running a single collector is inadvisable. Here is a proven topology:\n\n```\nServices → Agent Collectors (DaemonSet) → Gateway Collectors (Deployment) → Tempo\n (local buffering) (sampling, processing) (storage)\n```\n\nThe agent collectors run as a DaemonSet in Kubernetes. They receive spans over localhost (no network hop), do basic batching, and forward to the gateway. The gateway collectors do the heavy processing: tail-based sampling, attribute manipulation, and export to the backend.\n\n```yaml\n# Agent collector config (lightweight)\nreceivers:\n otlp:\n protocols:\n grpc:\n endpoint: 0.0.0.0:4317\nexporters:\n otlp:\n endpoint: otel-gateway:4317\n compression: zstd\nprocessors:\n batch:\n timeout: 1s\n send_batch_size: 1024\n\n# Gateway collector config (heavy processing)\nreceivers:\n otlp:\n protocols:\n grpc:\n endpoint: 0.0.0.0:4317\nprocessors:\n tail_sampling:\n # ... sampling policies\n attributes:\n # ... attribute processing\nexporters:\n otlp:\n endpoint: tempo:4317\n```\n\nThis separation lets you scale the gateway independently from the agents, and if the gateway goes down, the agents buffer locally without dropping spans from your services.\n\nRunning OpenTelemetry in production isn't hard. Running it *well*, with proper sampling, cost control, meaningful custom spans, and a topology that scales, that's where the craft is. Start with auto-instrumentation, add custom spans for business operations, set up tail-based sampling immediately, and pick a storage backend that won't bankrupt you. Everything else is optimization."},{"heading":"Frequently asked questions","level":2,"url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#questions","content":""},{"heading":"Should I use auto-instrumentation or manual instrumentation?","level":3,"url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#question-1","content":"Start with auto-instrumentation to get baseline visibility, then layer manual spans for business-critical paths. Auto-instrumentation covers HTTP, gRPC, and database calls well but misses domain-specific logic."},{"heading":"Jaeger or Grafana Tempo?","level":3,"url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#question-2","content":"Tempo if you're already in the Grafana ecosystem and want to minimize costs, it uses object storage (S3/GCS) instead of Elasticsearch. Jaeger if you need a standalone solution with mature tooling and don't mind running Elasticsearch or Cassandra."},{"heading":"How do I control OpenTelemetry costs in production?","level":3,"url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#question-3","content":"Three levers: head-based sampling to reduce volume at the source, tail-based sampling at the collector to keep interesting traces, and attribute limits to cap span size. We cut our costs 90% with a 10% head sample rate plus tail-based error sampling."},{"heading":"Tips","level":2,"url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#article-tips","content":"1. **Tip:** Set ` OTEL_TRACES_SAMPLER=parentbased_traceidratio ` with a ratio of 0.1 in production, then use tail-based sampling at the collector to catch 100% of errors regardless.\n2. **Tip:** Never put PII in span attributes. Use ` baggage ` for cross-service context and redact at the collector level with the ` attributes ` processor.\n3. **Tip:** Grafana Tempo with S3 backend costs us ~$180/mo for 2TB of traces. Jaeger with Elasticsearch was $2,400/mo for the same volume. Do the math.\n4. **Tip:** Use ` span.SetStatus(codes.Error, err.Error()) ` religiously, without it, your error traces are invisible in most UIs.\n\n- opentelemetry\n- distributed-tracing\n- observability\n- jaeger\n- tempo\n- microservices\n- otel\n- spans\n- instrumentation\n\n[Back to Blog](<https://lucioduran.com/blog>)"}],"links":[{"text":"Español","url":"https://lucioduran.com/es/blog/opentelemetry-distributed-tracing-microservices"},{"text":"OpenTelemetry","url":"https://opentelemetry.io/"},{"text":"Back to Blog","url":"https://lucioduran.com/blog"}],"structuredData":[{"@context":"https://schema.org","@graph":[{"@type":"BlogPosting","@id":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices#article","mainEntityOfPage":{"@type":"WebPage","@id":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices"},"headline":"OpenTelemetry in Production: Distributed Tracing Implementation and Cost Control","description":"Auto-instrumentation gotchas, custom spans that actually help, baggage propagation patterns, Jaeger vs Tempo showdown, trace-based testing, and how to not go bankrupt on observability costs.","datePublished":"2025-12-11","author":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"publisher":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"image":"https://lucioduran.com/favicons/og-default.png","url":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices","inLanguage":"en","isPartOf":{"@id":"https://lucioduran.com/blog#blog"},"articleSection":"Observability","keywords":"opentelemetry, distributed-tracing, observability, jaeger, tempo, microservices, otel, spans, instrumentation","wordCount":1469,"timeRequired":"PT8M","workTranslation":[{"@type":"BlogPosting","inLanguage":"es","url":"https://lucioduran.com/es/blog/opentelemetry-distributed-tracing-microservices"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://lucioduran.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://lucioduran.com/blog"},{"@type":"ListItem","position":3,"name":"OpenTelemetry in Production: Distributed Tracing Implementation and Cost Control","item":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices"}]},{"@type":"FAQPage","mainEntity":[{"@type":"Question","name":"Should I use auto-instrumentation or manual instrumentation?","acceptedAnswer":{"@type":"Answer","text":"Start with auto-instrumentation to get baseline visibility, then layer manual spans for business-critical paths. Auto-instrumentation covers HTTP, gRPC, and database calls well but misses domain-specific logic."}},{"@type":"Question","name":"Jaeger or Grafana Tempo?","acceptedAnswer":{"@type":"Answer","text":"Tempo if you're already in the Grafana ecosystem and want to minimize costs, it uses object storage (S3/GCS) instead of Elasticsearch. Jaeger if you need a standalone solution with mature tooling and don't mind running Elasticsearch or Cassandra."}},{"@type":"Question","name":"How do I control OpenTelemetry costs in production?","acceptedAnswer":{"@type":"Answer","text":"Three levers: head-based sampling to reduce volume at the source, tail-based sampling at the collector to keep interesting traces, and attribute limits to cap span size. We cut our costs 90% with a 10% head sample rate plus tail-based error sampling."}}]}]}],"contentHash":"2a96eadcff7b71b97bcf03151f583e882a01fb56c2c4d25f4f351bea6539e231","provenance":{"method":"Deterministic serialization of the server-rendered main element","source":"https://lucioduran.com/blog/opentelemetry-distributed-tracing-microservices","hashAlgorithm":"sha256","hashScope":"JSON document fields before contentHash and provenance","sourceHTMLBytes":67626,"contentBytes":14542}}