OpenTelemetry in Production: Distributed Tracing Implementation and Cost Control

The auto-instrumentation trap

The typical first step is applying the auto-instrumentation agent without further configuration.

For Node.js with OpenTelemetry, the initial setup appears simple:

// tracing.js, loaded before anything else via --require
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');

const sdk = new NodeSDK({
 traceExporter: new OTLPTraceExporter({
 url: 'http://otel-collector:4317',
 }),
 instrumentations: [getNodeAutoInstrumentations({
 '@opentelemetry/instrumentation-fs': { enabled: false }, // Trust me
 '@opentelemetry/instrumentation-http': {
 ignoreIncomingRequestHook: (req) => {
 return req.url === '/healthz' || req.url === '/readyz';
 },
 },
 })],
});

sdk.start();

See that instrumentation-fs disabled? A common discovery: the collector falls over because the fs instrumentation creates a span for every single file read, including config files, node_modules, temp files, everything. Trace volume jumps from 50MB/min to 12GB/min.

For Go services, the approach is different because Go doesn't have a runtime agent model:

package main

import (
 "context"
 "go.opentelemetry.io/otel"
 "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
 "go.opentelemetry.io/otel/sdk/resource"
 sdktrace "go.opentelemetry.io/otel/sdk/trace"
 semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
)

func initTracer() (*sdktrace.TracerProvider, error) {
 exporter, err := otlptracegrpc.New(context.Background(),
 otlptracegrpc.WithEndpoint("otel-collector:4317"),
 otlptracegrpc.WithInsecure(),
 )
 if err != nil {
 return nil, err
 }

 tp := sdktrace.NewTracerProvider(
 sdktrace.WithBatcher(exporter,
 sdktrace.WithMaxQueueSize(2048),
 sdktrace.WithMaxExportBatchSize(512),
 sdktrace.WithBatchTimeout(5*time.Second),
 ),
 sdktrace.WithResource(resource.NewWithAttributes(
 semconv.SchemaURL,
 semconv.ServiceNameKey.String("checkout-service"),
 semconv.ServiceVersionKey.String("1.4.2"),
 semconv.DeploymentEnvironmentKey.String("production"),
 )),
 sdktrace.WithSampler(sdktrace.ParentBased(
 sdktrace.TraceIDRatioBased(0.1),
 )),
 )

 otel.SetTracerProvider(tp)
 return tp, nil
}

The WithBatcher config is crucial. The defaults are fine for development, but in production with high throughput, you'll hit queue overflows. Services can drop 40% of spans silently because the batch queue was full and the default behavior is to drop, not block.

Custom spans that actually help

Auto-instrumentation gives you the skeleton. Custom spans give you the meat. But teams sometimes go overboard and create spans for every function call, which just turns your traces into unreadable noise.

A good rule: create custom spans for business operations and external dependencies that auto-instrumentation doesn't cover.

func (s *CheckoutService) ProcessOrder(ctx context.Context, order *Order) error {
 ctx, span := s.tracer.Start(ctx, "checkout.process_order",
 trace.WithAttributes(
 attribute.String("order.id", order.ID),
 attribute.Int("order.items_count", len(order.Items)),
 attribute.String("order.payment_method", order.PaymentMethod),
 attribute.Float64("order.total", order.Total),
 ),
 )
 defer span.End()

 // Validate inventory, this talks to another service
 if err := s.validateInventory(ctx, order); err != nil {
 span.SetStatus(codes.Error, "inventory validation failed")
 span.RecordError(err, trace.WithAttributes(
 attribute.StringSlice("failed_skus", getFailedSKUs(err)),
 ))
 return fmt.Errorf("inventory check: %w", err)
 }

 // Process payment, external gateway
 paymentSpan := trace.SpanFromContext(ctx)
 paymentSpan.AddEvent("payment.initiated", trace.WithAttributes(
 attribute.String("gateway", s.paymentGateway.Name()),
 ))

 charge, err := s.processPayment(ctx, order)
 if err != nil {
 span.SetStatus(codes.Error, "payment processing failed")
 span.RecordError(err)
 return fmt.Errorf("payment: %w", err)
 }

 span.SetAttributes(attribute.String("payment.charge_id", charge.ID))
 return nil
}

Notice a few things. The order.id is recorded as an attribute for trace searching by order ID. RecordError with additional attributes captures which specific SKUs failed. And span events mark significant moments within a single span (payment initiated vs. payment completed).

Baggage propagation: cross-service context done right

Baggage is the most underused feature of OpenTelemetry. It lets you attach key-value pairs to the context that propagate across service boundaries, without cluttering every span with the same attributes.

// At the API gateway, inject tenant context
import "go.opentelemetry.io/otel/baggage"

func gatewayMiddleware(next http.Handler) http.Handler {
 return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 tenantID := extractTenantID(r)
 region := resolveRegion(r)

 member1, _ := baggage.NewMember("tenant.id", tenantID)
 member2, _ := baggage.NewMember("region", region)
 bag, _ := baggage.New(member1, member2)

 ctx := baggage.ContextWithBaggage(r.Context(), bag)
 next.ServeHTTP(w, r.WithContext(ctx))
 })
}

// In ANY downstream service, read it without knowing how it got there
func (s *InventoryService) CheckStock(ctx context.Context, sku string) (int, error) {
 bag := baggage.FromContext(ctx)
 tenantID := bag.Member("tenant.id").Value()
 region := bag.Member("region").Value()

 // Route to tenant-specific inventory
 return s.repo.GetStock(ctx, tenantID, region, sku)
}

The collector can also read baggage and add it as span attributes automatically:

# otel-collector config
processors:
 baggage:
 rules:
 - from: baggage
 key: tenant.id
 action: insert
 - from: baggage
 key: region
 action: insert

This is incredibly powerful for multi-tenant systems. Instead of passing tenantID through every function signature, it rides the context through the entire request chain.

Jaeger vs Tempo: the real comparison

Here is an honest breakdown based on production experience.

Jaeger with Elasticsearch:

• Setup: Jaeger collector + Elasticsearch cluster (3 nodes, r6g.xlarge)
• Storage: ~2TB of traces, 7-day retention
• Monthly cost: ~$2,400 (mostly Elasticsearch)
• Query performance: Fast for trace ID lookup, slow for tag-based search at scale
• Operations burden: Elasticsearch index management, JVM tuning, shard rebalancing

Grafana Tempo with S3:

• Setup: Tempo (3 replicas in microservices mode) + S3 bucket
• Storage: ~2TB of traces, 30-day retention (yes, 4x longer)
• Monthly cost: ~$180 (S3 storage + a bit of compute)
• Query performance: Trace ID lookup is fast, tag-based search requires TraceQL
• Operations burden: Minimal, stateless ingesters, object storage does the heavy lifting

The cost difference is not a typo. Tempo uses object storage (S3, GCS, Azure Blob) instead of a full-text search engine. The trade-off is that you can't do arbitrary field searches, you need to use TraceQL or integrate with Grafana Loki for log-to-trace correlation.

# TraceQL, finding slow checkout traces
{ span.checkout.process_order && duration > 5s }

# With attribute filter
{ span.checkout.process_order
 && span.order.payment_method = "credit_card"
 && status = error }

The switch from Jaeger to Tempo is often a clear win. The observability bill goes from dreaded to barely noticed.

Trace-based testing: validation through observability

This is one of the most promising developments. Trace-based testing uses actual distributed traces to validate system behavior. Instead of mocking everything, you run real requests and assert on the trace structure.

Tracetest, which hooks into the OTEL collector, enables this:

# tracetest definition
type: Test
spec:
 name: "Checkout creates payment and sends confirmation"
 trigger:
 type: http
 httpRequest:
 url: http://api-gateway/api/v1/checkout
 method: POST
 body: '{"items": [{"sku": "WIDGET-1", "qty": 2}], "payment": "credit_card"}'
 specs:
 - selector: span[name="checkout.process_order"]
 assertions:
 - attr:order.items_count = 2
 - selector: span[name="payment.charge"]
 assertions:
 - attr:tracetest.span.duration < 2000ms
 - attr:payment.status = "captured"
 - selector: span[name="notification.send_email"]
 assertions:
 - attr:email.template = "order_confirmation"
 - attr:tracetest.span.duration < 500ms

This test fires a real HTTP request, waits for the trace to be collected, and then asserts on span attributes across multiple services. It's integration testing on steroids, you're validating not just the response, but the entire internal behavior of the system.

Running these in CI after deploying to a staging environment catches issues that traditional integration tests miss: a service that returns the right response but takes an extra database round-trip, a message queue consumer that processes events but doesn't propagate the trace context, an HTTP client that silently falls back to a default timeout.

Cost control: the survival guide

Observability costs can spiral out of control faster than you'd expect. Here is a proven three-layer approach:

Layer 1: Head-based sampling at the SDK

# Environment variable, simple and effective
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1

This samples 10% of traces at the source. The parentbased prefix is critical, it means child spans respect the parent's sampling decision. Without it, you get orphaned spans everywhere.

Layer 2: Tail-based sampling at the collector

# otel-collector config
processors:
 tail_sampling:
 decision_wait: 30s
 policies:
 - name: errors-always
 type: status_code
 status_code:
 status_codes: [ERROR]
 - name: slow-traces
 type: latency
 latency:
 threshold_ms: 5000
 - name: percentage-sample
 type: probabilistic
 probabilistic:
 sampling_percentage: 5

This catches 100% of error traces and slow traces, regardless of the head-based sampling decision. The collector needs to buffer spans for the decision_wait period (30s in this case), which requires memory, budget about 500MB per 10K spans/second.

Layer 3: Attribute limits

processors:
 attributes:
 actions:
 - key: http.request.body
 action: delete # Never store request bodies
 - key: db.statement
 action: hash # Hash SQL queries instead of storing them
 span:
 max_num_attributes: 64
 max_attribute_value_length: 256

The db.statement hashing prevents a scenario where full SQL queries including WHERE clause values, which may contain user emails, get logged. Security and cost in one config change.

With all three layers, tracing costs drop from ~$8,000/month to ~$800/month while actually improving the debugging experience (because 100% error capture means the traces that matter are never missed).

The collector topology that scales

Running a single collector is inadvisable. Here is a proven topology:

Services → Agent Collectors (DaemonSet) → Gateway Collectors (Deployment) → Tempo
 (local buffering) (sampling, processing) (storage)

The agent collectors run as a DaemonSet in Kubernetes. They receive spans over localhost (no network hop), do basic batching, and forward to the gateway. The gateway collectors do the heavy processing: tail-based sampling, attribute manipulation, and export to the backend.

# Agent collector config (lightweight)
receivers:
 otlp:
 protocols:
 grpc:
 endpoint: 0.0.0.0:4317
exporters:
 otlp:
 endpoint: otel-gateway:4317
 compression: zstd
processors:
 batch:
 timeout: 1s
 send_batch_size: 1024

# Gateway collector config (heavy processing)
receivers:
 otlp:
 protocols:
 grpc:
 endpoint: 0.0.0.0:4317
processors:
 tail_sampling:
 # ... sampling policies
 attributes:
 # ... attribute processing
exporters:
 otlp:
 endpoint: tempo:4317

This separation lets you scale the gateway independently from the agents, and if the gateway goes down, the agents buffer locally without dropping spans from your services.

Running OpenTelemetry in production isn't hard. Running it well, with proper sampling, cost control, meaningful custom spans, and a topology that scales, that's where the craft is. Start with auto-instrumentation, add custom spans for business operations, set up tail-based sampling immediately, and pick a storage backend that won't bankrupt you. Everything else is optimization.