Abstract

Head-based sampling - the decision to sample or drop a trace made at the first span - is operationally simple but throws away information about exactly the requests you most want to keep: errors, slow outliers, retries. Tail-based sampling defers that decision until the complete trace is assembled, so the collector can inspect outcome data like final status codes and end-to-end latency before deciding whether to forward or drop. The OpenTelemetry Collector’s tailsampling processor makes this viable in production, but running it correctly at scale requires understanding its stateful nature, the consequences of span fanout across multiple collector replicas, and the memory budgeting that prevents the collector itself from becoming a reliability risk.

How the Tail Sampling Processor Works

The tailsampling processor buffers incoming spans in memory, keyed by trace ID, for a configurable decision wait time - typically 10 to 30 seconds, long enough for the slowest downstream service in the call graph to emit its spans. At decision time, the processor evaluates a chain of policies against the complete (or near-complete) span set: status_code policy to capture errors, latency policy to capture p99+ outliers, string_attribute policy to keep spans for specific user IDs or tenant IDs, and a probabilistic policy that falls through to sample a percentage of everything else. The chain evaluates in order and the first matching policy determines the outcome. Policies are composable with and and or operators; a common pattern is (error OR slow) OR (random 1%) so that the bulk of healthy traffic is sampled at low rate while full coverage is retained for anything abnormal.

The Load Balancing Constraint

The processor is stateful: all spans for a trace must arrive at the same collector instance or the decision cannot be made correctly. A naive round-robin load balancer in front of a collector pool violates this constraint and produces partial traces, incorrect drop decisions, or both. The solution is the loadbalancing exporter in the collector itself - it hashes the trace ID and routes spans consistently to the same downstream collector replica. This means deploying a two-tier collector topology: a first tier of stateless collectors that receive spans from SDKs and forward them through the load balancing exporter, and a second tier of stateful tail-sampling collectors. Kubernetes makes this topology straightforward with two separate Deployments behind two Services, but the configuration YAML grows substantially and must be tested under partition scenarios to confirm that collector restarts do not cause trace ID remapping that drops complete traces.

Memory Budgeting and the Spill Problem

Each in-flight trace consumes memory for the duration of the decision window. At high request rates - say, 50,000 requests per second across a fleet of 200 services - the collector’s heap grows rapidly. The tailsampling processor exposes max_traces_per_second and the decision window together determine peak memory. A rough formula: peak spans = (rps) x (average spans per trace) x (decision window in seconds). At 50k rps, 20 spans per trace average, and a 15-second window, that is 15 million spans in flight simultaneously. At 1-2 KB per span, that is 15-30 GB of heap pressure. Horizontal scaling of the tail-sampling tier and careful tuning of the decision window - trading completeness for memory - are both necessary. The Grafana Tempo project’s in-process tail sampling (as opposed to collector-side) shifts this memory cost into the application tier where it is more predictable, but at the cost of requiring the Tempo SDK rather than the standard OTel SDK.