Abstract
Distributed tracing gives you a timeline of spans across services, but it rarely tells you why a span was slow. Continuous profiling fills that gap by correlating CPU flame graphs and heap profiles directly with trace data, turning a slow span from an opaque latency number into a stack of identifiable function calls. Tools like Grafana Pyroscope, Parca, and the Polar Signals Cloud have made always-on profiling practical at production scale, and their integration with OpenTelemetry trace context is reshaping how teams debug latency regressions in 2025.
Why Tracing Alone Falls Short
A trace tells you that a gRPC call took 340 ms. It does not tell you whether those 340 ms were spent on garbage collection, a tight CPU loop, excessive lock contention, or a slow database query that tracing already captured in a child span. Teams routinely spend hours adding ad-hoc profiling runs, trying to reproduce the latency in a lower environment, or instrumenting code with additional spans until the culprit emerges. The feedback loop is long and the signal is noisy, especially for latency spikes that appear intermittently in production.
How Profiling and Tracing Integrate
Pyroscope’s exemplars mechanism attaches profiling metadata to OpenTelemetry spans via baggage propagation. When a request enters a service, the profiler tags the goroutine or thread with the active trace ID and span ID. At flush time, Pyroscope groups profiling samples by those labels, so a Grafana query can pivot from a slow span directly into the flame graph captured during that exact request window. Parca takes a similar approach using pprof over eBPF, which eliminates the need for language-specific agents in Go, Rust, or JVM services. The result is a linked view: click a span, see the profile. No separate reproduction step is needed.
Operational Tradeoffs
Always-on profiling at 100 Hz per thread adds roughly 1-3% CPU overhead in typical Go and Java workloads, based on Grafana Labs’ published guidance for Pyroscope. The storage cost is significant but compressible; flame graphs deduplicate well and most vendors store them as stacked symbol tables rather than raw samples. Sampling rate tuning is the primary lever: dropping to 19 Hz (a prime number that avoids aliasing with common scheduler ticks) cuts overhead below 1% while retaining enough resolution to identify hot paths. The important operational decision is whether to run the profiler as a sidecar, as a node-level DaemonSet using eBPF, or as an in-process SDK. eBPF-based collection wins on low overhead and language agnosticism, but requires kernel 5.8 or later and appropriate seccomp/AppArmor profiles on hardened clusters.
Rollout Patterns for Platform Teams
Platform teams shipping this capability to product engineers should expose it through the same trace query interface developers already use. The Grafana Tempo-Pyroscope integration accomplishes this natively as of Grafana 11, linking the two data sources so that the span detail panel shows a “View Profile” button when a matching profile exists. Alerting can then fire on p99 CPU profiles exceeding a threshold during a deployment window, giving a deploy-correlated performance signal that complements error rate and latency SLOs. Teams that adopt this pattern typically cut their mean time to identify CPU regressions from hours to under 15 minutes, not because the tooling is magic, but because the data that used to require a separate investigation step is now present alongside the trace that triggered the alert.