eBPF-based continuous profiling in Kubernetes: flame graphs without the overhead

Meta description: Deploy Parca or Pyroscope as a Kubernetes DaemonSet for always-on CPU and latency flame graphs. Correlate hot paths with HPA events — zero code changes, zero APM tax.

---

TL;DR

Traditional APM agents extract a 15–20% CPU and memory tax just to tell you something is slow. eBPF-based continuous profiling — via Parca or Pyroscope — runs at the kernel level, requires zero application instrumentation, and delivers always-on flame graphs you can correlate directly with Kubernetes HPA scaling events. For Go and JVM microservices serving real mobile traffic, this is how you find the hot paths that only emerge under production load.

---

What most teams get wrong about profiling

Most teams profile in development. They fire up async-profiler or pprof locally, generate a flame graph, ship a fix, and move on. The problem is that hot paths under a mobile client’s real request patterns — bursty, geographically distributed, payload-heavy — almost never surface in a local benchmark.

A JVM service under sustained mobile traffic at 2,000 RPS will exhibit lock contention and GC pressure that simply doesn’t appear at 50 RPS on a developer laptop. These aren’t edge cases; they’re the norm. You need always-on profiling. And you need it without instrumentation overhead that defeats the purpose.

---

The eBPF advantage

eBPF programs run in the Linux kernel’s sandboxed VM. A profiling agent using eBPF — specifically perf_event_open syscalls — can sample CPU stack traces across every process on a node at a fixed interval (typically 19–99 Hz) with overhead in the single-digit percentages, not 15–20%.

Approach	Overhead	Code Changes	Always-On	Kernel Version
Agent-based APM (Datadog, New Relic)	15–20% CPU+Mem	Yes (SDK)	Yes	Any
async-profiler / pprof manual	Negligible	Yes (endpoint)	No	Any
eBPF (Parca / Pyroscope)	1–3% CPU	None	Yes	≥ 4.9
eBPF with DWARF unwinding	2–5% CPU	None	Yes	≥ 5.8 (recommended)

For JVM workloads, Pyroscope’s Java agent uses async-profiler under the hood with eBPF-assisted frame unwinding. For Go, both tools leverage native frame pointers, which Go preserves by default since 1.12.

---

Deployment: Parca as a DaemonSet

Parca Agent runs as a DaemonSet — one pod per node — using a privileged security context to load its eBPF programs.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: parca-agent
  namespace: observability
spec:
  selector:
    matchLabels:
      app: parca-agent
  template:
    spec:
      hostPID: true
      hostNetwork: true
      containers:
      - name: parca-agent
        image: ghcr.io/parca-dev/parca-agent:latest
        securityContext:
          privileged: true
        args:
        - --node=$(NODE_NAME)
        - --remote-store-address=parca.observability.svc:7070
        - --remote-store-insecure
        volumeMounts:
        - mountPath: /sys/fs/cgroup
          name: cgroup
        - mountPath: /sys/kernel/debug
          name: kerneldebug
      volumes:
      - name: cgroup
        hostPath: { path: /sys/fs/cgroup }
      - name: kerneldebug
        hostPath: { path: /sys/kernel/debug }

The Parca server stores profiles in a time-series-aware format backed by object storage. Its query UI is Prometheus-compatible, which plugs into the correlation workflow below.

---

Correlating flame graphs with HPA scaling events

This is the part that actually earns its keep. When your HPA scales out a deployment — say, from 4 to 12 replicas under a mobile traffic surge — you want to know why CPU spiked, not just that it did.

Parca exposes a gRPC Query API. With a simple annotation on your HPA events (Kubernetes audit logs → your observability stack), you can timestamp-align a scaling event against the flame graph delta. In practice:

# Fetch profiles for a 5-minute window around a known HPA scale-out
parcactl query \
  --start 2026-11-09T14:30:00Z \
  --end   2026-11-09T14:35:00Z \
  --label-selector 'app=payment-service' \
  --profile-type process_cpu:cpu:nanoseconds:cpu:nanoseconds

In my experience building production systems, the most expensive surprises are rarely in application logic. They’re in serialization (Jackson, Protobuf reflection), connection pool contention, and G1GC humongous allocations that only appear past a certain concurrency threshold. Flame graphs timed to HPA events expose exactly this.

---

JVM-specific: enabling frame pointers

JVMs traditionally omit frame pointers for performance. Without them, eBPF unwinding falls back to DWARF — slower and less accurate. Add this JVM flag:

-XX:+PreserveFramePointer

This carries a measured 1–3% throughput cost. That’s well below what you recover by dropping APM agents entirely, so it’s not really a trade-off.

---

Where to start

Three concrete changes:

1. Deploy Parca or Pyroscope as a DaemonSet. The privileged context requirement is a one-time ops conversation; what you get back is permanent visibility into production behavior without modifying a single line of application code.

2. Annotate HPA events into your observability pipeline. Correlating scale-out timestamps with flame graph snapshots turns autoscaling from a reactive measure into a diagnostic tool — you see the root cause of load-induced CPU growth, not just the symptom.

3. Enable -XX:+PreserveFramePointer on all JVM services. The 1–3% overhead pays for itself immediately. Accurate kernel-level unwinding across your entire fleet, under real traffic, is the foundation everything else builds on.

---

Tags: kubernetes backend microservices devops architecture