eBPF Observability for Mobile Backends: No Code, No APM Bill

SEO Meta: Skip the APM tax. eBPF kernel probes deliver automatic HTTP/gRPC tracing, syscall latency, and CPU flame graphs — zero instrumentation.

---

TL;DR

eBPF lets you attach kernel-level probes to your backend processes and capture HTTP/gRPC spans, syscall latency, and per-container CPU profiles — without modifying a single line of application code. For mobile backends processing millions of requests daily, this eliminates both the instrumentation burden and the commercial APM cost (Datadog, New Relic, Dynatrace regularly run $2k–$5k/month at scale). The tradeoff is operational complexity at the kernel layer, but the economics and signal quality justify it.

---

The problem with traditional APM at scale

SDK-based APM adds 2–8ms per request, version coupling, and a five-figure annual bill. eBPF eliminates all three.

In my experience building production systems that serve mobile clients at high throughput, observability costs follow a predictable curve: they start reasonable, then become a line-item conversation at the executive level. The financial cost is visible. The hidden cost — library coupling, SDK version drift, misconfigured spans generating alert fatigue — rarely makes it into the postmortem.

Approach	Instrumentation Effort	Avg Overhead	Monthly Cost (10B spans/month)*
Datadog APM (SDK)	High	4–8ms/req	$3,000–$6,000
OpenTelemetry (manual)	Medium-High	2–5ms/req	Infrastructure only
eBPF (kernel probes)	Near-zero	<0.5ms/req	Infrastructure only
No observability	None	0ms	$0 now, enormous later

Datadog pricing also scales with host count, not span volume alone — actual bills vary significantly by deployment size.

---

Prerequisites

Kernel requirements: eBPF CO-RE (Compile Once, Run Everywhere) and BTF (BPF Type Format) support require Linux kernel ≥ 5.8. Verify with uname -r and confirm BTF is enabled via /sys/kernel/btf/vmlinux. Most modern cloud provider managed node pools (EKS 1.27+, GKE 1.26+) ship with compliant kernels by default.

---

How eBPF span capture actually works

eBPF programs run inside the Linux kernel in a sandboxed VM. For HTTP/gRPC observability, the relevant probe points are:

• tcp_sendmsg / tcp_recvmsg — capture request/response bytes at the socket layer
• sys_enter_read / sys_exit_read — attribute syscall latency per process/container
• sched_switch — build CPU flame graphs without perf overhead

// Simplified kprobe on tcp_sendmsg to tag spans
SEC("kprobe/tcp_sendmsg")
int trace_tcp_send(struct pt_regs *ctx) {
    struct sock *sk = (struct sock *)PT_REGS_PARM1(ctx);
    u64 pid_tgid = bpf_get_current_pid_tgid();
    // extract cgroup id for container attribution
    u64 cgroup_id = bpf_get_current_cgroup_id();
    // emit to ring buffer for userspace collector
    bpf_ringbuf_output(&events, &event, sizeof(event), 0);
    return 0;
}

A lightweight userspace collector (Pixie, Beyla, or a custom Go daemon) reads the ring buffer, reconstructs spans, and exports to your backend — Jaeger, Tempo, or an OTLP endpoint.

---

Wiring it to a mobile backend

Here’s the architecture for a typical Kotlin/Spring or Go gRPC backend serving mobile clients.

Mobile Client
    │
    ▼
API Gateway (nginx/envoy)
    │
    ▼
Backend Service (no APM SDK)
    │ ← eBPF probes attach here at kernel level
    ▼
eBPF Collector (node-level DaemonSet)
    │
    ▼
OTLP Exporter → Grafana Tempo / Jaeger

The collector runs as a Kubernetes DaemonSet — one per node, not per pod. This matters: you are not paying the per-container memory overhead of a full APM agent on every replica. Note that eBPF collectors require privileged pods or CAP_BPF — scope your RBAC policies accordingly and treat node-level access with the same scrutiny you would apply to any privileged workload.

For gRPC specifically, HTTP/2 framing means you need to parse the binary protocol. Tools like Pixie and Grafana Beyla handle this out of the box. Beyla in particular has first-class Kotlin/JVM support via bytecode-level uprobes when kernel probes alone cannot reconstruct application-layer context.

---

Flame graphs without perf overhead

Continuous CPU profiling via eBPF (perf_event + bpf_get_stackid) adds roughly 0.1–1% CPU overhead versus 5–15% for traditional sampling profilers. For JVM-based backends (Kotlin/Spring), combine kernel stacks with async-profiler output to get full mixed-mode flame graphs — kernel frames and JVM frames in a single view, without a separate profiling agent.

---

Three takeaways

1. Start with Grafana Beyla or Pixie before writing custom eBPF — both support automatic HTTP/gRPC span capture for JVM and Go backends with a single DaemonSet deploy, and both handle gRPC binary framing without manual protocol parsing.
2. Run eBPF collectors at the node level, not as per-pod sidecars, to keep memory and scheduling overhead minimal in Kubernetes environments — and audit the privileged access they require before rolling to production.
3. Combine kernel probes with OTLP export to stay vendor-neutral — your data goes to Tempo or Jaeger today, and you never pay a per-span ingestion fee to a commercial APM vendor regardless of traffic growth.

---

Tags: backend api grpc microservices devops