eBPF observability: zero-code latency tracing in K8s

Meta description: Learn how eBPF kernel programs capture inter-service latency, TCP retransmits, and syscall profiles inside Kubernetes pods—no sidecars, no code changes, production-ready on a budget.

---

TL;DR

eBPF lets you attach lightweight programs to kernel tracepoints and capture deep observability signals—latency, TCP retransmits, syscall profiles—from inside any container without modifying application code, deploying Envoy sidecars, or paying the overhead tax. This covers the BPF map architecture, CO-RE portability, and how to pipe raw kernel traces into Grafana/Tempo for real observability at startup cost.

---

The problem with sidecar-first observability

Most teams reach for Envoy or Istio before understanding the cost. A service mesh sidecar adds 50–200ms cold-start latency, ~50MB of memory per pod, and CPU overhead that compounds as you scale horizontally. For a startup running 40 microservices on a mid-tier Kubernetes cluster, that overhead is not trivial—it is a budget line item.

eBPF changes the calculus entirely.

---

How eBPF captures what you actually need

eBPF (Extended Berkeley Packet Filter) programs run inside the Linux kernel in a sandboxed JIT-compiled VM. They attach to tracepoints, kprobes, and uprobes—giving you direct access to network events, scheduler decisions, and syscall execution without any userspace instrumentation.

For inter-service latency tracing, you attach to tcp_sendmsg and tcp_recvmsg kernel functions and correlate timestamps using BPF maps—shared memory structures readable from both kernel and userspace.

// Simplified BPF program: capture TCP send timestamp
SEC("kprobe/tcp_sendmsg")
int trace_tcp_send(struct pt_regs *ctx) {
    u64 pid = bpf_get_current_pid_tgid();
    u64 ts  = bpf_ktime_get_ns();
    bpf_map_update_elem(&send_start, &pid, &ts, BPF_ANY);
    return 0;
}

SEC("kretprobe/tcp_recvmsg")
int trace_tcp_recv(struct pt_regs *ctx) {
    u64 pid   = bpf_get_current_pid_tgid();
    u64 *tsp  = bpf_map_lookup_elem(&send_start, &pid);
    if (tsp) {
        u64 latency_ns = bpf_ktime_get_ns() - *tsp;
        bpf_perf_event_output(ctx, &events, BPF_F_CURRENT_CPU,
                              &latency_ns, sizeof(latency_ns));
    }
    return 0;
}

The kernel emits events into a perf ring buffer; a Go or Rust userspace daemon reads them and exports metrics. No application change required.

---

BPF map architecture for multi-tenant pods

BPF maps are typed, bounded, and kernel-managed. For multi-service Kubernetes environments, a hash map keyed by (pid, container_id) lets you correlate latency to a specific pod without cross-contamination.

Map type	Use case	Lookup complexity
BPF_MAP_TYPE_HASH	PID → timestamp correlation	O(1) average
BPF_MAP_TYPE_PERF_EVENT_ARRAY	Streaming events to userspace	O(1) per CPU
BPF_MAP_TYPE_RINGBUF	High-throughput trace export	Lock-free, lower overhead than perf
BPF_MAP_TYPE_LRU_HASH	Connection tracking at scale	Auto-evicts stale entries

For TCP retransmit detection, attach to the tcp_retransmit_skb tracepoint and aggregate counts per (src_ip, dst_ip, dport) in an LRU hash. You get network-layer signal with zero application awareness.

---

CO-RE: compile once, run everywhere

The historical barrier to eBPF adoption was kernel version fragmentation. CO-RE (Compile Once, Run Everywhere), enabled by BTF (BPF Type Format) and libbpf, eliminates this. Your eBPF object file ships with type information and the loader relocates field offsets at load time against the running kernel’s BTF data.

In practice: compile against kernel headers once in CI, ship the .o file as a container image layer, and load it on kernels from 5.4 through 6.x without recompilation. This makes eBPF viable in heterogeneous cloud environments where node kernel versions drift across node pools. It is one of those features that sounds like a minor toolchain detail until you have actually managed a fleet with mixed kernel versions—then it becomes the whole reason the approach is practical.

---

Integrating with Grafana and Tempo on a budget

On tight infrastructure budgets, the most cost-effective pipeline looks like this:

1. eBPF daemon (Go + cilium/ebpf library) reads ring buffer events
2. OpenTelemetry Collector receives spans via OTLP, batches and compresses them
3. Grafana Tempo stores traces in object storage (S3/GCS) — roughly $0.02/GB/month vs managed APM at $0.10–$0.30/GB
4. Grafana renders service maps from trace data, no Jaeger or Zipkin dependency

Compared to a full Istio mesh, this stack runs in one DaemonSet pod per node with a typical footprint of ~30MB RAM and <0.5% CPU per core. You get 90% of the observability value at roughly 15% of the infrastructure cost. That gap is large enough that it is worth treating the eBPF path as the default and only reaching for a service mesh when you genuinely need its traffic management features.

---

Observability stack comparison

Approach	Code changes	Memory overhead	Kernel visibility	Setup complexity
Envoy sidecar (Istio)	None	~50MB/pod	L7 only	High
Manual SDK (OTel)	Yes	Minimal	App layer only	Medium
eBPF DaemonSet	None	~30MB/node	L3–L7 + syscalls	Medium
No instrumentation	None	None	None	N/A

---

Three takeaways

1. Start with cilium/ebpf in Go. It has the most mature CO-RE support, active maintenance, and clean APIs for map management and ring buffer consumption. You can ship your first latency histogram in under a day.

2. Key your BPF maps on (netns_ino, pid), not just PID, to safely handle container boundaries in shared-kernel Kubernetes nodes and avoid cross-pod data leakage.

3. Route eBPF traces through OpenTelemetry Collector before Tempo. It gives you batching, retry logic, and the ability to swap backends later without touching your eBPF code.

---

#backend #microservices #devops #kubernetes #docker