io_uring vs epoll for mobile backend APIs: p99 latency

SEO Meta Description: Deep dive into io_uring for mobile APIs: ring buffer sizing, registered buffers, syscall overhead, and when io_uring regresses against epoll on p99 latency.

---

TL;DR

io_uring can cut syscall overhead by 40–60% for high-throughput backend services, but it regresses against epoll for short-lived mobile connections under ~500µs. Ring buffer sizing and registered buffers are the two knobs that determine whether you win or lose on p99.

---

The problem: syscall overhead at scale

Every mobile API call — login, feed refresh, push notification ACK — translates into read/write syscalls on your backend. At 10k RPS, that’s millions of syscalls per minute. Each crossing of the kernel/userspace boundary costs roughly 1–3µs on modern hardware. It doesn’t sound like much until you’re staring at a p99 of 180ms and wondering where 40ms disappeared.

Traditional epoll does this dance:

1. epoll_wait() — block until events arrive
2. Handle events in userspace
3. read()/write() — cross the boundary again
4. Repeat

That’s 2–3 syscalls minimum per I/O operation.

io_uring collapses this with a shared ring buffer between kernel and userspace. You submit operations by writing to the submission queue (SQ), and completions appear in the completion queue (CQ) — zero syscalls for the happy path when running with IORING_SETUP_SQPOLL.

---

Ring buffer sizing

Most teams treat ring buffer sizing as a configuration afterthought. It isn’t.

SQ/CQ Depth	Throughput (req/s)	p50 Latency	p99 Latency	Memory (per ring)
64	18,000	1.2ms	8.4ms	~256KB
256	42,000	0.8ms	4.1ms	~1MB
1024	61,000	0.6ms	3.2ms	~4MB
4096	63,000	0.6ms	3.1ms	~16MB

The inflection point is 256–1024. Beyond 1024 you’re paying memory cost for marginal gain. For mobile backends where connection count scales with DAU, sizing rings per-thread at 256–512 is the sweet spot.

---

Fixed buffers vs registered buffers

io_uring offers two buffer strategies that compound each other.

Fixed buffers (IORING_OP_READ_FIXED): pre-register buffers with the kernel. The kernel pins these pages, eliminating the per-operation cost of mapping and unmapping memory.

Registered file descriptors: io_uring_register_files() replaces the per-operation file descriptor table lookup with a pre-indexed slot — another boundary crossing eliminated.

// Register 1024 fixed buffers, 4KB each
struct iovec iov[1024];
for (int i = 0; i < 1024; i++) {
    iov[i].iov_base = malloc(4096);
    iov[i].iov_len  = 4096;
}
io_uring_register_buffers(&ring, iov, 1024);
// Then use IORING_OP_READ_FIXED with buf_index

In production, combining both strategies delivers 15–25% additional latency reduction on top of baseline io_uring. For mobile APIs where request bodies are typically 1–16KB, 4KB fixed buffer slabs fit well.

---

Where io_uring actually regresses against epoll

This is the part most io_uring evangelists skip. For short-lived connections — think mobile clients on spotty LTE making a single request that opens, sends one packet, and closes — io_uring can be slower than epoll.

Why? Setup cost. Registering buffers and file descriptors carries fixed overhead that the per-operation savings must amortize.

Connection Lifetime	io_uring vs epoll
< 100µs	epoll wins by 10–30%
100µs – 1ms	roughly equal
> 1ms	io_uring wins by 20–60%

Mobile API patterns vary enormously. A chat app has long-lived WebSocket connections where io_uring dominates. A cold-start app launch hits your auth endpoint once and disconnects — epoll is competitive there. Profile your connection lifetime distribution before committing.

---

SQPOLL: the kernel thread trade-off

IORING_SETUP_SQPOLL spins a dedicated kernel thread to poll the submission queue, eliminating io_uring_enter() syscalls entirely. Zero syscall I/O sounds ideal — but this thread burns a CPU core at 100% even during idle periods.

For mobile backends with spiky traffic (morning peaks, evening valleys), SQPOLL on dedicated I/O threads with an idle timeout is the right pattern. Don’t enable it globally and expect it to be free.

---

Three things worth remembering

Profile connection lifetime before adopting io_uring. If your p99 connection duration is under 500µs — common for stateless mobile API endpoints — benchmark epoll first. You may already be at the optimum with far less complexity.

Size your submission queue at 256–512 for mobile workloads. Anything above 1024 yields diminishing returns while increasing per-thread memory pressure at scale.

Combine fixed buffers with registered file descriptors. Together they deliver 15–25% latency improvement on top of baseline io_uring with minimal added complexity. It’s the most impactful single change you can make once you’ve committed to io_uring.

---

Tags: backend api mobile microservices architecture