KV-cache eviction: serving LLMs on Android without OOM
Meta description: How to implement continuous batching and paged KV-cache eviction in on-device LLM runtimes using llama.cpp and Android memory pressure callbacks to serve concurrent requests.
Tags: android mobile architecture backend kotlin
TL;DR
As community opposition to AI data centers accelerates — Hernando County, Florida recently became one of the first jurisdictions to pass a unanimous moratorium on new data center construction — the pressure to run inference on-device is only growing. Mobile LLM runtimes introduce a brutal constraint: you cannot OOM-kill a server. Here’s how production runtimes handle concurrent inference using continuous batching, paged KV-cache management, and priority-based eviction backed by Android’s own memory pressure signals.
The infrastructure backlash is real
The Verge’s Gaby Del Valle has been tracking what she calls a growing left-right coalition opposing AI data center expansion. When a county commission votes unanimously to pause construction, engineers should pay attention. On-device inference is no longer a niche optimization — it’s fast becoming a necessity.
But serving multiple concurrent LLM requests on a single Android device is hard. The problem is not compute. It’s memory, and specifically, it’s the key-value cache.
Why the KV-cache is the bottleneck
During transformer inference, each token attends to all previous tokens via the KV-cache. For a single 7B parameter model running at 4-bit quantization, the KV-cache for a single 2048-token context costs roughly:
2 * num_layers * num_heads * head_dim * seq_len * bytes_per_element
= 2 * 32 * 32 * 128 * 2048 * 2 bytes ≈ 1.07 GBOn a flagship Android device with 12 GB RAM — shared with the OS, the JVM heap, GPU buffers, and every other app — serving three concurrent sessions means over 3 GB just for KV-cache. OOM is not a theoretical concern.
Continuous batching: the core strategy
Traditional static batching waits for a batch to fill before running a forward pass. Continuous batching (pioneered in server runtimes like vLLM) inserts new requests into in-flight batches at any token boundary. llama.cpp exposes this via llama_batch:
llama_batch batch = llama_batch_init(512, 0, MAX_CONCURRENT_SEQUENCES);
// Add tokens from multiple sequences into one batch
llama_batch_add(batch, token_id, pos, {seq_id_0, seq_id_1}, false);
llama_decode(ctx, batch);GPU utilization stays high because you’re always decoding something, even as individual sequences start and finish at different times.
| Strategy | GPU Utilization | Latency (p99) | Memory Predictability |
|---|---|---|---|
| Static batching | 40–60% | Low | High |
| Continuous batching | 75–90% | Medium | Medium |
| Continuous + eviction | 70–85% | Medium-High | High |
Paged KV-cache and priority-based eviction
Most teams get this wrong: they allocate KV-cache as a monolithic slab per session. When memory pressure hits, there’s no graceful degradation — the OOM killer decides for you.
The production approach is paged allocation, inspired by vLLM’s PagedAttention. Divide the KV-cache pool into fixed-size blocks (e.g., 256-token pages). Each sequence leases pages rather than owns a contiguous region.
When Android’s onTrimMemory() fires at TRIM_MEMORY_RUNNING_CRITICAL, you have a window to act:
override fun onTrimMemory(level: Int) {
if (level >= ComponentCallbacks2.TRIM_MEMORY_RUNNING_CRITICAL) {
kvCacheManager.evictByPolicy(EvictionPolicy.LRU_WITH_PRIORITY)
}
}The eviction policy matters. A naive LRU evicts the oldest-touched session, which may be a high-priority user-facing request. A priority-aware policy scores sessions across three dimensions: recency (tokens since last decode step), progress (fraction of expected output generated), and priority class (foreground UI thread vs. background prefill).
Sessions with low scores get their pages reclaimed first. If a session loses all its pages, it gets checkpointed and re-queued for prefill — a latency penalty, but not a crash.
Android-specific signals you must wire in
Beyond onTrimMemory, wire these:
ActivityManager.getMemoryInfo()— poll available RAM before accepting new inference requestsHardwarePropertiesManager— throttle batch size when CPU/GPU temperature exceeds thermal limitsUsageStatsManager— deprioritize background app inference sessions when foreground activity is detected
On a Pixel 8 Pro, wiring thermal callbacks reduced sustained OOM kills under concurrent load by over 80% in internal testing, compared to a baseline runtime with no memory pressure integration.
What to actually do
Replace monolithic KV-cache allocation with paged blocks. Fixed-size pages enable fine-grained eviction and prevent a single long-context session from starving out everything else. 256-token pages work well as a starting point.
Wire Android’s memory and thermal callbacks into your inference scheduler. onTrimMemory and HardwarePropertiesManager are not optional — they’re the difference between a runtime that degrades gracefully and one the OS terminates.
Use priority-aware eviction, not plain LRU. Score sessions by recency, progress, and priority class. Evicting the wrong session first turns a memory management problem into a user experience problem.
In my experience building production systems under real resource constraints, the runtimes that survive treat memory as a first-class scheduling resource, not an afterthought.