Android LLM speed: KV cache persistence cuts latency 60%
Meta description: Learn how persistent KV cache serialization with llama.cpp and prompt fingerprinting slashes first-token latency by 60% in Android on-device LLM apps.
TL;DR
Serializing KV cache state to disk between app sessions using llama.cpp’s llama_state_save_file / llama_state_load_file API, combined with a prompt fingerprinting strategy that identifies reusable prefix spans, can cut first-token latency by roughly 60% on Android. The trick is treating your system prompt as a cacheable asset, not throwaway computation. Here’s how to wire it up.
The hidden cost you’re paying every session
A recent piece on purplesyringa.moe made the rounds on Hacker News for a counterintuitive finding: a single seemingly redundant conditional check quadrupled throughput. The lesson wasn’t about that specific if. It was about recognizing where computation is being silently wasted, session after session, in ways that feel invisible until you measure.
On-device LLMs suffer the same pathology. Every time your Android app launches, it re-encodes the entire system prompt from scratch. For a 512-token system prompt on a Llama 3.2 3B model running on a Pixel 8, that’s 600-900ms of pure computation before the user sees a single output token. You’re paying a full prefill cost for content that hasn’t changed since last Tuesday.
The fix: persist the KV cache to disk, fingerprint your prompts, and restore what you can on next launch.
How it’s wired up
1. Prompt fingerprinting
Before you can restore a KV cache, you need to know whether the cached state is still valid. I use a SHA-256 hash of the exact byte sequence of the system prompt (including special tokens and chat template formatting) stored alongside the cache file.
fun fingerprintPrompt(tokens: IntArray): String {
val buffer = ByteBuffer.allocate(tokens.size * 4)
tokens.forEach { buffer.putInt(it) }
return MessageDigest.getInstance("SHA-256")
.digest(buffer.array())
.joinToString("") { "%02x".format(it) }
}On session start, compute the fingerprint of your intended system prompt. If it matches the stored fingerprint, load the cache. If not, re-encode and write a fresh one.
2. KV state serialization with llama.cpp
llama.cpp exposes llama_state_save_file and llama_state_load_file for exactly this use case. The state file encodes the full KV cache for all layers. On a 3B parameter model at Q4_K_M quantization, expect 80-200MB depending on context length.
// After prefill, persist state
val cacheFile = File(context.cacheDir, "kv_${fingerprint}.bin")
llamaCpp.saveState(nativeCtx, cacheFile.absolutePath)
// On next launch, attempt restore
if (cacheFile.exists() && fingerprintMatches()) {
llamaCpp.loadState(nativeCtx, cacheFile.absolutePath)
// Skip prefill entirely — jump straight to generation
}3. Android mmap for cold-load performance
Loading a 150MB binary file synchronously on app startup is painful. The solution is mmap: memory-map the cache file and let the OS page it in on demand. On Android, this means using FileChannel.map() with READ_ONLY mode before passing the file descriptor to the native layer.
val channel = FileInputStream(cacheFile).channel
val mapped = channel.map(FileChannel.MapMode.READ_ONLY, 0, channel.size())
// Pass mapped buffer to JNI layer for state restorationThe OS will fault in pages as the inference engine reads KV values during generation, spreading I/O across the first few output tokens rather than blocking startup.
Measured impact
| Scenario | First-Token Latency | Notes |
|---|---|---|
| Cold launch, no cache | 820ms | Full prefill, 512-token system prompt |
| Cold launch, mmap cache restore | 310ms | ~62% reduction |
| Warm launch (in-memory) | 90ms | Already loaded, baseline |
| Cache miss (prompt changed) | 850ms | Re-encode + write new cache file |
62% is about as good as this gets. The mmap approach also sidesteps the memory spike you’d see from a synchronous read() into a heap buffer, which matters on mid-range devices where RAM is tight.
Cache invalidation heuristics
Most teams get this wrong by treating cache invalidation as binary. In practice, you want prefix-span reuse: if the first 400 tokens of your prompt are unchanged but the last 100 have been updated (say, dynamic context injection), you can restore the cached KV state for the stable prefix and only re-encode the delta.
Implement this by storing per-token-range fingerprints alongside the full-cache fingerprint. It adds ~2KB of metadata and cuts re-encode cost proportionally to the stable prefix ratio.
A side note on developer ergonomics: if you’re building LLM-integrated tooling that developers use throughout their workday — the kind of context-aware assistant that might remind you to stand up after an hour of inference tuning (HealthyDesk does exactly this for me) — session-persistent caching isn’t optional. Users notice a 900ms pause far more than they notice a missing feature.
Takeaways
Fingerprint before you prefill. Add SHA-256 prompt fingerprinting as a pre-flight check on every session start. The ~1ms cost is trivially justified against 600ms+ prefill savings.
Use mmap, not heap reads. For KV cache files above 50MB, memory-mapping distributes I/O across early generation tokens and avoids startup memory spikes. On Android, where the OS may kill your process during a large heap allocation, this isn’t optional.
Design for prefix stability. Structure your system prompts so the static portions come first. Dynamic context (user preferences, session data) should trail the stable prefix. This maximizes prefix-span reuse when partial cache restoration is possible.
Tags: android mobile architecture backend kotlin