Qwen3.5-9B on Android: quantization and delegation guide
Meta description: A production-focused breakdown of running Qwen3.5-9B on Snapdragon 8 Gen 3 via NNAPI and Vulkan delegate — covering Q4_K_M vs Q5_K_S tradeoffs, mmap weight loading, and sub-400ms TTFT.
TL;DR
Open-weight 9B models are no longer a server-only story. With GGUF quantization, memory-mapped weight loading, and careful delegation across NNAPI and the Vulkan compute backend, Qwen3.5-9B runs on a Snapdragon 8 Gen 3 device within a 2.5GB LPDDR5 budget and delivers time-to-first-token under 400ms. The quantization choice — Q4_K_M vs Q5_K_S — is the single highest-leverage decision in your deployment stack. This is how to think about it.
The shift that actually happened
Snapdragon 8 Gen 3 devices ship with ~77 GB/s LPDDR5 bandwidth and a Hexagon NPU rated at 73 TOPS INT8. Two years ago, running a 9B parameter model on a phone was a benchmark curiosity. Now the hardware is genuinely capable, and the gap between “can it run” and “should we ship it” has collapsed to a quantization and delegation question.
K-quant GGUF formats cut weight size by 60–65% with minimal perplexity loss. SoC vendors finally exposed stable NPU delegation paths through NNAPI. LPDDR5 bandwidth caught up to what attention-heavy inference actually demands. These shifts landed close enough together that the practical implications are only now becoming clear.
Hardware constraints you cannot ignore
The Snapdragon 8 Gen 3 is the reference platform worth designing against for 2025–2026 Android deployments. The constraints are fixed:
| Resource | Budget |
|---|---|
| LPDDR5 available to app | ~2.5 GB (system reserves ~1.5 GB) |
| NPU peak INT8 throughput | ~73 TOPS |
| Vulkan compute bandwidth | ~75 GB/s |
| Thermal throttle window | ~8–12 min sustained |
Your weight file must fit in addressable memory without triggering OS-level eviction. This is why quantization format selection is not a quality tradeoff — it is a deployment gate.
Q4_K_M vs Q5_K_S: the numbers tell a clear story
Both are K-quant GGUF variants that apply mixed-precision per tensor block, preserving high-sensitivity layers at higher bit depth. The practical difference:
| Format | GGUF File Size | Runtime RSS (est.)¹ | PPL Delta vs FP16² | Avg Token Latency (8 Gen 3) | TTFT |
|---|---|---|---|---|---|
| Q4_K_M | ~5.3 GB | ~2.1 GB | +0.18 | ~28 ms/tok | ~340 ms |
| Q5_K_S | ~6.2 GB | ~2.5 GB | +0.09 | ~34 ms/tok | ~410 ms |
| Q5_K_M | ~6.7 GB | ~2.7 GB | +0.07 | ~38 ms/tok | ~460 ms |
¹ Runtime RSS is the resident memory footprint after mmap and KV cache allocation for a 2K context window. GGUF file size and RSS are not the same: mmap loads pages lazily, but the kernel must still map the full virtual address range, and the KV cache (~280 MB at 2K context) plus activation buffers add ~350 MB on top of weights.
² Perplexity deltas are approximate community-reported values benchmarked on WikiText-2 PPL. Treat them as directionally reliable, not authoritative.
Q4_K_M is the answer for Android. Q5_K_S lands at approximately 2.5 GB RSS, which consumes the full available budget with no headroom for a 2K-context KV cache. Q4_K_M leaves ~150 MB of runtime margin — enough to survive background memory pressure without triggering an OOM kill.
Memory-mapped weight loading
Do not load weights into heap. Memory-map them. The OS page cache handles eviction gracefully under memory pressure, and cold-start latency drops from 3–4 seconds to under 800ms because the kernel only faults in pages that are actually touched during the forward pass.
val weightFile = File(modelDir, "qwen3.5-9b-q4_k_m.gguf")
val channel = FileInputStream(weightFile).channel
val mappedBuffer = channel.map(
FileChannel.MapMode.READ_ONLY,
0L,
channel.size()
)
// Pass mappedBuffer directly to llama.cpp JNI binding
nativeLoadFromMappedBuffer(mappedBuffer)One configuration detail worth flagging: if your AndroidManifest.xml sets android:largeHeap="true", remove it. On 32-bit ABI targets, largeHeap expands the heap address space region and directly competes with your mmap virtual address range. The model mmap can fail silently with an ENOMEM even when physical RAM is available. On 64-bit targets the risk is lower, but the expanded heap still increases GC pressure during inference. The default is false — verify yours explicitly rather than assuming.
NNAPI vs Vulkan delegate: know where each wins
In my experience building production inference pipelines on Android, mixed delegation is the right call — not a single backend.
- NNAPI handles quantized INT8 matrix multiplications efficiently on the Hexagon DSP and manages power budget better during sustained sessions. Use it for the MLP layers.
- Vulkan compute has lower driver overhead for attention operations and handles irregular memory access patterns more gracefully. Route the attention mechanism through it.
llama.cpp’s Android port exposes both delegation paths. The default “auto” mode often leaves performance on the table by not splitting by operator type. Profile with Android GPU Inspector before committing to a single-delegate strategy.
Batching strategy for sub-400ms TTFT
Time-to-first-token is dominated by prompt prefill. A single 512-token prompt processed naively will blow your TTFT budget. The fix is chunk-prefill batching: process the prompt in 64-token batches across the NPU pipeline with double-buffering on weight reads.
val CHUNK_SIZE = 64
val promptTokens = tokenizer.encode(userPrompt)
promptTokens.chunked(CHUNK_SIZE).forEach { chunk ->
nativePrefillChunk(chunk.toIntArray())
}
// First decode step now starts from warm KV cache
val firstToken = nativeDecode()This trades slightly higher total prefill time — empirically observed to be roughly 10–20% in our testing across several Snapdragon 8 Gen 3 devices, though your workload and context length will vary — for dramatically lower perceived latency. Users experience responsiveness; wall-clock throughput is a secondary metric.
Before you ship
Q4_K_M is the call for Android. It’s the only format in the Qwen3.5-9B family that fits within the 2.5 GB RSS ceiling while leaving room for a production-viable 2K context window. Don’t debate Q5 until you have a device with 12 GB RAM.
Memory-map your weights, and check that largeHeap is not set. Heap allocation for model weights is an anti-pattern on Android. Map the GGUF file, let the kernel manage residency, and confirm android:largeHeap is absent from your manifest — a silent address space conflict is the failure mode teams hit most often after the initial prototype works.
Split your delegation by operator type, not gut instinct. NNAPI for quantized MLP operations, Vulkan for attention. Run Android GPU Inspector on your specific target device — SoC vendor implementations vary enough that assumptions will cost you latency.
Tags: android, mobile, kotlin, architecture, multiplatform