Zero-copy LLM inference: NNAPI + llama.cpp on Android

Meta description: Learn how to wire llama.cpp’s Android backend to NNAPI via TFLite shim, eliminate JNI marshaling on token tensors, and ensure your Snapdragon NPU actually runs your quantized model.

Tags: android mobile architecture kmp backend

---

TL;DR

Wiring llama.cpp to Android’s Neural Networks API looks straightforward until you profile it. JNI marshaling on each token tensor, silent CPU fallback from the NNAPI delegate, and missed AHardwareBuffer opportunities can erase the NPU’s latency advantage entirely. This post covers the full stack — delegate selection heuristics, shared memory allocation, and zero-copy tensor handoff — so your Snapdragon NPU actually does the work.

---

The problem most teams hit first

Most teams integrate llama.cpp, enable the NNAPI delegate flag, ship it, and assume the NPU is running their quantized model. It almost certainly isn’t.

The NNAPI delegate in the TFLite shim performs capability checks at runtime. If any operator in your model graph is unsupported — common with non-standard quantization schemes like Q4_K_M — the delegate silently partitions the graph, with unsupported ops falling back to CPU. In the worst case, the entire model runs on CPU with the added overhead of the delegation round-trip.

To check what’s actually running, enable NNAPI delegation logging:

// Enable NNAPI delegation logging
val options = Interpreter.Options().apply {
    addDelegate(
        NnApiDelegate(NnApiDelegate.Options().apply {
            setExecutionPreference(NnApiDelegate.Options.EXECUTION_PREFERENCE_SUSTAINED_SPEED)
            setAllowFp16(true)
            // Critical: log accelerator assignment
            setModelToken("llama_q4")
        })
    )
}

Check adb logcat | grep -i nnapi for fallback warnings. If you see OperationNotSupported, you’re paying delegate overhead for zero NPU benefit.

---

Delegate selection heuristics

The NNAPI stack on Snapdragon devices routes ops through the Hexagon DSP driver when available, falling back to GPU (via OpenCL or Vulkan compute), then CPU. The selection isn’t user-controlled — it’s determined by the HAL implementation.

Accelerator	Typical latency (prefill)	Power draw	Quantization support
Hexagon NPU	Lowest	Lowest	INT8, INT4 (varies by SoC)
Adreno GPU	Moderate	Moderate	FP16, INT8
CPU (NEON)	Baseline	Highest	All formats

One hard constraint worth knowing: INT4 (Q4_K) support at the NPU level is SoC-generation dependent. On older Snapdragon 8 Gen 1 devices, Q4_K ops partition to CPU. On Gen 3 and later, the Hexagon v75 architecture includes native INT4 support. For broad NPU coverage today, Q8_0 is the safer target.

---

Bypassing JNI: AHardwareBuffer and zero-copy tensors

The bigger gain comes from eliminating copy-on-transfer between the JVM heap and native memory. Every ByteBuffer.array() call you make on a tensor crosses the JNI boundary and forces a heap allocation.

The fix: allocate your input tensor buffers using AHardwareBuffer in native code, then share the handle to the Java layer via HardwareBuffer. This gives you a single memory region accessible by CPU, GPU, and DSP simultaneously — no copies, no JNI marshaling per token.

// Native side — allocate shared buffer for token embeddings
AHardwareBuffer_Desc desc = {
    .width  = kEmbeddingDim,
    .height = kMaxSeqLen,
    .layers = 1,
    .format = AHARDWAREBUFFER_FORMAT_BLOB,
    .usage  = AHARDWAREBUFFER_USAGE_CPU_WRITE_OFTEN |
              AHARDWAREBUFFER_USAGE_GPU_DATA_BUFFER |
              AHARDWAREBUFFER_USAGE_DSP_COMPUTE
};
AHardwareBuffer* buffer = nullptr;
AHardwareBuffer_allocate(&desc, &buffer);

// Map for CPU write (token IDs)
void* data = nullptr;
AHardwareBuffer_lock(buffer, AHARDWAREBUFFER_USAGE_CPU_WRITE_OFTEN,
                     -1, nullptr, &data);
memcpy(data, token_ids, token_count * sizeof(int32_t));
AHardwareBuffer_unlock(buffer, nullptr);

On the Kotlin side, wrap this in a HardwareBuffer and hand it directly to the NNAPI delegate input. Zero copy, zero JNI per-token cost.

---

Wiring it into llama.cpp

llama.cpp’s Android backend doesn’t natively expose an NNAPI path yet. The practical approach is a TFLite shim: export your GGUF model’s attention layers as TFLite flatbuffers, run them through the NNAPI delegate for prefill, and use llama.cpp’s native NEON path for autoregressive decode where NNAPI latency per-step hurts more than it helps.

This hybrid split — NNAPI for prefill, CPU for decode — is how most serious on-device LLM stacks are built. Prefill is a large matrix multiply, ideal for NPU batch throughput. Decode is memory-bound and sequential; NPU scheduling overhead dominates there, so it doesn’t pay off.

---

Three things to get right before shipping

1. Verify delegation. Run adb logcat | grep nnapi in CI against your target SoC. Silent CPU fallback is the most common hidden regression in on-device LLM deployments.

2. Use AHardwareBuffer for tensor memory. Allocate embedding and KV-cache buffers natively with AHARDWAREBUFFER_FORMAT_BLOB and DSP_COMPUTE usage flags. Per-token JNI crossings compound with sequence length — eliminate them entirely.

3. Split prefill and decode across accelerators. Route prefill through NNAPI (NPU/GPU) and autoregressive decode through llama.cpp’s NEON path. Each phase goes to the accelerator it actually benefits from.