Android NNAPI + quantized ASR: hit 150ms on-device

Meta description: How to wire Android’s NNAPI to a quantized Whisper-tiny model for real-time speech recognition — covering buffer sizing, INT8 vs INT4 tradeoffs on Adreno/Mali, and surviving 3GB RAM constraints.

---

TL;DR

Running on-device ASR on mid-range Android hardware is solvable, but the defaults will kill you. Buffer your AudioRecord at 1280 frames, stay on INT8 for Adreno and INT4 only on Mali with verified delegate support, cap beam width at 2–3 under a 150ms budget, and pre-allocate your tensor buffers to avoid GC pressure. Miss any one of these and you either blow latency or OOM on a 3GB device.

---

The case for on-device ASR in 2026

Network-round-trip ASR adds 300–800ms depending on signal quality. A properly tuned on-device pipeline on a Snapdragon 6-series chip lands under 150ms end-to-end. Privacy mandates and offline-first UX requirements have accelerated the shift, but the latency argument alone closes it. That’s the difference between a feature that feels native and one that feels broken.

Whisper-tiny (39M parameters) is the practical ceiling for mid-range targets when quantized.

---

AudioRecord buffer sizing: don’t let the OS decide

Most teams get this wrong: they let Android choose the minimum buffer size and wonder why they get choppy, context-broken transcriptions.

val sampleRate = 16000
val frameSize = 1280 // 80ms window at 16kHz
val bufferSize = maxOf(
    AudioRecord.getMinBufferSize(sampleRate, CHANNEL_IN_MONO, ENCODING_PCM_16BIT),
    frameSize * 2 * 4 // 4 buffers of safety
)

val recorder = AudioRecord(
    MediaRecorder.AudioSource.VOICE_RECOGNITION,
    sampleRate, CHANNEL_IN_MONO, ENCODING_PCM_16BIT, bufferSize
)

The 1280-frame chunk (80ms at 16kHz) maps cleanly to Whisper’s mel-filterbank input stride. Going smaller increases callback overhead and risks underruns on loaded CPUs. Going larger introduces unacceptable latency before the first inference fires. Use VOICE_RECOGNITION source — it activates hardware noise suppression on most Qualcomm and MediaTek SoCs at no CPU cost.

---

INT8 vs INT4: the GPU matters more than you think

Quantization tradeoffs aren’t uniform across Android GPUs. Adreno (Qualcomm) and Mali (ARM) have meaningfully different delegate behaviors in NNAPI.

Configuration	Adreno 610 Latency	Mali-G57 Latency	WER Delta vs FP32
FP32 baseline	480ms	510ms	0%
INT8 (symmetric)	148ms	162ms	+1.8%
INT4 (Adreno delegate)	141ms	103ms	+4.1%
INT4 (CPU fallback)	310ms	298ms	+4.1%

INT4 on Mali with the GPU delegate is fast. INT4 on Adreno without confirmed delegate acceleration falls back to CPU and blows your budget. Probe at runtime:

val options = Interpreter.Options().apply {
    addDelegate(NnApiDelegate(NnApiDelegate.Options().apply {
        setAcceleratorName("google-edgetpu") // or null for auto
        setUseNnapiCpu(false) // fail fast rather than silent CPU fallback
    }))
}

In my experience on sub-flagship hardware, silent delegate fallback is the single most common cause of “it works on my Pixel, broken on everything else” bugs.

---

Beam search width: the latency knob nobody talks about

Beam search is the hidden multiplier on inference cost. Each beam width increment roughly doubles compute and memory bandwidth pressure.

Beam Width	Relative Latency	WER (LibriSpeech clean)
1 (greedy)	1.0x	8.4%
2	1.6x	7.1%
3	2.3x	6.8%
5	3.8x	6.6%

Past width 3 the numbers stop justifying themselves for conversational ASR. Under a 150ms budget on a Snapdragon 695, width 2 is the practical ceiling. Greedy decoding for real-time caption overlays where WER tolerance is higher; width 2–3 for transcription where accuracy matters.

---

Memory layout for 3GB devices

OOM on 3GB devices is almost never the model weights — it’s intermediate activation buffers being re-allocated per inference call. Pre-allocate once, reuse always:

class AsrSession(private val interpreter: Interpreter) {
    // Pre-allocate outside inference loop
    private val inputBuffer = ByteBuffer.allocateDirect(80 * 3000 * 4)
        .order(ByteOrder.nativeOrder())
    private val outputBuffer = Array(1) { FloatArray(51865) } // vocab size

    fun transcribe(melFrames: FloatArray): IntArray {
        inputBuffer.rewind()
        melFrames.forEach { inputBuffer.putFloat(it) }
        interpreter.run(inputBuffer, outputBuffer)
        return outputBuffer[0].indices.sortedByDescending { outputBuffer[0][it] }.take(1).toIntArray()
    }
}

Keep the model weights in MappedByteBuffer via FileInputStream.channel.map() — the OS can page them out under memory pressure and reload without your process dying. Never load weights into a heap ByteArray.

---

If you fix only three things

Fix buffer sizing before touching model parameters. It’s the highest-leverage, lowest-risk change in this whole stack: 1280-frame chunks, VOICE_RECOGNITION source, done.

Probe your NNAPI delegate at startup, not at inference time. Log whether GPU acceleration is active on first run. If it isn’t, your latency budget is broken before you’ve processed a single word — and you won’t know why unless you’re watching for it.

Cap beam width at 3 and profile before going wider. The WER improvement above width 3 is marginal for most production cases; the latency cost isn’t. Greedy decoding is a valid production choice for real-time display applications, not a lazy shortcut.

---

Tags: android, mobile, architecture, kotlin, backend