On-Device Whisper: NNAPI, INT8, and latency ceilings

SEO Meta Description: Wire a quantized Whisper TFLite model to Android’s NNAPI delegate with the right INT8 tradeoffs, buffer sizing strategy, and runtime fallback chain — and actually hit real-time transcription on device.

Tags: android kotlin mobile architecture multiplatform

---

TL;DR

Running Whisper on-device via TFLite + NNAPI is achievable on mid-range hardware — but only if you get three things right: delegate selection at runtime, mel-spectrogram preprocessing off the main thread, and buffer chunk sizing tuned to your latency target. Get any one wrong and you either miss the real-time ceiling or drain the battery into the ground.

---

Why on-device transcription is worth the complexity

Cloud ASR is fast until it isn’t. Network latency, privacy concerns, and offline scenarios all make it untenable. OpenAI’s Whisper, quantized to INT8 and packaged as a TFLite flatbuffer, brings that capability to Android without a server round trip. The numbers: Whisper tiny.en at INT8 fits under 40 MB on disk and runs well within 300 ms per 30-second chunk on a Snapdragon 8 Gen 1 with the NNAPI DSP delegate active.

The hard part isn’t the model. It’s the pipeline around it.

---

Delegate selection: build the fallback chain at runtime

Most teams treat NNAPI as a binary — either it works or it doesn’t. In production, that breaks fast. You need a tiered fallback chain detected at runtime.

fun buildInterpreter(model: MappedByteBuffer): Interpreter {
    val nnapi = NnApiDelegate(
        NnApiDelegate.Options().apply {
            acceleratorName = "qti-dsp"   // Qualcomm DSP; null = auto
            useNnapiCpu = false
            allowFp16PrecisionForFp32 = true
        }
    )
    val gpu = GpuDelegate(
        GpuDelegate.Options().apply { precisionLossAllowed = true }
    )
    val options = Interpreter.Options().apply {
        try {
            addDelegate(nnapi)
        } catch (e: Exception) {
            try { addDelegate(gpu) } catch (ignored: Exception) {
                setNumThreads(4) // CPU fallback
            }
        }
    }
    return Interpreter(model, options)
}

Run android.os.Build checks and NnApiDelegate.getNnApiErrno() post-inference to catch silent delegation failures — NNAPI will fall back to CPU on unsupported ops without throwing, which is a fun bug to discover in production.

---

INT8 vs FP16: the tradeoff table

Precision	Model size	Avg latency (tiny, 30s audio)	WER delta vs FP32	DSP compatible
FP32	~150 MB	620 ms	baseline	No
FP16	~75 MB	340 ms	+0.3%	Partial
INT8 (PTQ)	~38 MB	190 ms	+1.1–1.8%	Yes

Post-training quantization at INT8 is the sweet spot for NNAPI DSP delegation. The WER increase is real — don’t let anyone tell you quantization is free — but it stays acceptable for most voice UI workloads. FP16 is your best option when targeting GPU delegation on Mali or Adreno without DSP support.

---

Mel-spectrogram preprocessing: keep it off the audio thread

The audio thread budget on Android is tight — typically 4–8 ms per callback at 16 kHz. Mel-spectrogram extraction for Whisper requires 80 mel bins over a 25 ms window with 10 ms hop, which is CPU-expensive. Don’t block the audio callback.

// AudioRecord callback → ring buffer → coroutine on IO dispatcher
audioRecord.setRecordPositionUpdateListener(object : AudioRecord.OnRecordPositionUpdateListener {
    override fun onPeriodicNotification(recorder: AudioRecord) {
        val chunk = ShortArray(CHUNK_SIZE)
        recorder.read(chunk, 0, CHUNK_SIZE)
        ringBuffer.offer(chunk) // lock-free hand-off
    }
    override fun onMarkerReached(recorder: AudioRecord) {}
}, audioHandler)

// Separate coroutine
launch(Dispatchers.Default) {
    for (chunk in ringBuffer) {
        val mel = computeMelSpectrogram(chunk, sampleRate = 16000)
        inferenceQueue.send(mel)
    }
}

---

Buffer chunk sizing: streaming vs. utterance-level batching

Whisper was trained on 30-second fixed-length audio. That’s the core tension for streaming inference.

At the simple end, 30-second utterance chunks give you the best accuracy with the least engineering overhead. Good fit for voice memo or dictation where latency above 1 second is acceptable.

For streaming, you’re looking at 320–480 ms chunks with ~50% overlap and silence detection to gate inference calls. You’ll fire the model 3–4x more often, but perceived latency drops below 500 ms. Use VAD — WebRTC VAD via JNI or SileroVAD TFLite — to suppress inference during silence.

For most production voice UIs, a 1.5-second VAD-gated chunk through the INT8 NNAPI pipeline lands comfortably under 250 ms. That’s the ceiling where transcription starts feeling instantaneous rather than merely responsive.

---

Putting it together

A few things that matter more than the docs suggest:

1. Build the delegate fallback chain on first launch, cache the result in SharedPreferences, and skip the detection cost on subsequent runs. NNAPI DSP is the target; GPU is the backup; CPU at 4 threads is the floor.

2. Quantize to INT8 with PTQ against a representative English speech calibration set. The WER penalty stays below 2% for tiny.en and base.en — acceptable for command-and-control and caption use cases.

3. Gate inference with VAD, not a fixed timer. Firing the model on silence wastes 190 ms of DSP time and heats the device for nothing. A lightweight VAD model costs under 5 ms and halves your inference call volume in real-world usage.