Fine-Tuning Whisper.cpp for On-Device Speech-to-Text in KMP: Quantization Strategies, Audio Preprocessing Pipelines, and the Streaming Architecture That Delivers Real-Time Transcription Without Cloud Costs

TL;DR

Whisper.cpp brings OpenAI’s Whisper model to mobile devices without cloud costs. Using Kotlin Multiplatform’s expect/actual pattern, you can unify transcription logic across Android and iOS while using platform-native audio capture. Int8 quantization is the right pick for mobile — 4x smaller than float16 with under 2% word error rate degradation. A coroutine-driven sliding-window architecture keeps memory under 200MB and delivers partial transcripts fast enough for 60fps UI updates.

---

Why on-device transcription matters now

Cloud speech-to-text APIs charge $0.006-$0.024 per 15 seconds. For any app with sustained audio input — voice notes, accessibility tools, real-time captioning — those costs add up. At 10,000 daily active users averaging 5 minutes of transcription each, you’re looking at $6,000-$24,000/month in API costs alone.

Whisper.cpp, a C/C++ port of OpenAI’s Whisper, runs inference entirely on-device. That means zero per-request cost, offline capability, and lower latency since you skip the network round-trip.

And sometimes the network isn’t even available. Astronauts on the far side of the Moon don’t get to call a REST endpoint. Your users on the subway don’t either. On-device is a cost optimization most of the time, but it’s the only option often enough that it matters.

The KMP audio capture layer

The first challenge is platform-specific audio capture. Kotlin Multiplatform’s expect/actual pattern separates the contract from implementation:

// commonMain
expect class AudioCaptureEngine {
    fun startCapture(sampleRate: Int = 16000, onChunk: (ShortArray) -> Unit)
    fun stopCapture()
}

On Android, the actual implementation wraps AudioRecord. On iOS, it delegates to AVAudioEngine via Kotlin/Native interop. Both feed 16kHz mono PCM frames — exactly what Whisper.cpp expects — into a shared processing pipeline.

I’ve found that keeping the audio format normalization at the platform boundary eliminates an entire class of bugs downstream. Do the conversion once, right at the edge, and everything after that just works.

Quantization: int8 vs int4 on mobile hardware

The mistake I see most often with quantization: teams chase the smallest model without measuring real-world accuracy on their target domain.

Metric	Float16	Int8 (Q8_0)	Int4 (Q4_0)
Model size (base)	148 MB	78 MB	42 MB
Peak RAM usage	~380 MB	~190 MB	~120 MB
Inference speed (Pixel 8)	1.0x	1.6x	2.1x
Inference speed (iPhone 15)	1.0x	1.8x	2.4x
WER delta vs float16	baseline	+1.2%	+4.8%

Int8 wins for production mobile apps. You get a 1.6-1.8x speedup with a barely measurable accuracy hit. Int4 only makes sense if you’re targeting devices with under 2GB available RAM or need the small model to fit where only base would otherwise.

Sliding-window chunked inference

Whisper processes 30-second audio windows. Naively buffering 30 seconds before inference creates unacceptable latency. The fix is a sliding window with overlap:

// commonMain
class ChunkedInferenceEngine(
    private val whisperContext: WhisperContext,
    private val windowSize: Int = 30 * 16000,  // 30s at 16kHz
    private val stepSize: Int = 5 * 16000       // 5s stride
) {
    private val buffer = RingBuffer(windowSize)

    fun feedSamples(samples: ShortArray): PartialTranscript? {
        buffer.write(samples)
        if (buffer.available >= stepSize) {
            val window = buffer.readWindow(windowSize)
            return whisperContext.transcribe(window)
        }
        return null
    }
}

Each 5-second stride triggers inference on the full 30-second window. The 25-second overlap ensures context continuity. Peak memory stays stable (no unbounded buffer growth) and you get partial results every 5 seconds.

Coroutine-based streaming architecture

The last piece connects audio capture to inference to UI rendering using structured concurrency:

fun CoroutineScope.launchTranscription(
    engine: AudioCaptureEngine,
    inference: ChunkedInferenceEngine
) {
    val audioChannel = Channel<ShortArray>(capacity = 64)

    launch(Dispatchers.Default) {
        engine.startCapture { chunk -> audioChannel.trySend(chunk) }
    }

    launch(Dispatchers.Default) {
        for (chunk in audioChannel) {
            inference.feedSamples(chunk)?.let { partial ->
                withContext(Dispatchers.Main) {
                    updateTranscriptUI(partial)  // 60fps-safe
                }
            }
        }
    }
}

The Channel decouples audio capture from inference. The producer never blocks on a slow consumer — trySend drops frames under pressure, which is the right behavior for real-time audio. You want to process what you can and let the rest go. Inference runs on Dispatchers.Default, and only the UI update hops to Main, keeping the render thread free.

Memory budget

Component	Allocation
Whisper int8 model	~78 MB
Inference working memory	~80 MB
Audio ring buffer (30s)	~1 MB
Channel + coroutine overhead	<1 MB
Total	~160 MB

Comfortably under the 200MB target, even on mid-range devices.

What to do with all this

Start with int8 quantization. It has the best accuracy-to-performance ratio on current mobile hardware. Only drop to int4 if memory profiling on your minimum-spec device demands it.

Use a 5-second stride with 30-second windows. You get partial transcripts frequently enough for responsive UI without sacrificing the context Whisper needs for accurate word boundaries.

Decouple capture, inference, and rendering with channels and dispatchers. Structured concurrency in KMP gives you backpressure handling and cancellation for free. Never block the audio thread on model inference.

The whole thing fits in ~160MB of RAM. That’s less than most photo filter apps.

---

Tags: kotlin, kmp, multiplatform, mobile, architecture