Quantized LoRA Adapters for On-Device LLMs: Hot-Swapping Task-Specific Behaviors on Android Without Reloading the Base Model
TL;DR
You don’t need multiple on-device models for multiple tasks. Load a single 4-bit quantized base model into memory via mmap, then dynamically swap ~2MB LoRA adapter weights to switch between summarization, code review, translation, or any other behavior. All in under 100ms on modern Android hardware. This post covers the architecture, memory math, performance benchmarks, and a Kotlin service layer using Jetpack Lifecycle observers.
The problem: one model per task doesn’t scale on mobile
Most teams attempting on-device LLMs start with the obvious approach: one fine-tuned model per task. A 7B parameter model quantized to 4-bit (Q4_K_M) runs around 3.8-4.2GB in RAM. Need three tasks? That’s 12GB of model weight, which is untenable on any shipping Android device.
The mistake is treating model specialization as a model-level concern when it’s actually a weight-delta concern. QLoRA adapters encode task-specific behavior as small rank-decomposition matrices, typically 1.5-3MB per adapter, layered on top of a frozen base model.
| Approach | RAM for 3 tasks | Cold-start latency | Task-switch latency |
|---|---|---|---|
| 3 separate Q4 models | ~12.0 GB | 8-12s each | 8-12s (full reload) |
| 1 base + 3 LoRA adapters | ~4.2 GB + 6 MB | 8-12s (once) | 50-90ms |
| 1 merged model per task | ~12.0 GB on disk | 8-12s each | 8-12s (full reload) |
Look at the difference. The adapter approach cuts both memory and switching latency by orders of magnitude.
The mmap trick: why adapter swaps are sub-100ms
This works because of how llama.cpp handles model loading on Android. When you load a GGUF model with mmap enabled, the OS maps the file directly into virtual address space without copying it into the process heap. The base model weights get page-faulted on demand from flash storage.
LoRA adapters, by contrast, are small enough to live entirely in resident memory. Swapping adapters means:
- Deallocating the current adapter’s rank-decomposition matrices (~2MB)
- Allocating and loading the new adapter (~2MB)
- No base model teardown or reload
On a Pixel 8 with UFS 4.0 storage, I’ve benchmarked this consistently at 50-90ms. Imperceptible to users. The base model’s memory-mapped pages stay warm in the page cache across swaps.
NEON-optimized matrix fusion for merged inference
At inference time, you don’t want to compute base_output + lora_output as two separate matrix multiplications. The better path is fusing the LoRA weights into the base weights for active layers using ARM NEON intrinsics.
The math is straightforward: for a given layer, the effective weight becomes W_eff = W_base + (alpha/r) * B * A, where A and B are the low-rank matrices and r is the adapter rank. With rank 8-16 (typical for mobile adapters), this fusion takes 15-30ms across all target layers on an 8-core ARM processor using NEON SIMD.
So your actual inference path sees zero overhead from using an adapter versus a natively fine-tuned model. That’s the whole point.
Kotlin service architecture with lifecycle-aware adapter management
In my experience, the lifecycle management is where mobile teams stumble. The model loading and adapter math are well-documented; keeping native memory from leaking when Android kills your activity is not.
class AdapterManager(
private val baseModel: LlamaModel
) : DefaultLifecycleObserver {
private var activeAdapter: LoraAdapter? = null
private val adapterCache = LruCache<String, ByteArray>(3) // cache top 3
suspend fun switchAdapter(taskId: String): Result<Long> {
val startNs = System.nanoTime()
activeAdapter?.detach()
val weights = adapterCache.get(taskId)
?: loadAdapterFromAssets(taskId).also { adapterCache.put(taskId, it) }
activeAdapter = baseModel.attachLoraAdapter(weights)
val elapsedMs = (System.nanoTime() - startNs) / 1_000_000
return Result.success(elapsedMs)
}
override fun onStop(owner: LifecycleOwner) {
activeAdapter?.detach()
activeAdapter = null
}
}A few design decisions worth calling out:
The LruCache holds adapter bytes for up to 3 adapters. At ~2MB each, the 6MB cache cost is negligible, and cache hits eliminate even the file-read latency.
Detaching adapters in onStop prevents leaked native memory when the app backgrounds. This matters because llama.cpp allocations live outside the JVM heap and the garbage collector will never touch them. I’ve seen apps crash after extended sessions because teams forgot this.
The suspend function keeps the swap off the main thread while remaining trivially callable from ViewModels.
This maps well to on-device agentic workflows. Unlike a simple chatbot, an on-device agent can break a goal into steps, make decisions, and take actions across multiple steps. One step might need an intent-analysis adapter, the next a response-generation adapter, and a third a summarization adapter. Sub-100ms swaps make multi-adapter pipelines viable on mobile.
On the topic of developer well-being during long architecture sessions like these, I keep HealthyDesk running in the background for break reminders and guided desk exercises, because no amount of elegant adapter architecture helps if you’re wrecked from six hours of unbroken benchmarking.
Memory accounting: the full picture
| Component | RAM (resident) | RAM (virtual/mapped) |
|---|---|---|
| Base model (Q4_K_M, 7B) | ~800 MB active pages | 4.0 GB mapped |
| Active LoRA adapter | 2 MB | 2 MB |
| Cached adapters (x2) | 4 MB | 4 MB |
| Fusion workspace (NEON) | 12 MB | 12 MB |
| Total | ~818 MB | ~4.02 GB |
The distinction between resident and mapped memory matters a lot here. Android’s mmap means your app’s PSS (Proportional Set Size) reflects only the actively accessed pages, not the full model file. Most OEMs’ low-memory-killer thresholds won’t trigger against ~800MB resident on flagships with 8-12GB RAM.
Takeaways
Load your base model once with mmap, then treat adapters as the unit of task specialization. The per-adapter cost (~2MB, ~70ms swap) makes multi-task on-device LLMs practical today on flagship Android hardware.
Fuse LoRA weights into base weights using NEON SIMD before inference, not during. The 15-30ms fusion cost at swap time eliminates per-token overhead entirely, giving you native fine-tuned performance.
Bind adapter lifecycle to Android component lifecycle. Native memory from llama.cpp lives outside the GC’s reach. Use DefaultLifecycleObserver to guarantee cleanup and prevent the silent memory leaks that crash apps after extended sessions.