Dynamic LoRA adapter loading for on-device LLMs on Android

TL;DR

You can run multiple task-specialized LLMs on a single device without reloading model weights by attaching LoRA adapters at runtime via llama.cpp’s adapter API. The base model loads once; adapters are kilobytes-to-megabytes of delta weights that hot-swap between tasks. The hard part is the memory scheduler and the JNI bridge that makes this ergonomic in Kotlin.

---

Why adapters, not multiple models

Most teams get this wrong the same way: package one fine-tuned model per use case, then wonder why their APK weighs 4 GB and the OOM killer is their most active background process.

The better architecture is one quantized base model (Q4_K_M, ~4 GB for a 7B), plus a set of LoRA adapters — each a few MB of delta weights that specialize the base for a task. Load the base once at app launch. Swap adapters as user context changes.

Rank, size, and overhead at a glance:

Rank	Use case	Adapter size (7B base)	Memory overhead
r=4	Ultra-mobile	~8 MB	Negligible
r=8	Balanced	~16 MB	Low
r=16	Capable	~32 MB	Moderate
r=32	Desktop-tier	~64 MB	High — avoid on mobile

For production mobile, r=8 is the sweet spot: meaningful specialization without blowing your adapter cache budget.

---

The GGUF adapter format

llama.cpp expects adapters as GGUF files with general.type = adapter and adapter.type = lora. Each adapter file encodes the A and B matrices per-layer at the specified rank. You ship these alongside your base model in assets/, or download them on demand to getFilesDir().

Adapter files are self-describing — rank, base model hash, and target architecture are all embedded. Validate the hash against your loaded base model at attachment time. A mismatch causes silent garbage output, not a crash, so don’t skip this check.

---

The JNI bridge design

The adapter lifecycle needs to be first-class in Kotlin. A thin JNI wrapper around three llama.cpp calls is all you need:

object AdapterHandle {
    external fun attach(modelPtr: Long, adapterPath: String, scale: Float): Long
    external fun detach(modelPtr: Long, adapterHandle: Long)
    external fun clear(modelPtr: Long)
}

On the C++ side, attach calls llama_model_apply_lora_from_file, capturing the returned handle. detach removes a single adapter; clear strips all active adapters, restoring the base model. Keep adapter handles as Long opaque pointers in Kotlin — don’t serialize or persist them across JVM restarts.

class AdapterCache(private val modelPtr: Long, private val maxSlots: Int = 3) {
    private val lru = LinkedHashMap<String, Long>(maxSlots, 0.75f, true)

    fun acquire(task: String, path: String): Long {
        return lru.getOrPut(task) {
            if (lru.size >= maxSlots) evictLru()
            AdapterHandle.attach(modelPtr, path, 1.0f)
        }
    }

    private fun evictLru() {
        val victim = lru.entries.first()
        AdapterHandle.detach(modelPtr, victim.value)
        lru.remove(victim.key)
    }
}

---

Priority-based swap scheduling

The subtle failure mode: a background summarization request holds an adapter slot when a foreground chat request arrives needing a different adapter. Without a scheduler, you either block the foreground or corrupt the inference context.

In my experience building production systems, a two-tier priority queue solves this cleanly. Foreground (UI-bound) requests can preempt any background adapter slot; background jobs — notifications, prefetch — use remaining slots and yield on contention.

class AdapterScheduler(private val cache: AdapterCache) {
    private val foreground = PriorityBlockingQueue<AdapterRequest>()
    private val background = PriorityBlockingQueue<AdapterRequest>()

    suspend fun submit(request: AdapterRequest): Flow<Token> {
        val queue = if (request.isForeground) foreground else background
        queue.offer(request)
        return awaitSlot(request) // suspends until a cache slot is available;
                                  // full coroutine dispatch omitted for brevity
    }
}

awaitSlot suspends the coroutine until the cache has a free or preemptible slot, then calls cache.acquire under a mutex before resuming the inference flow. When foreground demand arrives and all slots are full, it evicts the lowest-priority background adapter. The base model never reloads — only the ~16 MB delta weight swaps. Measured cold-swap latency on a Pixel 8 Pro lands under 50 ms at r=8, which is imperceptible behind a loading indicator.

---

Before you ship

Use r=8 for mobile. At ~16 MB per adapter it’s small enough to cache three simultaneously on devices with 6+ GB RAM, and the task specialization is real — not cosmetic.

Validate adapter-to-base compatibility at attach time. The base model hash is in the GGUF header; check it. Silent output degradation from a mismatched adapter is harder to debug than a hard failure.

Add priority-aware eviction before you go to production. A plain LRU cache works fine in development. It won’t hold up once foreground requests start contending with background prefetch — you need deterministic preemption to keep UI inference latency stable.

---

Tags: android kotlin mobile architecture llm