WorkManager + on-device LLM: surviving the memory ceiling

Meta description: Wire Android WorkManager to a quantized GGUF model for background inference. Learn constraint graphs, doze exemptions, and the 512MB OOM threshold that reshapes your batching strategy.

Tags: android kotlin mobile architecture cleanarchitecture

---

TL;DR

Running a quantized LLM in the background on Android is a scheduler problem, not a model problem. WorkManager’s constraint graph, doze-mode exemptions, and the 10-minute background execution wall each impose hard ceilings. The 512MB OOM threshold on mid-range devices forces a completely different batching strategy than what works on flagship hardware. Get the orchestration wrong and your inference job silently dies.

---

The problem most teams ignore

In my experience building production systems that run ML inference on Android, model quantization gets all the attention. Teams spend weeks tuning GGUF Q4_K_M vs Q5_K_S, then ship a CoroutineWorker that dies in doze mode on the first overnight test.

WorkManager is not a job queue — it is a constraint negotiation system. Treating it as a simple enqueueUniqueWork wrapper is how you end up with silent failures at 2 AM on a mid-range Samsung.

---

Constraint graphs and model loading

A GGUF model load is expensive. A 3B parameter Q4 model sits at roughly 1.8GB on disk and requires 512–700MB of resident memory depending on context length. The constraint graph must account for this before the Worker even starts.

// NOTE: setExpedited() here is a fallback pattern for short, latency-sensitive
// first-token requests only. For full generation jobs, use a long-running
// worker with getForegroundInfo() instead — see the section below.
val inferenceRequest = OneTimeWorkRequestBuilder<LlmInferenceWorker>()
    .setConstraints(
        Constraints.Builder()
            .setRequiresBatteryNotLow(true)
            .setRequiresStorageNotLow(true)
            .build()
    )
    .setExpedited(OutOfQuotaPolicy.RUN_AS_NON_EXPEDITED_WORK_REQUEST)
    .setInputData(workDataOf("prompt" to prompt, "max_tokens" to 256))
    .build()

setRequiresBatteryNotLow is not optional for inference workloads. On mid-range devices, thermal throttling under low battery can extend a 30-second inference to 4+ minutes, which collides directly with the background execution limit.

---

The 10-minute wall and foreground service thresholds

WorkManager’s long-running task API exists precisely because background execution limits cap standard workers. The threshold that matters:

Worker type	Max execution time	Foreground service required	Doze exempt
Standard Worker	~10 minutes	No	No
Expedited Worker	~3 minutes (quota-limited)	No	Partial
Long-Running Worker	Foreground service lifetime¹	Yes	Yes (with foreground service)

¹ The OS can still reclaim memory from foreground services under extreme pressure; “unbounded” is a simplification.

For inference jobs that exceed token budgets, CoroutineWorker with getForegroundInfo() is the only safe path in production:

class LlmInferenceWorker(ctx: Context, params: WorkerParameters) :
    CoroutineWorker(ctx, params) {

    override suspend fun getForegroundInfo(): ForegroundInfo {
        return ForegroundInfo(
            NOTIFICATION_ID,
            buildInferenceNotification()
        )
    }

    override suspend fun doWork(): Result {
        setForeground(getForegroundInfo())
        return runInference()
    }
}

Without this, Android 12+ will terminate any worker exceeding the background CPU time quota, and you will see no exception — just a silent Result.failure() in your WorkInfo observer.

---

The 512MB OOM threshold: batching strategy by device class

A mid-range device (Snapdragon 6 Gen 1, 4GB RAM) exposes roughly 512MB to a background process before the OOM killer activates. A Pixel 9 Pro with 16GB RAM gives you 1.2–1.8GB in the same scenario. That gap forces two distinct codepaths.

Mid-range (≤512MB budget):

• Load model once, process prompts in a single session
• Cap context window at 512 tokens
• Use Q2_K quantization if latency permits
• Batch size: 1 prompt per Worker invocation, chain with then()

Flagship (>1GB budget):

• Persistent model session across chained workers
• Context window up to 2048 tokens
• Q4_K_M for quality/speed balance
• Batch size: 4–8 prompts per invocation

// Mid-range safe: chain small jobs rather than batching
val chain = WorkManager.getInstance(context)
    .beginUniqueWork("llm_queue", ExistingWorkPolicy.APPEND_OR_REPLACE, firstRequest)
    .then(secondRequest)
    .then(thirdRequest)
    .enqueue()

Detect device class at runtime using ActivityManager.MemoryInfo.totalMem and select the strategy before enqueueing.

---

Doze mode and expedited work quotas

Expedited workers give you partial doze exemption, but the foreground service path in long-running workers is the only guarantee of full exemption. Android also limits expedited execution to roughly 10 minutes per 24-hour window on stock Android — OEMs vary significantly, and apps in lower standby buckets may receive less. Burn through this quota with large models and your afternoon batch runs at standard priority, straight into doze suppression.

The pattern that actually works: use expedited only for the first token (latency-sensitive), then downgrade to a long-running worker for the generation tail. This keeps quota intact while keeping perceived responsiveness high.

---

Conclusion

On-device LLM inference lives or dies on respecting Android’s resource boundaries at every layer — from the OOM killer up through WorkManager’s scheduler. Miss one constraint and the failure is silent by design.

Three takeaways

1. Profile your device fleet before choosing quantization. The 512MB OOM ceiling on mid-range hardware is a hard architectural constraint, not a tuning parameter. Build two batching codepaths from the start.

2. Always implement getForegroundInfo() for any inference worker. Any job that might exceed 3 minutes must be a long-running worker with a foreground service. Silent failures from the background execution limit are nearly impossible to debug in production.

3. Treat expedited work quota as a finite resource. Audit how your WorkManager chains consume the daily quota — it varies by OEM and standby bucket — and reserve it for user-facing latency, not bulk background processing.