MVP Factory
EN
ai startup development

Android MediaPipe LLM: session state & KV-cache

KW
Krystian Wiewiór · · 6 min read

Meta description: Deep dive into MediaPipe LLM session management, KV-cache reuse, and context budgeting to keep P99 latency flat on mid-range Android devices.

Tags: android kotlin mobile architecture jetpackcompose


TL;DR

MediaPipe LLM Inference API gives you on-device LLM power, but multi-turn conversations expose a sharp performance cliff. KV-cache invalidation, unbounded context growth, and naive LoRA switching will wreck your P99 latency on anything below a flagship chipset. This post shows you exactly where the budget breaks and how to manage session state to stay flat.


The problem nobody talks about at session init

Building a single-turn prompt wrapper with MediaPipe is straightforward. Building a multi-turn conversation engine that stays responsive across 20 exchanges on a Pixel 6a? That is an entirely different problem.

Most teams get this wrong in the same way: they treat the LlmInference session as a stateless HTTP endpoint and re-initialize it on every turn. That nukes your KV-cache on every call and forces the model to re-process the entire conversation prefix from scratch — every single time.


How MediaPipe manages session tokens

The LlmInference API maintains an internal KV-cache that stores key-value attention pairs for all previously processed tokens. MediaPipe does not expose a stateful “add turn” API — you pass the full accumulated prompt on each call. However, as long as you hold the session open, the runtime uses prefix cache matching: tokens the model has already processed at the same position are served from cache, and only net-new tokens at the tail of the prompt require full attention computation.

The distinction that matters: you pass the full history, but the session only processes the delta.

KV-cache invalidation triggers to avoid:

  • Calling close() and re-creating the session between turns
  • Swapping LoRA adapters mid-session without a full reset
  • Exceeding the model’s maxTokens context window (the cache is fully evicted)
  • Modifying the system prompt between turns
// WRONG: session torn down each turn — full re-inference on every call,
// no KV-cache prefix reuse possible
fun respondToUser(context: Context, options: LlmInference.Options, input: String): String {
    val session = LlmInference.createFromOptions(context, options)
    return session.generateResponse(input).also { session.close() }
}

// CORRECT: session held open across turns.
// MediaPipe requires the full accumulated prompt on each call,
// but the open session reuses KV-cache for all previously seen prefix tokens.
// Only the appended user turn triggers additional attention computation.
class ConversationEngine(context: Context, options: LlmInference.Options) {
    private val session: LlmInference = LlmInference.createFromOptions(context, options)
    private val history = mutableListOf<String>()

    fun respond(userInput: String): String {
        history.add("User: $userInput")
        // Full prompt passed — but prefix tokens are cache hits in the open session
        val prompt = history.joinToString("\n")
        val response = session.generateResponse(prompt)
        history.add("Model: $response")
        return response
    }

    fun close() = session.close()
}

Context window budget: the latency cliff

Context growth is not a linear problem — it is quadratic in attention cost. The table below shows illustrative order-of-magnitude estimates based on profiling patterns for a Gemma 2B INT4 model on a Snapdragon 778G (mid-range) versus a Snapdragon 8 Gen 2 (flagship). Your exact figures will vary with thermal state, memory bandwidth, and quantization scheme, but the shape of the degradation curve holds across chipset generations.

Context TokensTTFT (Flagship)TTFT (Mid-range)P99 Latency Risk
256~180ms~320msLow
512~260ms~610msModerate
1024~440ms~1,400msHigh
2048~850ms~3,200ms+Budget exceeded

If your app targets devices outside the flagship tier, your effective usable context budget is roughly 512–768 tokens before P99 latency crosses the 1-second user-perception threshold.


LoRA adapter switching: the hidden session killer

LoRA adapters let you fine-tune behavior per use case without swapping base models. The catch: switching adapters always invalidates the KV-cache. There is no partial cache preservation across adapter boundaries in the current MediaPipe contract.

sealed class ConversationMode {
    object General : ConversationMode()
    object CodeAssist : ConversationMode()
}

class AdaptiveSession(private val context: Context, private val baseOptions: LlmInference.Options) {
    private var currentMode: ConversationMode? = null
    private var session: LlmInference? = null

    fun ensureMode(mode: ConversationMode): LlmInference {
        if (mode != currentMode) {
            session?.close()
            session = buildSession(mode)
            currentMode = mode
        }
        return session!!
    }

    private fun buildSession(mode: ConversationMode): LlmInference =
        LlmInference.createFromOptions(context, baseOptions) // apply LoRA path per mode
}

Minimize adapter switches per conversation. If you must switch, batch all turns of a given mode together before transitioning.


Practical context window management

Use a sliding window with a hard token ceiling, not a turn count ceiling. Token counts vary wildly by message length.

MediaPipe’s LlmInference exposes a sizeInTokens() method — use it. The 4-chars-per-token heuristic is a reasonable fallback for Latin-script prose, but it breaks down for code-heavy or multilingual prompts where token density differs significantly.

fun buildPromptWithBudget(
    session: LlmInference,
    history: List<String>,
    budget: Int = 600
): String {
    val builder = ArrayDeque(history)
    // Prefer sizeInTokens() for accuracy; fall back to char heuristic if unavailable
    fun tokenCount(text: String) = try {
        session.sizeInTokens(text)
    } catch (_: UnsupportedOperationException) {
        text.length / 4
    }
    var tokenEstimate = tokenCount(builder.joinToString("\n"))
    while (tokenEstimate > budget && builder.size > 2) {
        tokenEstimate -= tokenCount(builder.removeFirst())
    }
    return builder.joinToString("\n")
}

Keep your system prompt pinned outside the sliding window and account for its token cost separately.


A note on developer ergonomics

In my experience building production systems with on-device inference, heap pressure from NNAPI delegate allocation is the silent killer of session stability. Use Android Studio’s Memory Profiler with the NNAPI heap category enabled to track native allocation growth across conversation turns. If you see the NNAPI heap climbing without bounds, you likely have a session lifecycle leak — not a context window problem.


Three things worth repeating

Hold your LlmInference session open across turns. Session re-creation destroys the prefix cache and is the single highest-impact latency regression most teams ship without realizing it. MediaPipe requires full prompt replay by design, but an open session turns the prefix into cache hits — that is where the performance is recovered.

Budget by tokens, not turns, and use sizeInTokens(). Set a hard ceiling of 512–768 tokens for mid-range device targets, implement a sliding window that evicts oldest turns first while preserving the system prompt, and prefer the API’s native token counter over character heuristics.

Treat LoRA adapter switches as session resets. Design your conversation flow to group turns by mode so cache invalidation is predictable and infrequent, not scattered throughout a session.


Share: Twitter LinkedIn