On-device LLM tool calling on Android: MediaPipe, JSON schema, and the 800ms budget
Meta description: How to wire Android’s MediaPipe LLM Inference API to a streaming function-calling pipeline, covering JSON schema validation, partial token dispatch, and KV-cache reuse on Pixel 9.
Tags: android kotlin mobile architecture api
TL;DR
Running structured function-calling on-device with MediaPipe’s LLM Inference API is achievable on mid-range hardware, but only if you manage three things precisely: JSON schema validation for tool signatures, early dispatch from partial token streams, and KV-cache reuse across multi-turn rounds. Get these right and you can hit sub-800ms round-trips on a Pixel 9.
The problem nobody talks about
MediaPipe’s LLM Inference API brings capable small models directly to Android, which cuts network latency and keeps user data local. Those are real benefits. The problem is that most teams treat on-device models like a stateless Q&A engine. That works fine until you need tool use: function calling, structured output, actual action dispatch. At that point the architecture gets a lot harder.
Here’s what it takes to make it work.
Defining tool signatures with JSON schema
The foundation of any function-calling pipeline is a clean contract between the model and your tools. With MediaPipe, you inject this schema into the prompt context yourself. There’s no native tool-use abstraction.
A minimal tool schema in Kotlin looks like this:
data class ToolSchema(
val name: String,
val description: String,
val parameters: Map<String, ParameterDef>
)
data class ParameterDef(
val type: String, // "string", "number", "boolean"
val description: String,
val required: Boolean = true
)
You serialize this to JSON and prepend it to the system prompt, instructing the model to emit a structured tool_call block when it needs to invoke a function. The schema definition is your single source of truth — validate against it both when parsing model output and when re-injecting tool results.
Validation matters here. A model running quantized on-device will occasionally emit malformed JSON. Build a lenient parser that can recover from trailing commas and unquoted keys before your strict validator runs.
Parsing partial token streams for early dispatch
Waiting for the model to finish generating before dispatching a tool call is a latency killer. Stream token-by-token and trigger dispatch the moment the JSON object is structurally complete, not when the full response finishes.
class StreamingToolParser {
private val buffer = StringBuilder()
private var depth = 0
fun feed(token: String): ToolCall? {
buffer.append(token)
for (char in token) {
when (char) {
'{' -> depth++
'}' -> {
depth--
if (depth == 0) return tryParse(buffer.toString())
}
}
}
return null
}
}
This brace-depth tracker fires as soon as the closing brace of the outermost JSON object lands, typically saving 80–150ms over waiting for the end-of-sequence token. On a Pixel 9, that margin is the difference between a snappy interaction and one that feels sluggish.
Latency budget: what the numbers look like
A realistic breakdown of a single tool-call round-trip on mid-range hardware:
| Stage | Typical Duration |
|---|---|
| Prompt encoding (cached prefix) | 15–30ms |
| Model generation to tool call emit | 300–450ms |
| JSON validation + tool dispatch | 5–15ms |
| Tool execution (local function) | 10–80ms |
| Result re-injection + re-encode | 20–40ms |
| Final response generation | 200–350ms |
| Total (optimistic) | ~550ms |
| Total (pessimistic) | ~965ms |
Generation time dominates, as expected. The surprise is re-encoding after tool result injection: it’s the hidden cost that blows budgets. That’s where KV-cache reuse becomes essential.
KV-cache reuse
Every time you inject a tool result and restart generation, the model re-encodes the entire conversation history from scratch. On a 1B–3B parameter model, that costs 150–300ms per turn. Across a multi-turn workflow, it compounds fast.
The fix is prefix caching. Keep the KV cache for the static portions of your prompt (system instructions, tool schemas) pinned across turns. Only re-encode the dynamic suffix: the new user message, tool call, and result. MediaPipe’s LlmInferenceSession supports session reuse; structure your prompts so the cacheable prefix stays stable.
// Stable prefix — encode once, reuse across turns
val systemPrefix = buildSystemPrompt(toolSchemas)
// Dynamic suffix — re-encode each turn
val dynamicSuffix = buildTurnContext(history, latestResult)
In practice, this reduces multi-turn re-encoding overhead by roughly 60–70%, which is what pulls worst-case latency back under 800ms on typical hardware.
Takeaways
-
Validate tool schemas bidirectionally. Define your JSON schema once and enforce it on both model output parsing and tool result re-injection. Add a lenient pre-pass for quantization-induced malformation before strict validation.
-
Dispatch on stream completion, not sequence end. Implement a brace-depth tracker to detect structurally complete JSON objects mid-stream. The 80–150ms savings matters within an 800ms budget.
-
Pin your KV-cache prefix. Separate system prompt and tool schemas (static) from conversation history (dynamic). Session reuse across turns is the highest-leverage optimization on MediaPipe’s on-device inference path.