NNAPI Function Calling: On-Device LLM Tool Use on Android

SEO Meta Description: Wire a quantized on-device LLM to Android’s NPU for real-time function calling. Covers schema injection, token budgets, fallback parsing, and 3B vs 7B tradeoffs.

---

TL;DR

Running structured output generation — function calling — from a quantized LLM on Android is achievable, but the path is narrower than most teams assume. The three constraints that determine whether you ship or stall: JSON schema injection at the prompt prefix, a hard token budget that enforces sub-300ms tool-call latency, and a fallback parser when the model drifts. Get all three right, and you have a viable on-device agent tier.

---

A note on NNAPI deprecation

Before going further: the NNAPI NDK APIs (ANeuralNetworks*) were deprecated as of Android 13 (API 33). If you are starting a new project today, do not build directly on top of them. The current production path is LiteRT (Google’s rebrand of TensorFlow Lite) or the MediaPipe LLM Inference API, both of which delegate to the NPU through stable, maintained abstractions. The architectural patterns in this article — schema injection, token budgets, fallback parsing — apply regardless of which runtime you use. Where I reference “NNAPI-delegated inference,” read that as the underlying hardware path, not the deprecated NDK surface.

---

Why on-device function calling is worth the pain

In my experience building production systems across mobile and backend, the gravitational pull toward the cloud is strong — and usually correct. But for agent workloads where the tool call itself involves sensitive user context (location, purchase history, sensor data), round-tripping to a remote LLM introduces both latency and a data-residency problem you do not want to own.

A single round-trip to a hosted model endpoint — including network, inference, and deserialization — realistically lands between 800ms and 2,500ms on a mid-tier mobile connection. NPU-delegated inference via LiteRT, with a properly quantized 3B model, can execute a single forward pass in the 150–250ms range. That delta is the difference between a responsive agent and a loading spinner.

---

The architecture: three layers that have to work together

Prompt Builder → [Schema Prefix + User Turn]
       ↓
NPU-Delegated Inference (INT4/INT8 quantized weights, via LiteRT)
       ↓
Token Budget Gate (hard stop at N tokens)
       ↓
Structured Output Parser → JSON tool call
       ↓
Fallback Heuristic Parser (regex + key extraction)

Layer 1: schema injection at the prompt prefix

The model does not know your tool schema unless you tell it — every time. Inject the schema as a system-level prefix before the user turn, serialized as compact JSON. Do not use verbose descriptions; token count directly affects your latency ceiling.

The example below uses the Llama 2 chat format ([INST]/[/INST] special tokens). If you are using Gemma, Phi, Mistral, or another model family, substitute the correct special tokens for your architecture — the structural principle is identical, but the delimiters are not interchangeable.

// Llama 2 chat format — replace [INST]/[/INST] with your model's special tokens
val schemaPrefix = """
[TOOLS]
{"name":"search_products","params":{"query":"string","category":"string|null"}}
[/TOOLS]
[INST]
""".trimIndent()

val fullPrompt = schemaPrefix + userMessage + "\n[/INST]"

Keep your tool schema under 80 tokens. Every token in the prefix is a token not available for reasoning, and every token generated is time on the clock.

Layer 2: token budget enforcement

This is what most teams get wrong. They treat max token generation as a model configuration detail rather than a latency contract. It is a hard constraint.

Model Size	Quantization	NPU Tier	Tokens/sec	Budget for <300ms
3B	INT4	Mid-range NPU	~40–60	~12–18 tokens
3B	INT8	High-end NPU	~55–80	~16–24 tokens
7B	INT4	Flagship NPU	~25–40	~7–12 tokens
7B	INT8	Flagship NPU	~30–50	~9–15 tokens

A tool call response — the JSON your agent needs to act on — must fit within that budget. This means your output schema must be minimal: no prose, no explanation, just the structured call.

// Inline the budget math directly — no abstraction needed
val tokensPerSec = 50.0 // measure on your target device
val maxTokens = (targetLatencyMs / 1000.0 * tokensPerSec).toInt()
inferenceSession.generate(prompt, maxNewTokens = maxTokens)

Layer 3: fallback parsing when the model drifts

Quantized models drift from their schema. This is not a flaw you eliminate — it is a distribution property you engineer around. Build a two-pass parser:

1. Strict JSON deserialization. If it succeeds, you are done.
2. Regex extraction of known field names and values. Extract "name" and any recognized parameter keys. Reconstruct a best-effort tool call.

If both passes fail, surface a graceful degradation — fall back to a keyword-based routing layer or escalate to a cloud call. Do not silently swallow parse failures.

---

3B vs 7B: the device tier decision

The model size question reduces to a single axis: target device tier.

A 3B INT4-quantized model fits comfortably in 2–2.5GB of RAM, which covers mid-range devices from the last three years. A 7B model demands 4–5GB, which effectively limits you to flagship hardware released in the past 18 months. For function calling specifically — where output length is intentionally short — the 3B model’s latency advantage compounds across multiple agent turns.

One factor that per-token benchmarks obscure: cold-start load time. A 3B INT4 model loading from eMMC storage can add 800ms–2s of latency on the first inference call. For agent workloads with infrequent invocations, this initialization cost can dominate total response time and argues for keeping the model warm in memory — or reconsidering whether cloud fallback makes more sense for low-frequency use cases.

---

Before you ship

1. Enforce a token budget as a first-class latency contract. Measure tokens-per-second on your target NPU tier, compute the budget floor for your latency SLA using floor(targetLatencyMs / 1000.0 * tokensPerSec), and set maxNewTokens accordingly before you write any other inference code.

2. Design your tool schemas for token efficiency, not human readability. Strip descriptions, abbreviate field names where the model can still generalize, and validate that the schema itself fits within 80 prompt tokens.

3. Ship a two-pass fallback parser from day one. Quantized models will drift. A regex-based extraction layer over known field names is 30 lines of Kotlin and will save you from silent failures in production.

---

#android #kotlin #mobile #architecture #llm