MVP Factory
EN
ai startup development

Wiring Android's MediaPipe LLM Inference API to a Retrieval-Augmented Generation Pipeline: Embedding Models, Vector Search, and the Context Window Budget That Fits On-Device

KW
Krystian Wiewiór · · 5 min read

TL;DR

Running RAG entirely on-device is now viable on mid-to-high-tier Android hardware. The winning stack: a quantized embedding model (INT8 all-MiniLM-L6-v2) for semantic search, sqlite-vec with an HNSW index as the vector store, and MediaPipe’s LLM Inference API driving a 4-bit Gemma 2B for generation. Keep chunks at 256 tokens, budget under 150ms for embedding latency, and stay within the 2,048-token context window. Total peak RAM: ~2.3 GB — achievable on devices with 6 GB+.


Why on-device RAG is worth the complexity

Consider the operational scale Uber and Zipline are targeting: one million drone deliveries per day across dozens of US cities by the end of 2029. The companion apps, dispatch tools, and customer-facing interfaces for a system that size cannot afford a round-trip to a remote LLM on every query. Connectivity in delivery corridors is intermittent. Latency is unacceptable. Costs compound at a million interactions per day. This is the scenario that forces the architecture decision: run the intelligence on-device, or don’t run it at all.

The same logic applies to any high-frequency, latency-sensitive mobile surface. On-device RAG removes the network as a failure point.


The architecture at a glance

No network calls anywhere in this stack:

User Query
    │
    ▼
[INT8 Embedding Model]  ←── all-MiniLM-L6-v2 (22M params)
    │  384-dim vector
    ▼
[SQLite-vec HNSW Index] ←── top-k = 5 nearest neighbors
    │  retrieved chunks
    ▼
[Context Builder]       ←── token budget enforcement
    │  augmented prompt ≤ 2,048 tokens
    ▼
[MediaPipe LLM API]     ←── Gemma 2B Q4_K_M
    │
    ▼
  Response

Weights, vector index, and chunk corpus all live on-device. Cold-start after index warm-up is dominated by LLM initialization (~1.8s on Pixel 8 Pro), not retrieval.


Chunk sizing: the first budget problem

Most teams treat chunking as an afterthought. It is your primary architectural constraint because it determines whether your context window survives contact with real queries.

Chunk Size (tokens)Retrieval PrecisionTokens Used (top-5)Remaining for Generation
128High (too narrow)6401,408
256Balanced1,280768
384Rich but risky1,920128
512DiffuseExceeds 2,048N/A

With Gemma 2B’s 2,048-token context limit, 256 tokens per chunk with top-k=5 is the defensible default. It leaves ~700 tokens for a system prompt plus generation headroom. At 384 tokens you are one verbose system prompt away from silent overflow.


Embedding latency budget

INT8-quantized all-MiniLM-L6-v2 benchmarks at:

DeviceEmbedding LatencyHNSW Search (10K docs)Total Retrieval
Pixel 8 Pro (Tensor G3)38ms5ms43ms
Galaxy S24 (SD 8 Gen 3)31ms4ms35ms
Mid-tier (SD 6 Gen 1)95ms8ms103ms

Acceptable UX threshold: < 150ms total retrieval latency before passing context to the LLM. All three tiers fit. sqlite-vec’s HNSW implementation doesn’t get nearly enough credit — it adds negligible overhead even at 50K documents.

val embeddingResult = embeddingModel.embedAsync(queryText).await()
// INT8, ~38ms on Pixel 8 Pro

val neighbors = vectorIndex.search(
    vector = embeddingResult.embedding,
    k = 5
) // ~5ms HNSW lookup in sqlite-vec

Memory ceiling: what actually fits

In my experience building production systems, memory planning is where on-device RAG projects fail silently — until they don’t, and the OOM killer arrives mid-session.

ComponentMemory (MB)
Gemma 2B Q4_K_M weights~1,600
MediaPipe LLM runtime~200
INT8 embedding model~90
sqlite-vec HNSW index (10K docs)~40
Android process baseline~350
Total peak~2,280

A 6 GB device typically has ~4.2 GB available to the foreground app. This pipeline fits with margin. On 4 GB devices, switch to Gemma 1.1B Q4_K_M (~900 MB) and accept the quality trade-off — it is a better decision than fighting the memory ceiling.


Wiring it together

val llmInference = LlmInference.createFromOptions(
    context,
    LlmInference.LlmInferenceOptions.builder()
        .setModelPath(modelPath)
        .setMaxTokens(2048)
        .setTopK(40)
        .setTemperature(0.7f)
        .build()
)

val augmentedPrompt = buildString {
    append(systemPrompt)           // ~100 tokens — measure this
    neighbors.forEach { chunk ->
        append("\n\n[Context]\n${chunk.text}")
    }
    append("\n\n[Question]\n$userQuery")
}

// Count tokens before submission — overflow is silent in some builds
check(tokenizer.countTokens(augmentedPrompt) <= 2000)

val response = llmInference.generateResponseAsync(augmentedPrompt).await()

Token counting before submission is non-negotiable. Some MediaPipe builds silently truncate on overflow rather than throwing — you will never know your context was truncated unless you check.


3 takeaways

  1. Lock chunk size at 256 tokens and top-k at 5 before tuning anything else. These two parameters control your context budget more than any other variable. Drift from this baseline and you need to re-measure both retrieval quality and token overflow risk simultaneously.

  2. Benchmark embedding latency on your actual target device tier, not a flagship. Mid-tier hardware at ~95ms still fits the 150ms retrieval budget — but only barely. Test with real corpus sizes (10K+ documents) before committing to a hardware target.

  3. Build your RAM budget table before selecting your model. If your component total exceeds ~2.4 GB, drop to Gemma 1.1B rather than optimizing your way around the OOM killer. The ceiling is a hard constraint; the model choice is a variable.

On-device RAG is no longer experimental. The constraint is not capability — it is discipline in managing three budgets simultaneously: tokens, latency, and memory.


Share: Twitter LinkedIn