Wiring Android's MediaPipe Image Embedding API to a Vector Store for Real-Time On-Device Visual Search: FAISS, Product Quantization, and the Memory Budget That Fits in 500MB

TL;DR

You can run a production-grade visual similarity search entirely on-device using MediaPipe’s ImageEmbedder, a JNI-bridged FAISS index, and Product Quantization. The right PQ tier (PQ32x8) compresses 1M float embeddings from 4GB to ~32MB, survives process death via mmap’d serialization, and clears >90% recall@10 — all within a 500MB RAM envelope.

---

The case for fully offline visual search

On-device inference has become the standard answer to data residency and privacy requirements. No network hop means no exfiltration surface. Visual search is a natural fit: product lookup, defect detection, inventory matching — all problems where milliseconds matter and query images may be sensitive.

The comparison isn’t subtle: network-dependent search pipelines introduce 80–300ms of latency on mobile. On-device, with the right index, you get sub-20ms queries.

---

Architecture overview

The pipeline:

CameraFrame / Bitmap
       ↓
MediaPipe ImageEmbedder  →  Float[1024] embedding
       ↓
FAISS IVFPQIndex (JNI)   →  Top-K candidate IDs
       ↓
Local SQLite / Room      →  Hydrated result objects

The embedding model (EfficientNet-Lite) runs on the CPU delegate. The FAISS index lives in native heap via JNI. SQLite stores the payload. No cloud required.

---

MediaPipe ImageEmbedder: latency profile

MediaPipe’s ImageEmbedder with the EfficientNet-Lite0 model produces 1,280-dimensional float embeddings. On a mid-tier Snapdragon 778G, the CPU delegate runs embedding in 14–22ms. The GPU delegate cuts that to 8–12ms, but adds ~400ms of first-run JIT overhead — fine for batch work, not for interactive search. The model itself is ~6MB on disk.

For search pipelines, I recommend the CPU delegate with quantized inference. The latency is acceptable and the thermal profile is far more sustainable over a session than GPU.

val embedder = ImageEmbedder.createFromOptions(
    context,
    ImageEmbedderOptions.builder()
        .setBaseOptions(BaseOptions.builder().build())
        .setQuantize(false) // keep float for FAISS cosine distance
        .build()
)
val result = embedder.embed(mpImage)
val embedding: FloatArray = result.embeddingResult()
    .embeddings()[0].floatEmbedding().values()

---

FAISS + Product Quantization: the memory math

PQ works by splitting each 1,280-dim vector into M sub-vectors, then quantizing each sub-vector to one of 2^nbits centroids. Storage per vector collapses to M × nbits / 8 bytes.

At 1M vectors, the numbers shake out like this:

Configuration	Bytes/Vector	1M Vectors	Est. Recall@10
Flat (float32)	5,120	4.9 GB	100%
IVFPQ — PQ64×8	64	61 MB	~96%
IVFPQ — PQ32×8	32	31 MB	~91%
IVFPQ — PQ16×8	16	16 MB	~84%

PQ32x8 is the sweet spot. You lose ~9% recall relative to brute force, gain 99.4% memory reduction, and stay comfortably within a 500MB process budget when factoring in the model, JNI overhead, and application heap.

---

Index serialization: surviving process death

Android kills processes. Your index must reload in under 500ms or users feel it. Serialize once, mmap on reload.

// Write (background thread, after index build/update)
fun persistIndex(index: FaissIndex, file: File) {
    file.outputStream().use { out ->
        index.serialize(out) // JNI call to faiss::write_index
    }
}

// Read (cold start — use FileChannel + MappedByteBuffer)
fun loadIndex(file: File): FaissIndex {
    val channel = FileInputStream(file).channel
    val buffer = channel.map(FileChannel.MapMode.READ_ONLY, 0, file.length())
    return FaissIndex.deserialize(buffer) // zero-copy via mmap
}

Mmap deserialization for a 31MB PQ32×8 index lands at ~35ms on cold start. Compared to re-embedding your entire catalog, this is effectively free.

For incremental updates, maintain a write-ahead delta index (flat, small) and merge nightly in a background worker. Atomic file-swap prevents a torn read.

---

Hitting >90% recall@10

The IVF structure introduces a coarse quantizer: nlist Voronoi cells. At query time, nprobe cells are searched. The trade-off:

• nlist = 1024, nprobe = 64 → ~91% recall@10, ~4ms query latency
• nlist = 1024, nprobe = 128 → ~95% recall@10, ~7ms query latency

Start at nprobe = nlist / 16 and tune upward until recall meets your SLA. In my experience building production systems, most visual search use cases tolerate 91% recall — a missed near-duplicate is far less damaging than a 300ms UI freeze.

---

Conclusion

Fully offline visual search on Android is a solved problem, if you respect the memory budget from day one.

A few things worth locking in before you ship:

1. Start with PQ32×8. It hits >90% recall@10 for 1M vectors at 31MB. Go flatter only if you have fewer than 50K vectors and can afford the RAM.
2. Mmap your serialized index. A 35ms mmap load versus a 4-second re-index isn’t a marginal gain — it’s the difference between a shippable product and a broken one.
3. Separate embedding from retrieval in your latency budget. Allocate 20ms to MediaPipe, 10ms to FAISS, and 5ms to result hydration. If either blows its budget, optimize that layer independently — don’t collapse the pipeline into a black box.