Android Room: FTS5 + on-device embeddings hybrid search

Meta description: Wire Android Room to on-device embeddings with FTS5 for a hybrid retrieval pipeline hitting sub-100ms on mid-range hardware.

---

TL;DR

Pure SQL keyword search fails users. Semantic search without structure kills your memory budget. The answer is a hybrid retrieval pipeline: FTS5 for recall, quantized on-device embeddings for intent rewriting, and cosine similarity ranking against a local vector index, all coordinated by a query planner that keeps you in the 50ms tier, not the 300ms tier.

---

The problem with pure SQL search

Most teams get this wrong about local search on Android: they treat Room as a key-value store with a WHERE clause, then wonder why users can’t find anything.

Keyword queries are brittle. A user typing “meeting notes from last Tuesday about the budget” will return zero results if the stored record says “Q3 planning session - finance review.” Same semantic content, zero lexical overlap.

FTS5 improves recall with BM25 ranking, but it still operates on surface tokens. The moment you introduce synonyms, abbreviations, or natural-language phrasing, you are back to zero.

The fix is semantic query rewriting: intercept the user’s raw query, project it into an embedding space, and use that vector to either rewrite the SQL or rerank the FTS5 candidates.

---

The architecture: three layers of retrieval

User Query
    │
    ▼
[Query Rewriter — On-Device Embedding Model]
    │                    │
    ▼                    ▼
[FTS5 Recall]    [Vector Similarity Ranking]
    │                    │
    └──────────┬──────────┘
               ▼
        [Hybrid Ranker]
               ▼
          Room Cursor

Layer 1: FTS5 virtual tables

Define your FTS5 table alongside your standard Room entity. FTS5 is your high-recall, low-latency first pass.

@Entity(tableName = "notes")
data class NoteEntity(
    @PrimaryKey val id: Long,
    val title: String,
    val body: String,
    val createdAt: Long
)

// ftsVersion = FTS_VERSION_5 required — @Fts4 defaults to FTS4 without it
@Fts4(contentEntity = NoteEntity::class, ftsVersion = FtsOptions.FTS_VERSION_5)
@Entity(tableName = "notes_fts")
data class NoteFts(
    val title: String,
    val body: String
)

FTS5 with BM25 on a 100K-row corpus on a mid-range device (Snapdragon 6-series) completes in the 15-40ms window. That is your floor.

Layer 2: On-device embedding model

The embedding model runs via ONNX Runtime or MediaPipe on-device. Quantize to INT8: this takes a typical 90MB MiniLM-style model down to ~23MB and cuts inference from ~180ms to ~55ms on CPU.

class EmbeddingEngine(context: Context) {
    private val session: OrtSession = OrtEnvironment
        .getEnvironment()
        .createSession(loadModel(context, "minilm_int8.onnx"))

    suspend fun embed(text: String): FloatArray = withContext(Dispatchers.Default) {
        val tokens = tokenizer.encode(text, maxLength = 128)
        val inputTensor = OnnxTensor.createTensor(env, tokens)
        session.run(mapOf("input_ids" to inputTensor))
            .get("last_hidden_state")
            .meanPooling()
            .l2Normalize()
    }
}

Store embeddings as BLOB columns in Room (quantized to INT8 vectors at write time). At 128 dimensions INT8, each stored vector costs 128 bytes, negligible at scale.

Layer 3: The hybrid ranker

Run FTS5 for candidate recall (top-50), then re-rank using cosine similarity against the query embedding. This is faster than a full vector scan because the cosine pass only touches the candidate set, not the full corpus.

suspend fun hybridSearch(query: String): List<NoteEntity> {
    val queryVec = embeddingEngine.embedWithCache(query)
    val ftsResults = noteDao.ftsSearch(query, limit = 50)

    return ftsResults
        .map { note ->
            val storedVec = note.embedding.dequantize()
            val score = cosineSimilarity(queryVec, storedVec)
            note to score
        }
        .sortedByDescending { it.second }
        .take(10)
        .map { it.first }
}

---

The query planner trap: 50ms vs 300ms

Strategy	P50 Latency (Snapdragon 6xx)	Memory Overhead	Recall Quality
Pure FTS5 BM25	20-40ms	~2MB index	Low-Medium
Pure Vector Scan (full corpus)	280-400ms	50-200MB	High
Hybrid (FTS5 + re-rank top-50)	55-95ms	~25MB (INT8 model)	High
Hybrid + result cache	5-15ms (cache hit)	+8MB LRU	High

Skip the FTS5 recall phase and the query planner destroys you. A full in-memory cosine scan over 100K vectors runs 280ms before you even add model inference. The hybrid pattern keeps you in the 50-100ms tier.

The other trap is running embedding inference on the main thread. Always dispatch to Dispatchers.Default and debounce the query input with a 200ms delay — this alone cuts unnecessary embedding calls by ~60% on incremental search UIs.

Caching query embeddings

Repeated and near-duplicate queries are common in local search UIs. A five-line LRU cache eliminates the inference step entirely on hits:

private val queryCache = object : LinkedHashMap<String, FloatArray>(16, 0.75f, true) {
    override fun removeEldestEntry(eldest: Map.Entry<String, FloatArray>) = size > 100
}

suspend fun embedWithCache(query: String): FloatArray {
    val key = query.trim().lowercase()
    return queryCache.getOrPut(key) { embed(key) }
}

A 100-entry cache at 128-dimension INT8 vectors adds roughly 8MB overhead and collapses cache-hit latency to single-digit milliseconds, which is exactly the last row of the table above.

---

Conclusion

The hybrid FTS5 + on-device embedding pipeline ships on mid-range Android hardware today, fits within a 30MB memory budget with INT8 quantization, and delivers retrieval quality that pure SQL can’t match.

Three things worth internalizing before you ship:

1. Use FTS5 as your recall layer (top-50 candidates), not your final ranking signal. Pairing it with a vector re-ranker keeps latency under 100ms while preserving semantic accuracy.

2. Quantize your embedding model to INT8 before shipping. The quality drop is marginal (typically <2% on MTEB benchmarks), the memory savings are 4x, and inference time on CPU drops by 60-70%.

3. Cache query embeddings aggressively. A simple LRU cache keyed on the normalized query string eliminates embedding inference on hits, collapsing latency to single-digit milliseconds with minimal memory cost.

---

android kotlin architecture mobile cleanarchitecture