TFLite INT8 + ML Kit: on-device ML routing for Android

Meta description: Build a two-tier on-device inference pipeline on Android — ML Kit handles common cases, TFLite INT8 catches edge cases, with confidence routing and a strict per-frame CameraX budget.

Tags: android mobile architecture jetpackcompose cleanarchitecture

---

TL;DR

ML Kit is fast and easy, but it surrenders control on edge cases. A two-tier pipeline — ML Kit as the primary path, a quantized INT8 TFLite model as fallback — keeps inference fast for the common case while handling edge cases on-device, all within a hard per-frame budget enforced at the camera layer.

---

What most teams get wrong about on-device ML

Most Android engineers reach for ML Kit because it ships pre-optimised, integrates in minutes, and offloads model management entirely. That’s a reasonable default. The mistake is treating it as a terminal decision rather than a first-pass filter.

ML Kit’s bundled models are tuned for recall on well-lit, high-quality input. In production, you’ll see confidence scores collapse on motion blur, partial occlusion, or atypical orientations. When that happens, teams either ignore the low-confidence result or round-trip to a cloud endpoint. Both are wrong answers for a latency-sensitive pipeline.

The right architecture is a confidence-gated fallback to a custom TFLite INT8 model you control entirely.

---

The two-tier architecture

CameraX Frame → ML Kit Inference
                    │
             confidence ≥ 0.85? ──YES──▶ emit result
                    │
                   NO
                    │
             TFLite INT8 Model
                    │
             confidence ≥ 0.60? ──YES──▶ emit result
                    │
                   NO
                    ▶ drop frame / fallback label

The routing layer is intentionally simple: a single floating-point threshold comparison. Keep it that way. The complexity lives in model training and delegate selection, not in the router.

---

Delegate selection: NNAPI vs GPU

This is where device tier matters. INT8 quantized models are a prerequisite for NNAPI acceleration — most NNAPI implementations reject FP32 graphs on mid-range silicon. The GPU delegate handles FP16 and FP32 but offers weaker INT8 support pre-Android 12.

Delegate	Best For	INT8 Support	Notes
NNAPI	Mid-range (Snapdragon 6xx, Dimensity 7xx)	Strong post-Android 10	Requires NnApiDelegate with allowFp16PrecisionForFp32=true
GPU	Flagship (Snapdragon 8xx, Tensor G3+)	Partial pre-Android 12	Lower driver consistency across OEMs
CPU (XNNPACK)	Fallback, all devices	Full	Most predictable, highest latency

In my experience building production systems, the safest heuristic is: probe NNAPI at startup, time a warm-up inference, and fall back to XNNPACK if the median latency exceeds your frame budget threshold. Do this once on first launch and cache the result in SharedPreferences.

fun selectDelegate(context: Context): List<Delegate> {
    return try {
        val nnapi = NnApiDelegate()
        // warm-up probe — if it throws or is slow, fall through
        listOf(nnapi)
    } catch (e: Exception) {
        emptyList() // XNNPACK via default interpreter
    }
}

---

Enforcing the per-frame budget with CameraX

At 30 fps you have 33 ms per frame. ML Kit + TFLite serial inference on a Dimensity 700-class device will routinely exceed that if you let every frame through.

Drop frames explicitly at the ImageAnalysis layer:

val analysis = ImageAnalysis.Builder()
    .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
    .setTargetRotation(Surface.ROTATION_0)
    .build()

analysis.setAnalyzer(inferenceExecutor) { imageProxy ->
    if (frameGate.tryAcquire()) { // token bucket, refill every 33ms
        runInferencePipeline(imageProxy)
    } else {
        imageProxy.close() // explicit drop
    }
}

STRATEGY_KEEP_ONLY_LATEST alone isn’t sufficient — it buffers one frame, not zero. Pair it with a token-bucket gate seeded to your target frame rate. On a mid-range device I benchmark this holding 28–30 fps with the two-tier pipeline active, versus 18–22 fps without frame gating.

---

Confidence threshold calibration

Don’t pick 0.85 and 0.60 out of thin air. Evaluate your model on a held-out validation set and plot precision-recall curves at each threshold. The ML Kit threshold should sit at the knee of your precision curve — you want near-zero false positives passing through, because the fallback model should own uncertain cases, not ML Kit’s reject pile.

For the TFLite INT8 fallback, a lower threshold is acceptable because this model is specifically trained on your hard negatives. If both models return low confidence, emit a “no result” signal and surface it to the UI rather than guessing.

---

The inference budget as a first-class product constraint

Treat your per-frame budget the same way you treat memory ceilings: a hard contract, not a guideline. I run HealthyDesk (a break-reminder app for developers) on mid-range test devices specifically to validate that background processing stays within battery and CPU envelopes that real users will accept. Profile on Snapdragon 680 or Dimensity 700 hardware, not on a flagship emulator.

---

Before you ship

Gate your inference loop at the camera layer. STRATEGY_KEEP_ONLY_LATEST plus a token-bucket frame dropper is the combination that works — neither alone is enough.

NNAPI performance varies widely across OEMs, so probe and cache delegate selection at startup. Measure a warm-up inference, store the winner, and revisit on app updates that change model architecture.

The 0.85 / 0.60 split is a starting point, not a law. Plot your precision-recall curves and choose thresholds that match the cost asymmetry of your specific false-positive vs false-negative trade-off.