CameraX + TFLite vision encoder under 40ms

Meta description: Wire CameraX ImageAnalysis to a quantized TFLite CLIP encoder. Covers YUV conversion, executor isolation, and frame-drop strategies with Pixel 8 benchmarks.

---

TL;DR

Connecting CameraX’s ImageAnalysis use case to a quantized TFLite vision encoder is doable under 40ms — but only if you eliminate allocation spikes in YUV-to-RGB conversion, isolate inference on a dedicated executor, and drop frames instead of queuing them when the model falls behind. Skip any one of these and you will either stutter the UI or overwhelm the inference thread.

---

The problem most teams get wrong

The usual mistake with real-time on-device vision: treat the camera pipeline and the ML pipeline as independent concerns and bolt them together with a shared thread pool. The result is latency jitter, dropped UI frames, and an inference queue that grows unbounded under load.

The architecture has to be deliberate from the start.

---

The pipeline architecture

The pipeline has five stages:

CameraX ImageAnalysis
        │
        ▼
  YUV→RGB Converter  (pre-allocated ByteBuffer)
        │
        ▼
  TFLite Inference   (single-threaded dedicated executor)
        │
        ▼
  Result Channel     (conflated — drops stale frames)
        │
        ▼
  UI / Compose State

The key constraint: the camera produces frames faster than a quantized encoder can consume them. On a Pixel 8, CameraX at 30 fps gives you ~33ms per frame. An INT8-quantized CLIP-style ViT-B/32 via TFLite runs roughly 28–35ms on the Pixel 8’s NPU delegate, and 55–80ms on a mid-range Snapdragon 6-series without NPU acceleration. You cannot process every frame. You should not try.

---

YUV-to-RGB without allocation spikes

ImageProxy delivers frames in YUV_420_888. The naive path — converting via Bitmap.createBitmap() on every frame — allocates approximately 3MB of YUV input and produces a ~6MB RGB Bitmap per frame at 1080p. At 30 fps that is upwards of 270MB/s of GC pressure. You will see it immediately in the allocation profiler as sawtooth spikes. Buffer reuse eliminates the YUV-side allocation entirely and recycles the Bitmap, reducing steady-state allocation to near zero after warmup.

The fix is pre-allocation:

class YuvToRgbConverter(private val inputSize: Int) {
    private val rgbBuffer = ByteBuffer
        .allocateDirect(inputSize * inputSize * 3)
        .also { it.order(ByteOrder.nativeOrder()) }

    // reusableBitmap is allocated once at construction and recycled on every frame
    val reusableBitmap: Bitmap = Bitmap.createBitmap(inputSize, inputSize, Bitmap.Config.ARGB_8888)

    fun convert(image: ImageProxy) {
        rgbBuffer.rewind()
        val yPlane = image.planes[0].buffer
        val uPlane = image.planes[1].buffer
        val vPlane = image.planes[2].buffer
        // nativeYuvToRgb() is a JNI convenience; teams avoiding JNI can substitute
        // CameraX's androidx.camera.core.internal.utils.ImageUtil or a RenderScript
        // YuvToRgb kernel — the pre-allocation pattern remains identical.
        nativeYuvToRgb(yPlane, uPlane, vPlane, rgbBuffer, inputSize)
        reusableBitmap.copyPixelsFromBuffer(rgbBuffer.also { it.rewind() })
    }
}

Reuse both the ByteBuffer and the target Bitmap. Allocate once at startup, reuse on every frame.

---

Executor threading strategy

Two executors. No sharing.

val cameraExecutor = Executors.newSingleThreadExecutor()
val inferenceExecutor = Executors.newSingleThreadExecutor()

The ImageAnalysis use case runs its analyze() callback on cameraExecutor. Inside that callback, gate on inference availability using an AtomicBoolean and submit to inferenceExecutor only if the previous inference has completed. The camera thread never blocks.

private val inferenceRunning = AtomicBoolean(false)
private val converter = YuvToRgbConverter(inputSize = 224)

override fun analyze(image: ImageProxy) {
    if (!inferenceRunning.compareAndSet(false, true)) {
        image.close() // drop frame — do not queue
        return
    }
    converter.convert(image)
    image.close()

    inferenceExecutor.submit {
        try {
            val result = tfliteModel.run(converter.reusableBitmap)
            _sceneState.value = result
        } finally {
            inferenceRunning.set(false)
        }
    }
}

Drop, don’t queue. A LinkedBlockingQueue as the executor’s backing store will build up frames and introduce compounding latency. A conflated StateFlow on the result side ensures the UI always sees the latest embedding, never a stale queued one.

---

Latency breakdown: Pixel 8 vs. mid-range

Stage	Pixel 8 (NPU delegate)	Snapdragon 6s (CPU)
YUV → RGB conversion	~1.5ms	~3ms
TFLite inference (INT8 ViT-B/32)	~28ms	~68ms
StateFlow emission + Compose recompose	~1ms	~1.5ms
Total end-to-end	~30.5ms	~72.5ms

The Pixel 8 lands comfortably under 40ms with the NNAPI/NPU delegate enabled. The mid-range device cannot process every frame at 30fps — at ~72.5ms per inference cycle, effective throughput is roughly 1 in every 2.2 frames (~14fps). That is still a workable rate for scene understanding; the pipeline just skips frames the model cannot absorb. Whether that is acceptable depends on your use case, but the drop-not-queue architecture at least keeps it predictable.

---

Bottom line

1. Pre-allocate your YUV buffers and Bitmap at startup. A reused ByteBuffer and Bitmap pair eliminates the ~3MB YUV input allocation and the ~6MB RGB Bitmap allocation per frame that cause GC sawtooth spikes. Measure with Android Studio’s allocation profiler before shipping.

2. Use two dedicated single-thread executors, never a shared pool. Camera callbacks and inference must be isolated. The AtomicBoolean gate on the camera thread is the simplest correct mechanism for frame dropping without queue buildup.

3. Enable the NNAPI delegate with CPU fallback and benchmark on your actual device tier. NPU acceleration is the difference between under-40ms and over-60ms. Ship two profiling builds — one with and one without the delegate — and gate on device capability at runtime.

---

#android #kotlin #mobile #architecture #jetpackcompose