CameraX + MiDaS v3 Small: Real-time depth at 30fps on Android

A GPU delegate that silently falls back to CPU will cost you 40ms per frame and show up only as a 12fps complaint in a Play Store review. This post is about preventing that — and the two other pipeline killers that stand between you and sustained 30fps depth estimation on mid-range Android hardware.

---

TL;DR

Connecting CameraX to a quantized MiDaS v3 Small (256×256 input) depth model at 30fps on mid-range Android hardware is doable — but only if you own every millisecond of the pipeline. The three killers: YUV-to-RGB allocation churn, a silently-falling-back GPU delegate, and unbounded point cloud growth. Solve those and you stay under 28ms.

---

The pipeline at a glance

CameraX ImageAnalysis → YUV→RGB → Resize/Normalize → TFLite GPU (MiDaS v3 Small) → Depth Map → Ring Buffer

Frame budget on a Snapdragon 7-series class device targeting MiDaS v3 Small at 256×256:

Stage	CPU Path	GPU Path	Target Budget
YUV→RGB conversion	6–10ms	2–4ms	≤4ms
Resize + preprocess	2–3ms	1–2ms	≤2ms
TFLite inference	30–50ms	8–15ms	≤18ms
Depth normalization	<1ms	<1ms	≤1ms
Point cloud update	1–3ms	1–2ms	≤3ms
Total	~40–67ms	~13–24ms	≤28ms

GPU inference isn’t optional. CPU alone blows the 33ms budget before normalization even starts. These figures are specific to the Small variant — MiDaS v3 Large (384×384 input) adds 30–60% to inference time and requires a higher-tier device to hit 30fps.

---

YUV-to-RGB: the hidden tax

CameraX ImageAnalysis delivers YUV_420_888 frames. RenderScript.ScriptIntrinsicYuvToRGB was the standard approach, but it’s deprecated as of API 31. The fastest modern replacement is direct plane extraction into a pre-allocated ByteBuffer, fed to a lightweight GPU compute path or RenderEffect on API 31+.

What most teams get wrong: calling ImageProxy.toBitmap() in the analyzer callback. That allocates a new Bitmap on every frame — on a 4GB device running a depth pipeline, you’re looking at an OOM event within minutes.

// Allocate ONCE at construction — never inside the callback
val rgbBuffer: ByteBuffer = ByteBuffer.allocateDirect(MODEL_WIDTH * MODEL_HEIGHT * 3)

analyzer.setAnalyzer(executor) { imageProxy ->
    convertYuvToRgb(imageProxy, rgbBuffer) // writes into pre-allocated buffer
    imageProxy.close()                     // critical — CameraX stalls if omitted
    runInference(rgbBuffer)
}

---

TFLite GPU delegate: the silent fallback problem

In my experience, this is the most common source of invisible performance regressions. If your input ByteBuffer isn’t direct-allocated and natively ordered, the GPU delegate silently falls back to CPU. You get no exception — just 4× slower inference and a frame rate that never reaches 30fps.

val gpuDelegate = GpuDelegate(GpuDelegate.Options().apply {
    isPrecisionLossAllowed = true  // enables FP16; 30–40% latency reduction on Adreno/Mali
})
val options = Interpreter.Options().addDelegate(gpuDelegate)
val interpreter = Interpreter(loadModelFile(context), options)

Pre-allocate the input tensor buffer with explicit native byte order:

val inputBuffer = ByteBuffer
    .allocateDirect(1 * MODEL_HEIGHT * MODEL_WIDTH * 3 * Float.SIZE_BYTES)
    .order(ByteOrder.nativeOrder()) // 4-byte alignment guarantee for GPU delegate

Detecting silent fallback in production. The Tensor.device() string API exists in some TFLite bindings but its return values aren’t guaranteed stable across library versions — don’t gate production logic on it. The reliable approach is a latency probe at interpreter startup: run two warmup inferences and measure wall time. On a 256×256 model, GPU-accelerated inference consistently completes under 20ms on Adreno 6xx and Mali-G7x hardware; CPU execution reliably exceeds 35ms. If your warmup latency exceeds that threshold, log a warning and surface it in your analytics pipeline:

// Illustrative — tune thresholds to your target device tier
val warmupMs = measureTimeMillis { repeat(2) { interpreter.run(inputBuffer, outputBuffer) } } / 2
if (warmupMs > GPU_LATENCY_THRESHOLD_MS) {
    Log.w(TAG, "TFLite GPU delegate may have fallen back to CPU (${warmupMs}ms avg)")
    analytics.logEvent("tflite_gpu_fallback_suspected")
}

Enable isPrecisionLossAllowed unconditionally on quantized MiDaS models — FP16 on supported Adreno and Mali GPUs cuts inference time by 30–40% with negligible impact on depth accuracy at scene-reconstruction quality.

---

Depth map normalization

MiDaS v3 outputs inverse relative depth — higher values indicate closer geometry. Normalize per-frame to [0, 1] to keep point cloud accumulation coherent:

val min = outputArray.min()
val max = outputArray.max()
val range = (max - min).coerceAtLeast(1e-5f)  // avoid division by zero in static scenes
outputArray.forEachIndexed { i, v ->
    normalizedDepth[i] = (v - min) / range
}

Don’t apply a global running normalization across frames unless the scene is stationary. Temporal drift from a global normalizer introduces flickering that makes ring-buffer accumulation incoherent.

---

Ring buffer point cloud accumulation

A 256×256 depth map at 30fps generates roughly 2 million depth values per second. An unbounded accumulation list will exhaust heap memory on any device under 6GB within seconds of continuous capture.

class DepthRingBuffer(private val capacity: Int) {
    private val frames = ArrayDeque<FloatArray>(capacity)

    fun push(depthMap: FloatArray) {
        if (frames.size >= capacity) frames.removeFirst()
        frames.addLast(depthMap)
    }

    fun snapshot(): List<FloatArray> = frames.toList()
}

For scene-level reconstruction, downsample depth maps to 64×64 before accumulation. The difference is stark:

Resolution	Floats/frame	Bytes/frame	60-frame buffer
256×256 (full)	65,536	262KB	~15MB
64×64 (downsampled)	4,096	16KB	~960KB

That’s 4,096 floats × 4 bytes × 60 frames = 983,040 bytes ≈ 960KB downsampled versus 65,536 × 4 × 60 ≈ 15MB full-resolution — a 16× reduction. On a mid-range device with a ~300MB app memory budget, that gap determines whether your app survives a five-minute session.

---

Three things that actually matter

1. Pre-allocate everything at startup. Every ByteBuffer.allocateDirect() or Bitmap allocation inside an analyzer callback is a GC pressure event. Allocate once, reuse across every frame.

2. Use a latency probe to detect GPU fallback, not a string check. Tensor.device() isn’t stable across TFLite versions. A warmup inference that exceeds your GPU threshold is more reliable, and it surfaces device-tier issues you’ll never catch in your own test lab.

3. Downsample before accumulating. A 64×64 ring buffer at 60 frames stays under 1MB. Full-resolution unbounded accumulation hits 15MB in under two seconds and will trigger the OOM killer mid-session on mid-range hardware.

---

Tags: android, kotlin, mobile, architecture