Real-Time AR Occlusion on Android: CameraX + TFLite GPU Under 33ms

Meta description: Wire CameraX ImageAnalysis to a quantized MiDaS/Depth Anything V2 model via TFLite GPU delegate for real-time AR occlusion meshes — covering delegate selection, YUV→RGB cost, and ring-buffer recovery.

---

TL;DR

Monocular depth for AR occlusion is solvable on-device at 30fps if you treat the pipeline as a single GPU memory budget problem. Use CameraX ImageAnalysis in non-blocking mode, convert YUV→RGB on the GPU, run INT8-quantized MiDaS or Depth Anything V2 via the TFLite GPU delegate, and feed the resulting depth map into SceneView/ARCore as an occlusion mesh — all within a 33ms frame budget.

---

The problem: geometry without a depth sensor

Most Android devices lack a hardware depth sensor. ARCore gives you a sparse point cloud, but per-pixel occlusion — the kind that makes virtual objects convincingly hide behind real furniture — requires a dense depth map every frame. The solution is a monocular depth estimation model running entirely on-device.

The catch: inference latency must stay under 33ms to hold 30fps, and the depth map must arrive in time to influence the current SceneView render pass.

---

Pipeline architecture

CameraX ImageAnalysis
    │ (YUV_420_888, non-blocking)
    ▼
YUV → RGB Conversion (GPU Bitmap or shader)
    │
    ▼
TFLite Interpreter (GPU Delegate)
  └── INT8 quantized MiDaS / Depth Anything V2
    │
    ▼
Depth Map Buffer (ring buffer, 3 frames)
    │
    ▼
SceneView / ARCore Occlusion Mesh

Each stage competes for the same GPU bus. Getting this wrong — even one stage on the wrong executor — blows your frame budget.

---

Delegate selection: GPU vs. NNAPI vs. CPU

Most teams benchmark delegates in isolation, not under the concurrent load of a live camera stream and SceneView rendering. That’s where you get burned. NNAPI looks attractive on paper but introduces unpredictable scheduling latency when the GPU is also busy.

Delegate	Avg Inference (INT8)	Jitter (p99)	Notes
CPU (4 threads)	~90ms	High	Unusable at 30fps
NNAPI	~28ms	High under load	Driver-dependent, risky
GPU Delegate	~18–22ms	Low	Consistent, composable

The GPU delegate wins because it shares memory space with the render pipeline — no cross-bus copies. Use GpuDelegateV2 with GpuDelegateV2.Options set to setPrecisionLossAllowed(true) and setInferencePriority1(INFERENCE_PRIORITY_MIN_LATENCY).

---

YUV→RGB: the hidden tax

CameraX delivers YUV_420_888 frames. Converting to the RGB float tensor your model expects is not free. On CPU this costs 5–12ms per frame — enough to break your budget before inference even starts.

The right approach: use a RenderScript intrinsic (ScriptIntrinsicYuvToRGB) or, on API 31+, a hardware-accelerated ImageReader with USAGE_GPU_SAMPLED_IMAGE. The goal is to land the RGB data directly in GPU memory so the TFLite GPU delegate can read it without a host-side copy.

// API 31+ path: zero-copy GPU texture hand-off
val imageReader = ImageReader.newInstance(
    width, height,
    ImageFormat.YUV_420_888, 3,
    HardwareBuffer.USAGE_GPU_SAMPLED_IMAGE or HardwareBuffer.USAGE_CPU_READ_RARELY
)

On pre-31 devices, fall back to RenderScript. CPU-side Bitmap conversion is a last resort — benchmark it first on your minimum target device.

---

Frame-drop recovery with a ring buffer

Under thermal throttling or during SceneView’s heavy render frames, the depth pipeline will occasionally miss its slot. Without a recovery strategy, you get flickering occlusion meshes as stale geometry snaps to new camera poses.

A three-slot ring buffer solves this. The depth consumer always reads the most recent completed depth map, and the producer overwrites the oldest slot. This decouples inference timing from render timing:

class DepthRingBuffer(size: Int = 3) {
    private val slots = Array(size) { FloatArray(WIDTH * HEIGHT) }
    private val writeIdx = AtomicInteger(0)
    fun write(depth: FloatArray) {
        val slot = writeIdx.getAndIncrement() % slots.size
        depth.copyInto(slots[slot])
    }
    fun readLatest(): FloatArray = slots[writeIdx.get() % slots.size]
}

The render thread calls readLatest() and applies it to the occlusion mesh regardless of inference completion. Missed frames reuse the prior map — imperceptible at 30fps unless the camera moves fast.

---

GPU memory budget that separates 30fps from jank

On a mid-range SoC (Adreno 6xx, Mali G78), the shared GPU memory budget for a camera + AR workload is roughly 200–300MB before the OS starts evicting. Your depth model (INT8 MiDaS 384×384) occupies ~15MB in the delegate’s tensor arena. SceneView’s shadow maps and environment textures consume another 80–120MB.

This leaves little margin. Quantize aggressively: INT8 over FP32 cuts model size by ~4x with minimal accuracy loss for occlusion (you do not need millimeter precision — you need correct layering order). Avoid FP16 intermediate activations in the GPU delegate if your driver does not handle them efficiently; measure with adb shell dumpsys gfxinfo.

---

3 actionable takeaways

Profile delegate latency under real load, not in a synthetic benchmark. Run CameraX and SceneView simultaneously before committing to NNAPI vs. GPU delegate. NNAPI’s p99 latency is its weakness and you will not see it until everything is running together.

Fix the YUV→RGB CPU path before touching anything else. This single change can recover 8–12ms of frame budget before you touch the model or delegate configuration — the highest-leverage optimization in the pipeline.

Ship a ring buffer from day one. Thermal throttling and driver hiccups are production realities, not edge cases. Decoupling inference timing from render timing costs 45KB of memory and prevents visible occlusion flicker.

---

#android #mobile #architecture #jetpackcompose #kotlin