CameraX + DeepLab v3+: Real-time segmentation on Android
Meta description: Wire CameraX ImageAnalysis to a quantized DeepLab v3+ model via TFLite GPU delegate. Learn tensor layout, stride bitmaps, and memory trade-offs.
Tags: android kotlin mobile architecture jetpackcompose
TL;DR
Running on-device background replacement at 30fps on Android requires CameraX’s ImageAnalysis delivering YUV_420_888 frames, a stride-correct conversion to NHWC-layout bitmaps, and a quantized DeepLab v3+ model behind the TFLite GPU delegate — all three working in lockstep. Get any one wrong and you blow your 33ms per-frame budget, or worse, OOM on mid-range devices.
The problem with naive pipelines
Most teams treat on-device segmentation as a model problem. It’s actually a data plumbing problem. The GPU delegate is fast. DeepLab v3+ quantized is compact. CameraX is reliable. Connecting them exposes a minefield of layout mismatches, unnecessary allocations, and synchronization gaps that destroy throughput before inference even starts.
Tensor input layout: NHWC, or you’re wasting cycles
DeepLab v3+ expects input in NHWC format — [1, H, W, 3] — with pixel values in the quantized range appropriate to your model variant. TFLite’s uint8 quantization path uses [0, 255]; the newer full-integer int8 path uses [-128, 127]. The GPU delegate handles scale and zero-point remapping internally either way, but you must match the buffer type to the model’s quantization scheme or your masks will be silently wrong.
The model’s default input resolution is 513×513, but the 257×257 variant offers a meaningful latency reduction on tighter hardware.
// Correct: pre-allocate a reusable ByteBuffer
val inputBuffer = ByteBuffer.allocateDirect(1 * 513 * 513 * 3)
.order(ByteOrder.nativeOrder())Never allocate per-frame. A single allocateDirect at startup and manual rewind() per frame is the difference between 28ms and 45ms average inference on a Pixel 6.
Stride-aligned bitmap conversion
CameraX ImageProxy in YUV_420_888 format has a nasty property: the Y plane row stride often does not equal the image width. Ignoring this produces a sheared, corrupted input tensor.
fun ImageProxy.toStrideCorrectedBitmap(): Bitmap {
val yPlane = planes[0]
val yBuffer = yPlane.buffer
val rowStride = yPlane.rowStride
val width = width
val height = height
// Allocate only the true pixel data — rowStride may include padding
val yData = ByteArray(height * width)
for (row in 0 until height) {
yBuffer.position(row * rowStride)
yBuffer.get(yData, row * width, width)
}
// Build grayscale ARGB_8888 bitmap; color planes handled separately for RGB
val pixels = IntArray(width * height) { i ->
val y = yData[i].toInt() and 0xFF
0xFF000000.toInt() or (y shl 16) or (y shl 8) or y
}
val bitmap = Bitmap.createBitmap(pixels, width, height, Bitmap.Config.ARGB_8888)
return Bitmap.createScaledBitmap(bitmap, 513, 513, false)
}Always read rowStride from the plane, not image.width. This loop copies only the valid pixel columns per row, discarding stride padding. Note this produces a luminance-only bitmap suitable for grayscale input; a full YUV→RGB conversion follows the same per-row stride pattern on the U and V planes. This alone eliminates a class of silent accuracy regressions that are easy to miss on flagship devices where stride happens to equal width.
GPU delegate configuration
val gpuDelegate = GpuDelegate(
GpuDelegate.Options().apply {
inferencePreference = INFERENCE_PREFERENCE_SUSTAINED_SPEED
precisionLossAllowed = true // enables FP16 path on supported hardware
}
)
val interpreter = Interpreter(
loadModelFile(context, "deeplab_v3_plus_int8.tflite"),
Interpreter.Options().addDelegate(gpuDelegate)
)precisionLossAllowed = true enables the FP16 execution path on Adreno and Mali GPUs. Combined with INT8 quantization on the model side, this is the primary lever for staying under 33ms on mid-range silicon.
Memory budget: the real constraint
The 400MB ceiling reflects Android’s ActivityManager.getLargeMemoryClass() on a typical mid-range device. What consumes that budget in a live pipeline:
| Component | Approx. memory |
|---|---|
| FP32 DeepLab v3+ (MobileNetV2 backbone) | ~8–9 MB |
| FP32 DeepLab v3+ (Xception backbone) | >200 MB |
| INT8 quantized model (MobileNetV2) | ~2.5–3 MB |
| GPU delegate tensor buffers | ~30–60 MB |
| CameraX preview surface | ~25–40 MB |
| Output mask + compositing | ~15–20 MB |
| Framework overhead | ~80–120 MB |
The Xception backbone is a non-starter on mobile without aggressive pruning. For production, the MobileNetV2-backbone INT8 variant is the right baseline — comfortable within budget on 4GB devices, with headroom for the rest of your app stack.
Per-frame latency budget
At 30fps you have 33ms per frame. A realistic breakdown on a mid-range Snapdragon 778G:
- YUV→NHWC conversion: ~4–6ms
- GPU delegate inference (INT8, 513×513): ~18–24ms
- Mask post-processing + compositing: ~4–6ms
That leaves 1–5ms of slack — enough for one dropped frame in a GC pause, not two. If compositing uses Canvas.drawBitmap in software mode, you’ve already lost. Use RenderEffect (API 31+) or a GLSL shader via SurfaceTexture to keep compositing on the GPU.
Before you ship
Fix stride first. Every accuracy and throughput investigation should start by verifying rowStride handling in your ImageProxy converter. It’s the most common silent failure in CameraX-to-TFLite pipelines, and it manifests as accuracy degradation rather than a crash — which makes it easy to miss.
Default to INT8 quantization, not as a fallback. The MobileNetV2 INT8 variant is the production-correct choice for the devices that make up the majority of your install base. Benchmark on a Pixel 6a or Galaxy A54, not a Pixel 9 Pro, and verify your quantization scheme matches your ByteBuffer type.
Keep all tensor operations GPU-side end to end. The moment you copy a buffer back to CPU for compositing, you serialize the pipeline and forfeit the delegate’s latency advantage. Invest in GLES compositing early — retrofitting it is expensive.