Real-time sign language recognition on Android under 30ms

Meta description: Wire CameraX to MediaPipe Hands and a quantized TFLite INT8 classifier with GPU delegate. Full pipeline, memory layout, and EMA smoothing for sub-30ms latency on mid-range Android.

Tags: android kotlin mobile architecture jetpackcompose

---

TL;DR

CameraX → MediaPipe Hands landmark extraction → quantized TFLite gesture classifier with GPU delegate can hit sub-30ms end-to-end latency on mid-range devices (Snapdragon 7xx class). The decisions that actually matter: ImageAnalysis backpressure strategy, ByteBuffer memory layout matching TFLite’s expected input tensor, and a lightweight exponential moving average smoother over landmark sequences. INT8 quantization costs ~2–3% accuracy on hand keypoints but saves ~40% inference time. That tradeoff is almost always worth it.

---

The pipeline at a glance

What most teams get wrong about on-device vision pipelines: they treat each stage as independent, then wonder why total latency is 80ms when each stage measures at 15ms individually. The overhead lives in the handoffs.

The full pipeline:

CameraX ImageAnalysis
    └─► ImageProxy (YUV_420_888) → Bitmap conversion
        └─► MediaPipe Hands (CPU, landmark extraction)
            └─► 21 × (x, y, z) keypoints → FloatArray
                └─► Temporal smoother (EMA, α=0.6)
                    └─► TFLite INT8 classifier (GPU delegate)
                        └─► Gesture label + confidence

Total budget: 30ms.

---

Stage 1: CameraX ImageAnalysis — backpressure is everything

val analysisUseCase = ImageAnalysis.Builder()
    .setTargetResolution(Size(640, 480))
    .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
    .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_YUV_420_888)
    .build()

STRATEGY_KEEP_ONLY_LATEST is non-negotiable. With STRATEGY_BLOCK_PRODUCER, a slow inference frame causes the camera queue to back up and you’ll be processing stale frames — the worst possible outcome for gesture recognition. Drop frames aggressively; temporal smoothing handles the gaps.

---

Stage 2: MediaPipe Hands landmark extraction

MediaPipe Hands produces 21 3D landmarks per hand. On a Pixel 6a (Tensor G2), extraction alone runs at ~12ms. On a Snapdragon 778G (mid-range), expect ~16–18ms. This is the most expensive stage, and it runs on CPU.

One thing that trips people up: pass the Bitmap and FrameMetadata to the Hands object via hands.send(). Avoid re-encoding to JPEG — that single mistake adds 8–12ms.

val frameMetadata = FrameMetadata.Builder()
    .setWidth(bitmap.width)
    .setHeight(bitmap.height)
    .setRotation(rotationDegrees)
    .build()
hands.send(bitmap, frameMetadata, SystemClock.uptimeMillis())

---

Stage 3: Temporal smoothing — don’t skip this

Raw landmark sequences are noisy. A single dropped or occluded frame produces a landmark spike that maps to the wrong gesture. An exponential moving average over the last N frames costs ~0.1ms and eliminates most false positives:

class LandmarkSmoother(private val alpha: Float = 0.6f) {
    private var prev: FloatArray? = null

    fun smooth(current: FloatArray): FloatArray {
        val p = prev ?: current.copyOf()
        val smoothed = FloatArray(current.size) { i -> alpha * current[i] + (1 - alpha) * p[i] }
        prev = smoothed
        return smoothed
    }
}

α = 0.6 balances responsiveness and stability. Lower values (0.3–0.4) work better for slower, deliberate signs. Higher values (0.8+) suit fast fingerspelling.

---

Stage 4: TFLite INT8 classifier with GPU delegate

The classifier is a 3-layer MLP: 63 float inputs (21 landmarks × x, y, z) → Dense(128, ReLU) → Dense(64, ReLU) → Dense(num_classes, Softmax). Intentionally shallow — deeper models add latency without meaningful accuracy gains on a 63-feature input space, and the flat landmark vector gives you no spatial hierarchy to exploit with convolutions.

Config	Inference time (Snapdragon 778G)	Top-1 accuracy (ASL 26-class)
FP32, CPU	11.2ms	97.4%
INT8, CPU	6.8ms	95.1%
INT8, GPU delegate	3.1ms	95.1%
INT8, NNAPI	4.4ms	94.8%

GPU delegate on INT8 wins. Skip NNAPI — it introduces driver inconsistency across OEMs, and I’ve seen 2x variance on the same chipset across firmware versions.

GPU delegate initialization fails silently on roughly 10% of devices due to broken OEM drivers. Always wrap it:

val options = Interpreter.Options()
try {
    options.addDelegate(GpuDelegate())
} catch (e: Exception) {
    // Fall back to INT8 CPU — still 6.8ms, well within budget
}
val interpreter = Interpreter(modelBuffer, options)

Memory layout is where silent failures hide. TFLite’s GPU delegate requires the input ByteBuffer to be direct-allocated with float values interleaved in [landmark_index][x, y, z] order. Pass a heap-allocated buffer or the wrong stride and you’ll get wrong predictions with no exception thrown — just quietly bad results.

val inputBuffer = ByteBuffer.allocateDirect(63 * 4).order(ByteOrder.nativeOrder())
landmarks.forEach { lm ->
    inputBuffer.putFloat(lm.x)
    inputBuffer.putFloat(lm.y)
    inputBuffer.putFloat(lm.z)
}

---

End-to-end latency budget

Stage	Mid-range (778G)
CameraX frame delivery	~2ms
YUV → Bitmap	~3ms
MediaPipe Hands	~17ms
EMA smoothing	~0.1ms
TFLite INT8 + GPU	~3.1ms
Total	~25ms

That leaves ~5ms of headroom on Snapdragon 7xx hardware before hitting the 30ms budget — enough to absorb GC pauses without dropping user-visible frames. On the CPU fallback path, total latency rises to ~29ms, which still clears the target.

---

Before you ship

Set STRATEGY_KEEP_ONLY_LATEST unconditionally for any real-time vision ImageAnalysis pipeline. Backpressure accumulation is the silent killer of perceived latency.

Quantize to INT8 and use the GPU delegate together, but always provide a CPU fallback. The ~2% accuracy delta is acceptable for most gesture vocabularies, and you recover 8ms of inference budget. The fallback covers you on the ~10% of devices with broken GPU delegate drivers.

Add temporal smoothing before the classifier, not after. Post-classification smoothing on labels introduces jitter at gesture boundaries. Smoothing the landmark sequence itself produces cleaner transitions and higher effective accuracy without touching the model.