Real-time AR makeup on Android: CameraX + MediaPipe Face Landmarker under 40ms

SEO Meta Description: Wire CameraX ImageAnalysis to MediaPipe Face Landmarker v2 for real-time AR makeup on Android. GPU delegate, 478-landmark mesh, blend shapes, and frame pacing under 40ms on Snapdragon 7-series.

---

TL;DR

Shipping real-time AR makeup on Android requires three components working in lockstep: CameraX’s ImageAnalysis use case feeding frames at controlled cadence, MediaPipe Face Landmarker v2 producing a 478-point mesh with blend shape coefficients, and a GPU delegate pipeline that keeps end-to-end latency under 40ms on mid-range hardware. Get any one of these wrong and you either drop frames or melt the battery. This is the architecture that holds together in production.

---

The stack at a glance

Layer	Component	Role
Camera input	CameraX ImageAnalysis	Frame delivery, YUV→RGB
Landmark inference	MediaPipe Face Landmarker v2	478 points + blend shapes
GPU acceleration	TFLite GPU Delegate	Delegated mesh inference
Render	OpenGL ES 3.0 + tiled texture	Makeup layer compositing
Frame budget	Choreographer + fence sync	Pacing under 40ms

---

Why 40ms

At 30fps, you have 33ms per frame. At 60fps, 16.6ms. The 40ms target is a pragmatic ceiling for mid-range Snapdragon 7-series devices — it covers inference, compositing, and display pipeline overhead without triggering thermal throttling within a typical five-minute session. Exceed it consistently and you see jitter; exceed it under thermal pressure and the GPU governor steps down clocks, compounding the problem.

MediaPipe Face Landmarker v2 with GPU delegate runs at roughly 8–12ms on Snapdragon 7-series. That leaves ~28ms for frame acquisition, format conversion, makeup compositing, and display submission — tight but achievable if you’re disciplined about where the time actually goes.

---

Wiring CameraX to the inference pipeline

The ImageAnalysis use case is the right entry point. Avoid Preview for inference — it gives you no backpressure control.

val imageAnalysis = ImageAnalysis.Builder()
    .setTargetResolution(Size(640, 480))
    .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
    .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888)
    .build()

imageAnalysis.setAnalyzer(inferenceExecutor) { imageProxy ->
    val bitmap = imageProxy.toBitmap()
    landmarker.detectAsync(
        MPImage.fromBitmap(bitmap),
        imageProxy.imageInfo.timestamp
    )
    imageProxy.close()
}

STRATEGY_KEEP_ONLY_LATEST is non-negotiable. Under inference backpressure, you drop frames rather than queue them. Queuing is how you accumulate 300ms of lag that users perceive as the makeup “chasing” their face.

---

The 478-landmark mesh and blend shape budget

MediaPipe Face Landmarker v2 produces a canonical 478-point mesh covering the full facial surface including the iris contours (landmarks 468–477). For makeup, you primarily operate on three sub-regions:

• Lips: landmarks 61–291 (outer contour + inner vermilion boundary)
• Eyes/lids: landmarks 33–263 (with dedicated lid-crease indices)
• Cheeks: landmarks 50, 280, 330, 100 as anchor quads for blush polygon fill

Blend shape coefficients give you 52 expression weights (ARKit-compatible naming). For makeup specifically, eyeBlinkLeft, eyeBlinkRight, jawOpen, and mouthSmile* are the ones worth reacting to — they drive the deformation of lipstick and eyeshadow layers when the user speaks or smiles.

val result: FaceLandmarkerResult = // from callback
val blendShapes = result.faceBlendshapes().get()[0]
val smileCoeff = blendShapes
    .find { it.categoryName() == "mouthSmileLeft" }
    ?.score() ?: 0f

// Deform lip mesh UVs proportionally
lipMeshUVs = deformLipUVs(baseLipUVs, smileCoeff)

---

GPU delegate memory layout and tiled texture rendering

Most teams get this wrong: they let TFLite allocate its own textures and then copy results back to CPU for rendering. That round-trip kills your frame budget.

The correct approach is to configure the delegate with serialized model caching and keep compositing entirely on-GPU:

val gpuOptions = GpuDelegateFactory.Options().apply {
    isPrecisionLossAllowed = true   // INT8 activations, ~2x throughput
    inferencePreference = GpuDelegateFactory.Options
        .INFERENCE_PREFERENCE_SUSTAINED_SPEED
}

For makeup compositing, use tiled textures — one 512×512 atlas covering lip colors, blush gradients, and eyeshadow variants. A single glBindTexture + UV remap per frame beats multiple draw calls by a wide margin on mobile tile-based deferred renderers (Adreno, Mali).

---

Frame pacing

Use Choreographer.FrameCallback to synchronize inference submission with vsync, not a raw executor loop:

Choreographer.getInstance().postFrameCallback { frameTimeNanos ->
    if (latestFrame != null) submitInference(latestFrame!!)
    Choreographer.getInstance().postFrameCallback(this)
}

Combine this with an EGL sync fence after the render pass — eglCreateSyncKHR — so you never submit a new frame while the GPU is still compositing the previous one. This eliminates the tearing artifacts that appear when lipstick renders half-updated during a blink.

---

Before you ship

Three things worth saying twice:

1. Use STRATEGY_KEEP_ONLY_LATEST without exception. Frame queuing under inference backpressure destroys perceived latency. Drop frames, never buffer them.

2. Keep compositing on-GPU end-to-end. Avoid CPU readbacks between inference and rendering. A texture atlas + UV deformation driven by blend shape coefficients is both simpler and faster than per-frame CPU mesh reconstruction.

3. Budget blend shapes before you ship. Not all 52 coefficients are cheap to react to. Profile mouthSmile* and jawOpen deformations on your target Snapdragon tier before adding expression-driven makeup layers — blend shape processing on CPU is where frame budgets quietly die.

---

#android #mobile #architecture #jetpackcompose #kotlin