Gemma 3n on Android: Sub-2GB multimodal inference

SEO Meta Description: Deploy Google’s Gemma 3n multimodal model on Android under 2GB RAM using ExecuTorch’s XNNPACK and NNAPI delegates — covering export, quantization, image preprocessing, and threading.

---

TL;DR

Gemma 3n is Google’s multimodal on-device model built for constrained hardware. Paired with ExecuTorch’s NNAPI and XNNPACK delegates, you can serve image+text inference on mid-range Android devices without a network call. The pipeline is non-trivial — model export, quantization, delegate selection, ImageReader preprocessing, and threading all require deliberate engineering choices. This post walks through each layer.

---

Why Gemma 3n and ExecuTorch, not TFLite?

Most teams reach for TFLite out of habit, then discover its quantization story for large generative models is painful and its multimodal graph support is immature.

ExecuTorch — Meta’s production on-device inference runtime — takes a different approach. It delegates computation at the operator level, which means XNNPACK handles float/quantized CPU math while NNAPI offloads eligible ops to NPU or GPU accelerators. Gemma 3n’s architecture is explicitly designed for this kind of hardware-aware partitioning.

Framework	Multimodal Support	NNAPI Delegation	Quantization Tooling	Maturity
TFLite	Limited	Yes (coarse)	Good (int8/float16)	Stable
ExecuTorch	Native	Yes (op-level)	Excellent (GPTQ, PT2E)	Production-ready
ONNX Runtime Mobile	Partial	Via QNN EP	Moderate	Growing

Op-level delegation means more ops reach the NPU. Gemma 3n’s MatMul-heavy transformer blocks are exactly what NPU accelerators are built for — that gap becomes obvious once you profile on real hardware.

---

The export pipeline

Start with the PyTorch checkpoint. ExecuTorch uses torch.export with a two-phase flow:

import torch
from executorch.exir import to_edge, EdgeCompileConfig

# Export with dynamic image+text shapes
exported = torch.export.export(
    model,
    args=(image_tensor, input_ids, attention_mask),
    dynamic_shapes={
        "image_tensor": {0: torch.export.Dim("batch")},
        "input_ids": {1: torch.export.Dim("seq_len", max=512)},
    }
)

edge_program = to_edge(
    exported,
    compile_config=EdgeCompileConfig(_check_ir_validity=True)
)

The to_edge step lowers to ExecuTorch’s portable IR. From there, apply your delegate backends before serializing to .pte.

---

Quantization for the sub-2GB target

Gemma 3n in bf16 lands well above the 2GB ceiling for mid-range devices. Post-training quantization via PT2E (PyTorch 2 Export Quantization) with per-channel int8 weights and dynamic activations gets you there:

from torch.ao.quantization.quantize_pt2e import prepare_pt2e, convert_pt2e
from executorch.backends.xnnpack.quantizer import XNNPACKQuantizer

quantizer = XNNPACKQuantizer().set_global(
    get_symmetric_quantization_config(is_per_channel=True, is_dynamic=True)
)
prepared = prepare_pt2e(exported_program, quantizer)
# Run calibration with representative image+text pairs
converted = convert_pt2e(prepared)

Int8 weights with dynamic int8 activations on linear layers typically cut memory by ~4x versus fp32. Combined with Gemma 3n’s architecture-level efficiency, staying under 2GB on a 4GB device is achievable while leaving headroom for the app runtime.

---

Image preprocessing via Android’s ImageReader

Android doesn’t have CVPixelBuffer. The equivalent pipeline runs through ImageReader → Bitmap → normalized FloatBuffer:

val imageReader = ImageReader.newInstance(width, height, ImageFormat.YUV_420_888, 2)

imageReader.setOnImageAvailableListener({ reader ->
    val image = reader.acquireLatestImage() ?: return@setOnImageAvailableListener
    val bitmap = image.toBitmap() // extension via ImageUtils
    val tensor = bitmap.toNormalizedFloatTensor(mean = IMAGENET_MEAN, std = IMAGENET_STD)
    image.close()
    inferenceQueue.offer(tensor)
}, backgroundHandler)

The YUV→RGB conversion is the expensive step. Keep it off the main thread.

---

The threading model

The architecture that keeps inference off the UI compositor:

Main Thread        ──→  UI updates only
ImageReader Thread ──→  YUV decode + normalization → CoroutineChannel
Inference Thread   ──→  ExecuTorch .forward() (pinned, high priority)
Result Thread      ──→  Dispatches to Main via Dispatchers.Main.immediate

In Kotlin:

private val inferenceDispatcher = Executors.newSingleThreadExecutor { thread ->
    thread.apply { priority = Thread.MAX_PRIORITY - 1 }
}.asCoroutineDispatcher()

suspend fun runInference(imageTensor: FloatArray, tokens: IntArray): String =
    withContext(inferenceDispatcher) {
        module.forward(imageTensor, tokens)
    }

Pinning inference to a single high-priority thread avoids lock contention on the ExecuTorch module and prevents the scheduler from interleaving inference work with compositor frames.

---

Before you ship

A few things consistently catch teams off guard:

Quantization scheme locks in at export time. PT2E quantization must be applied pre-delegation — retrofitting it after the .pte is serialized means re-exporting from scratch. On large multimodal models, that’s an expensive loop to end up in.

NNAPI delegation coverage is worth profiling explicitly. Use adb shell dumpsys nnapi and ExecuTorch’s op partitioning logs to verify what percentage of ops actually land on the NPU. CPU fallback for unsupported ops silently kills latency targets, and there’s no warning when it happens.

The ImageReader pipeline is a real bottleneck, not just scaffolding. YUV conversion and tensor normalization are consistently underestimated. Benchmark end-to-end latency from camera frame to model output — not just model execution time — to find your actual ceiling.

---

Tags: android mobile architecture kotlin multiplatform