CameraX + Quantized CLIP: Zero-shot vision at 30fps

---

TL;DR

You can run a quantized CLIP vision encoder on-device in Android, fed by CameraX frame data, computing cosine similarity against a pre-built label embedding matrix accelerated by XNNPACK — and hold 30fps without triggering the OOM killer. Memory layout, quantization strategy, and label set discipline are what actually determine whether you succeed.

---

The problem most teams get wrong

Most teams treat on-device zero-shot classification as a model problem when it’s actually a pipeline problem. Swapping in a smaller model doesn’t save you if your frame-to-tensor conversion stalls on the main thread, or if your embedding matrix lives in heap memory that GC can fragment at the worst moment.

At 30fps, you have a 33ms budget per frame. A quantized CLIP vision encoder in INT8 typically costs 18–22ms on a mid-range Snapdragon. That leaves 11–15ms for everything else — CameraX buffer acquisition, preprocessing, similarity scoring, and UI dispatch. That margin is not a suggestion; it’s a hard ceiling.

---

Architecture overview

The pipeline has five stages.

CameraX ImageAnalysis
        │
        ▼
  YUV→RGB Conversion (libyuv, prefer over deprecated RenderScript on API 31+)
        │
        ▼
  TFLite Interpreter (CLIP Vision Encoder, INT8, XNNPACK delegate)
        │
        ▼
  L2-Normalized Image Embedding [1×512]
        │
        ▼
  Cosine Similarity Scores → ArgMax → Label

The label embedding matrix is computed once at startup — you run your text prompts through the CLIP text encoder, L2-normalize each vector, and freeze the result as a [N×512] float32 buffer. Enable the XNNPACK delegate when constructing your Interpreter so that TFLite automatically pre-packs weight matrices into its tiled layout before any inference call:

val xnnpackDelegate = XNNPackDelegate(
    XNNPackDelegate.Options().apply { numThreads = 2 }
)
val interpreter = Interpreter(
    modelBuffer,
    Interpreter.Options().addDelegate(xnnpackDelegate)
)

With the delegate active, weight pre-packing happens once at model load time, not per inference.

---

Quantization strategy

Precision	Model Size	Inference (Pixel 7)	Accuracy Drop
FP32	~350MB	~55ms	Baseline
FP16	~175MB	~38ms	<0.5%
INT8 (PTQ)	~88MB	~19ms	1–3%
INT8 (QAT)	~88MB	~19ms	<1%

Post-training quantization (PTQ) gets you to the right ballpark. Quantization-aware training (QAT) recovers most of the accuracy regression and is worth the training cost if you control the model. For CLIP specifically, the vision encoder quantizes well; the text encoder is only run offline, so its precision is irrelevant to the runtime budget.

---

The XNNPACK pre-packing detail that actually matters

When the XNNPACK delegate is enabled and your model is loaded, TFLite pre-packs the weight tensors — including your frozen label embedding matrix — into a tiled memory layout optimized for XNNPACK’s matrix multiplication kernels. Passing the delegate at Interpreter construction time is sufficient; no manual packing API is required or available through the public TFLite interface.

What you must not do: recreate the interpreter between frames, or reload the model mid-session. Either will trigger re-packing and cost you 4–8ms plus GC pressure. In my experience building production systems with on-device ML, interpreter lifetime should match session lifetime — allocate once, release only when the use case is torn down.

---

CameraX integration

Use ImageAnalysis with STRATEGY_KEEP_ONLY_LATEST — you want the freshest frame, not a queue of stale ones. setTargetResolution is deprecated as of CameraX 1.3; use ResolutionSelector with ResolutionStrategy instead:

val resolutionSelector = ResolutionSelector.Builder()
    .setResolutionStrategy(
        ResolutionStrategy(
            Size(224, 224),
            ResolutionStrategy.FALLBACK_RULE_CLOSEST_HIGHER_THEN_LOWER
        )
    )
    .build()

val imageAnalysis = ImageAnalysis.Builder()
    .setResolutionSelector(resolutionSelector)
    .setBackpressureStrategy(STRATEGY_KEEP_ONLY_LATEST)
    .setOutputImageFormat(OUTPUT_IMAGE_FORMAT_RGBA_8888)
    .build()
    .also { it.setAnalyzer(executor, ::analyzeFrame) }

Run the analyzer on a dedicated Executor backed by a single thread — frame analysis must not compete with the main thread or Compose recomposition.

---

Memory discipline under 512MB

• Keep the TFLite interpreter allocated for the session lifetime — interpreter creation costs ~40ms and triggers GC pressure
• Allocate input/output tensors as ByteBuffer.allocateDirect() — off-heap, GC-invisible
• XNNPACK’s pre-packed weight buffers live in native memory; they do not count against your Java heap

The OOM killer on Android targets the largest contiguous Java heap allocation first. Native buffers are your friend here.

---

What actually matters

1. Enable the XNNPACK delegate at interpreter construction time and never recreate the interpreter mid-session. Pre-packing happens once at load; any interpreter teardown forces a repack cycle that burns 4–8ms and spikes GC pressure at the worst possible moment.

2. Use INT8 PTQ as your baseline, validate accuracy on your target label distribution, and invest in QAT only if the regression exceeds your product threshold. Model size and latency gains are identical between the two; accuracy is the only variable.

3. Freeze your label embeddings at init and treat them as immutable native memory. Any runtime reallocation of that matrix will introduce GC pauses that spike your 99th-percentile frame time and cause visible stutters at exactly the wrong moment.

---

#android #mobile #architecture #kotlin