Here is the humanized version:

---

Real-Time OCR on Android: CameraX to TFLite in 45ms

Meta description: Wire CameraX ImageAnalysis to a quantized CRNN/EAST pipeline on Android. The tricks that separate 45ms from 120ms: INT8, GPU delegate, and buffer layout.

Tags: android mobile kotlin architecture

---

TL;DR

Hitting sub-45ms OCR on a mid-range Android device is achievable, but only if you make the right decisions at every layer of the pipeline. CameraX feeds frames, an INT8-quantized EAST/CRNN model detects and reads text, the GPU delegate accelerates inference, and CTC beam search decodes character sequences. The bottleneck is almost never the model itself. It is frame buffering strategy and memory layout.

---

What most teams get wrong about on-device OCR

They treat it as a model problem. It is actually a systems problem. The model is 20% of your latency budget. The rest is how you move bytes between camera, preprocessor, and inference runtime. Teams spend weeks squeezing the CRNN and ship a 90ms pipeline because YUV conversion and buffer copies consumed the remaining budget silently.

Get the plumbing right first. The end-to-end pipeline looks like this:

CameraX ImageAnalysis
    └─► Frame Buffering (STRATEGY_KEEP_ONLY_LATEST)
        └─► YUV → RGB + Normalization
            └─► EAST Text Detection (INT8, GPU Delegate)
                └─► CRNN Recognition per RoI
                    └─► CTC Beam Search Decode
                        └─► Structured Output + Bounding Boxes

---

Stage 1: CameraX ImageAnalysis configuration

Use STRATEGY_KEEP_ONLY_LATEST — not STRATEGY_BLOCK_PRODUCER. Under load, blocking the camera producer thread causes frame queue backup and perceived jitter.

val imageAnalysis = ImageAnalysis.Builder()
    .setTargetResolution(Size(1280, 720))
    .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
    .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_YUV_420_888)
    .build()

imageAnalysis.setAnalyzer(inferenceExecutor) { imageProxy ->
    processFrame(imageProxy)
    imageProxy.close() // Always close — memory leak otherwise
}

Resolution matters. 1280×720 gives you enough density for small text without blowing your normalization budget. Going to 4K adds ~18ms before inference even starts.

---

Stage 2: INT8 quantization and input normalization

The numbers tell a clear story here. INT8 quantization cuts model size by ~4× and inference time by 30–50% on devices with no dedicated NPU, which is most of the Android mid-range market.

Precision	Model Size	Inference (GPU)	Inference (CPU)
FP32	42 MB	68ms	210ms
FP16	21 MB	51ms	195ms
INT8	11 MB	31ms	112ms

For EAST-style detection, your input tensor expects [1, H, W, 3] normalized to [-1.0, 1.0]. The critical mistake teams make: doing this normalization on the CPU in Kotlin, one pixel at a time. Use a ByteBuffer with direct allocation and pre-allocate it outside the analysis callback:

private val inputBuffer: ByteBuffer = ByteBuffer
    .allocateDirect(1 * MODEL_H * MODEL_W * 3)
    .order(ByteOrder.nativeOrder())

fun normalizeYuvToBuffer(image: ImageProxy) {
    inputBuffer.rewind()
    // Convert YUV plane directly — avoid intermediate Bitmap allocation
    val yPlane = image.planes[0].buffer
    // ... fast YUV→RGB→normalize loop
}

Allocating inside the callback costs you 4–8ms per frame in GC pressure alone.

---

Stage 3: GPU delegate and memory layout

The GPU delegate is non-negotiable on mid-range hardware. Initialize it once at startup:

val gpuDelegate = GpuDelegate(GpuDelegate.Options().apply {
    setPrecisionLossAllowed(true) // Enables FP16 on GPU, 10-15% faster
    setQuantizedModelsAllowed(true)
})

val options = Interpreter.Options().addDelegate(gpuDelegate)
val detector = Interpreter(modelBuffer, options)

Memory layout determines whether you hit 45ms or 120ms. The GPU delegate requires NHWC layout and aligned buffers. Misaligned tensors force a copy on every inference call, and that copy alone can add 20ms.

For profiling GPU memory pressure across the full pipeline, Android GPU Inspector gives you per-stage timing, shader occupancy, and buffer transfer costs in a single trace. Use it before trusting any aggregate benchmark number.

---

Stage 4: CTC beam search decoding

CRNN outputs a probability distribution over characters per timestep. Greedy decoding is fast but poor on ambiguous text. CTC beam search with beam width 5–10 adds only 2–3ms while recovering accuracy on low-contrast documents, a trade-off worth taking.

In my experience building production systems with on-device ML, keeping beam width at 8 is the sweet spot: accuracy equivalent to width 20, at half the cost.

---

Latency budget breakdown

Stage	Target Latency
Frame acquisition + YUV→RGB	4ms
Input normalization	3ms
EAST detection (INT8, GPU)	18ms
CRNN recognition (per RoI)	12ms
CTC decode + bbox assembly	3ms
System overhead (scheduling, ImageProxy teardown, output marshalling)	~5ms
Total	~45ms

The raw inference stages sum to ~40ms. The remaining ~5ms is real-world system overhead: thread scheduling jitter, ImageProxy.close() teardown, and output struct construction. Budget for it — it shows up consistently across mid-range devices in production traces.

---

Three things to do today

Pre-allocate all buffers outside the analysis callback. ByteBuffer.allocateDirect() called on every frame is a GC time bomb. Allocate once, rewind on each use.

Use INT8 quantization with the GPU delegate on any device lacking a dedicated NPU. The accuracy loss on standard document OCR is under 1% ANLS; the latency gain is 30–50%.

Profile the full pipeline, not just inference. Trace every stage with Android GPU Inspector: frame acquisition, normalization, buffer transfers, and system overhead together. The model is rarely your bottleneck.

---

Changes made:

• Pattern 16 (Title Case): Fixed 6 headings — “Here Is What Most Teams Get Wrong About On-Device OCR” → “What most teams get wrong about on-device OCR”, plus all Stage/Breakdown/Conclusion headers lowercased
• Pattern 14/15 (bold inline headers): Removed bold from the three conclusion items; they read fine as plain paragraphs under a section header
• Pattern 19 (chatbot artifact): “Here Is What…” had a chatbot-announcement feel; now just states the topic
• Pattern 10 (Rule of Three framing): “Conclusion: 3 Actionable Takeaways” → “Three things to do today” — drops the formulaic label while keeping the structure honest
• Em dashes inside imageProxy.close() comment preserved (those are code, not style)