TFLite Delegates on Android: GPU, NNAPI, Fallback Chains

Meta: GPU delegates crash off-thread. NNAPI silently falls back to CPU. Emulators mislead by 3–5x. A production delegate selection strategy for Android.

Tags: android, mobile, architecture, kotlin, cleanarchitecture

---

Tested Environment TensorFlow Lite 2.14 · Kotlin 1.9 · minSdk 24 · Benchmarks run on physical hardware (Pixel 7, Galaxy S22)

---

TL;DR

TFLite delegate selection is not plug-and-play. GPU delegates crash when called from the wrong thread. NNAPI silently falls back to CPU on unsupported devices without throwing an exception. Emulator benchmarks mislead by a factor of 3–5x.

---

The problem you hit on day two

You wire up a GpuDelegate, run your integration test, ship to QA — and half your test devices crash on cold start. The other half are mysteriously slow. Your emulator numbers looked great.

Most teams fall into the same trap: treating delegate selection as a configuration problem when it’s fundamentally a runtime environment problem. The delegate you initialize is only as good as the hardware underneath it, the API level it runs on, and the thread it was born on.

---

GPU delegate: the thread safety trap

The GPU delegate wraps an OpenGL ES or OpenCL context. That context is thread-local. Create it on one thread, call interpreter.run() from another — you get a silent crash or undefined behavior.

The fix is non-negotiable: initialize your Interpreter and GpuDelegate together, on the inference thread.

// Must run entirely on the same thread
val gpuDelegate = GpuDelegate(
    GpuDelegate.Options().apply {
        isPrecisionLossAllowed = true  // enables fp16, ~1.4x faster on most Mali/Adreno
    }
)

val options = Interpreter.Options().apply {
    addDelegate(gpuDelegate)
}

val interpreter = Interpreter(model, options)

If you are using coroutines, pin this to a dedicated Dispatcher backed by a single thread — not Dispatchers.IO, which is a pool.

---

NNAPI: the compatibility matrix you must carry

NNAPI is available from API level 27, but the number that matters in production is API 28, where op coverage became meaningful. Below that, NnApiDelegate will initialize without error and accelerate nothing — or worse, produce partial graph acceleration with overhead that exceeds the CPU baseline (observed in our testing on API 27 emulation layers).

API Level	NNAPI Status	Practical Acceleration
< 27	Not available	None — must skip
27	Available, sparse ops	Marginal or negative (in our testing)
28	Improved op set	Conv layers, basic MobileNet
29+	Full acceleration profile	Most standard architectures
31+	NNAPI 1.3, int8 support	Production-grade for quantized models

The key option that buys you real throughput on fp32 models targeting API 28+:

val nnApiDelegate = NnApiDelegate(
    NnApiDelegate.Options().apply {
        allowFp16PrecisionForFp32 = true
        executionPreference =
            NnApiDelegate.Options.EXECUTION_PREFERENCE_FAST_SINGLE_ANSWER
    }
)

---

Building a crash-free fallback chain

GPU wins on latency, CPU wins on predictability. Your production code needs both.

fun buildDelegate(context: Context): Delegate? = runCatching {
    GpuDelegate(GpuDelegate.Options().apply { isPrecisionLossAllowed = true })
}.getOrElse {
    if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.P) {
        runCatching { NnApiDelegate() }.getOrNull()
    } else null
    // null = Interpreter falls through to multithreaded CPU
    // CPU inference is thread-safe by default — no single-thread pinning required here
}

Log which delegate was actually selected. Silent CPU fallback is the leading cause of “why is inference slow on that device” bugs in production.

---

The benchmarking harness that doesn’t lie

Emulators do not have GPU or NNAPI. Every millisecond you measure there is fiction. Use the TFLite Benchmark Tool on physical hardware:

adb shell /data/local/tmp/benchmark_model \
  --graph=/data/local/tmp/model.tflite \
  --use_gpu=true \
  --num_runs=50 \
  --warmup_runs=5

Representative numbers from a MobileNetV3-Small classification model across delegates — measured on physical hardware:

Delegate	Pixel 7 (ms)	Galaxy S22 (ms)	Emulator (ms)
CPU (4 threads)	46	58	190
GPU (fp16)	11	16	N/A
NNAPI (API 31)	14	20	820

The emulator NNAPI number is not a typo — it routes through a software emulation layer that is catastrophically slow. Teams that skip the physical device harness ship with 3–4x worse latency than they measured in CI.

---

Takeaways

1. Pin your inference thread. GpuDelegate must be created and used on the same dedicated thread — newSingleThreadContext or HandlerThread both work. CPU fallback is thread-safe by default, so this is a GPU-only concern.
2. Gate NNAPI on API 28+ and always log which delegate is actually active. Silent CPU fallback is a latency bug in disguise.
3. Benchmark on physical hardware only, using the TFLite Benchmark Tool with --warmup_runs. Emulator numbers aren’t just inaccurate — they actively lie about delegate performance.