CoreML in iOS 18: ANE vs GPU vs CPU fallback chains

Meta description: CoreML’s compute unit scheduler silently demotes ops to CPU with no warning. Learn to detect CPU fallback at runtime with MLComputePlan, instrument ANE dispatch, and control delegation in iOS 18.

Tags: ios swift mobile architecture

---

TL;DR

CoreML’s automatic compute unit selection isn’t magic — it’s a graph-partitioning algorithm that silently demotes unsupported ops to CPU, often tripling inference latency with zero runtime warning. In iOS 18, detecting silent CPU fallback with MLComputePlan, instrumenting ANE dispatch, and constraining delegation at compile time is the difference between a real-time pipeline that ships and one that misses frame deadlines by 40ms.

---

The compute unit hierarchy

CoreML exposes four compute unit policies via MLComputeUnits:

Policy	What it actually does
.all	ANE preferred, GPU fallback, CPU last resort
.cpuAndNeuralEngine	ANE + CPU only, no GPU
.cpuAndGPU	GPU + CPU, ANE explicitly excluded
.cpuOnly	Deterministic, slowest, no delegation

The default .all sounds optimal. In practice it’s a black box that hands your op graph to a scheduler you can’t observe without deliberate instrumentation.

---

How iOS 18 schedules the ANE

When CoreML compiles your model, it partitions the computation graph into subgraphs based on operator support tables per compute unit. The ANE supports a well-defined but underdocumented op set — primarily quantized convolutions, matrix multiplications, and activation functions in specific data layouts.

The critical failure mode: a single unsupported op forces an entire subgraph onto the CPU, not just that op. No exception. No warning in the logs. It happens silently at model load time.

Common ANE-incompatible patterns:

• Dynamic shapes without a fixed upper bound
• Custom MLCustomLayer implementations
• Certain gather/scatter ops with non-static indices
• bfloat16 activations on older ANE generations

---

Detecting CPU fallback at runtime with MLComputePlan

Most teams assume .all means ANE. It doesn’t. MLComputePlan, available since iOS 17, lets you enumerate per-operation compute device assignments before you run a single inference pass. This is the right tool for verifying ANE delegation — not guesswork from conversion logs, not hoping Instruments will surface the problem after the fact.

import CoreML

// iOS 17+: enumerate per-op compute assignments before inference runs
let config = MLModelConfiguration()
config.computeUnits = .all

let modelURL = Bundle.main.url(forResource: "MyModel", withExtension: "mlmodelc")!

MLComputePlan.load(contentsOf: modelURL, configuration: config) { result in
    guard case .success(let plan) = result,
          case .program(let program) = plan.modelStructure else { return }

    for (functionName, function) in program.functions {
        for block in function.blocks {
            for op in block.operations {
                guard let usage = plan.computeDeviceUsage(for: op) else { continue }
                switch usage.preferredComputeDevice {
                case .cpu:
                    print("[CPU FALLBACK] \(op.operator.name) in '\(functionName)'")
                case .gpu:
                    print("[GPU] \(op.operator.name)")
                case .neuralEngine:
                    break // expected path
                @unknown default:
                    break
                }
            }
        }
    }
}

Run this in your CI validation pass, not just locally. A model that delegates cleanly on a simulator may fall back on physical hardware due to ANE microarchitecture differences between A-series generations.

---

Instrumenting inference with os_signpost

Pair MLComputePlan analysis with signpost instrumentation to isolate dispatch overhead from compute time in Instruments:

import os.signpost

let log = OSLog(subsystem: "com.yourapp.inference", category: .pointsOfInterest)

func runInference(input: MLFeatureProvider) throws -> MLFeatureProvider {
    let id = OSSignpostID(log: log)
    os_signpost(.begin, log: log, name: "CoreML Inference", signpostID: id)
    defer { os_signpost(.end, log: log, name: "CoreML Inference", signpostID: id) }
    return try model.prediction(from: input)
}

Open Instruments → Core ML Instrument. Per-subgraph timelines are labelled by compute unit. Any CPU segments in a model configured as .all confirm a fallback — and you now have the MLComputePlan output to pinpoint the offending op.

---

Constraining delegation at conversion time

In coremltools, set your deployment target accurately before the .mlpackage reaches the device:

import coremltools as ct

model = ct.convert(
    source_model,
    compute_units=ct.ComputeUnit.ALL,
    minimum_deployment_target=ct.target.iOS18
)

# coremltools does not expose delegation decisions at conversion time.
# Use MLComputePlan on-device (iOS 17+) to inspect per-op assignments.
model.save("MyModel.mlpackage")

minimum_deployment_target matters more than most teams realize. Setting it to iOS18 unlocks ANE op support unavailable when targeting earlier OS versions — even on iOS 18 hardware running a model compiled for iOS 16. If you’ve left this at iOS15 and can’t figure out why latency is worse than expected, that’s probably why.

---

The latency gap is real

On an iPhone 15 Pro (A17 Pro) running a MobileNetV3-style vision model, measured inference latencies look roughly like this:

Compute path	Latency (approx., iPhone 15 Pro / A17 Pro)
ANE	8–12ms
GPU	15–25ms
CPU	60–120ms

A real-time pose estimation pipeline at 30fps has roughly 33ms per frame. One subgraph falling back to CPU blows past that ceiling before you catch it in testing. Transformer-heavy models and attention-based layers will land at different absolute numbers, but the gap between ANE and CPU is consistent across model families on Apple silicon.

---

Actionable takeaways

1. Assert zero CPU fallback in CI. Run the MLComputePlan enumeration snippet against every model update and fail the build if any op on your critical path resolves to .cpu with .all configured.

2. Set minimum_deployment_target to your actual floor. Mismatched targets silently suppress ANE ops your hardware fully supports. The fix is one argument in your conversion script.

3. Isolate known-incompatible ops into a separate .cpuOnly model. If your graph contains a custom layer or unsupported op you can’t remove, partition it out explicitly so it can’t contaminate subgraph scheduling for the rest of your ANE-bound pipeline.