Real-time CoreML in SwiftUI with Swift 6

Meta description: Master CoreML async prediction with SwiftUI using priority queues, ANE batch scheduling, and Swift 6 structured concurrency to maintain 60fps on-device.

---

TL;DR

CoreML’s MLPredictionOptions and the Apple Neural Engine give you serious on-device inference throughput — but only if you respect the ANE’s memory budget, avoid concurrent model instance contention, and pipe requests through a structured priority queue. This post walks through the full architecture using Swift 6 async sequences and structured concurrency to keep your SwiftUI frame budget intact.

---

On-device inference is the right default now. Privacy, latency, and infra costs all point the same direction: if the model fits on the ANE, run it there. The hard part is doing it without wrecking your frame rate.

---

The ANE memory budget is your ceiling

Most teams treat CoreML like a background HTTP call. It isn’t. The Apple Neural Engine has a fixed memory budget per compiled model load, and running multiple model instances simultaneously doesn’t parallelize — it contends.

Configuration	ANE Utilization	Avg Latency (MobileNetV3)	Frame Drop Risk
1 instance, sequential	~60%	4.2 ms	Low
2 instances, concurrent	~95%	9.8 ms	High
1 instance, batched (batch=4)	~75%	6.1 ms	Low
CPU fallback (no ANE)	N/A	31 ms	Very High

Measured on iPhone 15 Pro, iOS 17.4, MobileNetV3-Small compiled to Float16 via coremltools. Results will vary by device generation and model precision.

One instance with batch scheduling beats concurrent instances on both latency and stability. The ANE scheduler does not thank you for parallelism.

---

Designing the priority queue dispatch layer

The architecture I use in production: a single MLModel instance behind an actor-isolated scheduler, fed by an AsyncStream that accepts prioritized requests from the SwiftUI layer.

TaskPriority doesn’t conform to Comparable in Swift 6, so sort on rawValue directly — higher values map to higher OS priority:

actor ANEScheduler {
    private let model: MLModel
    private var queue: [PredictionRequest] = []
    private var isDraining: Bool = false

    struct PredictionRequest {
        let pixelBuffer: CVPixelBuffer
        let priority: TaskPriority
        let continuation: CheckedContinuation<MLFeatureProvider, Error>
    }

    init(modelURL: URL) throws {
        let config = MLModelConfiguration()
        config.computeUnits = .cpuAndNeuralEngine
        self.model = try MLModel(contentsOf: modelURL, configuration: config)
    }

    func enqueue(_ buffer: CVPixelBuffer, priority: TaskPriority) async throws -> MLFeatureProvider {
        try await withCheckedThrowingContinuation { continuation in
            queue.append(.init(pixelBuffer: buffer, priority: priority, continuation: continuation))
            queue.sort { $0.priority.rawValue > $1.priority.rawValue }
            if !isDraining {
                Task { await self.drain() }
            }
        }
    }

    private func drain() async {
        guard !isDraining else { return }
        isDraining = true
        defer { isDraining = false }
        while let request = queue.first {
            queue.removeFirst()
            do {
                let options = MLPredictionOptions()
                options.usesCPUOnly = false
                let input = try MLDictionaryFeatureProvider(dictionary: ["image": request.pixelBuffer])
                let result = try model.prediction(from: input, options: options)
                request.continuation.resume(returning: result)
            } catch {
                request.continuation.resume(throwing: error)
            }
        }
    }
}

Two correctness details worth calling out. First, isDraining prevents redundant drain tasks from spawning on every enqueue call — the flag is checked before creating the Task, and the guard inside drain() is a safety net for any re-entrant path. Second, the while loop replaces the original single-item dequeue: without it, items queued while a drain is in progress stall until the next enqueue call triggers another task. That’s a subtle production bug under bursty load.

---

Wiring it to SwiftUI with Swift 6 async sequences

In SwiftUI, the camera feed arrives as a CMSampleBuffer stream. The bridge uses AsyncStream to decouple capture rate from inference rate — you need this to hold 60fps rendering while inference runs at 15–30fps.

struct ClassifierView: View {
    @State private var label: String = "Analyzing..."
    let predictionStream: AsyncStream<String>

    var body: some View {
        Text(label)
            .task {
                for await result in predictionStream {
                    label = result
                }
            }
    }
}

The stream is produced by a CameraCoordinator that submits frames to the ANEScheduler at a throttled rate using Clock.sleep. Decoupling frame capture from inference dispatch is the single most effective change I’ve seen teams make to eliminate jank.

---

Batch scheduling: when to group requests

MLPredictionOptions exposes batch prediction through MLArrayBatchProvider. Batching pays off when you’re running classification over a video timeline — seek preview, gallery thumbnails — rather than a live camera feed.

let batchProvider = MLArrayBatchProvider(array: inputs)
let options = MLPredictionOptions()
options.usesCPUOnly = false
let results = try model.predictions(fromBatch: batchProvider, options: options)

For live camera inference, don’t batch. The added latency per frame exceeds the throughput gain. For offline or scrubbing workloads, batch sizes of 4–8 are the sweet spot before diminishing ANE returns.

---

Avoiding the most expensive mistake

Loading MLModel on the main thread during view initialization. In my experience building production systems, this causes more launch-time ANE failures than anything else. Always load asynchronously at app startup using a Task in @main, cache the ANEScheduler in your SwiftUI environment, and never reload the model per-view.

---

Takeaways

• Single model instance, actor-isolated. One MLModel behind an actor scheduler with an isDraining loop eliminates ANE contention and the subtle stall bug from single-item drain calls. Never load two instances of the same model concurrently expecting a speedup.

• Decouple capture rate from inference rate. Use AsyncStream as a buffer between your camera pipeline and your CoreML scheduler. Let SwiftUI render at 60fps; let inference run at whatever rate the ANE sustains without frame pressure.

• Batch only for offline workloads. MLArrayBatchProvider is powerful for gallery or timeline inference. For real-time classification, sequential single-frame requests through a priority queue will outperform batching in both latency and frame-rate stability.

---

#ios #swiftui #mobile #architecture