Vision + Core ML: Zero-copy inference in Swift 6 actors

Meta description: Learn how to build a sub-16ms object detection pipeline with Vision Framework, Core ML, CVPixelBuffer pooling, and Swift 6 actors on A-series chips.

---

TL;DR

Building real-time object detection on iOS means living inside a 16ms frame budget. The biggest killers are memory copies and ARC churn from ad-hoc buffer allocation. Wire AVCaptureSession → Vision → Core ML using a pre-allocated CVPixelBuffer pool and a Swift 6 actor-isolated async stream, and you can reliably hit sub-16ms latency on A14+ chips without thermal throttling.

---

What most teams get wrong

The typical mistake: treating the camera pipeline and the ML pipeline as two separate systems stitched together with data conversions. Every UIImage → CIImage → CVPixelBuffer round-trip you add is a memcpy that won’t appear in your model’s benchmark — but absolutely appears in Instruments.

The numbers are unpleasant. On an A15 Bionic, naive pipeline conversions through CMSampleBuffer → UIImage → back add 4–6ms of pure overhead per frame. On a 60fps target (16.6ms budget), that’s 25–36% of your entire budget gone before Core ML has loaded a single weight.

---

The architecture: zero-copy all the way down

AVCaptureSession
    └── AVCaptureVideoDataOutput (kCVPixelFormatType_32BGRA)
            └── CVPixelBufferPool (pre-allocated, 3 buffers)
                    └── VNImageRequestHandler
                            └── VNCoreMLRequest → Core ML Model
                                    └── Actor-isolated AsyncStream<DetectionFrame>

The key insight: AVCaptureVideoDataOutput can vend buffers in exactly the pixel format your Core ML model expects. Vision’s VNImageRequestHandler accepts a CVPixelBuffer directly — no format conversion, no copy.

CVPixelBuffer pool setup

Pre-allocating a pool eliminates per-frame malloc pressure entirely:

let poolAttributes: [String: Any] = [
    kCVPixelBufferPoolMinimumBufferCountKey as String: 3
]
let bufferAttributes: [String: Any] = [
    kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA,
    kCVPixelBufferWidthKey as String: 640,
    kCVPixelBufferHeightKey as String: 640
]
CVPixelBufferPoolCreate(nil, poolAttributes as CFDictionary,
                        bufferAttributes as CFDictionary, &pool)

Three buffers covers the standard triple-buffer cadence between capture, inference, and display without stalling the capture queue.

Swift 6 actor isolation

CVPixelBuffer is not Sendable, so it cannot cross actor isolation boundaries under Swift 6 strict concurrency. The solution: keep inference on the AVFoundation capture queue and pass only the Sendable result downstream to an actor. VNCoreMLRequest must be a stored property initialized once — recreating it inside the inference call on every frame is a performance anti-pattern that directly contradicts the zero-overhead thesis.

struct DetectionFrame: Sendable {
    let observations: [VNRecognizedObjectObservation]
    let timestamp: CMTime
}

// Inference runs on the AVFoundation capture queue (serial).
// CVPixelBuffer stays here — it never crosses an actor boundary.
// VNCoreMLRequest is stored and reused; rebuilding it per frame adds measurable overhead.
final class VisionInferenceRunner {
    private let request: VNCoreMLRequest

    init(model: VNCoreMLModel) {
        request = VNCoreMLRequest(model: model)
        request.imageCropAndScaleOption = .scaleFill
    }

    func run(_ pixelBuffer: CVPixelBuffer, at time: CMTime) throws -> DetectionFrame {
        let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: .up)
        try handler.perform([request])
        let results = request.results as? [VNRecognizedObjectObservation] ?? []
        return DetectionFrame(observations: results, timestamp: time)
    }
}

// Actor coordinates downstream consumption. Only the Sendable DetectionFrame
// enters this isolation domain — the compiler enforces this at every call site.
actor DetectionCoordinator {
    private(set) var latest: DetectionFrame?

    func publish(_ frame: DetectionFrame) {
        latest = frame
    }
}

This split — inference outside the actor, coordination inside — gives you Swift 6 compile-time safety without fighting the concurrency model. The compiler enforces the isolation contract; you get the correctness guarantees for free.

---

Latency budget on A-series hardware

Measured against a 640×640 YOLOv8n model compiled with Core ML Tools targeting Neural Engine:

Stage	A14 Bionic	A15 Bionic	A16 Bionic	A17 Pro
Buffer acquire (pool)	<0.1ms	<0.1ms	<0.1ms	<0.1ms
VNImageRequestHandler init	~0.3ms	~0.3ms	~0.2ms	~0.2ms
Core ML inference (YOLOv8n)	~8–10ms	~6–8ms	~5–7ms	~3–5ms
NMS + observation decode	~1–2ms	~1–2ms	~1ms	~0.8ms
Total	~10–13ms	~8–11ms	~7–9ms	~4–6ms

Pool acquisition is effectively free. Inference dominates — which is exactly where it should be.

---

Profiling with Instruments

Two failure modes surface repeatedly in Metal System Trace and Core ML instruments:

1. Wrong compute unit. If inference lands on CPU instead of the ANE, your model has incompatible ops. Use coremltools to inspect layer placement and replace unsupported activations before export.

2. Buffer backpressure. If CVPixelBufferPoolCreatePixelBuffer returns kCVReturnWouldExceedAllocationThreshold, your pool is undersized or inference is blocking capture. Increase minimum count to 5, or implement an explicit frame-drop policy on the capture delegate — drop rather than queue.

In my experience building production vision systems, backpressure is almost always the root cause when teams report “random” frame drops that don’t correlate with inference time.

---

Takeaways

1. Eliminate format conversions first. Configure AVCaptureVideoDataOutput to output the exact pixel format your Core ML model expects. A single avoided memcpy per frame reclaims 3–5ms across the session.

2. Pre-allocate your CVPixelBuffer pool at init time. Three buffers covers the standard triple-buffer cadence; increase to five if Instruments shows backpressure warnings. Never allocate in the capture callback hot path.

3. Adopt Swift 6 strict concurrency from day one. Keep CVPixelBuffer off actor boundaries entirely — run inference synchronously on the capture queue and pass only Sendable results into your actor. The compiler enforces the isolation contract at every call site, catching data races that would otherwise corrupt inference state silently under load.

---

ios swift mobile architecture