MLX Swift: Run Fine-Tuned LLMs Without the CoreML Tax

Meta description: Skip CoreML compilation overhead. MLX Swift bindings give you direct Metal-backed access to run quantized fine-tuned LLMs on Apple Silicon with Swift 6 concurrency.

Tags: swift swiftui mobile architecture backend

---

TL;DR

CoreML imposes real latency costs: model compilation on first load, ANE scheduling overhead, and limited control over KV-cache. MLX Swift bindings give you direct access to the Metal-backed compute graph on M-series chips, cutting time-to-first-token for quantized LLMs. This post covers model loading via swift-transformers, MLX array operations, and KV-cache management using Swift 6 actors — with benchmark context for where CoreML wins and where MLX dominates.

---

Why CoreML becomes a bottleneck for fine-tuned LLMs

When engineers reach for on-device LLM inference on Apple hardware, CoreML is the default assumption. It has solid tooling, tight OS integration, and works well for vision and classification tasks. But for autoregressive text generation — particularly fine-tuned models you’re loading from Hugging Face — CoreML’s strengths become liabilities.

The core issue: CoreML requires a compiled .mlmodelc package. First-load compilation for a 3B parameter model can take 15–45 seconds on M2. You can pre-compile and ship the artifact, but now your app bundle balloons, and every fine-tune iteration requires a new compilation cycle. For teams iterating on fine-tuned models, this friction compounds fast.

MLX, Apple’s array framework for machine learning on Apple Silicon, sidesteps this entirely.

---

The MLX advantage: direct Metal, lazy evaluation, unified memory

MLX operates on a lazy computation graph backed by Metal. Operations on MLXArray are not executed until explicitly evaluated, allowing the framework to fuse kernels and optimize memory layout across the unified memory architecture that M-series chips use.

import MLX
import MLXRandom

// Lazy — no compute happens here
let weights = MLXArray(converting: loadedFloatArray)
let input = MLXArray(tokenIds, dtype: .int32)

// Compute graph is built, then evaluated in one fused pass
let logits = model(input, weights)
MLX.eval(logits)

This is different from CoreML’s eager execution model, where each layer boundary is a potential synchronization point.

---

Loading fine-tuned models from Hugging Face

The swift-transformers library handles tokenization and config parsing, while MLX handles tensor operations. Weights should be converted to .safetensors format — the mlx-lm Python toolchain handles this conversion — and loaded using MLX’s NumpyFile-based utilities or the weight-loading helpers in the mlx-swift examples repository.

Note: The snippet below is illustrative of the overall pattern. Exact API signatures depend on the version of mlx-swift and swift-transformers you pin — consult the respective repositories for current method names, as this ecosystem is evolving quickly.

import Transformers
import MLX

// Tokenizer loaded via swift-transformers
let tokenizer = try await AutoTokenizer.from(pretrained: "your-org/your-finetuned-model")

// Weights loaded from locally converted .safetensors checkpoint
// Actual loading uses mlx-swift's NumpyFile or equivalent weight utilities
let weightsURL = localCheckpointURL.appendingPathComponent("weights.safetensors")
let arrays = try MLX.loadArrays(url: weightsURL) // illustrative; see mlx-swift docs

You convert weights once — no per-device compilation. The same checkpoint loads on M1, M2, and M3 with no changes.

---

Platform reality: macOS first, iOS with caveats

MLX Swift targets macOS on Apple Silicon as its primary deployment environment. iOS support exists but carries significant constraints:

• Memory budget: A 7B model at 4-bit quantization requires approximately 4GB of RAM. This eliminates virtually every iPhone currently in the field — even the iPhone 15 Pro tops out at 8GB, leaving little headroom alongside the OS and app stack.
• Thermal limits: Sustained generation on iPhone triggers aggressive throttling far sooner than on MacBook or Mac mini hardware.
• ANE exposure: MLX runs GPU-primary, which on iOS trades ANE efficiency for flexibility. For battery-sensitive workloads, this matters.

In practice, MLX Swift is the right choice for macOS apps and developer tooling. If you need on-device LLM inference on iPhone today, a heavily quantized sub-2B model with CoreML or a dedicated framework like llama.cpp via C interop is more realistic than a full MLX pipeline.

---

KV-cache management with Swift 6 actors

Autoregressive generation requires careful KV-cache handling to avoid redundant computation. In Swift 6, actors give you the isolation guarantees you need:

actor KVCache {
    private var keys: [MLXArray] = []
    private var values: [MLXArray] = []

    func append(key: MLXArray, value: MLXArray) {
        keys.append(key)
        values.append(value)
    }

    func concatenated() -> (MLXArray, MLXArray) {
        return (MLX.concatenated(keys, axis: 1),
                MLX.concatenated(values, axis: 1))
    }
}

The actor boundary prevents data races across the generation loop while keeping the cache on-device without copying back to CPU.

---

Benchmark comparison: MLX vs CoreML on M-series

For generation-heavy workloads, the gap is real. These figures are representative of community benchmarks reported across the mlx-swift and llm-benchmark repositories; results vary meaningfully by thermal state, chip generation, and memory pressure — treat them as directional, not absolute.

Metric	CoreML (ANE)	MLX (Metal GPU)
First-load compile (3B model)	15–45s	0s (no compile)
Time-to-first-token (4-bit quant)	~120ms	~60–80ms
Tokens/sec (sustained, 7B 4-bit)	~25 tok/s	~40–55 tok/s
Fine-tune iteration cycle	Recompile required	Reload weights only
ANE utilization	High	Low (GPU-primary)

Methodology note: Representative figures from community benchmarks run on M2 Pro (macOS 14.x), Mistral-7B-Instruct at 4-bit quantization, single-sequence generation (batch size 1), measured wall-clock from prompt submission to first token (TTFT) and averaged over 50 generation steps (throughput). Your results will vary by chip SKU, thermal state, and model architecture.

CoreML retains an edge for inference pipelines that benefit from the ANE — particularly smaller vision models and batch classification. But for LLM generation with dynamic sequence lengths and frequent model updates, MLX wins on latency and developer velocity.

---

Where CoreML still wins

In my experience building production systems, the right answer is usually “both, for different workloads.” CoreML excels when:

• The model is static and ships with the app (no iteration cycle)
• You need ANE for battery efficiency on sustained inference
• You’re targeting older A-series chips or iOS devices where MLX’s memory footprint is prohibitive

For dynamic fine-tuned LLM workflows updated from a server and running on macOS, MLX is the better path.

---

Actionable takeaways

1. Convert fine-tuned weights to .safetensors via mlx-lm once and load directly. This eliminates the CoreML compilation bottleneck from your iteration cycle and keeps your checkpoint format portable across chip generations.

2. Wrap KV-cache in a Swift 6 actor. You get Sendable conformance and data-race safety for free; the isolation cost is negligible compared to generation latency.

3. Profile time-to-first-token, not just throughput. For interactive applications, TTFT is the user-perceived metric that matters — this is where MLX’s lazy graph evaluation delivers the most visible win over CoreML’s eager dispatch. Instrument it before committing to an inference backend.