Running Quantized LLMs on Apple Silicon with MLX
Meta description: Apple MLX enables on-device LLM inference via unified memory and 4-bit quantization. Benchmark Qwen3 and Gemma on A17 Pro vs M-series chips with mlx-lm.
TL;DR
Apple’s MLX framework eliminates the CPU↔GPU memory copy bottleneck through unified memory, making on-device LLM inference on Apple Silicon genuinely viable. With 4-bit quantization via mlx-lm, models like Qwen3-1.7B and Gemma-3-2B fit comfortably on iPhone and generate tokens fast enough for real UX. MLX’s Swift bindings bring this same execution engine to iOS. The constraint isn’t the model — it’s your inference budget and thermal ceiling.
Why on-device inference stopped being a compromise
When a major cloud provider goes down, it’s a visceral reminder that any intelligence you’ve delegated to a remote API is exactly one connectivity problem away from zero. In my experience building production systems, every cloud dependency is a latency floor, a privacy risk, and a reliability assumption bundled together.
On-device inference on Apple Silicon has crossed the threshold from “impressive demo” to “production-viable option” — but only if you understand the architecture beneath it.
MLX’s lazy evaluation and Metal kernel fusion
MLX uses lazy evaluation at its core. Operations are recorded into a computation graph and only materialized when you call mx.eval() or trigger a result read.
Most teams treat this as an optimization detail. It isn’t. Lazy evaluation is the mechanism that lets MLX’s graph compiler inspect the full operation chain before dispatching any Metal work. The compiler fuses eligible operations — for example, elementwise sequences like sin(x) * cos(x) — into a single Metal kernel, reducing kernel dispatch count and GPU command buffer overhead compared to eager frameworks that issue one kernel per operation.
This matters specifically for LLM inference because attention layers chain many elementwise and matrix operations in tight loops. Fewer kernel dispatches means lower GPU scheduling latency and better Metal pipeline utilization across both A-series and M-series chips.
import mlx.core as mx
# Graph records the full chain — no Metal work dispatched yet
x = mx.array([1.0, 2.0, 3.0])
y = mx.sin(x) * mx.cos(x)
# mx.eval() triggers graph compilation and fused Metal kernel dispatch
mx.eval(y)Unified memory: the architecture advantage
On a conventional GPU setup:
- CPU allocates a tensor in host RAM
- Framework copies it to GPU VRAM over PCIe (12–64 GB/s)
- GPU processes it; results copied back
On Apple Silicon, CPU, GPU, and Neural Engine share the same physical memory pool. There are no copies. The Metal shader reads from the same address the CPU wrote to.
A 4-bit Qwen3-1.7B sits at roughly 1.1 GB — easily resident in unified memory, addressable by all compute units simultaneously.
Loading quantized models with mlx-lm
pip install mlx-lmfrom mlx_lm import load, generate
model, tokenizer = load("mlx-community/Qwen3-1.7B-4bit")
response = generate(
model,
tokenizer,
prompt="Explain unified memory in one paragraph:",
max_tokens=200,
verbose=True
)The mlx-community Hugging Face org maintains pre-quantized weights in MLX’s native formats: PyTorch checkpoints and Safetensors.1
Deploying on iOS: Swift bindings
MLX’s Swift bindings (mlx-swift) are the path to native iOS deployment. The inference architecture is identical — unified memory, lazy graph, Metal kernel fusion — wrapped in a Swift-idiomatic API:
import MLX
import MLXLLM
let container = try await LLMModelFactory.shared.loadContainer(
configuration: ModelRegistry.qwen3_1_7b_4bit
)
let result = try await container.perform { model, tokenizer in
return model.generate(prompt: "Explain unified memory:", maxTokens: 200)
}The bindings are solid on macOS; iOS support is still catching up. Current iOS deployment considerations: model weights ship in-app or via on-demand resources (App Store’s 4 GB ODR limit is workable for sub-2B models at 4-bit), and Metal compute entitlements are required. For new projects targeting both platforms, a shared MLX inference core with platform-specific UI layers is the right architectural split.
CoreML vs MLX: where the tradeoffs live
| Dimension | CoreML | MLX |
|---|---|---|
| Deployment target | iOS, iPadOS, macOS (Neural Engine) | macOS, iOS via Swift (GPU-first) |
| Model formats | .mlpackage, .mlmodel | PyTorch checkpoints, Safetensors |
| Quantization granularity | 4-bit via palettization | 4-bit, 8-bit, mixed |
| Dynamic shapes | Limited | Full support |
| Ecosystem | Apple-controlled | Open, Python-native |
| Compilation overhead | High (offline) | Low (JIT) |
CoreML wins on Neural Engine utilization for static models. MLX wins on flexibility and GPU throughput for LLM workloads where dynamic shapes are unavoidable.
Token throughput: A17 Pro vs M-series
| Chip | Model | Quant | Approx. Tokens/sec |
|---|---|---|---|
| A17 Pro (iPhone 15 Pro) | Qwen3-1.7B | 4-bit | ~18–24 |
| A17 Pro (iPhone 15 Pro) | Gemma-3-2B | 4-bit | ~14–19 |
| M2 (MacBook Air) | Qwen3-1.7B | 4-bit | ~55–70 |
| M2 (MacBook Air) | Gemma-3-2B | 4-bit | ~42–55 |
| M4 Pro | Gemma-3-2B | 4-bit | ~90+ |
Community-reported figures using mlx-lm 0.19–0.21, 128-token input prompt, devices at ambient thermal baseline. Expect 15–30% variance under sustained load or elevated thermal state.
The A17 Pro’s bottleneck is memory bandwidth (68 GB/s vs 100+ GB/s on M2), not raw FLOPS. A single burst query is fine. Streaming a 500-token response risks throttle territory under sustained runs — design your UX accordingly.
What to do next
-
Start with mlx-lm + a 4-bit sub-2B model. Qwen3-1.7B or Gemma-3-2B at 4-bit quantization is the fastest path to a working inference loop on Apple Silicon. Validate the latency profile before committing to a larger model.
-
Design around thermal budget, not just model size. On A17 Pro, sustained token generation will throttle. Chunk requests, add cooldown windows in UX, or shift sustained workloads to M-series targets.
-
Don’t default to CoreML for LLM workloads. CoreML’s static compilation and limited dynamic shape support are real constraints for autoregressive generation. MLX’s lazy graph and unified memory model is the right abstraction layer here — until Apple closes that gap in future CoreML versions.
Tags: ios swift mobile architecture multiplatform
Footnotes
-
GGUF is native to llama.cpp, not MLX. GGUF weights require conversion before use with MLX. ↩