CoreML model compression: cut on-device latency by 50%
Meta description: Learn how coremltools palettization and pruning APIs cut on-device inference latency by 50%+ on Neural Engine targets without sacrificing model accuracy.
Tags: ios swift mobile architecture
TL;DR
Apple’s coremltools 7.x palettization and unstructured pruning APIs can cut model size by 4–8x and inference latency by 40–60% on Neural Engine targets. But only if you profile layer-by-layer sensitivity first and avoid the activation range calibration traps that silently wreck post-training quantization quality.
The problem with shipping raw models to mobile
A typical MobileNet-class float32 model at ~14MB runs at roughly 8ms on the A17 Neural Engine. INT8 quantization via coremltools drops that to ~5ms. INT4 palettization applied selectively gets you to ~3.5ms — below the perceptual threshold for interactive use. That’s a 56% latency reduction from a single compression pass, and most teams aren’t doing it.
The mistake I see repeatedly: teams optimize for model accuracy on a GPU cluster, then ship the same weights to a Neural Engine with a 4ms latency budget and wonder why it falls apart. On-device inference is now table stakes for competitive consumer apps. Whether you can compress models aggressively enough to run them in real time, without degrading user experience, is what separates good on-device AI from bad.
Profiling first: identify which layers tolerate compression
Never compress blindly. The workflow that actually works in production starts with sensitivity analysis.
import numpy as np
import coremltools as ct
from coremltools.optimize.coreml import get_weights_metadata
model = ct.models.MLModel("MyModel.mlpackage")
metadata = get_weights_metadata(model, weight_threshold=1024)
for name, info in metadata.items():
w = info.val
sparsity = np.mean(np.abs(w) < 1e-6)
print(f"{name}: shape={w.shape}, sparsity={sparsity:.2%}, dtype={w.dtype}")Computing near-zero sparsity directly from info.val gives you ground truth on which layers already trend sparse — a reliable proxy for compression tolerance. Convolutional layers in early blocks typically tolerate 70–80% induced sparsity with less than 0.5% accuracy drop. Attention layers and final classification heads are where you pay the real cost. Keep those at float16 or conservative INT8.
INT4 vs INT8: where each belongs
| Quantization | Size Reduction | Latency Gain (Neural Engine) | Latency Gain (GPU) | Accuracy Risk |
|---|---|---|---|---|
| Float16 (baseline) | 2x vs FP32 | — | — | Negligible |
| INT8 (linear) | 4x vs FP32 | ~35–40% | ~20–25% | Low if calibrated |
| INT4 (palettized) | 8x vs FP32 | ~50–60% | ~10–15% | Medium — layer-dependent |
| Mixed INT4/INT8 | 5–6x vs FP32 | ~45–55% | ~18–22% | Low with profiling |
INT4 palettization benefits Neural Engine execution disproportionately because the ANE’s memory bandwidth bottleneck is the primary constraint, not compute. On GPU execution units, INT4 gains shrink and you risk latency regressions from dequantization overhead. Mixed precision is almost always the right production choice.
The conversion pipeline that doesn’t break calibration
The activation range calibration trap hits hardest in PyTorch → ONNX → CoreML pipelines. If you run post-training quantization (PTQ) on the ONNX graph before conversion, you lose the operator-level visibility coremltools needs to set accurate activation ranges per layer.
The correct order:
import torch
import coremltools as ct
from coremltools.optimize.torch.quantization import PostTrainingQuantizer, PostTrainingQuantizerConfig
# Step 1: Quantize in PyTorch space BEFORE tracing
config = PostTrainingQuantizerConfig.from_dict({
"global_config": {
"weight_dtype": "int8",
"activation_dtype": "int8"
}
})
quantizer = PostTrainingQuantizer(model, config)
quantized_model = quantizer.compress(dataloader=calibration_loader, num_batches=128)
# Step 2: Trace the quantized model
example_input = torch.rand(1, 3, 224, 224)
traced = torch.jit.trace(quantized_model, example_input)
# Step 3: Convert — ranges are already embedded
mlmodel = ct.convert(traced, inputs=[ct.ImageType(shape=example_input.shape)])
mlmodel.save("CompressedModel.mlpackage")128 calibration batches is the practical floor. Below 64, activation range estimates drift enough to cause silent accuracy degradation that only surfaces on edge-case inputs — exactly the kind of bug that won’t show up in standard eval benchmarks.
Unstructured pruning: the underused lever
In my experience, teams default to quantization and skip pruning entirely. That’s leaving performance on the table.
from coremltools.optimize.coreml import OpThresholdPrunerConfig, prune_weights
pruner_config = OpThresholdPrunerConfig(
threshold=1e-3,
minimum_sparsity_percentile=0.4,
maximum_sparsity_percentile=0.8,
weight_threshold=1024
)
pruned_model = prune_weights(model, config=pruner_config)Unstructured pruning at 50–70% sparsity on feed-forward layers compounds with INT8 quantization — you get both memory bandwidth and compute wins. The Neural Engine handles sparse computation efficiently; the GPU less so. INT4 palettization alone gets you ~50% latency reduction. Adding 60% unstructured sparsity on compatible layers pushes that to 60–65%. That’s the difference between a demo and something shippable.
Three things worth remembering
Profile before you compress. Compute per-layer sparsity from get_weights_metadata to identify high-tolerance layers. Blindly applying INT4 to attention heads is how you ship a broken model to production.
Quantize in PyTorch space, not ONNX. Activation range calibration integrity only survives if you compress before tracing. Post-ONNX PTQ fails silently on tail inputs — the kind of failure that clears your eval suite and then blows up in production.
Combine pruning and quantization on Neural Engine targets. Mixed INT4/INT8 palettization paired with 60% unstructured sparsity on feed-forward layers delivers 60–65% latency reduction with manageable accuracy trade-offs, but only when sensitivity analysis decides which layers get which treatment. Skip that step and you’re guessing.