Speculative decoding on mobile: 2-3x faster LLM inference
Meta description: Speculative decoding cuts mobile LLM latency 2-3x. Tune acceptance rate, manage dual-model memory, and structure token trees for Neural Engine hardware.
TL;DR
Speculative decoding pairs a small draft model with a large verifier to generate multiple tokens per forward pass. On mobile hardware — Apple’s Neural Engine and Android’s Snapdragon NPU — this technique can reduce mean token latency by 2-3x, but only if you tune acceptance rate, manage the dual-model memory budget carefully, and structure your token trees for the hardware’s parallel execution model.
The problem with autoregressive inference on mobile
Every autoregressive token on mobile loads the full weight matrix from DRAM. That’s the bottleneck, not compute. Autoregressive generation means every token requires one full forward pass through your model, and on mobile silicon, that forward pass is memory-bound: you load weights once per token, generate one token, repeat.
A 7B parameter model at INT4 quantization sits at roughly 3.5 GB. Each forward pass touches all of it. Even on an Apple A17 Pro with 68 GB/s memory bandwidth, you’re budget-constrained by the time you account for the KV cache, the activation buffer, and OS overhead.
The result is mean token latency of 50-120ms for 7B models on current flagship hardware — acceptable for some use cases, genuinely painful for real-time conversational experiences.
How speculative decoding changes the equation
Let me walk you through the architecture. Speculative decoding introduces two models:
- Draft model (1B parameters, INT4): Fast, cheap, runs in ~5ms per token
- Verifier model (7B parameters, INT4): Accurate, expensive, runs in ~80ms per token
The draft model generates K candidate tokens speculatively. The verifier then evaluates all K+1 positions in a single forward pass — transformer attention is parallelizable across the sequence dimension, which is what makes this work. If the draft tokens match the verifier’s distribution (within a tunable acceptance threshold), you keep all K tokens. If not, you truncate at the first rejection and emit a corrected token.
The theoretical speedup ceiling is 1 / (1 - α) where α is the acceptance rate. At α = 0.7, that ceiling is ~3.3x — but this assumes draft forward passes are free, which they aren’t. A more accurate model is:
speedup ≈ (α·K + 1) / (1 + r·(K + 1))where r is the draft-to-verifier cost ratio. With r = 0.06 (5ms draft / 80ms verifier), K = 7, and α = 0.7, the realized speedup is closer to 2.1x — still a meaningful gain, but don’t expect the theoretical ceiling in practice.
Token trees: beyond linear draft sequences
Linear drafting wastes the verifier’s parallel capacity. Token trees fix this.
Instead of generating a single chain of K tokens, the draft model generates a tree of possible continuations — branching at positions where token probability is distributed across multiple candidates. The verifier evaluates all branches simultaneously in one attention pass using a structured attention mask.
The tree below represents a depth-3 structure with branching factor 2. Each level is a speculative continuation; the verifier scores all leaf paths in a single pass and accepts the highest-probability valid path.
Draft token tree — depth 3, branching factor 2
(each indented level is a speculative next token)
[START]
├── "The"
│ ├── "quick"
│ │ ├── "fox"
│ │ └── "dog"
│ └── "slow"
└── "A"
└── "fast"Note: if your CMS strips indentation from code blocks, read this as a tree rooted at [START] with two first-level branches (“The” and “A”), “The” expanding to “quick” and “slow”, and “quick” further branching to “fox” and “dog”.
The verifier scores every leaf in parallel. You accept the highest-probability path that falls within your acceptance threshold. On hardware that supports batched attention efficiently — Apple’s ANE, Qualcomm’s Hexagon NPU — tree verification can increase accepted tokens per verifier call by 40-60% compared to linear drafts.
Memory vs. speed: the mobile budget problem
Running two models simultaneously on mobile is the hard constraint. A realistic INT4 memory budget:
| Model | Params | INT4 Weight Size | KV Cache (2K ctx) | Total |
|---|---|---|---|---|
| Draft (1B) | 1B | ~0.5 GB | ~0.1 GB | ~0.6 GB |
| Verifier (7B) | 7B | ~3.5 GB | ~0.4 GB | ~3.9 GB |
| Combined | ~4.5 GB |
On iOS, the Neural Engine memory pool on A-series chips typically allows 6 GB for model execution on high-end devices. On Android, device memory headroom on Snapdragon 8 Gen 3 hardware reaches 4-6 GB in practice — but this depends on LPDDR5 allocation policies set by OEMs, not NNAPI itself, which is an inference dispatch API rather than a memory allocator.
The practical implication: you cannot use speculative decoding on mid-range hardware without aggressive shared-weight architectures or model distillation. Design for your p25 device, not your p75.
Tuning acceptance rate in production
Acceptance rate α is not fixed — it depends on the prompt distribution your users send. A 1B draft model trained on general text may achieve α = 0.65 on conversational prompts and drop to α = 0.40 on domain-specific technical content.
The levers to tune:
- Match draft and verifier sampling temperatures. Mismatched temperatures tank acceptance rate.
- Fine-tune the draft model on your specific prompt distribution. A domain-adapted 1B model can match a general 3B model in acceptance rate at half the memory cost.
- Dynamically adjust draft length based on rolling acceptance rate. If recent α < 0.5, reduce K to 3. If α > 0.75, increase K to 8.
Before you ship
Profile your acceptance rate before committing to a draft model. Run your production prompt sample through a candidate draft-verifier pair and measure α. If it falls below 0.55, either fine-tune the draft model or select a larger draft that better tracks the verifier’s distribution.
Use token trees on NPU-capable hardware, linear drafts on GPU fallback. Tree verification pays off only when the hardware supports efficient masked attention. Implement a runtime capability check and fall back gracefully.
Size your draft model to your verifier’s memory headroom, not to a benchmark. The correct draft model is the largest one that fits your device’s memory budget after the verifier and KV cache are allocated — not the one with the best standalone benchmark score.
Tags: mobile android ios architecture productengineering