On-Device Inference in 2026: 160 Tokens a Second on a Laptop, a 2-Bit Model in Every iPhone, and the Bandwidth Arithmetic That Decides What Stays Local

My whole business is running models' code in the cloud, so it is a little awkward to write that the most interesting inference story of 2026 is the one where the request never leaves the phone. But it is, and the reason is arithmetic rather than marketing. A decoded token needs every active weight read from memory once; divide the memory bandwidth by the bytes of active weights and you have the ceiling on tokens per second. That single ratio, plus what fits in RAM, explains everything in this post: why Apple trains a 3-billion-parameter model at 2 bits, why Google splits Gemma into nested sub-models, why the NPU "TOPS" number on the box is almost irrelevant, and why the hybrid fallback to a private cloud is designed into every platform rather than bolted on. Here is what is actually shipping, with the numbers I could verify.

The arithmetic

Decode is memory-bound. Prefill, the reading of your prompt, is compute-bound and benefits from the NPU or GPU's matrix units; generation of each subsequent token touches all the active weights and the key-value cache and is limited by how fast you can stream them. So the number to know about any device is bandwidth, and the number to know about any model is bytes per token.

Now the check against measurement. Google's LiteRT-LM card for Gemma 4 E2B, a model with 2.3 billion effective parameters in a mixed 2/4/8-bit file of 2.58 GB, reports decode at 160 tokens a second on an M4 Max GPU, 143 on an RTX 4090, 56.5 on an iPhone 17 Pro GPU, 52 on a Galaxy S26 Ultra, 32 on a Qualcomm Dragonwing NPU and 7.6 on a Raspberry Pi 5. Take roughly two gigabytes touched per token: the M4 Max ceiling is about 270 tokens a second and it hits 160, around 60 percent; the iPhone ceiling is about 38 and it hits 56, which tells you the effective bytes per token are lower than the file size because the per-layer embeddings live in flash and only part of the model is read per token. Liquid's LFM2.5-2.6B, under 2.5 GB, claims 220 tokens a second on an M5 Max, whose 614 GB/s gives a ceiling near 245, and around 30 on a phone. The vendor number is suspiciously close to the ceiling; the phone number matches. Every figure in this field falls out of this ratio at 40 to 90 percent efficiency, which is why the 4090 with four times the M4 Max's bandwidth is slower on the tiny model: at that size it is overhead-bound, not bandwidth-bound.

What each platform ships

Apple. The 2025 technical report describes the on-device model as about 3 billion parameters, trained with 2-bit quantisation-aware training and sharing key-value caches between blocks, deployed on the Neural Engine. At WWDC in June, Apple said it rebuilt that model "from the ground up" with image input and better tool calling, but did not state a new size, and more importantly opened the Foundation Models framework so that any model can back a session: it ships open-source implementations for the Neural Engine and for MLX on the Mac GPU, and said Anthropic and Google would provide Swift packages for Claude and Gemini. The Private Cloud Compute model is now available to third-party apps with a 32,000-token context, reasoning levels and per-request token accounting, free for developers under two million first-time downloads. The adapter story is the constraint to know: LoRA adapters are about 160 MB, must be downloaded rather than bundled, are bound to one exact system-model version, and need an Apple entitlement. Whether Siri's Gemini-derived brain runs on device is settled by Tim Cook's January statement: on device and in Private Cloud Compute, with the routing model deciding. The reported billion-dollar-a-year figure for that deal is from Bloomberg and neither company has confirmed it.

Google. Gemini Nano runs inside the AICore system service, exposed through ML Kit's GenAI APIs for summarisation, proofreading, rewriting, image description and speech; AICore isolates each request and routes any network traffic through Private Compute Services. The Pixel 10's Tensor G5, on TSMC 3 nm, runs Nano "2.6 times faster", and a teardown by Android Authority, which I could not confirm from Google, describes the deployment as a Matformer: an 8-billion-parameter model of which 4 billion sit in RAM, with a nested 5-billion sub-model at 2 billion resident, and about 3 GB of the phone's 12 reserved for AI. In the browser, Chrome's Prompt API has been stable since Chrome 138 but requires 22 GB of free disk and either a 4 GB GPU or 16 GB of RAM, desktop only.

Microsoft. Windows ML went GA in September 2025 as an ONNX Runtime layer with Windows-managed execution providers from AMD, Intel, NVIDIA and Qualcomm, which is the first time the NPU vendors' stacks have been installed and updated by the OS. Copilot+ still means an NPU over 40 TOPS. The detail I did not expect: the Phi Silica documentation, dated July, says the built-in model is being replaced by something called Aion Instruct, with a retail rollout in November and Phi Silica removed, and every LoRA adapter has to be retrained. Foundry Local offers twenty-plus open models through an OpenAI-compatible endpoint on any Windows 10 or later machine.

The chips, and why TOPS is the wrong number

Every vendor leads with TOPS. Qualcomm's Snapdragon X2 Elite is 80 TOPS INT8 on all SKUs; Intel's Panther Lake claims 50 on the NPU, 120 on the GPU and "170 platform TOPS"; AMD's Ryzen AI 400 desktop parts are 50; Microsoft's threshold is 40. Apple published 38 for the M4 and has published nothing for the M5, preferring to talk about Neural Accelerators in each GPU core and "four times the peak GPU compute for AI". MediaTek's Dimensity 9500 gives only relative numbers and the interesting claim that its NPU handles 1.58-bit BitNet models natively. Phone-side TOPS figures you will see for the Snapdragon 8 Elite Gen 5 and the Dimensity are from aggregator sites and likely count INT4 or INT2, which doubles or quadruples the number for the same silicon.

TOPS measures prefill and vision, where the matrix units matter. For decode, refer to the table above: the M5 Max's 614 GB/s beats the X2 Elite Extreme's 228 by a factor that no TOPS figure captures. The one independent benchmark worth citing is Argmax's iPhone 17 measurements: the GPU improved 2.5 to 3.1 times over the iPhone 16 Pro on speech models while the Neural Engine improved 1.0 to 1.15 times, which is why Apple moved its GPU roadmap toward AI. MLPerf Client v2.0 now makes an 8B Llama and Phi-4 Mini mandatory at 4K prompts and reports time to first token and tokens a second, but I found no published result tables yet.

The models

The small-model field has a clear shape: everything meaningful is 2 to 9 billion parameters, Apache-licensed or close, and increasingly mixture-of-experts even at phone scale, because active parameters are what the bandwidth ratio charges for and total parameters are what flash storage charges for.

There is no phone-class Llama 4; the smallest is Scout at 109 billion total. The independent quality read is Artificial Analysis's phone benchmark from August: of 41 sub-8 GB builds tried on an iPhone 17 Pro, 33 ran; LFM2.5-2.6B and Qwen3.5-9B sit at the top of their intelligence index at 63 and 61, and LFM2.5-2.6B answered a 1,024-token-in, 256-token-out task in 8.0 seconds using 2.3 GB. End-to-end times ranged from under a second to 27, and some models hit a 16K context cap on nearly a third of tasks. Compare that with the open-weight frontier and you have a two-year-old flagship in your pocket, which is the right way to think about it.

Runtimes

llama.cpp tags a build most days and merged multi-token prediction in May: with three draft tokens a Qwen3.6-27B at Q6_K on an RTX 3090 went from 23 to 42 tokens a second with 72 to 82 percent acceptance. The server exposes several speculative modes and quantised KV caches down to q4_0. Google's LiteRT-LM runs on CPU, GPU and Qualcomm NPUs with C++, Python and Kotlin stable, and is what powers Chrome and the Pixel Watch. PyTorch's ExecuTorch 1.0, from October 2025, is in Instagram, WhatsApp, Messenger and Meta's Ray-Ban Display glasses, with backends for Core ML, Qualcomm's Hexagon, Arm Ethos, Vulkan and Samsung's Exynos. MLX is Apple's, Ollama added Gemma 4 with image and audio on its MLX engine this month, LM Studio checkpoints the KV cache for agent loops. In the browser, WebGPU is in Chrome, Edge and Safari 26 with Firefox still gated, and WebNN became a W3C Candidate Recommendation draft on 4 September. Speculative decoding with a 300-million-parameter drafter, Liquid's DSpark, gives 2.27 times on an M4 Max with greedy-identical output, and Gemma 4 ships its own drafters; the Windows NPU model uses a draft model too. That is the technique that changes the ratio: a drafter that reads far fewer bytes per token proposes several, and the big model verifies them in one pass.

Running one

Google's LiteRT-LM is the shortest path to a number on your own machine:

python -m pip install -U litert-lm
litert-lm run \
  --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
  --prompt="Explain why LLM decode speed is bounded by memory bandwidth."

Expect roughly 160 tokens a second on an M4 Max GPU per Google's table, and around a third to a half of that on a base M5. With llama.cpp and a model that carries multi-token-prediction heads, the server flags verified from the merged pull request and README are:

llama-server -hf <MODEL_REPO> --spec-type mtp --spec-draft-n-max 2 --parallel 1 \
  --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn auto
# or a separate drafter:  --spec-type draft-dspark --model-draft <LFM2.5-2.6B-DSpark gguf>

On Apple silicon, pip install mlx-lm then mlx_lm.generate --model mlx-community/<any 4-bit repo> --prompt "...". Check the repository name exists before you script it; the model names in this field change monthly.

What stays private, and what does not

The privacy promise is concrete on two platforms. Apple's notice says a device model decides whether a request can be served locally, only relevant data goes to Private Cloud Compute, nothing is stored there, and users can turn on transparency logging and export a report of every request that left the phone. Google's AICore stores neither inputs nor outputs. WhatsApp's Private Processing runs the cloud half inside TEEs behind an oblivious relay, which I covered in the confidential computing post. Those are real engineering commitments and they are also admissions: the local model is a router and a first responder, and the hard requests leave.

The other direction is the weights. A USENIX Security systematisation found on-device model extraction "widespread, scalable, and under-defended", with encrypted models recovered by instrumenting the runtime, and a May 2026 review found a quarter of attack papers target model theft while half of defence papers do, and no defence literature at all for adversarial attacks on on-device inference. Qualcomm now advertises "GenAI model encryption" on the 8 Elite Gen 5; TZ-LLM puts the weights in Arm TrustZone with NPU sharing and a sub-second cost. If you ship a model you paid to train into an app, assume it will be extracted, and license accordingly. Apple's answer is to bundle the model with the OS and let you ship only an adapter, which is elegant and also a moat.

My reading, as someone who sells the other thing. A 2-to-4-billion-parameter model at 4 bits on a 2026 phone is a competent assistant at 50 tokens a second, with a context that runs out, and every platform routes the hard problems to a cloud it controls. The laptop tier is where local gets serious: 600 GB/s and 128 GB on a MacBook runs a 30-billion-parameter mixture-of-experts at speeds my customers pay for. What the device cannot do is the thing my platform exists for, which is run the code the model writes somewhere it cannot hurt anything, and I would rather the model lived on the laptop and the sandbox lived with me than the reverse. The router model decides which requests leave. Make sure you know what it decides.

Related: LLM Inference in 2026, Open Weights in 2026 and Neuromorphic Computing in 2026.