Five chips, two runtimes · TTFT and TPOT

Median time to first token and time per output token, from 60 to 90 benchmark prompts per cell (GSM8K, IFEval, BFCL). Prompts run 19 to 772 tokens, median 92.

Purpose

Show how fast five small models answer on five phone chips, on the two runtimes the app ships, so a reader can pick a model and runtime for a given phone. Time to first token is what a user waits before the reply starts; time per output token is how fast it then streams.

How it was graded

Not graded here: this page is timing only. The same 90 prompts (30 GSM8K, 30 IFEval, 30 BFCL; Gemma 3 skips BFCL, having no tool syntax) ran greedy with thinking off, and the median prefill and decode time of each cell is plotted. The pass rates for these runs are in the repository's benchmark matrix.

How long it ran

One pass per cell on 2026-09-03 and 04. Generation time per cell ran from 6 minutes (LFM2.5-1.2B on ExecuTorch, 8 Elite) to about 100 minutes (SmolLM3 on llama.cpp, Tensor G5); 29 hours of phone time in all, plus loading and Test Lab queueing.

Latency map

Each line joins one chip's two runtimes. Lower and left is faster. Both axes are logarithmic.

All models, all chips

The same numbers as a grid, both columns on logarithmic scales. llama.cpp ExecuTorch. The model picked above is banded.

Not matched conditions. The Dimensity 9400 ran in hand under a desk fan. The Snapdragon 8 Gen 3 ran on Qualcomm Device Cloud and the 8 Elite, Tensor G5 and Exynos 2400 on Firebase Test Lab, racked, where phones throttle within about 40 seconds.

Two open gaps. Tensor G5 and Exynos 2400 decode at about half speed on llama.cpp against ExecuTorch, cause unknown. SmolLM3 on ExecuTorch prefills at a third of llama.cpp's rate on every chip, which is the compiled model rather than the phones.

Files. llama.cpp ran Q4_K_M GGUFs, Qwen3 at Q8_0. ExecuTorch ran the published XNNPACK INT8/INT4 exports, LFM2.5 as 8da4w and Llama 3.2 as SpinQuant. Runs on 2026-09-03 and 04, single pass, no repeats.