docs: add M5 — head_dim padding is worth ~5x on laptops (vs ~2x on Ultras)

M5 MacBook Pro (10-core GPU, 16GB) reproduces the head_dim=56 silent fallback
and shows the fix matters far MORE on narrow GPUs: padding 56->64 gives 4.9x
(1,16,4096), 5.1x (1,16,2048), 3.2x (4096,2,64). Raw fused-vs-unfused at
4096/fp32 is 0.14x on M5 vs ~0.53x on M3 Ultra — an M5 laptop matches an M1
Ultra on fused attention but is ~4x slower unfused. This reframes the fix as a
laptop optimization, not a big-iron one.
This commit is contained in:
modelbeast 2026-07-16 23:54:49 +10:00
parent df28940d81
commit fd00c31b1d

View File

@ -1,7 +1,7 @@
--- ---
title: "M-series fleet ablation: M3 Ultra / M1 Ultra / M4 Pro" title: "M-series fleet ablation: M3 Ultra / M1 Ultra / M4 Pro"
date: 2026-07-16 date: 2026-07-16
device: "M3 Ultra 256GB · M1 Ultra 128GB (64-core GPU) · M4 Pro Mac mini 24GB" device: "M3 Ultra 256GB · M1 Ultra 128GB · M4 Pro mini 24GB · M5 MacBook Pro 16GB (10-core)"
script: scripts/bench_optimizations.py (unmodified) + engine-level tiled×compile extension script: scripts/bench_optimizations.py (unmodified) + engine-level tiled×compile extension
--- ---
@ -44,6 +44,21 @@ Apple Silicon range. Contributed from a 3-Mac render-farm setup; happy to run fo
| (1, 16, 4096, ·) | 3.8 / 7.2 ms → 1.9× | 7.5 / 11.1 ms → 1.5× | | (1, 16, 4096, ·) | 3.8 / 7.2 ms → 1.9× | 7.5 / 11.1 ms → 1.5× |
| (4096, 2, 64, ·) | 1.3 / 2.5 ms → 1.9× | 1.5 / 2.6 ms → 1.7× | | (4096, 2, 64, ·) | 1.3 / 2.5 ms → 1.9× | 1.5 / 2.6 ms → 1.7× |
**…and on M5 it's worth ~5×, i.e. this fix matters MOST for laptop users.** Same
experiment on an M5 MacBook Pro (10-core GPU, 16 GB) — hdim 56 vs 64, fp32:
| shape | hdim 56 (today) | hdim 64 (padded) | **speedup from padding** |
|---|---|---|---|
| (1, 16, 4096) | 44.00 ms *(fallback: 1.05× vs unfused)* | **8.91 ms** *(fast path: 0.21×)* | **4.9×** |
| (1, 16, 2048) | 11.44 ms *(1.06×)* | **2.23 ms** *(0.22×)* | **5.1×** |
| (4096, 2, 64) | 16.72 ms *(0.97×)* | **5.29 ms** *(0.25×)* | **3.2×** |
Why bigger on the laptop: the fused kernel's advantage over unfused scales inversely
with GPU width. At seq 4096 fp32 (hdim 64) fused-vs-unfused is **7.1× on M5** vs
~1.9× on M3 Ultra — a 10-core GPU cannot brute-force the materialized N×N path the
way an 80-core Ultra can. So the head_dim-56 fallback costs Studio owners ~2× and
**MacBook owners ~5×**. Padding is not a big-iron optimization; it's a laptop one.
Suggested changes: (a) pad qkv projections to head_dim 64 (at minimum for the global Suggested changes: (a) pad qkv projections to head_dim 64 (at minimum for the global
blocks) so sdpa's fast kernel actually engages — M1 benefits *more* than M3; (b) until blocks) so sdpa's fast kernel actually engages — M1 benefits *more* than M3; (b) until
then, default `use_sdpa=False` (it is currently overhead-only). For MLX upstream: a then, default `use_sdpa=False` (it is currently overhead-only). For MLX upstream: a
@ -85,6 +100,22 @@ Apple Silicon range. Contributed from a 3-Mac render-farm setup; happy to run fo
| 1024 | 247.0 ms | 350.6 ms | 1149.2 ms | 610.7 ms | | 1024 | 247.0 ms | 350.6 ms | 1149.2 ms | 610.7 ms |
| 2048 | 1750.2 ms | 3527.3 ms | 23147 ms (swap) | 4984.7 ms | | 2048 | 1750.2 ms | 3527.3 ms | 23147 ms (swap) | 4984.7 ms |
### Raw SDPA kernel scaling (pure MLX, no CorridorKey) — fused vs unfused, hdim 64, fp32
Useful context for *why* the head-dim fix matters differently per machine. Fused-vs-unfused
ratio (lower = fused is winning by more):
| seq | M3 Ultra (80c) | M1 Ultra (64c) | M5 (10c laptop) |
|---|---|---|---|
| 1024 | 0.46× | 0.54× | 0.28× |
| 2048 | 0.53× | 0.71× | **0.21×** |
| 4096 | 0.53× | 0.59× | **0.14×** |
Absolute fused times at 4096/fp32: M3 3.86 ms · M1 7.19 ms · **M5 7.22 ms** — an M5
laptop matches an M1 Ultra on *fused* attention while being ~4× slower unfused (51.5 ms
vs 12.2 ms). The narrower the GPU, the more the fast path is worth. (M5 numbers taken on
a machine in active desktop use; treat as indicative, not lab-clean.)
## Full ablation tables ## Full ablation tables
### M3 Ultra 256GB ### M3 Ultra 256GB