docs: add M5 — head_dim padding is worth ~5x on laptops (vs ~2x on Ultras)
M5 MacBook Pro (10-core GPU, 16GB) reproduces the head_dim=56 silent fallback and shows the fix matters far MORE on narrow GPUs: padding 56->64 gives 4.9x (1,16,4096), 5.1x (1,16,2048), 3.2x (4096,2,64). Raw fused-vs-unfused at 4096/fp32 is 0.14x on M5 vs ~0.53x on M3 Ultra — an M5 laptop matches an M1 Ultra on fused attention but is ~4x slower unfused. This reframes the fix as a laptop optimization, not a big-iron one.
This commit is contained in:
parent
df28940d81
commit
fd00c31b1d
@ -1,7 +1,7 @@
|
|||||||
---
|
---
|
||||||
title: "M-series fleet ablation: M3 Ultra / M1 Ultra / M4 Pro"
|
title: "M-series fleet ablation: M3 Ultra / M1 Ultra / M4 Pro"
|
||||||
date: 2026-07-16
|
date: 2026-07-16
|
||||||
device: "M3 Ultra 256GB · M1 Ultra 128GB (64-core GPU) · M4 Pro Mac mini 24GB"
|
device: "M3 Ultra 256GB · M1 Ultra 128GB · M4 Pro mini 24GB · M5 MacBook Pro 16GB (10-core)"
|
||||||
script: scripts/bench_optimizations.py (unmodified) + engine-level tiled×compile extension
|
script: scripts/bench_optimizations.py (unmodified) + engine-level tiled×compile extension
|
||||||
---
|
---
|
||||||
|
|
||||||
@ -44,6 +44,21 @@ Apple Silicon range. Contributed from a 3-Mac render-farm setup; happy to run fo
|
|||||||
| (1, 16, 4096, ·) | 3.8 / 7.2 ms → 1.9× | 7.5 / 11.1 ms → 1.5× |
|
| (1, 16, 4096, ·) | 3.8 / 7.2 ms → 1.9× | 7.5 / 11.1 ms → 1.5× |
|
||||||
| (4096, 2, 64, ·) | 1.3 / 2.5 ms → 1.9× | 1.5 / 2.6 ms → 1.7× |
|
| (4096, 2, 64, ·) | 1.3 / 2.5 ms → 1.9× | 1.5 / 2.6 ms → 1.7× |
|
||||||
|
|
||||||
|
**…and on M5 it's worth ~5×, i.e. this fix matters MOST for laptop users.** Same
|
||||||
|
experiment on an M5 MacBook Pro (10-core GPU, 16 GB) — hdim 56 vs 64, fp32:
|
||||||
|
|
||||||
|
| shape | hdim 56 (today) | hdim 64 (padded) | **speedup from padding** |
|
||||||
|
|---|---|---|---|
|
||||||
|
| (1, 16, 4096) | 44.00 ms *(fallback: 1.05× vs unfused)* | **8.91 ms** *(fast path: 0.21×)* | **4.9×** |
|
||||||
|
| (1, 16, 2048) | 11.44 ms *(1.06×)* | **2.23 ms** *(0.22×)* | **5.1×** |
|
||||||
|
| (4096, 2, 64) | 16.72 ms *(0.97×)* | **5.29 ms** *(0.25×)* | **3.2×** |
|
||||||
|
|
||||||
|
Why bigger on the laptop: the fused kernel's advantage over unfused scales inversely
|
||||||
|
with GPU width. At seq 4096 fp32 (hdim 64) fused-vs-unfused is **7.1× on M5** vs
|
||||||
|
~1.9× on M3 Ultra — a 10-core GPU cannot brute-force the materialized N×N path the
|
||||||
|
way an 80-core Ultra can. So the head_dim-56 fallback costs Studio owners ~2× and
|
||||||
|
**MacBook owners ~5×**. Padding is not a big-iron optimization; it's a laptop one.
|
||||||
|
|
||||||
Suggested changes: (a) pad qkv projections to head_dim 64 (at minimum for the global
|
Suggested changes: (a) pad qkv projections to head_dim 64 (at minimum for the global
|
||||||
blocks) so sdpa's fast kernel actually engages — M1 benefits *more* than M3; (b) until
|
blocks) so sdpa's fast kernel actually engages — M1 benefits *more* than M3; (b) until
|
||||||
then, default `use_sdpa=False` (it is currently overhead-only). For MLX upstream: a
|
then, default `use_sdpa=False` (it is currently overhead-only). For MLX upstream: a
|
||||||
@ -85,6 +100,22 @@ Apple Silicon range. Contributed from a 3-Mac render-farm setup; happy to run fo
|
|||||||
| 1024 | 247.0 ms | 350.6 ms | 1149.2 ms | 610.7 ms |
|
| 1024 | 247.0 ms | 350.6 ms | 1149.2 ms | 610.7 ms |
|
||||||
| 2048 | 1750.2 ms | 3527.3 ms | 23147 ms (swap) | 4984.7 ms |
|
| 2048 | 1750.2 ms | 3527.3 ms | 23147 ms (swap) | 4984.7 ms |
|
||||||
|
|
||||||
|
### Raw SDPA kernel scaling (pure MLX, no CorridorKey) — fused vs unfused, hdim 64, fp32
|
||||||
|
|
||||||
|
Useful context for *why* the head-dim fix matters differently per machine. Fused-vs-unfused
|
||||||
|
ratio (lower = fused is winning by more):
|
||||||
|
|
||||||
|
| seq | M3 Ultra (80c) | M1 Ultra (64c) | M5 (10c laptop) |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 1024 | 0.46× | 0.54× | 0.28× |
|
||||||
|
| 2048 | 0.53× | 0.71× | **0.21×** |
|
||||||
|
| 4096 | 0.53× | 0.59× | **0.14×** |
|
||||||
|
|
||||||
|
Absolute fused times at 4096/fp32: M3 3.86 ms · M1 7.19 ms · **M5 7.22 ms** — an M5
|
||||||
|
laptop matches an M1 Ultra on *fused* attention while being ~4× slower unfused (51.5 ms
|
||||||
|
vs 12.2 ms). The narrower the GPU, the more the fast path is worth. (M5 numbers taken on
|
||||||
|
a machine in active desktop use; treat as indicative, not lab-clean.)
|
||||||
|
|
||||||
## Full ablation tables
|
## Full ablation tables
|
||||||
|
|
||||||
### M3 Ultra 256GB
|
### M3 Ultra 256GB
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user