From fd00c31b1d76bc876895d94e8f5e03d31473e2f0 Mon Sep 17 00:00:00 2001 From: modelbeast Date: Thu, 16 Jul 2026 23:54:49 +1000 Subject: [PATCH] =?UTF-8?q?docs:=20add=20M5=20=E2=80=94=20head=5Fdim=20pad?= =?UTF-8?q?ding=20is=20worth=20~5x=20on=20laptops=20(vs=20~2x=20on=20Ultra?= =?UTF-8?q?s)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit M5 MacBook Pro (10-core GPU, 16GB) reproduces the head_dim=56 silent fallback and shows the fix matters far MORE on narrow GPUs: padding 56->64 gives 4.9x (1,16,4096), 5.1x (1,16,2048), 3.2x (4096,2,64). Raw fused-vs-unfused at 4096/fp32 is 0.14x on M5 vs ~0.53x on M3 Ultra — an M5 laptop matches an M1 Ultra on fused attention but is ~4x slower unfused. This reframes the fix as a laptop optimization, not a big-iron one. --- ...6-07-16-m-series-fleet-ablation-results.md | 33 ++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/docs/2026-07-16-m-series-fleet-ablation-results.md b/docs/2026-07-16-m-series-fleet-ablation-results.md index 48ea661..074255f 100644 --- a/docs/2026-07-16-m-series-fleet-ablation-results.md +++ b/docs/2026-07-16-m-series-fleet-ablation-results.md @@ -1,7 +1,7 @@ --- title: "M-series fleet ablation: M3 Ultra / M1 Ultra / M4 Pro" date: 2026-07-16 -device: "M3 Ultra 256GB · M1 Ultra 128GB (64-core GPU) · M4 Pro Mac mini 24GB" +device: "M3 Ultra 256GB · M1 Ultra 128GB · M4 Pro mini 24GB · M5 MacBook Pro 16GB (10-core)" script: scripts/bench_optimizations.py (unmodified) + engine-level tiled×compile extension --- @@ -44,6 +44,21 @@ Apple Silicon range. Contributed from a 3-Mac render-farm setup; happy to run fo | (1, 16, 4096, ·) | 3.8 / 7.2 ms → 1.9× | 7.5 / 11.1 ms → 1.5× | | (4096, 2, 64, ·) | 1.3 / 2.5 ms → 1.9× | 1.5 / 2.6 ms → 1.7× | + **…and on M5 it's worth ~5×, i.e. this fix matters MOST for laptop users.** Same + experiment on an M5 MacBook Pro (10-core GPU, 16 GB) — hdim 56 vs 64, fp32: + + | shape | hdim 56 (today) | hdim 64 (padded) | **speedup from padding** | + |---|---|---|---| + | (1, 16, 4096) | 44.00 ms *(fallback: 1.05× vs unfused)* | **8.91 ms** *(fast path: 0.21×)* | **4.9×** | + | (1, 16, 2048) | 11.44 ms *(1.06×)* | **2.23 ms** *(0.22×)* | **5.1×** | + | (4096, 2, 64) | 16.72 ms *(0.97×)* | **5.29 ms** *(0.25×)* | **3.2×** | + + Why bigger on the laptop: the fused kernel's advantage over unfused scales inversely + with GPU width. At seq 4096 fp32 (hdim 64) fused-vs-unfused is **7.1× on M5** vs + ~1.9× on M3 Ultra — a 10-core GPU cannot brute-force the materialized N×N path the + way an 80-core Ultra can. So the head_dim-56 fallback costs Studio owners ~2× and + **MacBook owners ~5×**. Padding is not a big-iron optimization; it's a laptop one. + Suggested changes: (a) pad qkv projections to head_dim 64 (at minimum for the global blocks) so sdpa's fast kernel actually engages — M1 benefits *more* than M3; (b) until then, default `use_sdpa=False` (it is currently overhead-only). For MLX upstream: a @@ -85,6 +100,22 @@ Apple Silicon range. Contributed from a 3-Mac render-farm setup; happy to run fo | 1024 | 247.0 ms | 350.6 ms | 1149.2 ms | 610.7 ms | | 2048 | 1750.2 ms | 3527.3 ms | 23147 ms (swap) | 4984.7 ms | +### Raw SDPA kernel scaling (pure MLX, no CorridorKey) — fused vs unfused, hdim 64, fp32 + +Useful context for *why* the head-dim fix matters differently per machine. Fused-vs-unfused +ratio (lower = fused is winning by more): + +| seq | M3 Ultra (80c) | M1 Ultra (64c) | M5 (10c laptop) | +|---|---|---|---| +| 1024 | 0.46× | 0.54× | 0.28× | +| 2048 | 0.53× | 0.71× | **0.21×** | +| 4096 | 0.53× | 0.59× | **0.14×** | + +Absolute fused times at 4096/fp32: M3 3.86 ms · M1 7.19 ms · **M5 7.22 ms** — an M5 +laptop matches an M1 Ultra on *fused* attention while being ~4× slower unfused (51.5 ms +vs 12.2 ms). The narrower the GPU, the more the fast path is worth. (M5 numbers taken on +a machine in active desktop use; treat as indicative, not lab-clean.) + ## Full ablation tables ### M3 Ultra 256GB