diff --git a/docs/2026-07-16-m-series-fleet-ablation-results.md b/docs/2026-07-16-m-series-fleet-ablation-results.md index 1f3ce2a..6486023 100644 --- a/docs/2026-07-16-m-series-fleet-ablation-results.md +++ b/docs/2026-07-16-m-series-fleet-ablation-results.md @@ -1,7 +1,7 @@ --- -title: "M-series fleet ablation: M1 / M3 / M4 / M5 across 5 machines (Ultra → fanless Air)" +title: "M-series fleet ablation: 6 machines, M1 → M5 (8GB mini → 256GB Ultra)" date: 2026-07-16 -device: "M1 Ultra 128GB · M3 Ultra 256GB · M3 Air 16GB (fanless) · M4 Pro mini 24GB · M5 MacBook Pro 16GB" +device: "M1 Max 32GB (24c) · M1 Ultra 128GB (64c) · M3 Air 16GB (10c, fanless) · M3 Ultra 256GB (80c) · M4 Pro mini 24GB · M5 MacBook Pro 16GB (10c)" script: scripts/bench_optimizations.py (unmodified) + engine-level tiled×compile extension --- @@ -56,11 +56,30 @@ happy to run follow-ups on any of them. | (1, 16, 2048) | 11.44 ms *(1.06×)* | **2.23 ms** *(0.22×)* | **5.1×** | | (4096, 2, 64) | 16.72 ms *(0.97×)* | **5.29 ms** *(0.25×)* | **3.2×** | - Why bigger on the laptop: the fused kernel's advantage over unfused scales inversely - with GPU width. At seq 4096 fp32 (hdim 64) fused-vs-unfused is **7.1× on M5** vs - ~1.9× on M3 Ultra — a 10-core GPU cannot brute-force the materialized N×N path the - way an 80-core Ultra can. So the head_dim-56 fallback costs Studio owners ~2× and - **MacBook owners ~5×**. Padding is not a big-iron optimization; it's a laptop one. + **The fallback is universal; the size of the win is not.** Measured on six machines + (padding 56→64 at `(1,16,4096)` fp32). Every machine shows the fallback at hdim 56 + (fused/unfused 0.97–1.06×, i.e. the fast kernel never engages) — but the payoff varies + ~4× and **does not track GPU width monotonically**: + + | machine | GPU cores | hdim 56 | hdim 64 | **padding speedup** | fused/unfused @4096 | + |---|---|---|---|---|---| + | M5 MacBook Pro | 10c | 44.00 ms | 8.91 ms | **4.9×** | 0.14× | + | M1 Ultra | 64c | 150.5 ms | 53.8 ms | **2.7×** | 0.59× | + | M3 Air | 10c | 138.9 ms | 54.3 ms | **2.6×** | 0.40× | + | M3 Ultra | 80c | 63.2 ms | 28.6 ms | **2.2×** | 0.53× | + | **M1 Max** | **24c** | 20.16 ms | 15.79 ms | **1.3×** | **0.87×** | + + We initially read this as "narrower GPU ⇒ bigger win" (M5 10c gains most, M3 Ultra 80c + least). **The M1 Max refutes that**: it is mid-width (24c) yet gains the *least* of all. + The actual predictor is each chip's **fused-kernel quality relative to its own raw matmul + throughput** — the M1 Max's fused path is only ~0.87× its unfused path at every size + (i.e. MLX's SDPA kernel barely beats a plain matmul there), whereas the M5's is 0.14×. + Where the fused kernel is strong, missing it is expensive; where it's weak, missing it + barely matters. + + Practical upshot is unchanged and *strictly positive*: padding to 64 helps on **every + Apple GPU generation tested** — by 1.3× (M1 Max) to 4.9× (M5) — and hurts nowhere, + despite ~14 % more FLOPs. Suggested changes: (a) pad qkv projections to head_dim 64 (at minimum for the global blocks) so sdpa's fast kernel actually engages — M1 benefits *more* than M3; (b) until