diff --git a/docs/2026-07-16-m-series-fleet-ablation-results.md b/docs/2026-07-16-m-series-fleet-ablation-results.md index 77a18dc..855cdd3 100644 --- a/docs/2026-07-16-m-series-fleet-ablation-results.md +++ b/docs/2026-07-16-m-series-fleet-ablation-results.md @@ -1,18 +1,18 @@ --- -title: "M-series fleet ablation: 6 machines, M1 → M5 (8GB mini → 256GB Ultra)" +title: "M-series fleet ablation: 7 machines, M1 → M5 (16GB mini → 256GB Ultra)" date: 2026-07-16 -device: "M1 Max 32GB (24c) · M1 Ultra 128GB (64c) · M3 Air 16GB (10c, fanless) · M3 Ultra 256GB (80c) · M4 Pro mini 24GB · M5 MacBook Pro 16GB (10c)" +device: "M1 Max 32GB (24c) · M1 Ultra 128GB (64c) · M3 Air 16GB (10c, fanless) · M3 Ultra 256GB (80c) · M4 mini 16GB (10c) · M4 Pro mini 24GB · M5 MacBook Pro 16GB (10c)" script: scripts/bench_optimizations.py (unmodified) + engine-level tiled×compile extension --- # M-series Fleet Ablation Benchmarks Same methodology as the 2026-03-09 wave2 doc (`--sweep ablation --resolution 512 1024 2048`, -checkpoint weights, 3 warmup + 10 bench runs), executed across **six machines** spanning the +checkpoint weights, 3 warmup + 10 bench runs), executed across **seven machines** spanning the Apple Silicon range — **M1 Max 32GB (24c) · M1 Ultra 128GB (64c) · M3 Air 16GB (10c, fanless) · -M3 Ultra 256GB (80c) · M4 Pro mini 24GB · M5 MacBook Pro 16GB (10c)** — i.e. every current GPU -generation, Studio→fanless laptop, an M1 width ladder (24c→64c) and a matched-width M3-vs-M5 -pair. Contributed from a multi-Mac render-farm setup; happy to run follow-ups on any of them. +M3 Ultra 256GB (80c) · M4 mini 16GB (10c) · M4 Pro mini 24GB · M5 MacBook Pro 16GB (10c)** — +i.e. every current GPU generation, Studio→fanless laptop, an M1 width ladder (24c→64c) and a +**matched-width M3→M4→M5 generation ladder (all 10-core)**. Contributed from a multi-Mac render-farm setup; happy to run follow-ups on any of them. ## Headline findings @@ -67,8 +67,14 @@ pair. Contributed from a multi-Mac render-farm setup; happy to run follow-ups on | M1 Ultra | 64c | 150.5 ms | 53.8 ms | **2.7×** | 0.59× | | M3 Air | 10c | 138.9 ms | 54.3 ms | **2.6×** | 0.40× | | M3 Ultra | 80c | 63.2 ms | 28.6 ms | **2.2×** | 0.53× | + | M4 mini | 10c | 48.79 ms | 24.40 ms | **2.0×** | 0.50× | | **M1 Max** | **24c** | 20.16 ms | 15.79 ms | **1.3×** | **0.87×** | + Note how tightly the padding win tracks the fused/unfused column (0.14×→4.9×, + 0.40×→2.6×, 0.50×→2.0×, 0.87×→1.3×) and how poorly it tracks core count. The + M4 mini was measured *after* the correction below and fits it — the theory made a + prediction and held. + We initially read this as "narrower GPU ⇒ bigger win" (M5 10c gains most, M3 Ultra 80c least). **The M1 Max refutes that**: it is mid-width (24c) yet gains the *least* of all. The actual predictor is each chip's **fused-kernel quality relative to its own raw matmul @@ -139,16 +145,22 @@ vs 12.2 ms). **How much the fast path is worth is chip-specific, not width-speci (M1 Max 24c: 0.87×; M5 10c: 0.14×) — see the six-machine table above. (M5 numbers taken on a machine in active desktop use; treat as indicative, not lab-clean.) -### Generational delta at *matched* GPU width — M3 Air vs M5, both 10-core +### Generational delta at *matched* GPU width — M3 → M4 → M5, all 10-core -The Ultras confound generation with width. These two don't: both are 10-core laptop GPUs, -two generations apart (fused SDPA, hdim 64, fp32): +The Ultras confound generation with width. These three don't: all are 10-core GPUs, one +generation apart each (fused SDPA, hdim 64, fp32): -| seq | M3 Air (10c) | M5 (10c) | **M5 speedup** | -|---|---|---|---| -| 1024 | 2.44 ms | 0.80 ms | **3.1×** | -| 2048 | 17.51 ms | 2.32 ms | **7.5×** | -| 4096 | 55.77 ms | 7.22 ms | **7.7×** | +| seq | M3 Air (10c) | M4 mini (10c) | M5 (10c) | **M5 vs M3** | +|---|---|---|---|---| +| 1024 | 2.44 ms | 1.86 ms | 0.80 ms | **3.1×** | +| 2048 | 17.51 ms | 6.35 ms | 2.32 ms | **7.5×** | +| 4096 | 55.77 ms | 24.47 ms | 7.22 ms | **7.7×** | + +At hdim **56** (what the model actually runs today) the same ladder is 138.9 → 48.8 → +44.0 ms: the M4 nearly triples the M3, but the **M5's advantage is concentrated almost +entirely in the fused kernel** — its unpadded step is barely better than the M4's, while +its padded step is ~3× better. i.e. M5's headline MLX gains show up *only if you hit the +fast path*, which makes the head-dim fix especially valuable on the newest silicon. ~7.7× at identical core count — i.e. the M5's gain is architectural, not width. (Apple's own published MLX figures claim ~3.8× M4→M5 on FLUX image-gen; this attention workload shows more.)