corridorkey-mrp-mlx/docs/2026-07-16-m-series-fleet-ablation-results.md

title	date	device	script
M-series fleet ablation: 7 machines, M1 → M5 (16GB mini → 256GB Ultra)	2026-07-16	M1 Max 32GB (24c) · M1 Ultra 128GB (64c) · M3 Air 16GB (10c, fanless) · M3 Ultra 256GB (80c) · M4 mini 16GB (10c) · M4 Pro mini 24GB · M5 MacBook Pro 16GB (10c)	scripts/bench_optimizations.py (unmodified) + engine-level tiled×compile extension

M-series Fleet Ablation Benchmarks

Same methodology as the 2026-03-09 wave2 doc (--sweep ablation --resolution 512 1024 2048, checkpoint weights, 3 warmup + 10 bench runs), executed across seven machines spanning the Apple Silicon range — M1 Max 32GB (24c) · M1 Ultra 128GB (64c) · M3 Air 16GB (10c, fanless) · M3 Ultra 256GB (80c) · M4 mini 16GB (10c) · M4 Pro mini 24GB · M5 MacBook Pro 16GB (10c) — i.e. every current GPU generation, Studio→fanless laptop, an M1 width ladder (24c→64c) and a matched-width M3→M4→M5 generation ladder (all 10-core). Contributed from a multi-Mac render-farm setup; happy to run follow-ups on any of them.

Headline findings

1. sdpa is a ~4.5× regression on M1-class GPUs at 2048 — and free on M3. Single-toggle isolation at 2048 (median of 6, all-off baseline):

   toggle	M1 Ultra	M3 Ultra
   all off	4139 ms	1743 ms
   bf16 only	4120 ms (neutral!)	1766 ms
   sdpa only	18592 ms (4.5× slower)	1763 ms
   bf16+sdpa	6388 ms	1767 ms

   The suspected M1 danger (bf16 emulation — M1 lacks hardware bf16) is innocent on this workload.

   Root cause (follow-up, same day): it is not the SDPA kernel. Instrumenting a real 2048 forward shows the model's attention shapes are head_dim=56 (global blocks (1,8,16384,56) ×15, (1,16,4096,56) ×3; windowed (4096,·,·,56)). At head_dim 56, mx.fast.scaled_dot_product_attention silently falls back to the unfused path — fused == unfused within 1 % on every shape on both M1 and M3 Ultra. So use_sdpa=True currently buys zero kernel benefit on any machine, while its branch pays extra 5-D transpose/reshape choreography — which is what costs ~4.5× on M1 (layout-sensitive), ~free on M3.

   The actionable win: pad head_dim 56 → 64. The fast path then engages and beats unfused decisively — despite ~14 % more FLOPs (fp32, exact model shapes):

   shape (hdim 64)	M3 fused/unfused	M1 fused/unfused
   (1, 8, 16384, ·)	28.6 / 63.1 ms → 2.2×	53.8 / 144.5 ms → 2.7×
   (1, 16, 4096, ·)	3.8 / 7.2 ms → 1.9×	7.5 / 11.1 ms → 1.5×
   (4096, 2, 64, ·)	1.3 / 2.5 ms → 1.9×	1.5 / 2.6 ms → 1.7×

   On M5 it's worth ~5× — the largest win we measured. M5 MacBook Pro (10-core GPU, 16 GB), hdim 56 vs 64, fp32:

   shape	hdim 56 (today)	hdim 64 (padded)	speedup from padding
   (1, 16, 4096)	44.00 ms (fallback: 1.05× vs unfused)	8.91 ms (fast path: 0.21×)	4.9×
   (1, 16, 2048)	11.44 ms (1.06×)	2.23 ms (0.22×)	5.1×
   (4096, 2, 64)	16.72 ms (0.97×)	5.29 ms (0.25×)	3.2×

   The fallback is universal; the size of the win is not. Measured on six machines (padding 56→64 at (1,16,4096) fp32). Every machine shows the fallback at hdim 56 (fused/unfused 0.97–1.06×, i.e. the fast kernel never engages) — but the payoff varies ~4× and does not track GPU width monotonically:

   machine	GPU cores	hdim 56	hdim 64	padding speedup	fused/unfused @4096
   M5 MacBook Pro	10c	44.00 ms	8.91 ms	4.9×	0.14×
   M1 Ultra	64c	150.5 ms	53.8 ms	2.7×	0.59×
   M3 Air	10c	138.9 ms	54.3 ms	2.6×	0.40×
   M3 Ultra	80c	63.2 ms	28.6 ms	2.2×	0.53×
   M4 mini	10c	48.79 ms	24.40 ms	2.0×	0.50×
   M1 Max	24c	20.16 ms	15.79 ms	1.3×	0.87×

   Note how tightly the padding win tracks the fused/unfused column (0.14×→4.9×, 0.40×→2.6×, 0.50×→2.0×, 0.87×→1.3×) and how poorly it tracks core count. The M4 mini was measured after the correction below and fits it — the theory made a prediction and held.

   We initially read this as "narrower GPU ⇒ bigger win" (M5 10c gains most, M3 Ultra 80c least). The M1 Max refutes that: it is mid-width (24c) yet gains the least of all. The actual predictor is each chip's fused-kernel quality relative to its own raw matmul throughput — the M1 Max's fused path is only ~0.87× its unfused path at every size (i.e. MLX's SDPA kernel barely beats a plain matmul there), whereas the M5's is 0.14×. Where the fused kernel is strong, missing it is expensive; where it's weak, missing it barely matters.

   Practical upshot is unchanged and strictly positive: padding to 64 helps on every Apple GPU generation tested — by 1.3× (M1 Max) to 4.9× (M5) — and hurts nowhere, despite ~14 % more FLOPs.

   Suggested changes: (a) pad qkv projections to head_dim 64 (at minimum for the global blocks) so sdpa's fast kernel actually engages — M1 benefits more than M3; (b) until then, default use_sdpa=False (it is currently overhead-only). For MLX upstream: a warning (or doc note) when sdpa silently falls back on unsupported head dims would have made this obvious much sooner.

2. stage_gc is harmful on Ultra-class machines at every resolution — 0.53× at 512, 0.64× at 1024, 0.66× at 2048 on M3 Ultra, for a 1–5 % peak-memory saving. Wave2 measured a mild 0.79–0.99× on its reference hardware; on big-memory machines it's pure overhead. Suggestion: document as a low-memory-only flag.

3. Tiled + compile is the best 2048 config on both Ultras — upstream engine.py forced compile=False in tiled mode; tiles are fixed-shape, so fused compilation applies (patched in this fork, output bit-identical, verified against ground-truth alpha):

   engine config (2048 input)	M3 Ultra	M1 Ultra	peak	alpha MAE*
   full-frame 2048	2788 ms	5373 ms	27.9 GB	0.00906
   tiled 512/64	2760 ms	4576 ms	2.2 GB	0.00821
   tiled 512/64 + compile	2478 ms	4222 ms	2.3 GB	0.00821
   tiled 768/64 + compile	1949 ms	3275 ms	2.4 GB	0.00842
   tiled 1024/64 + compile	3475 ms	5864 ms	3.7 GB	0.00915

   *alpha MAE vs exact ground truth: synthetic 2048² green-screen plates (soft-alpha subject + motion-blur stripes + defocus disk composited over chroma green), hint = 8× downscaled truth. Tiled beats full-frame on accuracy as well as memory — the model runs at native tile scale over full-res input. Confirms wave2's tiled-768 pick and adds ~8–13 % from compiling the tile graph.

4. 24 GB Macs must tile at 2048. On the M4 Pro mini (24 GB), every full-frame 2048 config lands at 20–24 s/run — the ~26 GB working set swaps; toggle choice becomes noise. At ≤1024 the M4 is healthy (218 ms @512, 1149 ms @1024). Tiled 2048 runs in ~2.3 GB.

Cross-machine baselines (all-off)

res	M3 Ultra	M1 Ultra	M4 Pro 24GB	wave2 reference
512	53.3 ms	80.8 ms	218.1 ms	119.6 ms
1024	247.0 ms	350.6 ms	1149.2 ms	610.7 ms
2048	1750.2 ms	3527.3 ms	23147 ms (swap)	4984.7 ms

Raw SDPA kernel scaling (pure MLX, no CorridorKey) — fused vs unfused, hdim 64, fp32

Useful context for why the head-dim fix matters differently per machine. Fused-vs-unfused ratio (lower = fused is winning by more):

seq	M3 Ultra (80c)	M1 Ultra (64c)	M5 (10c laptop)
1024	0.46×	0.54×	0.28×
2048	0.53×	0.71×	0.21×
4096	0.53×	0.59×	0.14×

Absolute fused times at 4096/fp32: M3 3.86 ms · M1 7.19 ms · M5 7.22 ms — an M5 laptop matches an M1 Ultra on fused attention while being ~4× slower unfused (51.5 ms vs 12.2 ms). How much the fast path is worth is chip-specific, not width-specific (M1 Max 24c: 0.87×; M5 10c: 0.14×) — see the six-machine table above. (M5 numbers taken on a machine in active desktop use; treat as indicative, not lab-clean.)

Generational delta at matched GPU width — M3 → M4 → M5, all 10-core

The Ultras confound generation with width. These three don't: all are 10-core GPUs, one generation apart each (fused SDPA, hdim 64, fp32):

seq	M3 Air (10c)	M4 mini (10c)	M5 (10c)	M5 vs M3
1024	2.44 ms	1.86 ms	0.80 ms	3.1×
2048	17.51 ms	6.35 ms	2.32 ms	7.5×
4096	55.77 ms	24.47 ms	7.22 ms	7.7×

At hdim 56 (what the model actually runs today) the same ladder is 138.9 → 48.8 → 44.0 ms: the M4 nearly triples the M3, but the M5's advantage is concentrated almost entirely in the fused kernel — its unpadded step is barely better than the M4's, while its padded step is ~3× better. i.e. M5's headline MLX gains show up only if you hit the fast path, which makes the head-dim fix especially valuable on the newest silicon.

~7.7× at identical core count — i.e. the M5's gain is architectural, not width. (Apple's own published MLX figures claim ~3.8× M4→M5 on FLUX image-gen; this attention workload shows more.) The head_dim-56 fallback reproduces on the M3 Air too — padding to 64 gives 2.6× (138.91 → 54.26 ms at (1,16,4096)), so the finding now holds on M1, M3 Ultra, M3 Air, M4 Pro and M5 — every generation Apple currently ships.

Fanless sustained load: the M3 Air does not throttle on this workload (negative result)

Every number above is a burst measurement, so we checked whether a fanless chassis invalidates them. Sustained SDPA (1,16,2048,64) fp32 on the M3 Air (MacBook Air, no fan), per-30s medians over 5 minutes:

elapsed	30s	60s	150s	240s	300s
median	13.85 ms	14.91 ms	14.55 ms	14.43 ms	14.04 ms
vs first	1.00×	1.08×	1.05×	1.04×	1.01×

No meaningful throttle — ~4 % wobble, ending where it started. We expected a decay curve and did not find one; the burst numbers in this report are therefore not flattered by short runs. Caveat: attention at this shape may not be power-dense enough to reach the thermal ceiling — a full multi-minute diffusion pipeline could still behave differently.

Full ablation tables

M3 Ultra 256GB

Config	512	1024	2048	peak @2048
baseline	53.3	247.0	1750.2	26689 MB
slim+sdpa+bf16+fused_decode+gpu_preprocess	53.5	248.2	1790.1	27245 MB
slim+stage_gc+bf16+fused_decode+gpu_preprocess	95.4	382.7	2644.3	26689 MB
slim+stage_gc+sdpa+fused_decode+gpu_preprocess	96.3	385.9	2676.9	26661 MB
slim+stage_gc+sdpa+bf16+gpu_preprocess	99.4	372.1	2618.6	26661 MB
slim+stage_gc+sdpa+bf16+fused_decode	99.0	386.5	2688.0	26661 MB
stage_gc+sdpa+bf16+fused_decode+gpu_preprocess	100.1	391.4	2628.2	26661 MB
slim+stage_gc+sdpa+bf16+fused_decode+gpu_preprocess	101.0	390.9	2617.9	26661 MB

M1 Ultra 128GB

Config	512	1024	2048	peak @2048
baseline	80.8	350.6	3527.3	26689 MB
slim+sdpa+bf16+fused_decode+gpu_preprocess	79.1	353.3	33137.8	27245 MB
slim+stage_gc+bf16+fused_decode+gpu_preprocess	189.5	705.5	4919.0	26689 MB
slim+stage_gc+sdpa+fused_decode+gpu_preprocess	195.6	711.1	4939.2	26661 MB
slim+stage_gc+sdpa+bf16+gpu_preprocess	198.2	681.6	4853.8	26661 MB
slim+stage_gc+sdpa+bf16+fused_decode	197.7	714.1	4822.9	26661 MB
stage_gc+sdpa+bf16+fused_decode+gpu_preprocess	194.2	733.2	6274.1	26661 MB
slim+stage_gc+sdpa+bf16+fused_decode+gpu_preprocess	193.8	737.1	5128.2	26661 MB

Note the 33.1 s outlier: the only 2048 config without stage_gc but with sdpa — sdpa's slow path dominating once nothing throttles it (see headline 1 for the isolation). The sdpa cliff appears only at 2048; 512/1024 are unaffected.

M4 Pro Mac mini 24GB

Config	512	1024	2048 (swap-bound)
baseline	218.1	1149.2	23147.1
slim+sdpa+bf16+fused_decode+gpu_preprocess	221.4	1139.0	20994.5
slim+stage_gc+… (all stage_gc combos)	266–273	1268–1292	20233–24253

At 2048 the ~26 GB working set exceeds 24 GB unified memory; all configs swap and differences are not meaningful. ≤1024 full-frame or tiled-anything is the usable envelope.

Recommended settings by hardware

Hardware	2048 recommendation
M3-class (Max/Ultra)	full-frame or tiled 768/64 + compile; every toggle optional; avoid stage_gc
M1/M2-class	avoid sdpa; tiled 768/64 + compile; avoid stage_gc
≤24–32 GB any gen	tiled required (2.3 GB vs 26 GB); tiled 768/64 + compile

Environment: MLX (venv per uv sync --extra mlx), macOS 26.5, checkpoint v1.0.0. Quality harness (ground-truth plates + scoring) available on request — it's ~150 lines and reproduces the alpha-MAE column.