monster/modelbeast
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

benchmarks: 6-machine fleet MLX matrix — bf16-on-M1 answered, head_dim-64 padding critical on M4/M5

Browse Source 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
m3ultra 2026-07-19 00:31:55 +10:00
parent daa6124dcf
commit ed2ee8dc16
1 changed files with 21 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

21
BENCHMARKS.md
View File

@ -147,3 +147,24 @@ regression suite (12 checks, ~30s).
can make a node look 5–11x slower and it is invisible to load average (the M3 Ultra reads load
2.45 with its GPU pinned). Anything measured here should use min-of-runs and check
`ioreg … IOAccelerator` "Device Utilization %" *before* touching the GPU. See PERFCHECK.md.
## 2026-07-19 — fleet MLX capability matrix (mlx 0.32.0 everywhere)
Probe: 4096² matmul TFLOPS (fp16/bf16), 256MB elementwise GB/s, sdpa head_dim 56 vs 64 (B1·H8·L2048).
| host | chip | RAM | fp16 TF | bf16 TF | ~GB/s | sdpa d56/d64 |
|---|---|---|---|---|---|---|
| m3ultra | M3 Ultra | 256 | 23.48 | 23.54 | 400 | 1.51× |
| ultra | M1 Ultra | 128 | 16.31 | 13.16 | 297 | 0.94× |
| JING5 | M5 | 16 | 8.61 | 8.99 | 43* | **4.72×** |
| m4pro | M4 Pro | 24 | 5.24 | 3.86 | 225 | 1.29× |
| m4mini | M4 | 16 | 3.65 | 3.65 | 98 | 1.55× |
| AIRPRO | M3 | 16 | 1.55 | 1.64 | 87 | 0.95× |
*JING5 bandwidth measured under interactive load — retest idle before trusting.
Verdicts:
- **bf16-on-M1 question answered**: works, ~20% slower than fp16 (16.3→13.2). Ship fp16 on M1-class, bf16 free on M3 Ultra/M4/M5.
- **head_dim pad-to-64 matters MOST on newest chips** (M5 4.7×!, M4-class ~1.3-1.6×, M3 Ultra 1.5×) and ~zero on M1-class (no fused fast path there anyway — matches the corridorkey ablation).
- Fleet total ≈ 59 TFLOPS fp16; the two Ultras are 2/3 of it. TRELLIS.2-MLX lane assignment: m3ultra primary, ultra second (fp16), M4 Pro light ops.
- Probe script: reusable one-liner via `uv run --with mlx` — mlxprobe.py (this bench).