m3ultra 27564e80eb perfcheck: nightly fleet GPU canary + the contention lessons that shaped it

BENCHMARKS.md says what a model cost the day it was measured; nothing noticed if a
macOS/torch update or a thermal fault halved a node. perfcheck runs the whole fleet
in ~35s off godcheck's 03:30 cron on the m4mini and reports drift into
GODCHECK_LATEST.md.

Probes matmul (fp16/fp32), memory bandwidth, and SDPA at head_dim 64 vs 56 — the
latter turning CorridorKey's fast-path cliff into a permanent canary: it confirms
the padding win fleet-wide (1.96x-5.26x) and tells us if a future torch closes it.
Runs on venvs/rmbg/bin/python, already identical fleet-wide, so nothing new is
installed (nothing lands on the disk-tight m1max).

First cross-machine capability table for all 6 nodes. The M3 Ultra is ~2.2x the M1
Ultra on fp16 matmul, but they share ~625 GB/s — so bandwidth-bound stages run alike
while compute-bound ones scale. Both Ultras reach only ~78% of spec bandwidth on a
single kernel; the smaller Macs hit ~88%.

Measuring a fleet that is doing real work is the whole problem, and naive
benchmarking here is off by 11x:

- min, not median: a concurrent trellis_mac job dragged a median-of-5 matmul from
  ~24500 to ~2150 GFLOP/s, which reads exactly like a catastrophic regression.
- n=4096 not 2048: 2048 is dispatch-bound and swung 48% run-to-run; 4096 reproduces
  to 0.1% even while contended.
- sdpa 16x2048 not 8x1024: sub-ms probes are dispatch noise — 8x1024 gave ratios of
  0.79/3.95/2.35 on three runs of one machine, the first "proving" 56 is faster.
- busy nodes are excluded, not blamed: GPU contention is invisible to load average
  (M3 Ultra read load 2.45 with its GPU pinned), so bench.py samples ioreg GPU% before
  it touches the GPU — our own matmul pins the device, so ordering is the trick.
- baselines are the median of recent history, not a saved best: the M4 Pro also serves
  Ollama and is bimodal (~3200 vs ~5500 fp32), so a best-observed baseline pins to a
  lucky outlier and alerts forever.
- a regression must repeat before it is believed ([~] watching -> [!] CONFIRMED).

Validated by re-running the fleet against its own baselines: zero false alarms,
including a sweep where the M3 Ultra read 43 GB/s under load and was correctly
marked BUSY rather than reported as a 93% regression.

Also found: the m4mini is the only node with Tailscale SSH (RunSSH: true) and it does
NOT propagate remote exit codes — `ssh m4mini "exit 7"` returns 0, so any
`if ssh m4mini ...` test silently always passes. Test on output instead, which is
what godcheck already does (and why it is unaffected). It also cannot ssh to itself,
so run_fleet detects its own tailnet IP and benches the local node via the shell.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-07-17 18:03:03 +10:00

7.3 KiB

Raw Blame History

PERFCHECK — nightly GPU performance canary across the fleet

BENCHMARKS.md is a snapshot: what each model cost on the day it was measured. Perfcheck is the tripwire. A macOS point update, a torch/MLX bump, or a thermal fault can halve a node's throughput, and nothing else in the fleet would notice — you'd just quietly wait twice as long for every render until someone got suspicious months later. This catches it the next morning.

It runs as part of godcheck on the m4mini watchdog (~/godcheck/nightly.sh, cron 03:30), and appends a table + drift lines to ~/Documents/GODCHECK_LATEST.md on ultra. Whole fleet, 6 nodes, ~35s.

The numbers (torch 2.13.0 / MPS, 2026-07-17)

Every node runs MODELBEAST/venvs/rmbg/bin/python — the same torch is already installed fleet-wide, so nothing new gets installed (notably nothing lands on the disk-tight m1max).

node	chip	RAM	matmul fp16	matmul fp32	bandwidth	sdpa 64	sdpa 56	56/64
m3ultra	M3 Ultra	256GB	25202	22995	627 GB/s	0.98ms	4.27ms	4.35x
m1	M1 Ultra	128GB	~17100	~16000	639 GB/s	1.50ms	5.83ms	3.89x
studio	M1 Max	32GB	6996	6538	348 GB/s	3.48ms	10.73ms	3.09x
m4	M4 Pro	24GB	5678	3869	241 GB/s	6.59ms	12.88ms	1.96x
m4mini	M4	16GB	3774	3436	105 GB/s	5.46ms	27.83ms	5.10x
mini	M1 (2020)	8GB	2310	1781	61 GB/s	9.88ms	51.84ms	5.26x

GFLOP/s, higher better. m1/m3ultra are best observed rather than medians — both were serving real work all session (see "busy nodes" below), so their baselines are still warming up.

Bandwidth tracks spec closely on the small machines — mini 61/68 GB/s (89%), m4mini 105/120 (88%), m4 241/273 (88%), studio 348/400 (87%) — but both Ultras land near 625 GB/s against a spec of ~800, i.e. ~78%. A single add_ kernel doesn't saturate the Ultra fabric; that's a property of the interconnect, not a fault, and it is the number to beat if a workload ever needs to.

The 56/64 column is the point

CorridorKey's Hiera backbone uses head_dim=56, which misses the fused-attention fast path on Apple Silicon — padding to 64 is why our fork is 1.3x–4.9x faster (see CORRIDORKEY.md). This bench reproduces that cliff directly and independently, on every machine, every night:

It confirms the padding win is real fleet-wide (1.96x–5.26x, never below 1).
If a future torch closes the gap, this column collapses toward 1.0 and we find out — instead of carrying a workaround forever without noticing it became unnecessary.

The M4 Pro benefits least (1.96x); the small M4/M1 benefit most (>5x). Note these are pure microbenchmark ratios — a full CorridorKey run dilutes them with non-attention work, so they run higher than the end-to-end gains in CORRIDORKEY.md.

Usage

scripts/perfcheck/run_fleet.py              # bench fleet, compare to history, report
scripts/perfcheck/run_fleet.py --only m1,m4 # subset
scripts/perfcheck/run_fleet.py --show       # print current baselines, don't bench
scripts/perfcheck/deploy_to_watchdog.sh     # push to m4mini — RE-RUN AFTER EDITING bench.py

Exit 1 on a confirmed regression. State (history.jsonl, baselines.json, pending.json) is gitignored and per-runner; the authoritative copy lives on the m4mini.

Editing bench.py changes what the numbers mean. Bump SCHEMA in it (and in run_fleet.py) whenever a probe's shape or size changes — older history is then ignored rather than being median'd against a different benchmark and mistaken for a regression. Then re-run deploy_to_watchdog.sh, or the fleet keeps being measured by the old bench.

Why it's built the way it is

Everything below is a scar. The fleet runs real work while being measured, and that fact dominates the whole design.

Naive benchmarking here is off by 11x. The first version reported the M3 Ultra at 2154 GFLOP/s and 84 GB/s. The true figures are ~24900 and ~620. Two causes: n=2048 matmul is dispatch-bound rather than compute-bound (it swung 11700 vs 17361 across runs — 48%), and median lets a noisy neighbour through. n=4096 reproduced to 0.1% across runs while the GPU was contended. Sizes here are chosen for stability, not speed — shrinking them to make the bench faster turns it back into a random number generator.

min, not mean/median. A concurrent trellis_mac job dragged a median-of-5 matmul from ~24500 to ~2150 — a 5x error that looks exactly like a catastrophic regression. The minimum time is the least-interfered sample and the only estimate that survives a busy neighbour.

Sub-millisecond probes measure nothing. The obvious sdpa size (8 heads x 1024) is ~0.4ms and pure dispatch noise: three consecutive runs on one machine gave ratios of 0.79, 3.95 and 2.35 — the first would have "proven" head_dim=56 is faster, contradicting the real 8-machine result. At 16 x 2048 it lands ~1-4ms and reproduces to 0.6%.

Busy nodes are excluded, not blamed. GPU contention is invisible to load average — the M3 Ultra sat at load 2.45 with its GPU fully pinned by trellis_mac. So bench.py samples GPU utilization from ioreg before it touches the GPU (our own matmul pins the device at ~100%, so sampling during the run cannot separate us from a neighbour — the ordering is the whole trick). Above 50% the node is marked [~] BUSY, and its numbers raise no alert and never enter a baseline. Two quirks: the first ioreg read is garbage (it is since-last-read, and an idle m4mini reported 97% cold), and a GPU job alternates phases (100% CPU / 0% GPU one moment, 97% GPU the next) — hence discard-first and max-of-several.

Baselines are the median of recent history, not a saved "best". These machines have neighbours: the M4 Pro also serves Ollama and its fp32 is bimodal (~3200 vs ~5500 depending on whether a request is in flight); the M1 Ultra runs mrpgi on its GPU and swings 8301–15891 (1.9x). A best-observed baseline pins to a lucky outlier the node rarely reaches again — which means alerting every night forever. A median is what the machine actually does.

A regression must repeat before it's believed. Transient contention is far more common than real regression and doesn't recur on schedule. First sighting is [~] watching; only a second consecutive one is [!] CONFIRMED and fails the run. Getting faster is reported immediately — it's never an alarm.

Fleet gotcha found while building this

The m4mini is the only node with Tailscale SSH enabled (RunSSH: true), and it does not propagate remote exit codes. ssh m4mini "exit 7" returns 0; so does ssh m4mini "false". Every other node propagates correctly. This means any if ssh m4mini ...; then test silently always passes — a reach-check written that way will report every node healthy no matter what. Test on output instead (x=$(ssh … && echo YES)), which is what godcheck already does throughout, and why godcheck is unaffected.

It also means the watchdog cannot ssh to itself ("Host key verification failed"), so run_fleet.py detects its own tailnet IP and benches the local node through the shell instead.

Worth deciding: tailscale set --ssh=false on the m4mini would make it behave like the rest of the fleet, but it's a live config change on the always-on watchdog and hasn't been made.

7.3 KiB Raw Blame History Unescape Escape