27564e80eb
BENCHMARKS.md says what a model cost the day it was measured; nothing noticed if a macOS/torch update or a thermal fault halved a node. perfcheck runs the whole fleet in ~35s off godcheck's 03:30 cron on the m4mini and reports drift into GODCHECK_LATEST.md. Probes matmul (fp16/fp32), memory bandwidth, and SDPA at head_dim 64 vs 56 — the latter turning CorridorKey's fast-path cliff into a permanent canary: it confirms the padding win fleet-wide (1.96x-5.26x) and tells us if a future torch closes it. Runs on venvs/rmbg/bin/python, already identical fleet-wide, so nothing new is installed (nothing lands on the disk-tight m1max). First cross-machine capability table for all 6 nodes. The M3 Ultra is ~2.2x the M1 Ultra on fp16 matmul, but they share ~625 GB/s — so bandwidth-bound stages run alike while compute-bound ones scale. Both Ultras reach only ~78% of spec bandwidth on a single kernel; the smaller Macs hit ~88%. Measuring a fleet that is doing real work is the whole problem, and naive benchmarking here is off by 11x: - min, not median: a concurrent trellis_mac job dragged a median-of-5 matmul from ~24500 to ~2150 GFLOP/s, which reads exactly like a catastrophic regression. - n=4096 not 2048: 2048 is dispatch-bound and swung 48% run-to-run; 4096 reproduces to 0.1% even while contended. - sdpa 16x2048 not 8x1024: sub-ms probes are dispatch noise — 8x1024 gave ratios of 0.79/3.95/2.35 on three runs of one machine, the first "proving" 56 is faster. - busy nodes are excluded, not blamed: GPU contention is invisible to load average (M3 Ultra read load 2.45 with its GPU pinned), so bench.py samples ioreg GPU% before it touches the GPU — our own matmul pins the device, so ordering is the trick. - baselines are the median of recent history, not a saved best: the M4 Pro also serves Ollama and is bimodal (~3200 vs ~5500 fp32), so a best-observed baseline pins to a lucky outlier and alerts forever. - a regression must repeat before it is believed ([~] watching -> [!] CONFIRMED). Validated by re-running the fleet against its own baselines: zero false alarms, including a sweep where the M3 Ultra read 43 GB/s under load and was correctly marked BUSY rather than reported as a 93% regression. Also found: the m4mini is the only node with Tailscale SSH (RunSSH: true) and it does NOT propagate remote exit codes — `ssh m4mini "exit 7"` returns 0, so any `if ssh m4mini ...` test silently always passes. Test on output instead, which is what godcheck already does (and why it is unaffected). It also cannot ssh to itself, so run_fleet detects its own tailnet IP and benches the local node via the shell. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
121 lines
7.3 KiB
Markdown
121 lines
7.3 KiB
Markdown
# PERFCHECK — nightly GPU performance canary across the fleet
|
||
|
||
`BENCHMARKS.md` is a *snapshot*: what each model cost on the day it was measured. Perfcheck is
|
||
the *tripwire*. A macOS point update, a torch/MLX bump, or a thermal fault can halve a node's
|
||
throughput, and nothing else in the fleet would notice — you'd just quietly wait twice as long
|
||
for every render until someone got suspicious months later. This catches it the next morning.
|
||
|
||
It runs as part of godcheck on the m4mini watchdog (`~/godcheck/nightly.sh`, cron 03:30),
|
||
and appends a table + drift lines to `~/Documents/GODCHECK_LATEST.md` on ultra. Whole fleet,
|
||
6 nodes, **~35s**.
|
||
|
||
## The numbers (torch 2.13.0 / MPS, 2026-07-17)
|
||
|
||
Every node runs `MODELBEAST/venvs/rmbg/bin/python` — the same torch is already installed
|
||
fleet-wide, so nothing new gets installed (notably nothing lands on the disk-tight m1max).
|
||
|
||
| node | chip | RAM | matmul fp16 | matmul fp32 | bandwidth | sdpa 64 | sdpa 56 | **56/64** |
|
||
|---|---|---|---|---|---|---|---|---|
|
||
| **m3ultra** | M3 Ultra | 256GB | 25202 | 22995 | 627 GB/s | 0.98ms | 4.27ms | **4.35x** |
|
||
| **m1** | M1 Ultra | 128GB | ~17100 | ~16000 | 639 GB/s | 1.50ms | 5.83ms | **3.89x** |
|
||
| **studio** | M1 Max | 32GB | 6996 | 6538 | 348 GB/s | 3.48ms | 10.73ms | **3.09x** |
|
||
| **m4** | M4 Pro | 24GB | 5678 | 3869 | 241 GB/s | 6.59ms | 12.88ms | **1.96x** |
|
||
| **m4mini** | M4 | 16GB | 3774 | 3436 | 105 GB/s | 5.46ms | 27.83ms | **5.10x** |
|
||
| **mini** | M1 (2020) | 8GB | 2310 | 1781 | 61 GB/s | 9.88ms | 51.84ms | **5.26x** |
|
||
|
||
GFLOP/s, higher better. m1/m3ultra are *best observed* rather than medians — both were serving
|
||
real work all session (see "busy nodes" below), so their baselines are still warming up.
|
||
|
||
Bandwidth tracks spec closely on the small machines — mini 61/68 GB/s (89%), m4mini 105/120
|
||
(88%), m4 241/273 (88%), studio 348/400 (87%) — but **both Ultras land near 625 GB/s against a
|
||
spec of ~800**, i.e. ~78%. A single `add_` kernel doesn't saturate the Ultra fabric; that's a
|
||
property of the interconnect, not a fault, and it is the number to beat if a workload ever
|
||
needs to.
|
||
|
||
### The 56/64 column is the point
|
||
|
||
CorridorKey's Hiera backbone uses `head_dim=56`, which misses the fused-attention fast path on
|
||
Apple Silicon — padding to 64 is why our fork is 1.3x–4.9x faster (see `CORRIDORKEY.md`). This
|
||
bench reproduces that cliff directly and independently, on every machine, every night:
|
||
|
||
- It confirms the padding win is real fleet-wide (**1.96x–5.26x**, never below 1).
|
||
- **If a future torch closes the gap, this column collapses toward 1.0 and we find out** —
|
||
instead of carrying a workaround forever without noticing it became unnecessary.
|
||
|
||
The M4 Pro benefits least (1.96x); the small M4/M1 benefit most (>5x). Note these are pure
|
||
microbenchmark ratios — a full CorridorKey run dilutes them with non-attention work, so they
|
||
run higher than the end-to-end gains in `CORRIDORKEY.md`.
|
||
|
||
## Usage
|
||
|
||
```sh
|
||
scripts/perfcheck/run_fleet.py # bench fleet, compare to history, report
|
||
scripts/perfcheck/run_fleet.py --only m1,m4 # subset
|
||
scripts/perfcheck/run_fleet.py --show # print current baselines, don't bench
|
||
scripts/perfcheck/deploy_to_watchdog.sh # push to m4mini — RE-RUN AFTER EDITING bench.py
|
||
```
|
||
|
||
Exit 1 on a confirmed regression. State (`history.jsonl`, `baselines.json`, `pending.json`) is
|
||
gitignored and per-runner; the authoritative copy lives on the m4mini.
|
||
|
||
**Editing `bench.py` changes what the numbers mean.** Bump `SCHEMA` in it (and in
|
||
`run_fleet.py`) whenever a probe's shape or size changes — older history is then ignored rather
|
||
than being median'd against a different benchmark and mistaken for a regression. Then re-run
|
||
`deploy_to_watchdog.sh`, or the fleet keeps being measured by the old bench.
|
||
|
||
## Why it's built the way it is
|
||
|
||
Everything below is a scar. The fleet runs real work while being measured, and that fact
|
||
dominates the whole design.
|
||
|
||
**Naive benchmarking here is off by 11x.** The first version reported the M3 Ultra at 2154
|
||
GFLOP/s and 84 GB/s. The true figures are ~24900 and ~620. Two causes: `n=2048` matmul is
|
||
dispatch-bound rather than compute-bound (it swung 11700 vs 17361 across runs — 48%), and
|
||
`median` lets a noisy neighbour through. `n=4096` reproduced to **0.1%** across runs *while the
|
||
GPU was contended*. Sizes here are chosen for stability, not speed — shrinking them to make the
|
||
bench faster turns it back into a random number generator.
|
||
|
||
**min, not mean/median.** A concurrent `trellis_mac` job dragged a median-of-5 matmul from
|
||
~24500 to ~2150 — a 5x error that looks exactly like a catastrophic regression. The minimum
|
||
time is the least-interfered sample and the only estimate that survives a busy neighbour.
|
||
|
||
**Sub-millisecond probes measure nothing.** The obvious sdpa size (8 heads x 1024) is ~0.4ms
|
||
and pure dispatch noise: three consecutive runs on one machine gave ratios of 0.79, 3.95 and
|
||
2.35 — the first would have "proven" head_dim=56 is *faster*, contradicting the real 8-machine
|
||
result. At 16 x 2048 it lands ~1-4ms and reproduces to 0.6%.
|
||
|
||
**Busy nodes are excluded, not blamed.** GPU contention is invisible to load average — the M3
|
||
Ultra sat at **load 2.45 with its GPU fully pinned** by trellis_mac. So `bench.py` samples GPU
|
||
utilization from `ioreg` *before it touches the GPU* (our own matmul pins the device at ~100%,
|
||
so sampling during the run cannot separate us from a neighbour — the ordering is the whole
|
||
trick). Above 50% the node is marked `[~] BUSY`, and its numbers raise no alert and never enter
|
||
a baseline. Two quirks: the first `ioreg` read is garbage (it is since-last-read, and an idle
|
||
m4mini reported 97% cold), and a GPU job alternates phases (100% CPU / 0% GPU one moment, 97%
|
||
GPU the next) — hence discard-first and max-of-several.
|
||
|
||
**Baselines are the median of recent history, not a saved "best".** These machines have
|
||
neighbours: the M4 Pro also serves Ollama and its fp32 is bimodal (~3200 vs ~5500 depending on
|
||
whether a request is in flight); the M1 Ultra runs `mrpgi` on its GPU and swings 8301–15891
|
||
(1.9x). A best-observed baseline pins to a lucky outlier the node rarely reaches again — which
|
||
means alerting every night forever. A median is what the machine actually does.
|
||
|
||
**A regression must repeat before it's believed.** Transient contention is far more common than
|
||
real regression and doesn't recur on schedule. First sighting is `[~] watching`; only a second
|
||
consecutive one is `[!] CONFIRMED` and fails the run. Getting *faster* is reported immediately
|
||
— it's never an alarm.
|
||
|
||
## Fleet gotcha found while building this
|
||
|
||
**The m4mini is the only node with Tailscale SSH enabled (`RunSSH: true`), and it does not
|
||
propagate remote exit codes.** `ssh m4mini "exit 7"` returns **0**; so does `ssh m4mini
|
||
"false"`. Every other node propagates correctly. This means **any `if ssh m4mini ...; then`
|
||
test silently always passes** — a reach-check written that way will report every node healthy
|
||
no matter what. Test on *output* instead (`x=$(ssh … && echo YES)`), which is what godcheck
|
||
already does throughout, and why godcheck is unaffected.
|
||
|
||
It also means the watchdog cannot ssh to *itself* ("Host key verification failed"), so
|
||
`run_fleet.py` detects its own tailnet IP and benches the local node through the shell instead.
|
||
|
||
Worth deciding: `tailscale set --ssh=false` on the m4mini would make it behave like the rest of
|
||
the fleet, but it's a live config change on the always-on watchdog and hasn't been made.
|