# PERFCHECK — nightly GPU performance canary across the fleet

`BENCHMARKS.md` is a *snapshot*: what each model cost on the day it was measured. Perfcheck is
the *tripwire*. A macOS point update, a torch/MLX bump, or a thermal fault can halve a node's
throughput, and nothing else in the fleet would notice — you'd just quietly wait twice as long
for every render until someone got suspicious months later. This catches it the next morning.

It runs as part of godcheck on the m4mini watchdog (`~/godcheck/nightly.sh`, cron 03:30),
and appends a table + drift lines to `~/Documents/GODCHECK_LATEST.md` on ultra. Whole fleet,
6 nodes, **~35s**.

## The numbers (torch 2.13.0 / MPS, 2026-07-17)

Every node runs `MODELBEAST/venvs/rmbg/bin/python` — the same torch is already installed
fleet-wide, so nothing new gets installed (notably nothing lands on the disk-tight m1max).

| node | chip | RAM | matmul fp16 | matmul fp32 | bandwidth | sdpa 64 | sdpa 56 | **56/64** |
|---|---|---|---|---|---|---|---|---|
| **m3ultra** | M3 Ultra | 256GB | 25202 | 22995 | 627 GB/s | 0.98ms | 4.27ms | **4.35x** |
| **m1** | M1 Ultra | 128GB | ~17100 | ~16000 | 639 GB/s | 1.50ms | 5.83ms | **3.89x** |
| **studio** | M1 Max | 32GB | 6996 | 6538 | 348 GB/s | 3.48ms | 10.73ms | **3.09x** |
| **m4** | M4 Pro | 24GB | 5678 | 3869 | 241 GB/s | 6.59ms | 12.88ms | **1.96x** |
| **m4mini** | M4 | 16GB | 3774 | 3436 | 105 GB/s | 5.46ms | 27.83ms | **5.10x** |
| **mini** | M1 (2020) | 8GB | 2310 | 1781 | 61 GB/s | 9.88ms | 51.84ms | **5.26x** |

GFLOP/s, higher better. m1/m3ultra are *best observed* rather than medians — both were serving
real work all session (see "busy nodes" below), so their baselines are still warming up.

Bandwidth tracks spec closely on the small machines — mini 61/68 GB/s (89%), m4mini 105/120
(88%), m4 241/273 (88%), studio 348/400 (87%) — but **both Ultras land near 625 GB/s against a
spec of ~800**, i.e. ~78%. A single `add_` kernel doesn't saturate the Ultra fabric; that's a
property of the interconnect, not a fault, and it is the number to beat if a workload ever
needs to.

### The 56/64 column is the point

CorridorKey's Hiera backbone uses `head_dim=56`, which misses the fused-attention fast path on
Apple Silicon — padding to 64 is why our fork is 1.3x–4.9x faster (see `CORRIDORKEY.md`). This
bench reproduces that cliff directly and independently, on every machine, every night:

- It confirms the padding win is real fleet-wide (**1.96x–5.26x**, never below 1).
- **If a future torch closes the gap, this column collapses toward 1.0 and we find out** —
  instead of carrying a workaround forever without noticing it became unnecessary.

The M4 Pro benefits least (1.96x); the small M4/M1 benefit most (>5x). Note these are pure
microbenchmark ratios — a full CorridorKey run dilutes them with non-attention work, so they
run higher than the end-to-end gains in `CORRIDORKEY.md`.

## Usage

```sh
scripts/perfcheck/run_fleet.py              # bench fleet, compare to history, report
scripts/perfcheck/run_fleet.py --only m1,m4 # subset
scripts/perfcheck/run_fleet.py --show       # print current baselines, don't bench
scripts/perfcheck/deploy_to_watchdog.sh     # push to m4mini — RE-RUN AFTER EDITING bench.py
```

Exit 1 on a confirmed regression. State (`history.jsonl`, `baselines.json`, `pending.json`) is
gitignored and per-runner; the authoritative copy lives on the m4mini.

**Editing `bench.py` changes what the numbers mean.** Bump `SCHEMA` in it (and in
`run_fleet.py`) whenever a probe's shape or size changes — older history is then ignored rather
than being median'd against a different benchmark and mistaken for a regression. Then re-run
`deploy_to_watchdog.sh`, or the fleet keeps being measured by the old bench.

## Why it's built the way it is

Everything below is a scar. The fleet runs real work while being measured, and that fact
dominates the whole design.

**Naive benchmarking here is off by 11x.** The first version reported the M3 Ultra at 2154
GFLOP/s and 84 GB/s. The true figures are ~24900 and ~620. Two causes: `n=2048` matmul is
dispatch-bound rather than compute-bound (it swung 11700 vs 17361 across runs — 48%), and
`median` lets a noisy neighbour through. `n=4096` reproduced to **0.1%** across runs *while the
GPU was contended*. Sizes here are chosen for stability, not speed — shrinking them to make the
bench faster turns it back into a random number generator.

**min, not mean/median.** A concurrent `trellis_mac` job dragged a median-of-5 matmul from
~24500 to ~2150 — a 5x error that looks exactly like a catastrophic regression. The minimum
time is the least-interfered sample and the only estimate that survives a busy neighbour.

**Sub-millisecond probes measure nothing.** The obvious sdpa size (8 heads x 1024) is ~0.4ms
and pure dispatch noise: three consecutive runs on one machine gave ratios of 0.79, 3.95 and
2.35 — the first would have "proven" head_dim=56 is *faster*, contradicting the real 8-machine
result. At 16 x 2048 it lands ~1-4ms and reproduces to 0.6%.

**Busy nodes are excluded, not blamed.** GPU contention is invisible to load average — the M3
Ultra sat at **load 2.45 with its GPU fully pinned** by trellis_mac. So `bench.py` samples GPU
utilization from `ioreg` *before it touches the GPU* (our own matmul pins the device at ~100%,
so sampling during the run cannot separate us from a neighbour — the ordering is the whole
trick). Above 50% the node is marked `[~] BUSY`, and its numbers raise no alert and never enter
a baseline. Two quirks: the first `ioreg` read is garbage (it is since-last-read, and an idle
m4mini reported 97% cold), and a GPU job alternates phases (100% CPU / 0% GPU one moment, 97%
GPU the next) — hence discard-first and max-of-several.

**Baselines are the median of recent history, not a saved "best".** These machines have
neighbours: the M4 Pro also serves Ollama and its fp32 is bimodal (~3200 vs ~5500 depending on
whether a request is in flight); the M1 Ultra runs `mrpgi` on its GPU and swings 8301–15891
(1.9x). A best-observed baseline pins to a lucky outlier the node rarely reaches again — which
means alerting every night forever. A median is what the machine actually does.

**A regression must repeat before it's believed.** Transient contention is far more common than
real regression and doesn't recur on schedule. First sighting is `[~] watching`; only a second
consecutive one is `[!] CONFIRMED` and fails the run. Getting *faster* is reported immediately
— it's never an alarm.

## Fleet gotcha found while building this

**The m4mini is the only node with Tailscale SSH enabled (`RunSSH: true`), and it does not
propagate remote exit codes.** `ssh m4mini "exit 7"` returns **0**; so does `ssh m4mini
"false"`. Every other node propagates correctly. This means **any `if ssh m4mini ...; then`
test silently always passes** — a reach-check written that way will report every node healthy
no matter what. Test on *output* instead (`x=$(ssh … && echo YES)`), which is what godcheck
already does throughout, and why godcheck is unaffected.

It also means the watchdog cannot ssh to *itself* ("Host key verification failed"), so
`run_fleet.py` detects its own tailnet IP and benches the local node through the shell instead.

Worth deciding: `tailscale set --ssh=false` on the m4mini would make it behave like the rest of
the fleet, but it's a live config change on the always-on watchdog and hasn't been made.