27564e80eb
BENCHMARKS.md says what a model cost the day it was measured; nothing noticed if a macOS/torch update or a thermal fault halved a node. perfcheck runs the whole fleet in ~35s off godcheck's 03:30 cron on the m4mini and reports drift into GODCHECK_LATEST.md. Probes matmul (fp16/fp32), memory bandwidth, and SDPA at head_dim 64 vs 56 — the latter turning CorridorKey's fast-path cliff into a permanent canary: it confirms the padding win fleet-wide (1.96x-5.26x) and tells us if a future torch closes it. Runs on venvs/rmbg/bin/python, already identical fleet-wide, so nothing new is installed (nothing lands on the disk-tight m1max). First cross-machine capability table for all 6 nodes. The M3 Ultra is ~2.2x the M1 Ultra on fp16 matmul, but they share ~625 GB/s — so bandwidth-bound stages run alike while compute-bound ones scale. Both Ultras reach only ~78% of spec bandwidth on a single kernel; the smaller Macs hit ~88%. Measuring a fleet that is doing real work is the whole problem, and naive benchmarking here is off by 11x: - min, not median: a concurrent trellis_mac job dragged a median-of-5 matmul from ~24500 to ~2150 GFLOP/s, which reads exactly like a catastrophic regression. - n=4096 not 2048: 2048 is dispatch-bound and swung 48% run-to-run; 4096 reproduces to 0.1% even while contended. - sdpa 16x2048 not 8x1024: sub-ms probes are dispatch noise — 8x1024 gave ratios of 0.79/3.95/2.35 on three runs of one machine, the first "proving" 56 is faster. - busy nodes are excluded, not blamed: GPU contention is invisible to load average (M3 Ultra read load 2.45 with its GPU pinned), so bench.py samples ioreg GPU% before it touches the GPU — our own matmul pins the device, so ordering is the trick. - baselines are the median of recent history, not a saved best: the M4 Pro also serves Ollama and is bimodal (~3200 vs ~5500 fp32), so a best-observed baseline pins to a lucky outlier and alerts forever. - a regression must repeat before it is believed ([~] watching -> [!] CONFIRMED). Validated by re-running the fleet against its own baselines: zero false alarms, including a sweep where the M3 Ultra read 43 GB/s under load and was correctly marked BUSY rather than reported as a 93% regression. Also found: the m4mini is the only node with Tailscale SSH (RunSSH: true) and it does NOT propagate remote exit codes — `ssh m4mini "exit 7"` returns 0, so any `if ssh m4mini ...` test silently always passes. Test on output instead, which is what godcheck already does (and why it is unaffected). It also cannot ssh to itself, so run_fleet detects its own tailnet IP and benches the local node via the shell. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
39 lines
1.9 KiB
Bash
Executable File
39 lines
1.9 KiB
Bash
Executable File
#!/bin/zsh
|
|
# Push perfcheck to the m4mini watchdog, where the nightly godcheck cron runs it.
|
|
#
|
|
# Idempotent — re-run it whenever bench.py or run_fleet.py changes, otherwise the fleet keeps
|
|
# being measured by the old benchmark. Accumulated state (history/baselines/pending) is NEVER
|
|
# overwritten: on the first deploy the local history is seeded across so the baselines carry
|
|
# over, and after that the m4mini's copy is authoritative.
|
|
#
|
|
# Lives beside godcheck (~/godcheck/perfcheck) rather than in ~/MODELBEAST, to match how
|
|
# godcheck itself is a standalone script and to keep the cron independent of the repo state.
|
|
set -u
|
|
W=m4mini@100.124.220.31
|
|
DEST=godcheck/perfcheck
|
|
HERE=${0:A:h}
|
|
|
|
echo "==> deploying perfcheck to $W:~/$DEST"
|
|
ssh -o ConnectTimeout=10 -o BatchMode=yes "$W" "mkdir -p ~/$DEST"
|
|
scp -q "$HERE/bench.py" "$HERE/run_fleet.py" "$W:$DEST/"
|
|
ssh -o ConnectTimeout=10 -o BatchMode=yes "$W" "chmod +x ~/$DEST/run_fleet.py"
|
|
|
|
# Seed history only if the watchdog has none, so a redeploy never clobbers real history.
|
|
#
|
|
# Tested on OUTPUT, never on ssh's exit status: the m4mini is the one node in the fleet with
|
|
# Tailscale SSH enabled (`RunSSH: true`), and tailscaled handles the session itself rather than
|
|
# sshd — it does not propagate the remote exit code. `ssh m4mini "exit 7"` returns 0, so every
|
|
# `if ssh m4mini test ...` silently succeeds. Every other node propagates correctly. (godcheck
|
|
# is output-based throughout and is unaffected.)
|
|
have=$(ssh -o ConnectTimeout=10 -o BatchMode=yes "$W" "test -s ~/$DEST/history.jsonl && echo YES")
|
|
if [ "$have" = "YES" ]; then
|
|
echo "==> watchdog already has history — left untouched"
|
|
elif [ -s "$HERE/history.jsonl" ]; then
|
|
echo "==> seeding history.jsonl (first deploy)"
|
|
scp -q "$HERE/history.jsonl" "$W:$DEST/"
|
|
fi
|
|
|
|
echo "==> smoke test (one node)"
|
|
ssh -o ConnectTimeout=10 -o BatchMode=yes "$W" "cd ~/$DEST && /usr/bin/python3 run_fleet.py --only m4mini --quiet"
|
|
echo "==> done"
|