BENCHMARKS.md says what a model cost the day it was measured; nothing noticed if a
macOS/torch update or a thermal fault halved a node. perfcheck runs the whole fleet
in ~35s off godcheck's 03:30 cron on the m4mini and reports drift into
GODCHECK_LATEST.md.
Probes matmul (fp16/fp32), memory bandwidth, and SDPA at head_dim 64 vs 56 — the
latter turning CorridorKey's fast-path cliff into a permanent canary: it confirms
the padding win fleet-wide (1.96x-5.26x) and tells us if a future torch closes it.
Runs on venvs/rmbg/bin/python, already identical fleet-wide, so nothing new is
installed (nothing lands on the disk-tight m1max).
First cross-machine capability table for all 6 nodes. The M3 Ultra is ~2.2x the M1
Ultra on fp16 matmul, but they share ~625 GB/s — so bandwidth-bound stages run alike
while compute-bound ones scale. Both Ultras reach only ~78% of spec bandwidth on a
single kernel; the smaller Macs hit ~88%.
Measuring a fleet that is doing real work is the whole problem, and naive
benchmarking here is off by 11x:
- min, not median: a concurrent trellis_mac job dragged a median-of-5 matmul from
~24500 to ~2150 GFLOP/s, which reads exactly like a catastrophic regression.
- n=4096 not 2048: 2048 is dispatch-bound and swung 48% run-to-run; 4096 reproduces
to 0.1% even while contended.
- sdpa 16x2048 not 8x1024: sub-ms probes are dispatch noise — 8x1024 gave ratios of
0.79/3.95/2.35 on three runs of one machine, the first "proving" 56 is faster.
- busy nodes are excluded, not blamed: GPU contention is invisible to load average
(M3 Ultra read load 2.45 with its GPU pinned), so bench.py samples ioreg GPU% before
it touches the GPU — our own matmul pins the device, so ordering is the trick.
- baselines are the median of recent history, not a saved best: the M4 Pro also serves
Ollama and is bimodal (~3200 vs ~5500 fp32), so a best-observed baseline pins to a
lucky outlier and alerts forever.
- a regression must repeat before it is believed ([~] watching -> [!] CONFIRMED).
Validated by re-running the fleet against its own baselines: zero false alarms,
including a sweep where the M3 Ultra read 43 GB/s under load and was correctly
marked BUSY rather than reported as a 93% regression.
Also found: the m4mini is the only node with Tailscale SSH (RunSSH: true) and it does
NOT propagate remote exit codes — `ssh m4mini "exit 7"` returns 0, so any
`if ssh m4mini ...` test silently always passes. Test on output instead, which is
what godcheck already does (and why it is unaffected). It also cannot ssh to itself,
so run_fleet detects its own tailnet IP and benches the local node via the shell.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Tested single image -> I2V orbit -> colmap -> brush. COLMAP registered 26/49
frames but triangulated ZERO 3D points (vs 1006 for real iPad footage);
gauge fix failed, 'reconstruction' in 2.4s. Brush still emitted a 190MB .ply
from geometry-free input = garbage.
Cause: the subject isn't rigid (tail dissolves, orb appears, figure morphs).
Video models optimise temporal plausibility, not multi-view consistency.
You cannot synthesize your way out of capture. Documented so it isn't retried.
Also: TI2V-5B 704x704/49f/40steps = 337s.
runner: gpu AND cpu are now a node pool. Per-node cpu_slots (primary 3,
helpers 2, nodes.json-overridable); net stays primary-only. Python-less ops
(ffmpeg/ffprobe) now run remotely on the node's system python3. Verified: 9
concurrent ffmpeg_frames distributed 4 local / 2 m1 / 2 m4.
hunyuan3d_mlx: default to Studio-quality (octree 384, texture 4096, remesh
120k) — 4096 bake verified watchdog-free on M3 Ultra; big quality gain
(defined face, 120k faces). remesh_faces now a param. All param-overridable.
trellis_mac verified end-to-end: 18.4MB PBR GLB, clean coherent geometry on a
thin-ringed astrolabe (far beyond SF3D draft quality). ~15GB one-time download
then ~5min cached generation. All three local image->3D paths now live: sf3d
(fast draft), trellis_mac (SOTA local), plus fal cloud tier.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- run.py: force HF_HUB_DISABLE_XET=1 (xet chunked downloader fails on BFL repos
with 'Unable to parse string as hex hash value'; HTTP path is reliable)
- BENCHMARKS.md: warm generation times for all 5 local FLUX models on M3 Ultra
(Klein 4B 9.1s, schnell-4bit 18.5s, Klein 9B 18.7s, schnell 20.4s, dev 108s).
Verdict: Klein 9B = best hero-asset quality (beats schnell at same speed),
Klein 4B = volume workhorse.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
