BENCHMARKS.md says what a model cost the day it was measured; nothing noticed if a
macOS/torch update or a thermal fault halved a node. perfcheck runs the whole fleet
in ~35s off godcheck's 03:30 cron on the m4mini and reports drift into
GODCHECK_LATEST.md.
Probes matmul (fp16/fp32), memory bandwidth, and SDPA at head_dim 64 vs 56 — the
latter turning CorridorKey's fast-path cliff into a permanent canary: it confirms
the padding win fleet-wide (1.96x-5.26x) and tells us if a future torch closes it.
Runs on venvs/rmbg/bin/python, already identical fleet-wide, so nothing new is
installed (nothing lands on the disk-tight m1max).
First cross-machine capability table for all 6 nodes. The M3 Ultra is ~2.2x the M1
Ultra on fp16 matmul, but they share ~625 GB/s — so bandwidth-bound stages run alike
while compute-bound ones scale. Both Ultras reach only ~78% of spec bandwidth on a
single kernel; the smaller Macs hit ~88%.
Measuring a fleet that is doing real work is the whole problem, and naive
benchmarking here is off by 11x:
- min, not median: a concurrent trellis_mac job dragged a median-of-5 matmul from
~24500 to ~2150 GFLOP/s, which reads exactly like a catastrophic regression.
- n=4096 not 2048: 2048 is dispatch-bound and swung 48% run-to-run; 4096 reproduces
to 0.1% even while contended.
- sdpa 16x2048 not 8x1024: sub-ms probes are dispatch noise — 8x1024 gave ratios of
0.79/3.95/2.35 on three runs of one machine, the first "proving" 56 is faster.
- busy nodes are excluded, not blamed: GPU contention is invisible to load average
(M3 Ultra read load 2.45 with its GPU pinned), so bench.py samples ioreg GPU% before
it touches the GPU — our own matmul pins the device, so ordering is the trick.
- baselines are the median of recent history, not a saved best: the M4 Pro also serves
Ollama and is bimodal (~3200 vs ~5500 fp32), so a best-observed baseline pins to a
lucky outlier and alerts forever.
- a regression must repeat before it is believed ([~] watching -> [!] CONFIRMED).
Validated by re-running the fleet against its own baselines: zero false alarms,
including a sweep where the M3 Ultra read 43 GB/s under load and was correctly
marked BUSY rather than reported as a 93% regression.
Also found: the m4mini is the only node with Tailscale SSH (RunSSH: true) and it does
NOT propagate remote exit codes — `ssh m4mini "exit 7"` returns 0, so any
`if ssh m4mini ...` test silently always passes. Test on output instead, which is
what godcheck already does (and why it is unaffected). It also cannot ssh to itself,
so run_fleet detects its own tailnet IP and benches the local node via the shell.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
scripts/agent_token.py mints the 'agents' guest user + token non-interactively,
writes data/agent.env (0600, gitignored, never printed). Fixes _register_outputs
dropping user_id — guest job outputs were registered ownerless, so guests could
never see or download their own results.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixed confirmed findings before public exposure:
HIGH:
- upload filename path traversal → store.safe_name() strips to basename
- login rate-limit XFF bypass → key on request.client.host + per-username bucket;
auth.check_login() burns bcrypt time on unknown users (no enumeration)
- cross-user read access → per-user isolation: guests see/use/download/delete only
their own assets & jobs (owner sees all); WS job events scoped per-user
MEDIUM:
- unbounded upload read → bounded chunked streaming to the 1GB cap
- asset member path check → Path.is_relative_to boundary + ownership gate
- WS token-in-query-string leak → session-cookie-only WS auth
LOW:
- retry_job bypassed the per-user job cap → cap now checked on retry
- wholesale API-key injection → env_for_operator injects a paid key only to
operators that declare it (guest local jobs never receive fal/OpenRouter keys)
- session revocation → users.session_epoch, bumped on password change
- int() 500s → 400; net-lane defense-in-depth (guests blocked by requires_env AND
resources==net, so a mis-tagged paid op is still blocked)
+ public /api/health for serve.sh & proxy; docs/VPS.md; mb MB_TOKEN bearer auth
tests/smoke.sh: 34 checks passing incl. all new hardening.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New operator: briaai/RMBG-2.0 image->transparent cutout, fully local on Apple
Silicon (weights unlocked by owner HF license). Free local alternative to the
cloud fal_bg_remove; the recommended pre-pass before SF3D / image->3D. Verified:
clean 1024px cutout of the FLUX astrolabe (alpha 0-255, 86.6% removed, thin rings
preserved). Same OpenMP guardrails as sf3d. scripts/install_rmbg.sh; venvs/rmbg.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- mb: zero-dependency python CLI for the full API (ops/upload/run/wait/follow/
download incl. folder assets, retry/cancel, settings). Schema-aware -p k=v
coercion. Works from any tailnet machine via MB_HOST. Tested end-to-end
(upload -> blender_convert -> download).
- AGENTS.md: complete handover brief for other agents using this box as an
asset factory — endpoints, CLI reference, catalog, recipes, lanes/etiquette.
- scripts/serve.sh (headless start/restart) + scripts/install_launchagent.sh
(optional boot persistence, owner-run)
- flux_local upgraded to mflux 0.18 reality: flux2-klein-4b default (Apache,
UNGATED — runs with zero keys), klein-9b, schnell/schnell-4bit community
quant, dev/krea-dev. Research verdict: FLUX.1-dev no longer competitive
(Elo ~1027) vs klein ~1083-1119 vs nano-banana ~1154.
- openrouter_image rewritten to the dedicated Image API (POST /api/v1/images):
b64_json parsing, seed, resolution/aspect, exact usage.cost logging; model
enum: gemini-2.5-flash-image / 3.1-flash-image (NB2) / 3-pro-image
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- flux_local: prompt->image fully on-device via mflux (schnell/dev, steps,
quantize, seed, size). Both FLUX repos are HF-gated as of 2026-07 (schnell
included) — clean GatedRepoError surfaces with a hint; needs owner HF token.
- openrouter_image: prompt->image via OpenRouter chat/completions with
modalities [image,text]; parses data-URL images from the response; model
picker for nano-banana / nano-banana-pro. Gated on OPENROUTER_API_KEY.
- settings: openrouter_key added to vault (masked/redacted/env-injected)
- scripts/install_mflux.sh; venv installed
- A/B flow: run both with the same prompt, judge in Compare mode
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
