BENCHMARKS.md says what a model cost the day it was measured; nothing noticed if a
macOS/torch update or a thermal fault halved a node. perfcheck runs the whole fleet
in ~35s off godcheck's 03:30 cron on the m4mini and reports drift into
GODCHECK_LATEST.md.
Probes matmul (fp16/fp32), memory bandwidth, and SDPA at head_dim 64 vs 56 — the
latter turning CorridorKey's fast-path cliff into a permanent canary: it confirms
the padding win fleet-wide (1.96x-5.26x) and tells us if a future torch closes it.
Runs on venvs/rmbg/bin/python, already identical fleet-wide, so nothing new is
installed (nothing lands on the disk-tight m1max).
First cross-machine capability table for all 6 nodes. The M3 Ultra is ~2.2x the M1
Ultra on fp16 matmul, but they share ~625 GB/s — so bandwidth-bound stages run alike
while compute-bound ones scale. Both Ultras reach only ~78% of spec bandwidth on a
single kernel; the smaller Macs hit ~88%.
Measuring a fleet that is doing real work is the whole problem, and naive
benchmarking here is off by 11x:
- min, not median: a concurrent trellis_mac job dragged a median-of-5 matmul from
~24500 to ~2150 GFLOP/s, which reads exactly like a catastrophic regression.
- n=4096 not 2048: 2048 is dispatch-bound and swung 48% run-to-run; 4096 reproduces
to 0.1% even while contended.
- sdpa 16x2048 not 8x1024: sub-ms probes are dispatch noise — 8x1024 gave ratios of
0.79/3.95/2.35 on three runs of one machine, the first "proving" 56 is faster.
- busy nodes are excluded, not blamed: GPU contention is invisible to load average
(M3 Ultra read load 2.45 with its GPU pinned), so bench.py samples ioreg GPU% before
it touches the GPU — our own matmul pins the device, so ordering is the trick.
- baselines are the median of recent history, not a saved best: the M4 Pro also serves
Ollama and is bimodal (~3200 vs ~5500 fp32), so a best-observed baseline pins to a
lucky outlier and alerts forever.
- a regression must repeat before it is believed ([~] watching -> [!] CONFIRMED).
Validated by re-running the fleet against its own baselines: zero false alarms,
including a sweep where the M3 Ultra read 43 GB/s under load and was correctly
marked BUSY rather than reported as a 93% regression.
Also found: the m4mini is the only node with Tailscale SSH (RunSSH: true) and it does
NOT propagate remote exit codes — `ssh m4mini "exit 7"` returns 0, so any
`if ssh m4mini ...` test silently always passes. Test on output instead, which is
what godcheck already does (and why it is unaffected). It also cannot ssh to itself,
so run_fleet detects its own tailnet IP and benches the local node via the shell.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Tested single image -> I2V orbit -> colmap -> brush. COLMAP registered 26/49
frames but triangulated ZERO 3D points (vs 1006 for real iPad footage);
gauge fix failed, 'reconstruction' in 2.4s. Brush still emitted a 190MB .ply
from geometry-free input = garbage.
Cause: the subject isn't rigid (tail dissolves, orb appears, figure morphs).
Video models optimise temporal plausibility, not multi-view consistency.
You cannot synthesize your way out of capture. Documented so it isn't retried.
Also: TI2V-5B 704x704/49f/40steps = 337s.
Checkpoints are no longer uniform across the fleet (M3 keeps only the
LoRA-compatible SD1.5 model; M1 holds the full SDXL archive). The gpu pool
routes by operator, not by checkpoint, so a job asking for bigLust could land
on a node without it. Query the node's own ComfyUI /object_info and exit with
the available list + where the rest lives, instead of a cryptic 400.
Fills the gap flux_local can't (mflux is FLUX-only). Pure-stdlib run.py talks
to a resident ComfyUI on :8188 (auto-starts, keeps checkpoints cached — 4s
warm per 5122/20-step image on M3). Warns on the SD1.5-LoRA-on-SDXL trap.
No manifest python => uses the node's system python3 via the python-less
remote path, so it distributes across the gpu pool.
runner: gpu AND cpu are now a node pool. Per-node cpu_slots (primary 3,
helpers 2, nodes.json-overridable); net stays primary-only. Python-less ops
(ffmpeg/ffprobe) now run remotely on the node's system python3. Verified: 9
concurrent ffmpeg_frames distributed 4 local / 2 m1 / 2 m4.
hunyuan3d_mlx: default to Studio-quality (octree 384, texture 4096, remesh
120k) — 4096 bake verified watchdog-free on M3 Ultra; big quality gain
(defined face, 120k faces). remesh_faces now a param. All param-overridable.
scripts/agent_token.py mints the 'agents' guest user + token non-interactively,
writes data/agent.env (0600, gitignored, never printed). Fixes _register_outputs
dropping user_id — guest job outputs were registered ownerless, so guests could
never see or download their own results.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The gpu lane is now a NODE POOL (this Mac + remote workers from nodes.json) instead
of a single-Metal semaphore. A gpu job runs on whichever node is free:
- server/remote.py: ssh+rsync dispatch — mkdir remote dirs, rsync inputs out, run
the operator's run.py over ssh with repo-relative paths (HF_TOKEN sourced from the
node's .env.remote, off the process table), rsync outputs back, clean up. Cached
health checks; per-node operator allowlist.
- runner: _acquire_gpu_node picks the first free node that supports the op (local
runs anything; a remote must list it + be reachable → auto-fallback to local).
_run_job branches local/remote; the M1 never touches the M3's DB.
- sysinfo/Dashboard: gpu lane limit = pool size; per-node status cards.
- nodes.json (gitignored, primary-only): M1 worker, allowlist excludes trellis_mac
(bf16 unverified on M1) + brush_train (not installed there).
Verified: two FLUX jobs from one queue split M3(local, 10.3s) + M1(remote via ssh,
21.2s), both images rsync'd back and registered on the M3.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The gpu lane is now a pool of nodes -- this Mac plus any remote workers in
nodes.json (gitignored, primary-only). A gpu job acquires the first free node
that supports its operator; local runs as before, a remote node has its inputs
rsynced out, runs the operator's run.py over ssh (keys sourced from the node's
.env.remote, kept off the process table), then its outputs are rsynced back and
registered locally exactly as for a local job. One job per node (one Metal
device each).
- server/remote.py (new): node loading, cached health checks, prepare/cmd/collect
- runner: _acquire_gpu_node + a remote branch in _run_job. lane_of now treats
'gpu' as a real lane -- it was dropped from LANE_LIMITS (which holds only the
semaphore lanes cpu/net), so gpu jobs were silently falling back to the cpu
lane and the pool was never reached. A node is reserved before the health
await so two concurrent jobs can't grab the same one.
- sysinfo: gpu lane capacity = pool size, plus per-node status for the dashboard
Verified: all 7 gpu operators route to the pool; remote_cmd/node_supports and
nodes.json parsing correct; acquisition is race-free; a scratch-DB server boots
clean and logs the pool.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Manifest "python" can now be a repo-relative path (venvs/mflux/bin/python) that
resolves against db.ROOT, so the same operator code runs on any machine/checkout
(M3 Ultra, M1 Ultra worker). All 6 venv operators switched to relative paths.
Prep for the M1 Ultra second node.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fixed confirmed findings before public exposure:
HIGH:
- upload filename path traversal → store.safe_name() strips to basename
- login rate-limit XFF bypass → key on request.client.host + per-username bucket;
auth.check_login() burns bcrypt time on unknown users (no enumeration)
- cross-user read access → per-user isolation: guests see/use/download/delete only
their own assets & jobs (owner sees all); WS job events scoped per-user
MEDIUM:
- unbounded upload read → bounded chunked streaming to the 1GB cap
- asset member path check → Path.is_relative_to boundary + ownership gate
- WS token-in-query-string leak → session-cookie-only WS auth
LOW:
- retry_job bypassed the per-user job cap → cap now checked on retry
- wholesale API-key injection → env_for_operator injects a paid key only to
operators that declare it (guest local jobs never receive fal/OpenRouter keys)
- session revocation → users.session_epoch, bumped on password change
- int() 500s → 400; net-lane defense-in-depth (guests blocked by requires_env AND
resources==net, so a mis-tagged paid op is still blocked)
+ public /api/health for serve.sh & proxy; docs/VPS.md; mb MB_TOKEN bearer auth
tests/smoke.sh: 34 checks passing incl. all new hardening.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
T490s already runs Win11, so keep it: WSL2 gives clean Linux CUDA env with eGPU
passthrough where the Linux-first 3D repos build properly. Bare-metal Ubuntu kept
as the leaner Path B. Covers mirrored WSL networking, headless/lid + Windows Update
settings for an always-on worker.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ready-to-follow checklist for standing up a headless Linux + eGPU CUDA node to
offload the CUDA-only 3D/mocap jobs the Mac can't run (splat->mesh, GVHMR mocap).
Bare-metal Linux required (no VM GPU passthrough on Mac hosts). Planned remote-CUDA
operators: sugar_mesh, gvhmr_mocap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
trellis_mac verified end-to-end: 18.4MB PBR GLB, clean coherent geometry on a
thin-ringed astrolabe (far beyond SF3D draft quality). ~15GB one-time download
then ~5min cached generation. All three local image->3D paths now live: sf3d
(fast draft), trellis_mac (SOTA local), plus fal cloud tier.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dinov3-vitl16-pretrain-lvd1689m access granted (the natural-image variant TRELLIS.2
needs — note the approval email named the sat493m satellite variant, but lvd1689m
came through too). Added HF_HUB_DISABLE_XET=1 (same xet download bug as flux) +
KMP_DUPLICATE_LIB_OK/OMP_NUM_THREADS=1 guards (torch + Metal-kernel omp). First
local TRELLIS.2 generation now running.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two new local image-prep operators through the already-installed mflux 0.18 (no
new heavy install, ungated weights):
- seedvr2_upscale: mflux-upscale-seedvr2 (ByteDance SeedVR2, MIT) — faithful
local upscale, free alternative to cloud fal_upscale
- mflux_image_edit: mflux-generate-qwen-edit (Qwen-Image-Edit, Apache-2.0) —
local prompt-based image editing, free alternative to fal_image_edit
Research verdict on dinov3 (recorded): waiting for Meta approval is the only
legitimate path — TRELLIS.2's DiT is trained on DINOv3 ViT-L/16 features so the
encoder can't be swapped, and the only ungated mirrors are unauthorized
re-uploads. trellis_mac stays blocked on the pending gate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New operator: briaai/RMBG-2.0 image->transparent cutout, fully local on Apple
Silicon (weights unlocked by owner HF license). Free local alternative to the
cloud fal_bg_remove; the recommended pre-pass before SF3D / image->3D. Verified:
clean 1024px cutout of the FLUX astrolabe (alpha 0-255, 86.6% removed, thin rings
preserved). Same OpenMP guardrails as sf3d. scripts/install_rmbg.sh; venvs/rmbg.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
torch and the compiled texture_baker/uv_unwrapper each link a libomp. Allowing
the duplicate runtime clears 'OMP: Error #15' but the duplicate then corrupts the
multithreaded CPU LAPACK path (torch.linalg.svd MPS fallback) -> segfault with no
Python traceback. Forcing OMP_NUM_THREADS=1 + MKL_NUM_THREADS=1 removes the
threading collision. Verified: clean 1.3MB GLB (13298 verts) in ~5s on M3 Ultra,
9GB peak. SF3D local image->3D now works (weights unlocked by owner HF license).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- run.py: force HF_HUB_DISABLE_XET=1 (xet chunked downloader fails on BFL repos
with 'Unable to parse string as hex hash value'; HTTP path is reliable)
- BENCHMARKS.md: warm generation times for all 5 local FLUX models on M3 Ultra
(Klein 4B 9.1s, schnell-4bit 18.5s, Klein 9B 18.7s, schnell 20.4s, dev 108s).
Verdict: Klein 9B = best hero-asset quality (beats schnell at same speed),
Klein 4B = volume workhorse.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- mb: zero-dependency python CLI for the full API (ops/upload/run/wait/follow/
download incl. folder assets, retry/cancel, settings). Schema-aware -p k=v
coercion. Works from any tailnet machine via MB_HOST. Tested end-to-end
(upload -> blender_convert -> download).
- AGENTS.md: complete handover brief for other agents using this box as an
asset factory — endpoints, CLI reference, catalog, recipes, lanes/etiquette.
- scripts/serve.sh (headless start/restart) + scripts/install_launchagent.sh
(optional boot persistence, owner-run)
- flux_local upgraded to mflux 0.18 reality: flux2-klein-4b default (Apache,
UNGATED — runs with zero keys), klein-9b, schnell/schnell-4bit community
quant, dev/krea-dev. Research verdict: FLUX.1-dev no longer competitive
(Elo ~1027) vs klein ~1083-1119 vs nano-banana ~1154.
- openrouter_image rewritten to the dedicated Image API (POST /api/v1/images):
b64_json parsing, seed, resolution/aspect, exact usage.cost logging; model
enum: gemini-2.5-flash-image / 3.1-flash-image (NB2) / 3-pro-image
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- flux_local: prompt->image fully on-device via mflux (schnell/dev, steps,
quantize, seed, size). Both FLUX repos are HF-gated as of 2026-07 (schnell
included) — clean GatedRepoError surfaces with a hint; needs owner HF token.
- openrouter_image: prompt->image via OpenRouter chat/completions with
modalities [image,text]; parses data-URL images from the response; model
picker for nano-banana / nano-banana-pro. Gated on OPENROUTER_API_KEY.
- settings: openrouter_key added to vault (masked/redacted/env-injected)
- scripts/install_mflux.sh; venv installed
- A/B flow: run both with the same prompt, judge in Compare mode
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Operators (16 total):
- fal_hunyuan3d_v21: Hunyuan3D 2.1 single-image (live-verified dash variant,
~90s; v21 multi-view is broken on fal — v2 stays the multi-view path)
- fal_bg_remove (BiRefNet v2): subject cutout pre-pass — the biggest quality
lever before image->3D
- fal_upscale (SeedVR faithful upscale), fal_image_edit (nano-banana prompt
edits), fal_text_image (Ideogram v3 — first no-input operator)
- fal_common: collect='images' mode; recursive URL extractor now takes the
wanted extension set
UI: operator dropdown grouped by category (optgroup); operators with
accepts: [] run without a selected asset ('No input asset needed' note);
run-button/launch logic updated accordingly.
Review fixes (Opus Phase 1 commit):
- runner._run_lane crashed (TypeError) when a queued job was deleted before
the worker picked it up
- PUT /api/settings treated empty string as a masked placeholder, making
secrets impossible to clear from the UI
- store.register_file mutated the caller's meta dict via pop
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
