Movie SBS converges on the screen plane, so real content sits at NEGATIVE
disparity; minDisparity=0 clipped everything behind convergence to black.
Window is now ±num_disp/2 with 2-98 percentile normalization and a validity
mask. Verified on Big Buck Bunny (half-OU) + One Night In Hell (half-SBS):
structured depth + colored clouds that render in Blender.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The README still read as a fal.ai front-end with a local afterthought: the headline
`mb` example ran fal_trellis2, the "recommended chain for best image→3D" was four
fal_* calls, and the operator table listed 5 local ops against 9 cloud ones. That is
backwards now — 22 of the 31 operators run entirely on the fleet.
It also predated most of what we have: the whole audio/video/text suite (tts_local,
stt_local, voice_clone_local, music_local, lipsync_local, wan_video, llm_local,
motion_local), comfyui_sd, flux_local, mflux_image_edit, seedvr2_upscale and
bg_remove_local were all missing from the catalog entirely.
- lead with local; cloud demoted to its own opt-in "costs real money" section
- all 31 operators documented, grouped by what you're trying to make
- recommended chain is now local: bg_remove_local -> [seedvr2_upscale] -> trellis_mac
(quality) or hunyuan3d_mlx (faster, ~3x lighter, the only one that runs on M1)
- setup section reordered: most local ops need nothing, hunyuan3d_mlx needs no HF
login at all, only sf3d/trellis_mac/bg_remove_local are gated
- lanes corrected: gpu+cpu are a NODE POOL (gpu 1/node, cpu per-node cpu_slots),
not the old flat "gpu=1, cpu=3, net=6"
- manifest `python` documented as repo-relative (remote nodes can't resolve absolute)
- link PERFCHECK.md; drop the stale "on the M3 Ultra" framing — it's a fleet
- verified: every operator on disk appears, no phantom ids, no broken doc links
Also: .gitignore had `*.env` and `.env`, neither of which matches a suffixed name
like `.env.remote` — which is sitting un-ignored in the m1's checkout right now, one
`git add -A` away from being committed. Ignore the `.env.*` family (keeping
.env.example committable). Nothing has leaked: no .env file is tracked.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
These three were onboarded on 2026-07-17 and are live in nodes.json, but the docs
still described a three-machine fleet (M3 Ultra / M1 Ultra / M4 Pro). Anyone reading
CLUSTER.md to decide where a job should land was working from a stale map.
Records the M1 Max as a second full gen node (flux_local, hunyuan3d_mlx; NOT trellis
- M1 bf16), and the M4 mini / M1 mini as light bg_remove + cpu helpers. Also notes
the M1 Max login split: admin is `m1max`, but the checkout and farm run as `studio`,
which needs /opt/homebrew/bin on its login PATH.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
BENCHMARKS.md says what a model cost the day it was measured; nothing noticed if a
macOS/torch update or a thermal fault halved a node. perfcheck runs the whole fleet
in ~35s off godcheck's 03:30 cron on the m4mini and reports drift into
GODCHECK_LATEST.md.
Probes matmul (fp16/fp32), memory bandwidth, and SDPA at head_dim 64 vs 56 — the
latter turning CorridorKey's fast-path cliff into a permanent canary: it confirms
the padding win fleet-wide (1.96x-5.26x) and tells us if a future torch closes it.
Runs on venvs/rmbg/bin/python, already identical fleet-wide, so nothing new is
installed (nothing lands on the disk-tight m1max).
First cross-machine capability table for all 6 nodes. The M3 Ultra is ~2.2x the M1
Ultra on fp16 matmul, but they share ~625 GB/s — so bandwidth-bound stages run alike
while compute-bound ones scale. Both Ultras reach only ~78% of spec bandwidth on a
single kernel; the smaller Macs hit ~88%.
Measuring a fleet that is doing real work is the whole problem, and naive
benchmarking here is off by 11x:
- min, not median: a concurrent trellis_mac job dragged a median-of-5 matmul from
~24500 to ~2150 GFLOP/s, which reads exactly like a catastrophic regression.
- n=4096 not 2048: 2048 is dispatch-bound and swung 48% run-to-run; 4096 reproduces
to 0.1% even while contended.
- sdpa 16x2048 not 8x1024: sub-ms probes are dispatch noise — 8x1024 gave ratios of
0.79/3.95/2.35 on three runs of one machine, the first "proving" 56 is faster.
- busy nodes are excluded, not blamed: GPU contention is invisible to load average
(M3 Ultra read load 2.45 with its GPU pinned), so bench.py samples ioreg GPU% before
it touches the GPU — our own matmul pins the device, so ordering is the trick.
- baselines are the median of recent history, not a saved best: the M4 Pro also serves
Ollama and is bimodal (~3200 vs ~5500 fp32), so a best-observed baseline pins to a
lucky outlier and alerts forever.
- a regression must repeat before it is believed ([~] watching -> [!] CONFIRMED).
Validated by re-running the fleet against its own baselines: zero false alarms,
including a sweep where the M3 Ultra read 43 GB/s under load and was correctly
marked BUSY rather than reported as a 93% regression.
Also found: the m4mini is the only node with Tailscale SSH (RunSSH: true) and it does
NOT propagate remote exit codes — `ssh m4mini "exit 7"` returns 0, so any
`if ssh m4mini ...` test silently always passes. Test on output instead, which is
what godcheck already does (and why it is unaffected). It also cannot ssh to itself,
so run_fleet detects its own tailnet IP and benches the local node via the shell.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Tested single image -> I2V orbit -> colmap -> brush. COLMAP registered 26/49
frames but triangulated ZERO 3D points (vs 1006 for real iPad footage);
gauge fix failed, 'reconstruction' in 2.4s. Brush still emitted a 190MB .ply
from geometry-free input = garbage.
Cause: the subject isn't rigid (tail dissolves, orb appears, figure morphs).
Video models optimise temporal plausibility, not multi-view consistency.
You cannot synthesize your way out of capture. Documented so it isn't retried.
Also: TI2V-5B 704x704/49f/40steps = 337s.
Checkpoints are no longer uniform across the fleet (M3 keeps only the
LoRA-compatible SD1.5 model; M1 holds the full SDXL archive). The gpu pool
routes by operator, not by checkpoint, so a job asking for bigLust could land
on a node without it. Query the node's own ComfyUI /object_info and exit with
the available list + where the rest lives, instead of a cryptic 400.
Fills the gap flux_local can't (mflux is FLUX-only). Pure-stdlib run.py talks
to a resident ComfyUI on :8188 (auto-starts, keeps checkpoints cached — 4s
warm per 5122/20-step image on M3). Warns on the SD1.5-LoRA-on-SDXL trap.
No manifest python => uses the node's system python3 via the python-less
remote path, so it distributes across the gpu pool.
runner: gpu AND cpu are now a node pool. Per-node cpu_slots (primary 3,
helpers 2, nodes.json-overridable); net stays primary-only. Python-less ops
(ffmpeg/ffprobe) now run remotely on the node's system python3. Verified: 9
concurrent ffmpeg_frames distributed 4 local / 2 m1 / 2 m4.
hunyuan3d_mlx: default to Studio-quality (octree 384, texture 4096, remesh
120k) — 4096 bake verified watchdog-free on M3 Ultra; big quality gain
(defined face, 120k faces). remesh_faces now a param. All param-overridable.
scripts/agent_token.py mints the 'agents' guest user + token non-interactively,
writes data/agent.env (0600, gitignored, never printed). Fixes _register_outputs
dropping user_id — guest job outputs were registered ownerless, so guests could
never see or download their own results.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The gpu lane is now a NODE POOL (this Mac + remote workers from nodes.json) instead
of a single-Metal semaphore. A gpu job runs on whichever node is free:
- server/remote.py: ssh+rsync dispatch — mkdir remote dirs, rsync inputs out, run
the operator's run.py over ssh with repo-relative paths (HF_TOKEN sourced from the
node's .env.remote, off the process table), rsync outputs back, clean up. Cached
health checks; per-node operator allowlist.
- runner: _acquire_gpu_node picks the first free node that supports the op (local
runs anything; a remote must list it + be reachable → auto-fallback to local).
_run_job branches local/remote; the M1 never touches the M3's DB.
- sysinfo/Dashboard: gpu lane limit = pool size; per-node status cards.
- nodes.json (gitignored, primary-only): M1 worker, allowlist excludes trellis_mac
(bf16 unverified on M1) + brush_train (not installed there).
Verified: two FLUX jobs from one queue split M3(local, 10.3s) + M1(remote via ssh,
21.2s), both images rsync'd back and registered on the M3.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
