modelbeast/CLUSTER.md

MODELBEAST cluster — read this first

Policy: LOCAL FIRST. This is a multi-Mac render farm on Tailscale. Every generation should run on our own silicon ($0, private, unlimited) unless a cloud model is genuinely better for the job. Reach for fal_* cloud operators only when no local operator covers the need (e.g. Rodin hero-grade meshes, the multi-view volume bargain) — never as the default. Cost of a local gen: nothing. Cost of a fal gen: real money (see BENCHMARKS.md for per-call prices).

The order of preference for any task:

1. Local operator on whichever node is free (the queue picks it — see below).
2. fal cloud only for capabilities local can't match yet.

The fleet

Node	Host / Tailscale IP	RAM	Role
M3 Ultra (primary)	m3ultra.local · 100.89.131.57	256 GB	Runs the server + the central queue on :8777. Handles everything, especially the heavy 3D models (hunyuan3d_mlx, trellis_mac) and FLUX.
M1 Ultra (worker)	ultra.local · 100.91.239.7	128 GB	GPU worker in the pool and its own standalone instance. MLX-native — the right node for hunyuan3d_mlx (trellis's torch-MPS is unverified here). Also FLUX, sf3d, bg_remove_local.
M2 Max (m2max)	m2max@100.120.83.110	96 GB	Second full gen node (added 2026-08-03), 4 TB SSD — the fleet's spare-disk box. Allowlisted: flux_local, hunyuan3d_mlx, trellis_mac, trellis2_mlx, sf3d, seedvr2_upscale, mflux_image_edit, bg_remove_local, ffmpeg/ffprobe. M2 does bf16, so unlike the M1s it can run trellis. Laptop — often asleep; treat as overflow, not latency-critical. If it silently drops out of the pool, check its authorized_keys first (wiped once, 7/31→8/3).
M4 Pro (helper)	m4pro@100.69.21.128	26 GB	Helper for light gpu ops (bg_remove_local) + cpu ops (ffmpeg_frames, ffprobe; 2 slots) + a standalone Ollama LLM (qwen2.5:7b at http://100.69.21.128:11434). Too little RAM for the big 3D/diffusion models.
M1 Max (studio)	studio@100.92.78.24	32 GB	Second full gen node (added 2026-07-17). flux_local (Klein 4B, ~38s/image), hunyuan3d_mlx (fp16, ~2× M3 time), bg_remove_local, ffmpeg/ffprobe. Not trellis (M1 bf16). Shared with audio work (stupendo) → treat as overflow, not latency-critical.
M4 mini	m4mini@100.124.220.31	16 GB	Light helper (added 2026-07-17): bg_remove_local + cpu ops (ffmpeg_frames, ffprobe; 2 slots). Also the always-on godcheck node. Too small for FLUX/fp16-hunyuan.
M1 mini	mini@100.79.229.50	8 GB	Light node (added 2026-07-17): bg_remove_local + net-class overflow. Baseline/clean-room test box.

Git origin (shared by all nodes): ssh://git@100.71.119.27:222/monster/modelbeast.git Login note: the M1 Max's admin user is m1max (has Homebrew/ffmpeg); the MODELBEAST checkout + farm run under the studio user, which needs /opt/homebrew/bin on its login PATH (already added to ~/.bash_profile/~/.zprofile).

The central queue (how tasks route to the right machine)

The M3 primary runs one server with per-lane concurrency:

• gpu lane = a pool of nodes. Each Mac has one Metal GPU, so each node runs one gpu job at a time. When gpu jobs queue up, the primary hands each to the first free node that supports that operator — rsyncing inputs over SSH, running the operator's run.py on that node, and rsyncing outputs back. This is the "unified queue": submit to M3, it load-balances across M3 + M1 (+ M4 for light ops).
• cpu lane is also pooled (since 2026-07-16): cpu ops distribute across nodes too — each node runs cpu_slots at once (primary 3, helpers 2 by default; set per node in nodes.json). A cpu op routes to a node only if it's in that node's allowlist and the node has the needed binary — ffmpeg_frames/ffprobe need ffmpeg (M1+M4 have it); colmap_poses/blender_convert stay on nodes with COLMAP/Blender. Python-less ops run on the node's system python3.
• net lane stays primary-only (the cloud API keys live only on M3), concurrency 6.

Ollama LLM (M4): a standalone Qwen2.5-7B server runs on M4 at http://100.69.21.128:11434 (OpenAI-compatible at /v1), reachable across the tailnet. It is not a MODELBEAST operator — point any copilot/captioning client straight at that endpoint. Swap models with ollama pull <model> on M4.

Which node may run which operator is set in nodes.json (repo root, gitignored, primary-only) via each node's operators allowlist. Omit the allowlist to let a node run every gpu op. nodes.json is read at server startup — restart the server (scripts/serve.sh) after editing it.

Adding a machine to the fleet

1. git clone ssh://git@100.71.119.27:222/monster/modelbeast.git ~/MODELBEAST
2. Install uv (curl -LsSf https://astral.sh/uv/install.sh | sh) — the venvs use it.
3. Run the per-operator install scripts you want that node to serve (scripts/install_*.sh). Each builds its own vendor/*/.venv. (install_hunyuan3d_mlx.sh auto-clones our self-owned MLX fork — monster/Hunyuan3D-2.2-mrp-MLX on the Gitea — into vendor/, so every node runs the same patched code.)
4. Model weights: copy from a node that already has them (rsync -a ~/.cache/huggingface/hub/models--… newnode:~/.cache/huggingface/hub/) rather than re-downloading — faster and dodges HF rate limits. We have an HF token in Settings if a fresh download is unavoidable.
5. On the primary, add the node to nodes.json with an operators allowlist matched to its RAM, then restart the server.

Operator cheat-sheet (local first!)

• image → 3D mesh: local hunyuan3d_mlx (MLX, M1-capable, no HF login) or trellis_mac (sharper at defaults) → sf3d for fast drafts. fal only for Rodin hero-grade / multi-view. Always bg_remove_local (or fal_bg_remove) the input first.
• prompt/image → image: local flux_local (FLUX.2 Klein) → mflux_image_edit. fal/OpenRouter for nano-banana scene-storytelling.
• video → frames / poses: ffmpeg_frames, colmap_poses (CPU — great M4 work).
• Full matrix + prices: AGENTS.md · BENCHMARKS.md.
• Which Mac runs what (RAM/hardware guide): HARDWARE.md — min + recommended RAM per operator, per Apple Silicon tier (M1 8GB → Ultra 256GB).