modelbeast/CLUSTER.md

MODELBEAST cluster — read this first

Policy: LOCAL FIRST. This is a multi-Mac render farm on Tailscale. Every generation should run on our own silicon ($0, private, unlimited) unless a cloud model is genuinely better for the job. Reach for fal_* cloud operators only when no local operator covers the need (e.g. Rodin hero-grade meshes, the multi-view volume bargain) — never as the default. Cost of a local gen: nothing. Cost of a fal gen: real money (see BENCHMARKS.md for per-call prices).

The order of preference for any task:

1. Local operator on whichever node is free (the queue picks it — see below).
2. fal cloud only for capabilities local can't match yet.

The fleet

Node	Host / Tailscale IP	RAM	Role
M3 Ultra (primary)	m3ultra.local · 100.89.131.57	256 GB	Runs the server + the central queue on :8777. Handles everything, especially the heavy 3D models (hunyuan3d_mlx, trellis_mac) and FLUX.
M1 Ultra (worker)	ultra.local · 100.91.239.7	128 GB	GPU worker in the pool and its own standalone instance. MLX-native — the right node for hunyuan3d_mlx (trellis's torch-MPS is unverified here). Also FLUX, sf3d, bg_remove_local.
M4 Pro (helper)	m4pro.local · 100.69.21.128	24 GB	Lightweight helper. CPU work (ffmpeg_frames, ffprobe, colmap_poses), light image ops (bg_remove_local), and small LLMs via Ollama. Not the big 3D/diffusion models — too little RAM.

Git origin (shared by all nodes): ssh://git@100.71.119.27:222/monster/modelbeast.git

The central queue (how tasks route to the right machine)

The M3 primary runs one server with per-lane concurrency:

• gpu lane = a pool of nodes. Each Mac has one Metal GPU, so each node runs one gpu job at a time. When gpu jobs queue up, the primary hands each to the first free node that supports that operator — rsyncing inputs over SSH, running the operator's run.py on that node, and rsyncing outputs back. This is the "unified queue": submit to M3, it load-balances across M3 + M1 (+ M4 for light ops).
• cpu / net lanes run on the primary (M3) with simple concurrency limits (cpu 3, net 6).

Which node may run which operator is set in nodes.json (repo root, gitignored, primary-only) via each node's operators allowlist. Omit the allowlist to let a node run every gpu op. nodes.json is read at server startup — restart the server (scripts/serve.sh) after editing it.

Adding a machine to the fleet

1. git clone ssh://git@100.71.119.27:222/monster/modelbeast.git ~/MODELBEAST
2. Install uv (curl -LsSf https://astral.sh/uv/install.sh | sh) — the venvs use it.
3. Run the per-operator install scripts you want that node to serve (scripts/install_*.sh). Each builds its own vendor/*/.venv.
4. Model weights: copy from a node that already has them (rsync -a ~/.cache/huggingface/hub/models--… newnode:~/.cache/huggingface/hub/) rather than re-downloading — faster and dodges HF rate limits. We have an HF token in Settings if a fresh download is unavoidable.
5. On the primary, add the node to nodes.json with an operators allowlist matched to its RAM, then restart the server.

Operator cheat-sheet (local first!)

• image → 3D mesh: local hunyuan3d_mlx (MLX, M1-capable, no HF login) or trellis_mac (sharper at defaults) → sf3d for fast drafts. fal only for Rodin hero-grade / multi-view. Always bg_remove_local (or fal_bg_remove) the input first.
• prompt/image → image: local flux_local (FLUX.2 Klein) → mflux_image_edit. fal/OpenRouter for nano-banana scene-storytelling.
• video → frames / poses: ffmpeg_frames, colmap_poses (CPU — great M4 work).
• Full matrix + prices: AGENTS.md · BENCHMARKS.md.