b3de1a4e35
m2max (96GB/4TB, added 2026-08-03): second bf16/trellis box, runs trellis_mac + trellis2_mlx. m4pro corrected to 26GB and hands-on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
50 lines
6.3 KiB
Markdown
50 lines
6.3 KiB
Markdown
# MODELBEAST cluster — read this first
|
||
|
||
**Policy: LOCAL FIRST.** This is a multi-Mac render farm on Tailscale. Every generation should run on our own silicon ($0, private, unlimited) unless a cloud model is genuinely better for the job. Reach for `fal_*` cloud operators only when no local operator covers the need (e.g. Rodin hero-grade meshes, the multi-view volume bargain) — never as the default. Cost of a local gen: nothing. Cost of a fal gen: real money (see [BENCHMARKS.md](BENCHMARKS.md) for per-call prices).
|
||
|
||
The order of preference for any task:
|
||
1. **Local operator** on whichever node is free (the queue picks it — see below).
|
||
2. **fal cloud** only for capabilities local can't match yet.
|
||
|
||
## The fleet
|
||
|
||
| Node | Host / Tailscale IP | RAM | Role |
|
||
|---|---|---|---|
|
||
| **M3 Ultra** (primary) | `m3ultra.local` · 100.89.131.57 | 256 GB | Runs the server + the central queue on :8777. Handles everything, especially the heavy 3D models (`hunyuan3d_mlx`, `trellis_mac`) and FLUX. |
|
||
| **M1 Ultra** (worker) | `ultra.local` · 100.91.239.7 | 128 GB | GPU worker in the pool **and** its own standalone instance. MLX-native — the right node for `hunyuan3d_mlx` (trellis's torch-MPS is unverified here). Also FLUX, `sf3d`, `bg_remove_local`. |
|
||
| **M2 Max** (m2max) | `m2max@100.120.83.110` | 96 GB | Second full gen node (added 2026-08-03), 4 TB SSD — the fleet's spare-disk box. Allowlisted: `flux_local`, `hunyuan3d_mlx`, `trellis_mac`, `trellis2_mlx`, `sf3d`, `seedvr2_upscale`, `mflux_image_edit`, `bg_remove_local`, `ffmpeg`/`ffprobe`. M2 does bf16, so unlike the M1s it **can** run trellis. Laptop — often asleep; treat as overflow, not latency-critical. If it silently drops out of the pool, check its `authorized_keys` first (wiped once, 7/31→8/3). |
|
||
| **M4 Pro** (helper) | `m4pro@100.69.21.128` | 26 GB | Helper for light gpu ops (`bg_remove_local`) + **cpu ops** (`ffmpeg_frames`, `ffprobe`; 2 slots) + a standalone **Ollama LLM** (`qwen2.5:7b` at `http://100.69.21.128:11434`). Too little RAM for the big 3D/diffusion models. |
|
||
| **M1 Max** (studio) | `studio@100.92.78.24` | 32 GB | Second full gen node (added 2026-07-17). `flux_local` (Klein 4B, ~38s/image), `hunyuan3d_mlx` (fp16, ~2× M3 time), `bg_remove_local`, `ffmpeg`/`ffprobe`. **Not** trellis (M1 bf16). Shared with audio work (stupendo) → treat as overflow, not latency-critical. |
|
||
| **M4 mini** | `m4mini@100.124.220.31` | 16 GB | Light helper (added 2026-07-17): `bg_remove_local` + cpu ops (`ffmpeg_frames`, `ffprobe`; 2 slots). Also the always-on godcheck node. Too small for FLUX/fp16-hunyuan. |
|
||
| **M1 mini** | `mini@100.79.229.50` | 8 GB | Light node (added 2026-07-17): `bg_remove_local` + `net`-class overflow. Baseline/clean-room test box. |
|
||
|
||
Git origin (shared by all nodes): `ssh://git@100.71.119.27:222/monster/modelbeast.git`
|
||
Login note: the M1 Max's admin user is `m1max` (has Homebrew/ffmpeg); the MODELBEAST checkout + farm run under the `studio` user, which needs `/opt/homebrew/bin` on its login PATH (already added to `~/.bash_profile`/`~/.zprofile`).
|
||
|
||
## The central queue (how tasks route to the right machine)
|
||
|
||
The M3 primary runs one server with per-lane concurrency:
|
||
- **`gpu` lane = a pool of nodes.** Each Mac has one Metal GPU, so each node runs **one** gpu job at a time. When gpu jobs queue up, the primary hands each to the first *free* node that supports that operator — rsyncing inputs over SSH, running the operator's `run.py` on that node, and rsyncing outputs back. This is the "unified queue": submit to M3, it load-balances across M3 + M1 (+ M4 for light ops).
|
||
- **`cpu` lane is also pooled** (since 2026-07-16): cpu ops distribute across nodes too — each node runs `cpu_slots` at once (primary 3, helpers 2 by default; set per node in nodes.json). A cpu op routes to a node only if it's in that node's allowlist **and** the node has the needed binary — `ffmpeg_frames`/`ffprobe` need ffmpeg (M1+M4 have it); `colmap_poses`/`blender_convert` stay on nodes with COLMAP/Blender. Python-less ops run on the node's system `python3`.
|
||
- **`net` lane** stays primary-only (the cloud API keys live only on M3), concurrency 6.
|
||
|
||
**Ollama LLM (M4):** a standalone Qwen2.5-7B server runs on M4 at `http://100.69.21.128:11434` (OpenAI-compatible at `/v1`), reachable across the tailnet. It is not a MODELBEAST operator — point any copilot/captioning client straight at that endpoint. Swap models with `ollama pull <model>` on M4.
|
||
|
||
Which node may run which operator is set in `nodes.json` (repo root, gitignored, primary-only) via each node's `operators` allowlist. Omit the allowlist to let a node run every gpu op. **`nodes.json` is read at server startup — restart the server (`scripts/serve.sh`) after editing it.**
|
||
|
||
## Adding a machine to the fleet
|
||
|
||
1. `git clone ssh://git@100.71.119.27:222/monster/modelbeast.git ~/MODELBEAST`
|
||
2. Install `uv` (`curl -LsSf https://astral.sh/uv/install.sh | sh`) — the venvs use it.
|
||
3. Run the per-operator install scripts you want that node to serve (`scripts/install_*.sh`). Each builds its own `vendor/*/.venv`. (`install_hunyuan3d_mlx.sh` auto-clones our self-owned MLX fork — `monster/Hunyuan3D-2.2-mrp-MLX` on the Gitea — into `vendor/`, so every node runs the same patched code.)
|
||
4. Model weights: **copy from a node that already has them** (`rsync -a ~/.cache/huggingface/hub/models--… newnode:~/.cache/huggingface/hub/`) rather than re-downloading — faster and dodges HF rate limits. We have an HF token in Settings if a fresh download is unavoidable.
|
||
5. On the **primary**, add the node to `nodes.json` with an `operators` allowlist matched to its RAM, then restart the server.
|
||
|
||
## Operator cheat-sheet (local first!)
|
||
|
||
- **image → 3D mesh:** local `hunyuan3d_mlx` (MLX, M1-capable, no HF login) or `trellis_mac` (sharper at defaults) → `sf3d` for fast drafts. fal only for Rodin hero-grade / multi-view. Always `bg_remove_local` (or `fal_bg_remove`) the input first.
|
||
- **prompt/image → image:** local `flux_local` (FLUX.2 Klein) → `mflux_image_edit`. fal/OpenRouter for nano-banana scene-storytelling.
|
||
- **video → frames / poses:** `ffmpeg_frames`, `colmap_poses` (CPU — great M4 work).
|
||
- Full matrix + prices: [AGENTS.md](AGENTS.md) · [BENCHMARKS.md](BENCHMARKS.md).
|
||
- **Which Mac runs what (RAM/hardware guide): [HARDWARE.md](HARDWARE.md)** — min + recommended RAM per operator, per Apple Silicon tier (M1 8GB → Ultra 256GB).
|