monster/modelbeast
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

README: local-first rewrite + ignore the .env.* family

Browse Source 
The README still read as a fal.ai front-end with a local afterthought: the headline
`mb` example ran fal_trellis2, the "recommended chain for best image→3D" was four
fal_* calls, and the operator table listed 5 local ops against 9 cloud ones. That is
backwards now — 22 of the 31 operators run entirely on the fleet.

It also predated most of what we have: the whole audio/video/text suite (tts_local,
stt_local, voice_clone_local, music_local, lipsync_local, wan_video, llm_local,
motion_local), comfyui_sd, flux_local, mflux_image_edit, seedvr2_upscale and
bg_remove_local were all missing from the catalog entirely.

- lead with local; cloud demoted to its own opt-in "costs real money" section
- all 31 operators documented, grouped by what you're trying to make
- recommended chain is now local: bg_remove_local -> [seedvr2_upscale] -> trellis_mac
  (quality) or hunyuan3d_mlx (faster, ~3x lighter, the only one that runs on M1)
- setup section reordered: most local ops need nothing, hunyuan3d_mlx needs no HF
  login at all, only sf3d/trellis_mac/bg_remove_local are gated
- lanes corrected: gpu+cpu are a NODE POOL (gpu 1/node, cpu per-node cpu_slots),
  not the old flat "gpu=1, cpu=3, net=6"
- manifest `python` documented as repo-relative (remote nodes can't resolve absolute)
- link PERFCHECK.md; drop the stale "on the M3 Ultra" framing — it's a fleet
- verified: every operator on disk appears, no phantom ids, no broken doc links

Also: .gitignore had `*.env` and `.env`, neither of which matches a suffixed name
like `.env.remote` — which is sitting un-ignored in the m1's checkout right now, one
`git add -A` away from being committed. Ignore the `.env.*` family (keeping
.env.example committable). Nothing has leaked: no .env file is tracked.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
m3ultra 2026-07-17 22:24:18 +10:00
parent b9de2a4167
commit 4bc2f56d34
2 changed files with 119 additions and 37 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
.gitignore vendored
View File

@ -15,6 +15,11 @@ web/dist/
# Secrets & local config # Secrets & local config
*.env *.env
.env .env
# `.env` matches only the exact name and `*.env` only matches names ENDING in .env — neither
# catches a suffixed one like `.env.remote` (which sits un-ignored on the m1 today). Secrets
# live in .env files by convention here, so ignore the whole family; keep examples committable.
.env.*
!.env.example
server/.env server/.env
# Large model weights / caches # Large model weights / caches

151
README.md
View File

@ -1,71 +1,148 @@
# MODELBEAST # MODELBEAST
Local-first web app that turns videos, images, and 3D files into meshes, splats, mocap, and rigged characters on the M3 Ultra. See [PLAN.md](PLAN.md) for the full verified tool matrix and roadmap, and [HANDOFF.md](HANDOFF.md) for the agent build brief (phases 1–4 instructions). **Local-first generation farm for a fleet of Macs.** Video, image, audio, text or 3D in →
meshes, splats, images, video, speech, music, motion and text out. Everything below runs *on
your own machines* — nothing leaves the tailnet unless you explicitly pick a `fal_*` or
`openrouter_*` operator, and guests can't pick those at all.
> 🖥️ **Multi-Mac fleet? Read [CLUSTER.md](CLUSTER.md) first** — the local-first policy, the M3/M1/M4 node roles, and how the central queue routes jobs across machines. > 🖥️ **Multi-Mac fleet? Read [CLUSTER.md](CLUSTER.md) first** — the local-first policy, node roles, and how the central queue routes jobs across machines.
> 💾 **What runs on which Mac?** [HARDWARE.md](HARDWARE.md) — min/recommended RAM per operator across every Apple Silicon tier (8GB → 256GB). > 💾 **What runs on which Mac?** [HARDWARE.md](HARDWARE.md) — min/recommended RAM per operator across every Apple Silicon tier (8GB → 256GB).
> 📊 **Is the fleet healthy / how fast is it?** [PERFCHECK.md](PERFCHECK.md) — nightly GPU canary + per-node capability table. [BENCHMARKS.md](BENCHMARKS.md) — what each model actually costs.
> 🎬 **Green-screen keying → 3D:** [CORRIDORKEY.md](CORRIDORKEY.md) — CorridorKey status, our Ultra tuning, and the video→cutout→mesh/splat use-case map. > 🎬 **Green-screen keying → 3D:** [CORRIDORKEY.md](CORRIDORKEY.md) — CorridorKey status, our Ultra tuning, and the video→cutout→mesh/splat use-case map.
See [PLAN.md](PLAN.md) for the verified tool matrix and roadmap, [HANDOFF.md](HANDOFF.md) for the
agent build brief.
## Run ## Run
```bash ```bash
/opt/homebrew/bin/uv run uvicorn server.main:app --host 0.0.0.0 --port 8777 /opt/homebrew/bin/uv run uvicorn server.main:app --host 0.0.0.0 --port 8777
``` ```
Open http://localhost:8777 (or `http://<tailscale-ip>:8777` from any device on the tailnet). Prefer `scripts/serve.sh` — it runs headless and prints the one-time owner password on first start. Open http://localhost:8777 (or `http://<tailscale-ip>:8777` from any device on the tailnet).
Prefer `scripts/serve.sh` — it runs headless and prints the one-time owner password on first start.
Drag/drop/paste any video, image, or 3D file; or drop files into `data/inbox/` for auto-ingest. Pick an operator, tune its parameters, run. Add API keys / HuggingFace token under ⚙ Settings. Use ⊞ Compare to view several outputs side by side, and **📊 Dashboard** for a live CPU/RAM/GPU/queue snapshot. Drag/drop/paste any video, image, audio or 3D file; or drop files into `data/inbox/` for
auto-ingest. Pick an operator, tune its parameters, run. Several operators (`flux_local`,
`comfyui_sd`, `tts_local`, `llm_local`, `music_local`, `motion_local`) need **no input asset** —
just a prompt. Use ⊞ Compare to view outputs side by side, and **📊 Dashboard** for a live
CPU/RAM/GPU/queue snapshot.
## Accounts & sharing ## Accounts & sharing
Login is required. The first `scripts/serve.sh` creates an **owner** (`monster`) and prints a one-time password to `data/server.log` — change it in ⚙ Settings → Users (or `uv run python scripts/users.py passwd monster`). The owner can add **guest** accounts for friends: guests are **local-only** — they can run only on-device operators (never the owner's paid fal/OpenRouter keys, enforced server-side), see only their own assets/jobs, and have a per-user concurrent-job cap. So mates cost you electricity, nothing else. Public hosting via a VPS + Caddy is in [docs/VPS.md](docs/VPS.md); the CUDA-worker offload plan is in [docs/CUDA-WORKER.md](docs/CUDA-WORKER.md). Login is required. The first `scripts/serve.sh` creates an **owner** (`monster`) and prints a
one-time password to `data/server.log` — change it in ⚙ Settings → Users (or `uv run python
scripts/users.py passwd monster`). The owner can add **guest** accounts for friends: guests are
**local-only** — they can run only on-device operators (never the owner's paid keys, enforced
server-side), see only their own assets/jobs, and have a per-user concurrent-job cap. So mates
cost you electricity, nothing else. Public hosting via a VPS + Caddy is in
[docs/VPS.md](docs/VPS.md); the CUDA-worker offload plan is in [docs/CUDA-WORKER.md](docs/CUDA-WORKER.md).
## Headless / agents ## Headless / agents
Everything is driveable without the UI via the zero-dependency [`mb`](mb) CLI (works from any tailnet machine with `MB_HOST=http://100.89.131.57:8777`) or raw REST. **[AGENTS.md](AGENTS.md) is the complete handover brief** — endpoints, `mb` reference, operator catalog, recipes, job semantics, etiquette. Server ops: `scripts/serve.sh` (start/restart headless), `scripts/install_launchagent.sh` (optional boot persistence, owner-run). Everything is driveable without the UI via the zero-dependency [`mb`](mb) CLI (works from any
tailnet machine) or raw REST. **[AGENTS.md](AGENTS.md) is the complete handover brief** —
endpoints, `mb` reference, operator catalog, recipes, job semantics, etiquette. Server ops:
`scripts/serve.sh` (start/restart headless), `scripts/install_launchagent.sh` (optional boot
persistence, owner-run).
```bash ```bash
export MB_HOST=http://100.89.131.57:8777 MB_TOKEN=mbt_... # token from ⚙ Settings → Users export MB_HOST=http://100.89.131.57:8777 MB_TOKEN=mbt_... # token from ⚙ Settings → Users
./mb run fal_trellis2 --file photo.jpg -p resolution=1024 --wait --download out/ ./mb run flux_local -p prompt="a brass astrolabe on velvet" --wait --download out/
./mb run bg_remove_local --file photo.jpg --wait --download out/
./mb run hunyuan3d_mlx --file cutout.png --wait --download out/
``` ```
## Develop
- Backend: `server/` — FastAPI + SQLite (`data/modelbeast.db`), job runner runs operators as subprocesses across concurrency lanes (gpu=1, cpu=3, net=6). Settings/secrets in `server/settings.py` (env-injected, log-redacted).
- Frontend: `web/` — React + Vite + three.js + `@mkkellogg/gaussian-splats-3d`. After editing: `cd web && npm run build` (the server serves `web/dist`).
- Data: `data/assets/` (store), `data/jobs/` (job workdirs), `data/inbox/` (watch folder). Delete `data/` to reset. Tests use `MODELBEAST_DATA=<tmp>`.
- Tests: `./tests/smoke.sh` (12 framework checks). Benchmarks in [BENCHMARKS.md](BENCHMARKS.md).
- Heavy tools live in `vendor/` (repos) + `venvs/` (per-tool envs), both gitignored. Reinstall with `scripts/install_*.sh`.
## Setup for the local mesh generators (one-time)
`sf3d` and `trellis_mac` are installed but their weights are HuggingFace-gated. Accept the licenses (stabilityai/stable-fast-3d, facebook/dinov3-vitl16-pretrain-lvd1689m, briaai/RMBG-2.0), then `huggingface-cli login` or paste an HF token in Settings. Cloud `fal_*` operators need `FAL_KEY` in Settings.
## Operators ## Operators
Each subfolder of `server/operators/` with a `manifest.json` is an operator. The UI auto-renders its parameter form from `params_schema` (JSON Schema) and filters by the selected asset's kind (`accepts`). Each subfolder of `server/operators/` with a `manifest.json` is an operator. The UI auto-renders
its parameter form from `params_schema` (JSON Schema) and filters by the selected asset's kind
(`accepts`).
Contract: the runner invokes `<python> run.py --input <asset> --outdir <jobdir> --params '<json>'`. Write outputs into the outdir; optionally write `result.json` (`{"outputs": [{"path": ..., "name": ..., "meta": ...}]}`) to control what gets registered as assets. stdout/stderr become the job log. ### Local — free, offline, on-fleet
Manifest fields: `id, name, category, description, accepts, produces, resources` (gpu/cpu/net lane), `requires_env` (gates the operator until the key is set), `python` (absolute venv path for heavy tools), `params_schema`. Operators can tag output asset kind via `result.json` output `meta.kind` (e.g. `splat`, `colmap_dataset`). This is the default. Reach for cloud only when local genuinely can't do it.
Current operators:
| id | lane | what | | id | lane | what |
|---|---|---| |---|---|---|
| `ffprobe` | cpu | media inspection | | **image → 3D** | | |
| `ffmpeg_frames` | cpu | video → frames (fps, mpdecimate dedupe, blur cull, max cap) | | `hunyuan3d_mlx` | gpu | image → GLB + PBR (Hunyuan3D 2.1, **native MLX**). Weights public — no HF login. The only local 3D op that runs on M1. ~4.3 min |
| `blender_convert` | cpu | universal 3D format conversion via headless Blender | | `trellis_mac` | gpu | image → GLB + PBR (TRELLIS.2 MPS port). Best local quality — sharper than hunyuan at defaults. ~5 min *[HF-gated]* |
| `sf3d` | gpu | image → GLB in ~5s. Draft tier — good on solid objects, struggles on thin/open geometry *[HF-gated]* |
| **image gen / edit** | | |
| `flux_local` | gpu | prompt → image via mflux/MLX. FLUX.2 Klein 4B is Apache + ungated, ~9s/image |
| `comfyui_sd` | gpu | prompt → image via local SD/SDXL checkpoints **+ LoRAs** (ComfyUI on Metal). `flux_local` cannot load SD/SDXL — this is the one that can |
| `mflux_image_edit` | gpu | image + instruction → edited image (Qwen-Image-Edit, ungated) |
| `seedvr2_upscale` | gpu | image → faithful upscale (SeedVR2, MIT) — sharpens without inventing detail |
| `bg_remove_local` | gpu | image → subject cutout (RMBG-2.0). **The single biggest quality lever before any image→3D** |
| **video** | | |
| `wan_video` | gpu | prompt → MP4, or image + prompt → MP4 (Wan 2.2 TI2V-5B on the resident ComfyUI). 24fps, 1280×704 |
| **audio / speech** | | |
| `tts_local` | gpu | text → speech (Kokoro-82M, ~50 voices, Apache, ~350MB) |
| `stt_local` | gpu | audio → transcript + timestamps (Whisper large-v3-turbo on MLX) |
| `voice_clone_local` | gpu | ~10s reference voice + text → that voice speaking it (Chatterbox, MIT) |
| `music_local` | gpu | style tags (+ optional lyrics) → music WAV (ACE-Step 3.5B, Apache) |
| `lipsync_local` | cpu | speech WAV → viseme timing JSON (Rhubarb, fully offline) |
| **text / motion** | | |
| `llm_local` | gpu | prompt → text (Qwen3-30B-A3B-4bit MoE on MLX) |
| `motion_local` | cpu | text → character motion as BVH + mp4 preview (MoMask) |
| **scan → splat** | | |
| `ffmpeg_frames` | cpu | video → frames (fps sampling, mpdecimate dedupe, blur cull) |
| `colmap_poses` | cpu | frames → camera poses + sparse cloud (COLMAP 4.x + GLOMAP) | | `colmap_poses` | cpu | frames → camera poses + sparse cloud (COLMAP 4.x + GLOMAP) |
| `brush_train` | gpu | colmap dataset → 3D gaussian splat (Brush, native Metal) | | `brush_train` | gpu | colmap dataset → gaussian splat .ply (Brush, native Metal) |
| `sf3d` | gpu | image → GLB locally (Stable-Fast-3D, MPS) *[HF-gated]* | | **utility** | | |
| `trellis_mac` | gpu | image → GLB+PBR locally (TRELLIS.2 MPS port) *[HF-gated]* | | `ffprobe` | cpu | media inspection |
| `hunyuan3d_mlx` | gpu | image → GLB+PBR locally (Hunyuan3D 2.1, native MLX — runs on M1 too; weights public, no HF login) | | `blender_convert` | cpu | GLB/GLTF/OBJ/FBX/USD/STL/PLY/BLEND ⇄ conversion via headless Blender |
| `fal_trellis` / `fal_trellis2` / `fal_hunyuan3d` / `fal_hunyuan3d_v21` / `fal_rodin` | net | image → mesh via fal.ai API *[needs FAL_KEY]* |
| `fal_bg_remove` | net | image → subject cutout (BiRefNet v2) — run before any image→3D for a big quality jump |
| `fal_upscale` | net | image → faithful upscale (SeedVR) |
| `fal_image_edit` | net | image + instruction → edited image (nano-banana) |
| `fal_text_image` | net | prompt → image (Ideogram v3, readable text) — no input asset needed |
Recommended fal chain for best image→3D: `fal_bg_remove` → (`fal_upscale` if thin) → `fal_trellis2` or `fal_hunyuan3d_v21`. Note: Hunyuan v21 **multi-view** is broken on fal (verified 2026-07) — v21 is single-image only; use v2 for multi-view. **Recommended local image→3D chain:** `bg_remove_local` → (`seedvr2_upscale` if thin/blurry) →
`trellis_mac` (quality) or `hunyuan3d_mlx` (faster, ~3× lighter, runs on M1).
Heavy operators get their own uv venv (`"python": "<abs venv path>"` in the manifest). Remaining roadmap (object_capture, freemocap, retargeting, tripo/meshy character APIs, workflow presets, LLM copilot) is in [HANDOFF.md](HANDOFF.md) §5–8. ### Cloud — costs real money, opt-in, owner-only
Use when local can't, or for on-demand SOTA without the local wait.
| id | lane | what |
|---|---|---|
| `fal_trellis2` / `fal_hunyuan3d_v21` / `fal_hunyuan3d` / `fal_trellis` / `fal_rodin` | net | image → mesh via fal.ai *[FAL_KEY]* |
| `fal_bg_remove` / `fal_upscale` / `fal_image_edit` / `fal_text_image` | net | cutout / upscale / nano-banana edit / Ideogram v3 (readable text) *[FAL_KEY]* |
| `openrouter_image` | net | prompt → image via OpenRouter (nano-banana family; exact cost reported per job) *[OPENROUTER_API_KEY]* |
Note: fal's Hunyuan **v21 multi-view is broken** (verified 2026-07) — v21 is single-image only;
use v2 for multi-view. `fal_rodin` is the film/hero tier (quad topology, ~$0.40+).
## Setup for the gated local models (one-time)
Most local operators work out of the box. Only these need anything:
- `sf3d`, `trellis_mac` — weights are HuggingFace-gated. Accept the licences
(stabilityai/stable-fast-3d, facebook/dinov3-vitl16-pretrain-lvd1689m), then
`huggingface-cli login` or paste an HF token in ⚙ Settings.
- `bg_remove_local` — briaai/RMBG-2.0 licence, same deal.
- `hunyuan3d_mlx` — **nothing**. Weights are public.
- Cloud operators — `FAL_KEY` / `OPENROUTER_API_KEY` in ⚙ Settings.
Heavy tools live in `vendor/` (repos) + `venvs/` (per-tool envs), both gitignored — reinstall
with `scripts/install_*.sh`.
## Develop
- Backend: `server/` — FastAPI + SQLite (`data/modelbeast.db`), job runner runs operators as subprocesses across concurrency lanes. **gpu and cpu are a node pool**: gpu = 1 job per node, cpu = per-node slots (`cpu_slots` in `nodes.json`; default 3 local / 2 remote), net = 6 on the primary only. Settings/secrets in `server/settings.py` (env-injected, log-redacted).
- Frontend: `web/` — React + Vite + three.js + `@mkkellogg/gaussian-splats-3d`. After editing: `cd web && npm run build` (the server serves `web/dist`).
- Data: `data/assets/` (store), `data/jobs/` (job workdirs), `data/inbox/` (watch folder). Delete `data/` to reset. Tests use `MODELBEAST_DATA=<tmp>`.
- Tests: `./tests/smoke.sh` (12 framework checks).
### Writing an operator
Contract: the runner invokes `<python> run.py --input <asset> --outdir <jobdir> --params '<json>'`.
Write outputs into the outdir; optionally write `result.json`
(`{"outputs": [{"path": ..., "name": ..., "meta": ...}]}`) to control what gets registered as
assets. stdout/stderr become the job log; the runner stamps `[node: <name>]` as log line 1.
Manifest fields: `id, name, category, description, accepts, produces, resources` (gpu/cpu/net
lane — defaults to cpu if omitted), `requires_env` (gates the operator until the key is set),
`python` (venv path for heavy tools — **repo-relative**, so remote nodes resolve it), and
`params_schema`. Operators can tag output asset kind via `result.json` output `meta.kind` (e.g.
`splat`, `colmap_dataset`).
Remaining roadmap (object_capture, freemocap, retargeting, character APIs, workflow presets, LLM
copilot) is in [HANDOFF.md](HANDOFF.md) §5–8.