MODELBEAST

Local-first web app that turns videos, images, and 3D files into meshes, splats, mocap, and rigged characters on the M3 Ultra. See PLAN.md for the full verified tool matrix and roadmap, and HANDOFF.md for the agent build brief (phases 1–4 instructions).

🖥️ Multi-Mac fleet? Read CLUSTER.md first — the local-first policy, the M3/M1/M4 node roles, and how the central queue routes jobs across machines. 💾 What runs on which Mac? HARDWARE.md — min/recommended RAM per operator across every Apple Silicon tier (8GB → 256GB). 🎬 Green-screen keying → 3D: CORRIDORKEY.md — CorridorKey status, our Ultra tuning, and the video→cutout→mesh/splat use-case map.

Run

/opt/homebrew/bin/uv run uvicorn server.main:app --host 0.0.0.0 --port 8777

Open http://localhost:8777 (or http://<tailscale-ip>:8777 from any device on the tailnet). Prefer scripts/serve.sh — it runs headless and prints the one-time owner password on first start.

Drag/drop/paste any video, image, or 3D file; or drop files into data/inbox/ for auto-ingest. Pick an operator, tune its parameters, run. Add API keys / HuggingFace token under ⚙ Settings. Use ⊞ Compare to view several outputs side by side, and 📊 Dashboard for a live CPU/RAM/GPU/queue snapshot.

Accounts & sharing

Login is required. The first scripts/serve.sh creates an owner (monster) and prints a one-time password to data/server.log — change it in ⚙ Settings → Users (or uv run python scripts/users.py passwd monster). The owner can add guest accounts for friends: guests are local-only — they can run only on-device operators (never the owner's paid fal/OpenRouter keys, enforced server-side), see only their own assets/jobs, and have a per-user concurrent-job cap. So mates cost you electricity, nothing else. Public hosting via a VPS + Caddy is in docs/VPS.md; the CUDA-worker offload plan is in docs/CUDA-WORKER.md.

Headless / agents

Everything is driveable without the UI via the zero-dependency mb CLI (works from any tailnet machine with MB_HOST=http://100.89.131.57:8777) or raw REST. AGENTS.md is the complete handover brief — endpoints, mb reference, operator catalog, recipes, job semantics, etiquette. Server ops: scripts/serve.sh (start/restart headless), scripts/install_launchagent.sh (optional boot persistence, owner-run).

export MB_HOST=http://100.89.131.57:8777 MB_TOKEN=mbt_...   # token from ⚙ Settings → Users
./mb run fal_trellis2 --file photo.jpg -p resolution=1024 --wait --download out/

Develop

• Backend: server/ — FastAPI + SQLite (data/modelbeast.db), job runner runs operators as subprocesses across concurrency lanes (gpu=1, cpu=3, net=6). Settings/secrets in server/settings.py (env-injected, log-redacted).
• Frontend: web/ — React + Vite + three.js + @mkkellogg/gaussian-splats-3d. After editing: cd web && npm run build (the server serves web/dist).
• Data: data/assets/ (store), data/jobs/ (job workdirs), data/inbox/ (watch folder). Delete data/ to reset. Tests use MODELBEAST_DATA=<tmp>.
• Tests: ./tests/smoke.sh (12 framework checks). Benchmarks in BENCHMARKS.md.
• Heavy tools live in vendor/ (repos) + venvs/ (per-tool envs), both gitignored. Reinstall with scripts/install_*.sh.

Setup for the local mesh generators (one-time)

sf3d and trellis_mac are installed but their weights are HuggingFace-gated. Accept the licenses (stabilityai/stable-fast-3d, facebook/dinov3-vitl16-pretrain-lvd1689m, briaai/RMBG-2.0), then huggingface-cli login or paste an HF token in Settings. Cloud fal_* operators need FAL_KEY in Settings.

Operators

Each subfolder of server/operators/ with a manifest.json is an operator. The UI auto-renders its parameter form from params_schema (JSON Schema) and filters by the selected asset's kind (accepts).

Contract: the runner invokes <python> run.py --input <asset> --outdir <jobdir> --params '<json>'. Write outputs into the outdir; optionally write result.json ({"outputs": [{"path": ..., "name": ..., "meta": ...}]}) to control what gets registered as assets. stdout/stderr become the job log.

Manifest fields: id, name, category, description, accepts, produces, resources (gpu/cpu/net lane), requires_env (gates the operator until the key is set), python (absolute venv path for heavy tools), params_schema. Operators can tag output asset kind via result.json output meta.kind (e.g. splat, colmap_dataset).

Current operators:

id	lane	what
ffprobe	cpu	media inspection
ffmpeg_frames	cpu	video → frames (fps, mpdecimate dedupe, blur cull, max cap)
blender_convert	cpu	universal 3D format conversion via headless Blender
colmap_poses	cpu	frames → camera poses + sparse cloud (COLMAP 4.x + GLOMAP)
brush_train	gpu	colmap dataset → 3D gaussian splat (Brush, native Metal)
sf3d	gpu	image → GLB locally (Stable-Fast-3D, MPS) [HF-gated]
trellis_mac	gpu	image → GLB+PBR locally (TRELLIS.2 MPS port) [HF-gated]
hunyuan3d_mlx	gpu	image → GLB+PBR locally (Hunyuan3D 2.1, native MLX — runs on M1 too; weights public, no HF login)
fal_trellis / fal_trellis2 / fal_hunyuan3d / fal_hunyuan3d_v21 / fal_rodin	net	image → mesh via fal.ai API [needs FAL_KEY]
fal_bg_remove	net	image → subject cutout (BiRefNet v2) — run before any image→3D for a big quality jump
fal_upscale	net	image → faithful upscale (SeedVR)
fal_image_edit	net	image + instruction → edited image (nano-banana)
fal_text_image	net	prompt → image (Ideogram v3, readable text) — no input asset needed

Recommended fal chain for best image→3D: fal_bg_remove → (fal_upscale if thin) → fal_trellis2 or fal_hunyuan3d_v21. Note: Hunyuan v21 multi-view is broken on fal (verified 2026-07) — v21 is single-image only; use v2 for multi-view.

Heavy operators get their own uv venv ("python": "<abs venv path>" in the manifest). Remaining roadmap (object_capture, freemocap, retargeting, tripo/meshy character APIs, workflow presets, LLM copilot) is in HANDOFF.md §5–8.