72 lines
6.3 KiB
Markdown
72 lines
6.3 KiB
Markdown
# MODELBEAST
|
||
|
||
Local-first web app that turns videos, images, and 3D files into meshes, splats, mocap, and rigged characters on the M3 Ultra. See [PLAN.md](PLAN.md) for the full verified tool matrix and roadmap, and [HANDOFF.md](HANDOFF.md) for the agent build brief (phases 1–4 instructions).
|
||
|
||
> 🖥️ **Multi-Mac fleet? Read [CLUSTER.md](CLUSTER.md) first** — the local-first policy, the M3/M1/M4 node roles, and how the central queue routes jobs across machines.
|
||
> 💾 **What runs on which Mac?** [HARDWARE.md](HARDWARE.md) — min/recommended RAM per operator across every Apple Silicon tier (8GB → 256GB).
|
||
> 🎬 **Green-screen keying → 3D:** [CORRIDORKEY.md](CORRIDORKEY.md) — CorridorKey status, our Ultra tuning, and the video→cutout→mesh/splat use-case map.
|
||
|
||
## Run
|
||
|
||
```bash
|
||
/opt/homebrew/bin/uv run uvicorn server.main:app --host 0.0.0.0 --port 8777
|
||
```
|
||
|
||
Open http://localhost:8777 (or `http://<tailscale-ip>:8777` from any device on the tailnet). Prefer `scripts/serve.sh` — it runs headless and prints the one-time owner password on first start.
|
||
|
||
Drag/drop/paste any video, image, or 3D file; or drop files into `data/inbox/` for auto-ingest. Pick an operator, tune its parameters, run. Add API keys / HuggingFace token under ⚙ Settings. Use ⊞ Compare to view several outputs side by side, and **📊 Dashboard** for a live CPU/RAM/GPU/queue snapshot.
|
||
|
||
## Accounts & sharing
|
||
|
||
Login is required. The first `scripts/serve.sh` creates an **owner** (`monster`) and prints a one-time password to `data/server.log` — change it in ⚙ Settings → Users (or `uv run python scripts/users.py passwd monster`). The owner can add **guest** accounts for friends: guests are **local-only** — they can run only on-device operators (never the owner's paid fal/OpenRouter keys, enforced server-side), see only their own assets/jobs, and have a per-user concurrent-job cap. So mates cost you electricity, nothing else. Public hosting via a VPS + Caddy is in [docs/VPS.md](docs/VPS.md); the CUDA-worker offload plan is in [docs/CUDA-WORKER.md](docs/CUDA-WORKER.md).
|
||
|
||
## Headless / agents
|
||
|
||
Everything is driveable without the UI via the zero-dependency [`mb`](mb) CLI (works from any tailnet machine with `MB_HOST=http://100.89.131.57:8777`) or raw REST. **[AGENTS.md](AGENTS.md) is the complete handover brief** — endpoints, `mb` reference, operator catalog, recipes, job semantics, etiquette. Server ops: `scripts/serve.sh` (start/restart headless), `scripts/install_launchagent.sh` (optional boot persistence, owner-run).
|
||
|
||
```bash
|
||
export MB_HOST=http://100.89.131.57:8777 MB_TOKEN=mbt_... # token from ⚙ Settings → Users
|
||
./mb run fal_trellis2 --file photo.jpg -p resolution=1024 --wait --download out/
|
||
```
|
||
|
||
## Develop
|
||
|
||
- Backend: `server/` — FastAPI + SQLite (`data/modelbeast.db`), job runner runs operators as subprocesses across concurrency lanes (gpu=1, cpu=3, net=6). Settings/secrets in `server/settings.py` (env-injected, log-redacted).
|
||
- Frontend: `web/` — React + Vite + three.js + `@mkkellogg/gaussian-splats-3d`. After editing: `cd web && npm run build` (the server serves `web/dist`).
|
||
- Data: `data/assets/` (store), `data/jobs/` (job workdirs), `data/inbox/` (watch folder). Delete `data/` to reset. Tests use `MODELBEAST_DATA=<tmp>`.
|
||
- Tests: `./tests/smoke.sh` (12 framework checks). Benchmarks in [BENCHMARKS.md](BENCHMARKS.md).
|
||
- Heavy tools live in `vendor/` (repos) + `venvs/` (per-tool envs), both gitignored. Reinstall with `scripts/install_*.sh`.
|
||
|
||
## Setup for the local mesh generators (one-time)
|
||
|
||
`sf3d` and `trellis_mac` are installed but their weights are HuggingFace-gated. Accept the licenses (stabilityai/stable-fast-3d, facebook/dinov3-vitl16-pretrain-lvd1689m, briaai/RMBG-2.0), then `huggingface-cli login` or paste an HF token in Settings. Cloud `fal_*` operators need `FAL_KEY` in Settings.
|
||
|
||
## Operators
|
||
|
||
Each subfolder of `server/operators/` with a `manifest.json` is an operator. The UI auto-renders its parameter form from `params_schema` (JSON Schema) and filters by the selected asset's kind (`accepts`).
|
||
|
||
Contract: the runner invokes `<python> run.py --input <asset> --outdir <jobdir> --params '<json>'`. Write outputs into the outdir; optionally write `result.json` (`{"outputs": [{"path": ..., "name": ..., "meta": ...}]}`) to control what gets registered as assets. stdout/stderr become the job log.
|
||
|
||
Manifest fields: `id, name, category, description, accepts, produces, resources` (gpu/cpu/net lane), `requires_env` (gates the operator until the key is set), `python` (absolute venv path for heavy tools), `params_schema`. Operators can tag output asset kind via `result.json` output `meta.kind` (e.g. `splat`, `colmap_dataset`).
|
||
|
||
Current operators:
|
||
| id | lane | what |
|
||
|---|---|---|
|
||
| `ffprobe` | cpu | media inspection |
|
||
| `ffmpeg_frames` | cpu | video → frames (fps, mpdecimate dedupe, blur cull, max cap) |
|
||
| `blender_convert` | cpu | universal 3D format conversion via headless Blender |
|
||
| `colmap_poses` | cpu | frames → camera poses + sparse cloud (COLMAP 4.x + GLOMAP) |
|
||
| `brush_train` | gpu | colmap dataset → 3D gaussian splat (Brush, native Metal) |
|
||
| `sf3d` | gpu | image → GLB locally (Stable-Fast-3D, MPS) *[HF-gated]* |
|
||
| `trellis_mac` | gpu | image → GLB+PBR locally (TRELLIS.2 MPS port) *[HF-gated]* |
|
||
| `hunyuan3d_mlx` | gpu | image → GLB+PBR locally (Hunyuan3D 2.1, native MLX — runs on M1 too; weights public, no HF login) |
|
||
| `fal_trellis` / `fal_trellis2` / `fal_hunyuan3d` / `fal_hunyuan3d_v21` / `fal_rodin` | net | image → mesh via fal.ai API *[needs FAL_KEY]* |
|
||
| `fal_bg_remove` | net | image → subject cutout (BiRefNet v2) — run before any image→3D for a big quality jump |
|
||
| `fal_upscale` | net | image → faithful upscale (SeedVR) |
|
||
| `fal_image_edit` | net | image + instruction → edited image (nano-banana) |
|
||
| `fal_text_image` | net | prompt → image (Ideogram v3, readable text) — no input asset needed |
|
||
|
||
Recommended fal chain for best image→3D: `fal_bg_remove` → (`fal_upscale` if thin) → `fal_trellis2` or `fal_hunyuan3d_v21`. Note: Hunyuan v21 **multi-view** is broken on fal (verified 2026-07) — v21 is single-image only; use v2 for multi-view.
|
||
|
||
Heavy operators get their own uv venv (`"python": "<abs venv path>"` in the manifest). Remaining roadmap (object_capture, freemocap, retargeting, tripo/meshy character APIs, workflow presets, LLM copilot) is in [HANDOFF.md](HANDOFF.md) §5–8.
|