172 lines
21 KiB
Markdown
172 lines
21 KiB
Markdown
# MODELBEAST — Build Handoff Brief
|
||
|
||
**Audience:** the AI agent (Opus 4.8 / Claude Code) continuing this build. Read this file top to bottom, then read `PLAN.md`, then start at Phase 1. Do not re-research what is already verified here — every tool claim in this doc and PLAN.md was verified against primary sources on 2026-07-12 by research agents.
|
||
|
||
---
|
||
|
||
## 0. Mission and definition of done
|
||
|
||
MODELBEAST is a **local-first web app** on an M3 Ultra Mac Studio (256GB unified memory) that turns videos/images/3D files into meshes, splats, mocap, and rigged animated characters. Local Apple Silicon tools do the heavy lifting; cloud APIs (fal.ai, Tripo, Meshy) are optional quality tiers. The owner's priority is **capability and quality, not cost savings**. He films things, drops them in, and wants finished 3D assets out — with full parameter control at every step.
|
||
|
||
**Done means:** all four phases below shipped and each operator proven end-to-end through the real browser UI (not just curl), with PLAN.md/README.md kept current.
|
||
|
||
## 1. Current state (Phase 0 — shipped and verified 2026-07-12)
|
||
|
||
Working today:
|
||
- FastAPI backend (`server/`), SQLite (`data/modelbeast.db`), job runner executing operators as subprocesses with streamed logs over WebSocket.
|
||
- React + Vite + three.js frontend (`web/`), served from `web/dist` by FastAPI. Drag/drop/paste ingest, asset library with kind detection, GLB/OBJ/FBX/PLY/STL viewer (GLTFLoader etc. + OrbitControls + AnimationMixer), video/image/frames-grid/JSON previews, param forms auto-generated from operator JSON Schemas, live job dashboard.
|
||
- Operators, all tested: `ffprobe` (inspect), `ffmpeg_frames` (fps sampling + mpdecimate dedupe + blur cull + max cap), `blender_convert` (universal format conversion via headless Blender).
|
||
|
||
Run it:
|
||
```bash
|
||
/opt/homebrew/bin/uv run uvicorn server.main:app --host 0.0.0.0 --port 8777
|
||
```
|
||
(`.claude/launch.json` has this config, name `modelbeast`.) Frontend rebuild after editing `web/src`: `cd web && npm run build`.
|
||
|
||
File map:
|
||
```
|
||
server/main.py FastAPI routes: /api/assets, /api/jobs, /api/operators, /ws, static mount
|
||
server/db.py sqlite schema (assets, jobs), connect() — check_same_thread=False (required!)
|
||
server/store.py asset store: data/assets/<id>/<name>, kind inference by extension
|
||
server/registry.py scans server/operators/*/manifest.json
|
||
server/runner.py queue + subprocess execution + output registration (result.json contract)
|
||
server/operators/<id>/{manifest.json, run.py}
|
||
web/src/{App.jsx, Viewer.jsx, api.js, index.css}
|
||
data/{assets,jobs,inbox,outputs} (gitignore this whole dir)
|
||
PLAN.md verified tool matrix + roadmap — treat as source of truth
|
||
```
|
||
|
||
**Operator contract** (do not break it; extend it backward-compatibly):
|
||
- Runner invokes: `<python> run.py --input <asset path> --outdir <job dir> --params '<json>'`.
|
||
- Operator writes outputs into outdir. Optional `result.json`: `{"outputs": [{"path": ..., "name": ..., "meta": {...}}], "summary": {...}}`. Without it, every top-level file in outdir is registered as an asset. stdout/stderr = job log.
|
||
- `manifest.json` fields: `id, name, category, description, accepts (asset kinds), produces, entry, params_schema (JSON Schema), python (optional absolute venv python — omit to use the server's venv)`.
|
||
|
||
## 2. Environment facts (do not rediscover)
|
||
|
||
- Machine: M3 Ultra, 256GB unified memory, macOS (Darwin 25.x). ~400GB local disk free — model weights are large; prefer `~/models/` or an external SSD if the owner has attached one by now (ask via a settings field, don't block on it).
|
||
- Secondary machine: M4 Pro, 4TB SSD, Tailscale `100.69.21.128` (~20–80ms RTT). Cold storage/archive only — not a hot working drive.
|
||
- Installed: ffmpeg 8.1.2, Blender 5.1.2 at `/Applications/Blender.app/Contents/MacOS/Blender`, Python 3.14 (Homebrew — TOO NEW for ML wheels, never use it for tool venvs), `uv`, node/npm, Ollama (`qwen3:235b`, `qwen3:32b`, `qwen3:8b` pulled), Epic Games Launcher + UE 5.8 at `/Users/Shared/Epic Games/UE_5.8`. BlenderMCP configured in Blender.
|
||
- NOT installed yet: COLMAP, ComfyUI, Xcode (check `xcode-select -p`; Object Capture CLI needs full Xcode or at least Swift toolchain — verify).
|
||
- Not a git repo yet. **First action: `git init`, write `.gitignore` (`data/`, `web/node_modules/`, `web/dist/`, `.venv/`, `*.env`, `venvs/`, `models/`), commit Phase 0 as baseline.** Commit at every working milestone.
|
||
|
||
**Gotchas already hit (don't repeat):**
|
||
1. `sqlite3.connect` needs `check_same_thread=False` — FastAPI sync endpoints run in a threadpool.
|
||
2. Embedding params into generated Blender python: JSON `true/false/null` are not Python — use `params = json.loads(<double json.dumps>)`.
|
||
3. `.claude/launch.json` must use absolute `/opt/homebrew/bin/uv` (GUI-spawned processes don't get Homebrew PATH).
|
||
4. Blender 5.x: several third-party add-ons (SMPL-X, Rokoko) lag on 5.x compatibility — test add-ons headless before building operators around them; fall back to driving Blender 4.x LTS via a second install if an add-on hard-fails (ask owner before installing).
|
||
|
||
## 3. Non-negotiable working rules
|
||
|
||
1. **One tool = one operator = one isolated uv venv.** ML repos have violently conflicting deps. Create venvs under `venvs/<tool>/` with `uv venv --python 3.11` (or the version the repo pins; 3.10–3.12, never 3.14), install with `uv pip install --python venvs/<tool>/bin/python`. Set `"python": "<abs path>"` in the manifest.
|
||
2. **Verify every operator end-to-end through the browser UI** before moving on: upload real input → run with non-default params → watch log stream → confirm output asset registers and previews. Then commit.
|
||
3. **Never block the event loop.** All heavy work stays in subprocesses. If an operator needs a long-lived model server (e.g. trellis-mac warm weights), run it as a separate process the operator talks to — see §5.3.
|
||
4. **Fail loud, log everything.** Operators print progress lines (they stream to the UI). Exit non-zero on failure. Never swallow stderr.
|
||
5. **Don't gold-plate the framework.** No Celery, no Redis, no Postgres, no node-graph editor yet. The SQLite + subprocess architecture is deliberate; extend it only as specified in §4.
|
||
6. **CUDA-locked tools stay off this machine.** Do not attempt local installs of: TRELLIS v1 official, WHAM, TRAM, EasyMocap, ActionMesh, DuoMo, GEM-X, UniRig, SuGaR/2DGS, ComfyUI-3D-Pack, gsplat/splatfacto. PLAN.md documents why. Cloud/API is the path for those.
|
||
7. Persistent memory for this project lives at `~/.claude/projects/-Users-m3ultra-Documents-MODELBEAST/memory/` — read `MEMORY.md` there at session start; keep it updated as you learn things.
|
||
|
||
## 4. Framework upgrades (build these as they become needed, not all upfront)
|
||
|
||
### 4.1 Settings + API keys (needed by Phase 1)
|
||
- New table `settings (key TEXT PRIMARY KEY, value TEXT)`. Endpoints `GET/PUT /api/settings` (PUT accepts partial dict). Keys: `fal_key`, `tripo_key`, `meshy_key`, `replicate_token`, `models_dir`, `archive_host` (default `m4pro@100.69.21.128`), `archive_path`.
|
||
- Runner injects settings as env vars (`FAL_KEY`, `TRIPO_KEY`, …) into operator subprocesses. Secrets never go into params/logs — redact anything matching a key value from job logs before storing.
|
||
- Frontend: a Settings modal (gear icon in header). Password-type inputs. Show which operators are enabled/disabled by missing keys: add `"requires_env": ["FAL_KEY"]` to manifests; UI greys out operators whose requirements are unset.
|
||
|
||
### 4.2 Multi-input jobs (needed by Phase 3 retargeting)
|
||
- Extend jobs: `asset_ids` JSON array (keep `asset_id` working for single-input). Runner passes `--input <path>` repeated, in order; manifest declares `"inputs": [{"name": "mesh", "accepts": ["model"]}, {"name": "motion", "accepts": ["model"]}]` for multi-input ops. UI: when such an operator is selected, render one asset-picker per declared input.
|
||
|
||
### 4.3 Queue lanes (needed once heavy local ops exist, Phase 1)
|
||
- Manifest gains `"resources": "gpu" | "cpu" | "net"` (default `cpu`). Runner keeps three lanes: `gpu` concurrency 1 (Metal contention), `cpu` concurrency 3, `net` concurrency 6. Simple dict of semaphores; no framework.
|
||
|
||
### 4.4 Job management (Phase 1 polish)
|
||
- `POST /api/jobs/{id}/cancel` (kill process group — spawn with `start_new_session=True`), `DELETE /api/jobs/{id}` (remove row + job dir). UI: cancel button on running jobs, retry button on failed jobs (re-POST same operator/asset/params), collapsible live log (already exists; make it auto-scroll).
|
||
|
||
### 4.5 Workflow presets (Phase 2+)
|
||
- `server/workflows/<id>.json`: `{"id", "name", "description", "steps": [{"operator", "params", "input": "$source" | "$step0.output0"}]}`. Sequential executor in runner (a workflow run = chain of jobs; each step's registered assets feed the next). `GET /api/workflows`, `POST /api/workflows/{id}/run {asset_id, param_overrides}`. UI: "Workflows" section above operators — pick asset, pick workflow, override params, run; show step progress.
|
||
- Ship presets: `video_to_splat`, `video_to_scanned_mesh`, `image_to_mesh_compare`, `image_to_rigged_character` (as its operators land).
|
||
|
||
### 4.6 Compare view (Phase 1)
|
||
- The 256GB flex: fan one image across N mesh generators. Implement as a workflow that launches parallel jobs + a frontend "Compare" screen: grid of three.js viewers, one per output of the selected jobs, with operator name + wall-clock + file size captions. Keep it simple: a `?compare=jobId1,jobId2,…` route.
|
||
|
||
## 5. PHASE 1 — Mesh generation (local + API)
|
||
|
||
### 5.1 `fal_trellis`, `fal_trellis2`, `fal_hunyuan3d`, `fal_rodin` (API operators — build FIRST, they validate the whole pipeline cheaply)
|
||
- One shared helper module `server/operators/_lib/fal_helper.py` (operators may import via relative path or sys.path insert; keep it dependency-light). Use the `fal-client` pip package in the **server's own venv** (it's tiny and conflict-free): upload input image with `fal_client.upload_file`, `fal_client.subscribe(endpoint, arguments=...)`, poll with progress prints, download resulting GLB into outdir.
|
||
- Endpoints (verify exact IDs against https://fal.ai/3d-models before coding; prices verified 2026-07-12): `fal-ai/trellis` ($0.02), `fal-ai/trellis-2` ($0.25–0.35 by resolution), `fal-ai/hunyuan3d/v2` ($0.16 untextured / $0.48 textured), `fal-ai/hyper3d/rodin` ($0.40; HighPack $1.20), `tripo3d/tripo/v2.5` ($0.20–0.40).
|
||
- Params to surface (per model, from fal docs): resolution/steps/seed/texture toggles — expose everything fal accepts; that's the product's point. `accepts: ["image"]`, `produces: ["model"]`, `resources: "net"`, `requires_env: ["FAL_KEY"]`.
|
||
- Acceptance: upload a photo in the UI → run fal_trellis → GLB appears and renders in the viewer. Test with a real key from Settings.
|
||
|
||
### 5.2 `sf3d` (local fast draft — first local ML operator, proves the venv pattern)
|
||
- Repo: https://github.com/Stability-AI/stable-fast-3d — **official MPS support** (needs macOS 15.2+; on 256GB ignore their low-RAM CPU advice). License: Stability Community (free under $1M revenue — fine).
|
||
- `venvs/sf3d` with the Python the repo pins (likely 3.10/3.11). Follow their README install; HF weights need `huggingface-cli login` OR accept-terms — if weights are gated, print a clear job-log error telling the owner what to do, and add `hf_token` to settings.
|
||
- Wrap their `run.py` (single image → GLB with UV textures). Params: `texture_resolution`, `remesh` options if exposed. `resources: "gpu"`.
|
||
- Acceptance: same photo → mesh in seconds-to-a-minute on MPS; renders in viewer.
|
||
|
||
### 5.3 `trellis_mac` (local flagship — hardest install of Phase 1; budget real debugging time)
|
||
- Repo: https://github.com/shivampkumar/trellis-mac (MIT; PyTorch-MPS port of Microsoft TRELLIS.2 with Metal kernels: mtlgemm, mtldiffrast). ~15GB weights. Benchmarked M4 Pro 24GB: ~5min cold / ~3m20s warm, 18GB peak. **No M3 Ultra numbers exist — measure and record yours in PLAN.md §4.**
|
||
- Expect compile steps for the Metal extensions (needs Xcode CLT — present if Homebrew works). Known limits: hole-filling disabled, sparse attention ~80s bottleneck. If the repo has moved/broken, fallback: https://github.com/pedronaugusto/trellis2-apple (MLX backend).
|
||
- Because cold-start dominates, implement **warm mode**: the operator's run.py checks for a running trellis daemon (a tiny HTTP wrapper you write, listening on 127.0.0.1:8801, loaded once); if absent, either start it and wait, or run one-shot. A `keep_warm` boolean param + a settings toggle. Don't over-engineer: single-endpoint Flask/FastAPI file inside the trellis venv.
|
||
- Output: GLB with baked PBR. Params: resolution tier, texture size (512–2048), seed, steps.
|
||
- Acceptance: same photo through trellis_mac vs sf3d vs fal_trellis2 in the Compare view; record timings.
|
||
|
||
### 5.4 (stretch) `hunyuan3d_mac` — shape-only on MPS via https://github.com/Maxim-Lanskoy/Hunyuan3D-2-Mac or the ComfyUI-free fork Brainkeys/Hunyuan3D-2.1-mac (unverified — inspect before trusting). Texture stage is CUDA-locked in the official repo; ship shape-only and note it. Skip entirely if the forks are dead — fal_hunyuan3d covers it.
|
||
|
||
## 6. PHASE 2 — Scan track (video → splat / mesh)
|
||
|
||
### 6.1 `colmap_poses`
|
||
- `brew install colmap`. GLOMAP is merged into COLMAP as the global mapper — verify the subcommand in `colmap help` (expect `colmap global_mapper` or a `glomap` binary; the standalone repo is deprecated).
|
||
- Input: `frames` asset. Pipeline inside run.py: feature extraction (SIFT, CPU on Mac — fine) → matching (sequential matcher for video) → global mapper → export as a **nerfstudio/colmap-format dataset folder** (`produces: ["colmap_dataset"]` — add this asset kind). Params: matcher type, downscale, camera model, quality preset.
|
||
- Acceptance: 150–300 frame orbit video of an object (ask owner to film one, or use any test video; even a phone orbit of a chair works) → sparse reconstruction completes; log reports registered-image count.
|
||
|
||
### 6.2 `brush_train` (3DGS)
|
||
- https://github.com/ArthurBrussee/brush — native Metal, has a full CLI. Prefer a release binary; else `cargo build` (install rust via brew). Input: `colmap_dataset`. Output: splat `.ply` (`produces: ["splat"]` — new kind for `.ply`/`.spz`/`.splat`; note plain mesh PLY vs splat PLY ambiguity — tag via meta, don't fight extension detection).
|
||
- Params: iterations (7k/30k presets), sh-degree, max splats. `resources: "gpu"`.
|
||
- Frontend: add a splat viewer — npm `@mkkellogg/gaussian-splats-3d` (three.js-based) renders splat PLYs; wire into Viewer.jsx by kind.
|
||
- Fallback trainer if Brush misbehaves: OpenSplat (`cmake -DGPU_RUNTIME=MPS`). Speed watch: https://github.com/rayanht/msplat (pure-Metal, young).
|
||
|
||
### 6.3 `object_capture` (video/photos → textured mesh — the quality mesh path)
|
||
- Apple RealityKit `PhotogrammetrySession`. Build a small Swift CLI (Apple sample: "creating a photogrammetry command line app" in developer docs) — needs Xcode; if only CLT is present, ask the owner to install Xcode (one-time). Compile once into `bin/objcap`, commit the Swift source, gitignore the binary.
|
||
- Input: `frames` (or image folder). Output: USDZ + also run `blender_convert` logic to emit GLB. Params: detail level (reduced/medium/full/raw). macOS 26 shipped improved reconstruction — no code changes needed.
|
||
- Acceptance: same orbit frames through object_capture vs brush_train; mesh renders textured in viewer.
|
||
|
||
## 7. PHASE 3 — Motion track
|
||
|
||
### 7.1 `freemocap`
|
||
- `venvs/freemocap`, Python 3.10–3.12, `pip install freemocap` (v1.8+, official Mac support, AGPL — fine for internal tooling). Drive it headless (it has a CLI/python API; inspect `freemocap --help`). Input: video (monocular OK but approximate; multi-cam is its strength). Output: 3D keypoint data + Blender export via `freemocap_blender_addon` — final deliverable should be an FBX/BVH the retarget step can consume. If the Blender add-on fights Blender 5.1.2, run its export through Blender with `--factory-startup` and the add-on zip, or fall back to emitting raw keypoints + a note.
|
||
- Set expectations in the operator description: "previz-grade from single camera; research-grade needs GVHMR (cloud)."
|
||
|
||
### 7.2 `gvhmr_import` (best-quality mocap, hybrid)
|
||
- GVHMR itself is CUDA — do NOT install locally. Build the import half: accept a GVHMR results file (the owner runs the Colab/HF Space manually at first) and convert SMPL params → FBX/BVH. Tools: https://github.com/softcat477/SMPL-to-FBX or Meshcapade's SMPL Blender add-on (verify Blender 5 compat; MPI SMPL-X add-on requires free registration — flag to owner).
|
||
- Stretch: a `gvhmr_cloud` operator that submits to a rented GPU (RunPod API) — only if the owner provides a RunPod key; otherwise document the manual Colab flow in the operator description.
|
||
|
||
### 7.3 `blender_retarget` (multi-input: character model + motion)
|
||
- Uses §4.2 multi-input. Headless Blender: import rigged character (FBX/GLB) + BVH/FBX motion → retarget → export FBX/GLB with baked animation.
|
||
- Retargeting add-on order of preference: the "Retarget" extension from extensions.blender.org (Expy Kit successor, Blender 5+ native — best bet), Rokoko plugin (free, may lag Blender 5), keemap (manual mapping files, bulletproof fallback). Prototype interactively in Blender first (BlenderMCP is available), then script it headless. This is the fiddliest operator of the project — bone-name mapping presets (Mixamo, Rigify, SMPL, MetaHuman) should be a param enum with a JSON mapping file each.
|
||
|
||
### 7.4 UE MetaHuman bridge (thin, don't overbuild)
|
||
- The UE 5.8 markerless BODY plugin is **Windows-only** (verified; PLAN.md §Stage 4). Mac gets facial solve. Bridge = a watch folder: `data/inbox/` — a `watchfiles` task in the server auto-ingests any file dropped there (this is generically useful anyway). Owner exports FBX from UE (local Mac facial solves, or body solves from a Windows GPU VM) into that folder → lands as asset → feed `blender_retarget`.
|
||
|
||
## 8. PHASE 4 — Character factory + polish
|
||
|
||
- **`tripo_character` (API)**: the only end-to-end gen→T-pose→quad-retopo→auto-rig→animate pipeline (~$0.55–0.75/character, FBX out). API docs: docs.tripo3d.ai. Multi-step async API — poll task states, print progress. Params: all of it (T-pose toggle, quad, texture quality, rig, animation preset ids). This operator is the "money button" — make it robust.
|
||
- **`meshy_rig` (API)**: cheapest rig+animate loop (rig 5cr, anim 3cr) for meshes generated elsewhere. docs.meshy.ai.
|
||
- **Mesh2Motion bridge**: it's a local web app (MIT): `git clone https://github.com/Mesh2Motion/mesh2motion-app && npm i && npm run dev`, semi-automatic (user positions template skeleton). Don't automate it — add a "Open in Mesh2Motion" button in the asset detail view that copies the GLB path / opens the local app URL, and rely on the watch folder for the return trip.
|
||
- **`archive_to_m4pro`**: rsync over ssh to `settings.archive_host:archive_path`, `--partial --progress`, verify with size/count, then optionally delete local. Requires the owner to have ssh keys set up — if `ssh -o BatchMode=yes <host> true` fails, print exact instructions in the job log and exit 1. Never delete local data unless `delete_local: true` param was explicitly set.
|
||
- **LLM copilot pane** (optional, do last): chat sidebar hitting Ollama (`http://localhost:11434/api/chat`, model `qwen3:32b` default — NOT the 235b for interactive use). System prompt embeds the operator manifest list; the model suggests operator + params as JSON; a button applies the suggestion to the form. No agentic execution — suggestions only.
|
||
- **Thumbnails**: background task renders model thumbnails via headless Blender (small turntable PNG) for the asset list. Nice-to-have.
|
||
|
||
## 9. Testing & quality bar
|
||
|
||
- `tests/smoke.sh`: starts server on a scratch port with `MODELBEAST_DATA=<tmpdir>` (add this env override to `db.py` — small change), uploads fixtures (generate with ffmpeg testsrc2 + a Blender-exported GLB, as Phase 0 did), runs ffprobe + ffmpeg_frames + blender_convert via curl, asserts statuses `done` and output assets exist. Run before every commit.
|
||
- Each new operator: at minimum one real end-to-end browser run (screenshot-verified if you have browser tooling) + smoke-test coverage where inputs can be synthesized.
|
||
- Failure modes to test deliberately: missing API key (operator greyed out / clean error), cancel mid-run (process actually dies — check with `ps`), server restart with a running job (job re-queues; already implemented — keep it working), garbage input file (clean error, not a hung job).
|
||
- Performance discipline: record wall-clock + peak memory (`/usr/bin/time -l`) for every local ML operator's first successful run in a `BENCHMARKS.md` table. These are the first public M3 Ultra numbers for several of these tools — the owner cares.
|
||
|
||
## 10. Priority order & scope guard
|
||
|
||
Build order: **git init/baseline → 4.1 settings → 5.1 fal ops → 4.6 compare → 5.2 sf3d → 4.3 lanes + 4.4 job mgmt → 5.3 trellis_mac → 6.1 colmap → 6.2 brush + splat viewer → 6.3 object_capture → 4.5 workflows + presets → 7.1 freemocap → 4.2 multi-input → 7.3 retarget → 7.2 gvhmr_import → 7.4 watch folder → Phase 4 items by owner preference (ask which he wants first — tripo_character is the likely answer).**
|
||
|
||
If something is broken upstream (repo moved, weights gated, API renamed): timebox to ~30 minutes of fixing, then ship the operator in a clearly-disabled state (manifest `"status": "blocked"` + reason shown in UI) and move on. Momentum beats completeness; the owner would rather have 12 working operators and 2 blocked ones than 6 perfect ones.
|
||
|
||
Keep PLAN.md, README.md, and this file's checkboxes updated as you go. Update project memory with anything a future session must know (new gotchas, benchmark numbers, changed decisions).
|