# MODELBEAST — Master Flow & Build Plan

*Written 2026-07-12. Every tool/claim below was verified against primary sources (repos, docs, pricing pages) by research agents on this date — including corrections to the Gemini advice that seeded this project.*

## Mission

A **local-first web app running on the M3 Ultra** (reachable over Tailscale from any device) that turns videos, images, and 3D files into meshes, splats, mocap, and rigged animated characters. Local Apple Silicon tools do the heavy lifting; cloud APIs (fal.ai, Tripo, Meshy) are optional quality/convenience tiers — not necessities. Full parameter control at every stage.

```
                        ┌─────────────────────────────────────────────┐
                        │        MODELBEAST WEB UI  (React + three.js)│
                        │  drag/drop/paste: mp4 png jpg glb obj fbx   │
                        │  blend usdz ply · workflow picker · param   │
                        │  forms · job dashboard · 3D preview · export│
                        └──────────────────┬──────────────────────────┘
                                           │ REST/WebSocket
                        ┌──────────────────▼──────────────────────────┐
                        │   FastAPI ORCHESTRATOR  (asset DB + job DAG)│
                        └──┬──────┬──────┬──────┬──────┬──────┬───────┘
                           │      │      │      │      │      │
                 ┌─────────▼─┐ ┌──▼───┐ ┌▼─────┐ ┌────▼───┐ ┌▼──────────┐
                 │ INGEST    │ │MESH  │ │SCAN  │ │ MOCAP  │ │ RIG/ANIM  │
                 │ ffmpeg    │ │GEN   │ │3DGS+ │ │        │ │           │
                 │ frames,   │ │      │ │photo-│ │        │ │           │
                 │ scenes,   │ │      │ │gram  │ │        │ │           │
                 │ sharpness │ │      │ │      │ │        │ │           │
                 └───────────┘ └──────┘ └──────┘ └────────┘ └───────────┘
                    each stage = swappable "operators", local or API
                                           │
                        ┌──────────────────▼──────────────────────────┐
                        │  CONVERT/EXPORT: Blender headless (universal│
                        │  GLB/FBX/OBJ/USD/BLEND) · archive → M4 Pro  │
                        └─────────────────────────────────────────────┘
```

---

## 1. Verified capability matrix (July 2026)

### Stage 1 — Video ingest & preprocessing (LOCAL, trivial)
- **ffmpeg 8.1.2** (installed): frame extraction (`fps=2..3` for orbits), `mpdecimate` to drop near-dupes, `scdet`/PySceneDetect for scene splits, I-frame selection.
- **Sharpness culling**: Laplacian-variance scoring per chunk; a few blurry frames can poison COLMAP matching. Target 150–300 frames/scene.

### Stage 2 — Mesh generation from image(s)
**Gemini said "local is impractical, use fal.ai" — REFUTED since early 2026.**

| Tier | Tool | Runs | Output | Notes |
|---|---|---|---|---|
| **Local SOTA** | **trellis-mac** (TRELLIS.2 MPS port) github.com/shivampkumar/trellis-mac | M3 Ultra, ~18GB peak, est. ~3–5 min/gen | GLB + baked PBR, ~400K verts | The headline local capability. MIT. ~15GB weights |
| Local fast draft | **SF3D / SPAR3D** (Stability, official MPS + Metal kernels) | seconds-fast | GLB, UV textures | Draft/iteration tier |
| Local 2nd opinion | **Hunyuan3D 2.1 Mac forks** (Brainkeys, SimonTingle ComfyUI-Mac) | shape solid on MPS; texture stage shaky w/o CUDA | GLB/OBJ | Best open textures when it works |
| API cheap | fal `fal-ai/trellis` | $0.02/gen | textured GLB | 2024-tier quality |
| API strong | fal `trellis-2` $0.25–0.35 · fal `hunyuan3d/v2` $0.16–0.48 · Hunyuan 3.1 Pro $0.375+ | — | GLB/OBJ, PBR | Zero-setup SOTA |
| **API character pipeline** | **Tripo v3 API** — the only gen→T-pose→quad retopo→auto-rig→animate chain | ~$0.55–0.75/rigged char | **FBX**, quads, rigged | The "finished game asset" button |
| API hero assets | Rodin Gen-2 via fal `hyper3d/rodin` $0.40 (HighPack $1.20) | — | GLB/USDZ/FBX, quad, 4K | Film-grade fidelity |

Also open + interesting: **Hunyuan3D-Omni** (pose-conditioned generation — make the character already posed), **Hunyuan3D-Part / PartCrafter** (part decomposition), TripoSG (open shape-only). TripoSR runs on Mac but is obsolete quality.

### Stage 3 — Scanning: video → splat / mesh (LOCAL, strong)
**Gemini's "Nerfstudio has strong Mac support" is wrong for splats** — gsplat/splatfacto is CUDA-only on Mac.

- **Poses**: COLMAP via Homebrew with the **GLOMAP global mapper** (now merged into COLMAP — much faster than incremental). Optional: colmap-metal fork for Metal SIFT.
- **Splat training**: **Brush** (native Metal, active, best UX, splat .ply out) · **OpenSplat** (`-DGPU_RUNTIME=MPS`, the proven fallback) · **msplat** (new pure-Metal trainer, reportedly 5–6× gsplat speed — watch it).
- **Meshes from reality**: **Apple Object Capture** (`PhotogrammetrySession`, GUI in Reality Composer Pro, or a tiny Swift CLI we compile) — native, fast, free, and the correct answer since all 3DGS→mesh research tools (SuGaR/2DGS) are CUDA-locked. macOS 26 shipped an improved reconstruction algorithm.
- Capture front-ends: Scaniverse (free splat PLY/SPZ + mesh export), Polycam (paywalled formats).

### Stage 4 — Motion capture from video
**Gemini's claims here were mostly wrong**: WHAM is CUDA-locked (name also garbled — it's "World-grounded Humans with Accurate Motion"); EasyMocap has zero Mac support; **ActionMesh is real** (Meta CVPR 2026) but CUDA-only *and not mocap* (rig-free animated mesh, no skeleton).

| Tier | Tool | Runs | Output |
|---|---|---|---|
| **Filming (face)** | **UE 5.8 MetaHuman Animator on Mac** — facial solve (offline + realtime from iPhone/webcam) is officially supported on macOS in 5.8; M3 Ultra exceeds Epic's recommended M2 Ultra tier | UE 5.8 (installed) | MetaHuman facial anim → AnimSequence/FBX |
| **Filming (body) — VERDICT** | **UE MetaHuman Markerless Mocap plugin is WINDOWS-ONLY** (Epic docs: "available only on Windows for body or face and body capture"; Fab listing gates to Windows). Fully local, no API key, DX12-class GPU (AMD ok, no CUDA), ~1 min solve per second of 1080p60 footage, hands/fingers included, single performer, tripod required. **Mac workaround: rent a Windows GPU VM** (AWS g5/Paperspace/Shadow, DX12 + 8GB VRAM) only for solves, export FBX ("Performer Skeleton" option = retarget to non-MetaHuman rigs), pull back to Mac. Meshcapade's own cloud (meshcapade.me) was SHUT DOWN Apr 2026 post-acquisition — this plugin is the only outlet for the tech | Windows only | AnimSequence → FBX |
| Local turnkey | **FreeMoCap v1.8** (`pip install freemocap`, official Mac support) + its Blender add-on | native | rigged armature → FBX/BVH. Best with 2–3 cameras |
| Hybrid best-quality | **GVHMR** on Colab/HF-Space or rented RTX (~$0.30/hr) → SMPL import via CEB/Meshcapade Blender add-ons | cloud solve, local everything else | world-grounded SMPL |
| Local previz | MediaPipe/BlendArMocap | native, realtime | rough |
| API | Move One $18/mo · DeepMotion $17/mo · Autodesk Flow Studio (ex-Wonder Studio) $10–45/mo · Rokoko Video (free ≤15s FBX) | — | FBX/BVH |

Not Mac-runnable locally (CUDA-locked): WHAM, TRAM, EasyMocap, ActionMesh, GEM-X/GENMO, DuoMo.

### Stage 5 — Rigging & retargeting (LOCAL, free)
- **Mesh2Motion** (MIT, runs locally via npm/Docker): template-skeleton rigging (human/quadruped/bird) + animation library, GLB in/out. *Semi-automatic* — you position the skeleton, it auto-skins.
- **Mixamo**: still free, still works from a browser, but frozen (bipeds only).
- **Blender 5.1.2** (installed, + BlenderMCP): Rigify baseline · **Rokoko plugin** (free retargeting) · **Retarget extension** (Expy Kit successor, Blender 5+) · keemap for exotic rigs · SMPL/SMPL-X add-ons for mocap import.
- AI auto-rig: **UniRig** (open weights, CUDA-only → cloud job) · **Tripo rig API** (25cr) · **Meshy rig API** (5cr, cheapest).

### Stage 6 — LLM brain (LOCAL, already proven)
- Ollama with **qwen3:235b already pulled**. Mid-2026 pattern: small fast MoE (Qwen3.6-35B-A3B class) for pipeline logic/tool-calling + the big model on demand. LM Studio/MLX when MLX outruns Ollama.
- Used inside the app for: workflow suggestion, prompt-writing for gen models, batch naming/tagging, "explain this failure".

---

## 2. The MODELBEAST app

### Architecture decisions (defaults chosen, all swappable)
| Concern | Choice | Why |
|---|---|---|
| Backend | **FastAPI** (Python 3.12 via `uv`) | All ML tooling is Python; async job streaming via WebSocket |
| Tool isolation | **each operator = subprocess in its own uv venv** | These repos have violently conflicting deps (torch pins, numpy, python versions). Never share an env. Homebrew Python 3.14 is too new for ML wheels — per-tool venvs pin 3.10–3.12 |
| DB | SQLite (WAL) | assets, jobs, workflow runs, provenance. Zero ops |
| Job model | DAG of operator invocations, asyncio worker pool, one GPU-heavy job at a time (Metal), N CPU jobs parallel | 256GB lets several stay resident |
| Frontend | React + Vite + Tailwind + three.js | drag/drop/paste ingest, GLB/splat preview, param forms |
| Param forms | **auto-generated from operator JSON-Schema manifests** | "FULL CONTROL" without hand-building UI per tool — every flag a tool exposes becomes a form field with sane defaults |
| Conversion | **Blender headless** (`blender -b -P convert.py`) as universal GLB↔FBX↔OBJ↔USD↔BLEND node + trimesh for light ops | FBX especially needs Blender |
| Storage | `data/{inbox,assets,jobs,outputs}` content-addressed; archive operator → **M4 Pro over Tailscale (100.69.21.128, 4TB)** via rsync | hot on local/external SSD, cold on M4 Pro |
| Access | bind 0.0.0.0, reach via Tailscale from any device | drive the beast from phone/laptop |
| API keys | `.env` + settings page (fal, Tripo, Meshy, Replicate) | cloud operators light up when keys exist |

### The operator concept (the heart of "full control")
Every capability — local or cloud — is one **operator**: a folder with `manifest.json` (name, input types, output types, JSON-Schema params, venv spec, resource class) and a `run.py`/shell entry. The UI reads manifests to render workflow choices and parameter forms; the DAG runner chains them by matching input/output types. Adding a new model = dropping in a new operator folder. Presets (saved DAGs + params) give one-click workflows:

- **"Video → Rigged Character"**: ffmpeg best-frame → trellis-mac (or Tripo API) → Mesh2Motion/Tripo-rig → retarget mocap → FBX
- **"Video → Splat"**: ffmpeg → COLMAP/GLOMAP → Brush → .ply/.spz
- **"Video → Scanned Mesh"**: ffmpeg → Object Capture → GLB/USDZ
- **"Video → Mocap"**: FreeMoCap (or UE MetaHuman externally) → BVH/FBX
- **"Image → Mesh ×N"**: fan the same image across trellis-mac + SF3D + Hunyuan + fal APIs, compare side-by-side in the viewer, keep the winner ← *the 256GB flex: run several models resident simultaneously*

### Format I/O matrix (via Blender headless + native tool outputs)
In: mp4/mov, png/jpg/webp/heic, glb/gltf, obj, fbx, blend, usd(z), ply/spz.
Out: glb, fbx, obj, usd(z), blend, ply/spz, bvh, abc, png sequences/turntable mp4 renders.

---

## 3. Build roadmap

- **Phase 0 — Skeleton (now)**: FastAPI + SQLite + asset ingest (upload/paste) + job queue + React UI with asset library, three.js GLB viewer, job dashboard. Operators: `ffprobe`, `ffmpeg-frames`, `blender-convert`. *Proves the whole loop: drop file → run op → preview → export.*
- **Phase 1 — First real generators**: `sf3d` (fast local), `fal-trellis` + `fal-trellis2` (API, trivial), `trellis-mac` (the local flagship — expect real install debugging). Side-by-side compare view.
- **Phase 2 — Scan track**: COLMAP+GLOMAP operator, Brush operator, Object Capture Swift CLI operator. Splat viewer in UI.
- **Phase 3 — Motion track**: FreeMoCap operator, UE MetaHuman handoff (watch folder → ingest FBX back), GVHMR cloud-solve helper, Blender retarget operator (Rokoko/Retarget).
- **Phase 4 — Character factory**: Tripo/Meshy full API operators (rig+animate), Mesh2Motion embed/bridge, Hunyuan3D-Omni posed-gen, LLM copilot pane (Ollama), archive-to-M4Pro operator, batch/watch-folder mode.

## 4. Open items
- ~~UE MetaHuman markerless Mac verdict~~ **RESOLVED: Windows-only** (see Stage 4). Experimental plugin — Epic's pattern (Creator→Mac in 5.7, Animator facial→Mac in 5.8) suggests body could follow in 5.9+, but no commitment exists. Watch the Fab listing. Filming workflow for now: film 60fps HD/4K on tripod, single performer, form-fitting clothing; face-solve on Mac, body-solve on rented Windows GPU VM (batch design = pay only for solve hours).
- M3 Ultra benchmarks for trellis-mac don't exist publicly yet (only M4 Pro ~3.5–5 min) — we'll be among the first; measure and record in this doc.
- Disk: ~400GB free locally; weights alone (Trellis 15GB, Hunyuan ~50GB, Qwen 142GB…) justify the planned external SSD; M4 Pro is the archive tier.

## 5. Key sources
trellis-mac github.com/shivampkumar/trellis-mac · TRELLIS.2 github.com/microsoft/TRELLIS.2 · SF3D github.com/Stability-AI/stable-fast-3d · Hunyuan3D 2.1 github.com/tencent-hunyuan/hunyuan3d-2.1 · Tripo pricing docs.tripo3d.ai/get-started/pricing.html · fal 3D fal.ai/3d-models · Brush github.com/ArthurBrussee/brush · OpenSplat github.com/pierotofy/OpenSplat · msplat github.com/rayanht/msplat · Object Capture developer.apple.com/documentation/RealityKit/creating-a-photogrammetry-command-line-app · FreeMoCap github.com/freemocap/freemocap · GVHMR github.com/zju3dv/GVHMR · Mesh2Motion github.com/Mesh2Motion/mesh2motion-app · UniRig github.com/VAST-AI-Research/UniRig · CEB_4d_Humans github.com/carlosedubarreto/CEB_4d_Humans · Meshcapade SMPL addon github.com/Meshcapade/SMPL_blender_addon