139 lines
14 KiB
Markdown
139 lines
14 KiB
Markdown
# MODELBEAST — Master Flow & Build Plan
|
||
|
||
*Written 2026-07-12. Every tool/claim below was verified against primary sources (repos, docs, pricing pages) by research agents on this date — including corrections to the Gemini advice that seeded this project.*
|
||
|
||
## Mission
|
||
|
||
A **local-first web app running on the M3 Ultra** (reachable over Tailscale from any device) that turns videos, images, and 3D files into meshes, splats, mocap, and rigged animated characters. Local Apple Silicon tools do the heavy lifting; cloud APIs (fal.ai, Tripo, Meshy) are optional quality/convenience tiers — not necessities. Full parameter control at every stage.
|
||
|
||
```
|
||
┌─────────────────────────────────────────────┐
|
||
│ MODELBEAST WEB UI (React + three.js)│
|
||
│ drag/drop/paste: mp4 png jpg glb obj fbx │
|
||
│ blend usdz ply · workflow picker · param │
|
||
│ forms · job dashboard · 3D preview · export│
|
||
└──────────────────┬──────────────────────────┘
|
||
│ REST/WebSocket
|
||
┌──────────────────▼──────────────────────────┐
|
||
│ FastAPI ORCHESTRATOR (asset DB + job DAG)│
|
||
└──┬──────┬──────┬──────┬──────┬──────┬───────┘
|
||
│ │ │ │ │ │
|
||
┌─────────▼─┐ ┌──▼───┐ ┌▼─────┐ ┌────▼───┐ ┌▼──────────┐
|
||
│ INGEST │ │MESH │ │SCAN │ │ MOCAP │ │ RIG/ANIM │
|
||
│ ffmpeg │ │GEN │ │3DGS+ │ │ │ │ │
|
||
│ frames, │ │ │ │photo-│ │ │ │ │
|
||
│ scenes, │ │ │ │gram │ │ │ │ │
|
||
│ sharpness │ │ │ │ │ │ │ │ │
|
||
└───────────┘ └──────┘ └──────┘ └────────┘ └───────────┘
|
||
each stage = swappable "operators", local or API
|
||
│
|
||
┌──────────────────▼──────────────────────────┐
|
||
│ CONVERT/EXPORT: Blender headless (universal│
|
||
│ GLB/FBX/OBJ/USD/BLEND) · archive → M4 Pro │
|
||
└─────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## 1. Verified capability matrix (July 2026)
|
||
|
||
### Stage 1 — Video ingest & preprocessing (LOCAL, trivial)
|
||
- **ffmpeg 8.1.2** (installed): frame extraction (`fps=2..3` for orbits), `mpdecimate` to drop near-dupes, `scdet`/PySceneDetect for scene splits, I-frame selection.
|
||
- **Sharpness culling**: Laplacian-variance scoring per chunk; a few blurry frames can poison COLMAP matching. Target 150–300 frames/scene.
|
||
|
||
### Stage 2 — Mesh generation from image(s)
|
||
**Gemini said "local is impractical, use fal.ai" — REFUTED since early 2026.**
|
||
|
||
| Tier | Tool | Runs | Output | Notes |
|
||
|---|---|---|---|---|
|
||
| **Local SOTA** | **trellis-mac** (TRELLIS.2 MPS port) github.com/shivampkumar/trellis-mac | M3 Ultra, ~18GB peak, est. ~3–5 min/gen | GLB + baked PBR, ~400K verts | The headline local capability. MIT. ~15GB weights |
|
||
| Local fast draft | **SF3D / SPAR3D** (Stability, official MPS + Metal kernels) | seconds-fast | GLB, UV textures | Draft/iteration tier |
|
||
| Local 2nd opinion | **Hunyuan3D 2.1 Mac forks** (Brainkeys, SimonTingle ComfyUI-Mac) | shape solid on MPS; texture stage shaky w/o CUDA | GLB/OBJ | Best open textures when it works |
|
||
| API cheap | fal `fal-ai/trellis` | $0.02/gen | textured GLB | 2024-tier quality |
|
||
| API strong | fal `trellis-2` $0.25–0.35 · fal `hunyuan3d/v2` $0.16–0.48 · Hunyuan 3.1 Pro $0.375+ | — | GLB/OBJ, PBR | Zero-setup SOTA |
|
||
| **API character pipeline** | **Tripo v3 API** — the only gen→T-pose→quad retopo→auto-rig→animate chain | ~$0.55–0.75/rigged char | **FBX**, quads, rigged | The "finished game asset" button |
|
||
| API hero assets | Rodin Gen-2 via fal `hyper3d/rodin` $0.40 (HighPack $1.20) | — | GLB/USDZ/FBX, quad, 4K | Film-grade fidelity |
|
||
|
||
Also open + interesting: **Hunyuan3D-Omni** (pose-conditioned generation — make the character already posed), **Hunyuan3D-Part / PartCrafter** (part decomposition), TripoSG (open shape-only). TripoSR runs on Mac but is obsolete quality.
|
||
|
||
### Stage 3 — Scanning: video → splat / mesh (LOCAL, strong)
|
||
**Gemini's "Nerfstudio has strong Mac support" is wrong for splats** — gsplat/splatfacto is CUDA-only on Mac.
|
||
|
||
- **Poses**: COLMAP via Homebrew with the **GLOMAP global mapper** (now merged into COLMAP — much faster than incremental). Optional: colmap-metal fork for Metal SIFT.
|
||
- **Splat training**: **Brush** (native Metal, active, best UX, splat .ply out) · **OpenSplat** (`-DGPU_RUNTIME=MPS`, the proven fallback) · **msplat** (new pure-Metal trainer, reportedly 5–6× gsplat speed — watch it).
|
||
- **Meshes from reality**: **Apple Object Capture** (`PhotogrammetrySession`, GUI in Reality Composer Pro, or a tiny Swift CLI we compile) — native, fast, free, and the correct answer since all 3DGS→mesh research tools (SuGaR/2DGS) are CUDA-locked. macOS 26 shipped an improved reconstruction algorithm.
|
||
- Capture front-ends: Scaniverse (free splat PLY/SPZ + mesh export), Polycam (paywalled formats).
|
||
|
||
### Stage 4 — Motion capture from video
|
||
**Gemini's claims here were mostly wrong**: WHAM is CUDA-locked (name also garbled — it's "World-grounded Humans with Accurate Motion"); EasyMocap has zero Mac support; **ActionMesh is real** (Meta CVPR 2026) but CUDA-only *and not mocap* (rig-free animated mesh, no skeleton).
|
||
|
||
| Tier | Tool | Runs | Output |
|
||
|---|---|---|---|
|
||
| **Filming (face)** | **UE 5.8 MetaHuman Animator on Mac** — facial solve (offline + realtime from iPhone/webcam) is officially supported on macOS in 5.8; M3 Ultra exceeds Epic's recommended M2 Ultra tier | UE 5.8 (installed) | MetaHuman facial anim → AnimSequence/FBX |
|
||
| **Filming (body) — VERDICT** | **UE MetaHuman Markerless Mocap plugin is WINDOWS-ONLY** (Epic docs: "available only on Windows for body or face and body capture"; Fab listing gates to Windows). Fully local, no API key, DX12-class GPU (AMD ok, no CUDA), ~1 min solve per second of 1080p60 footage, hands/fingers included, single performer, tripod required. **Mac workaround: rent a Windows GPU VM** (AWS g5/Paperspace/Shadow, DX12 + 8GB VRAM) only for solves, export FBX ("Performer Skeleton" option = retarget to non-MetaHuman rigs), pull back to Mac. Meshcapade's own cloud (meshcapade.me) was SHUT DOWN Apr 2026 post-acquisition — this plugin is the only outlet for the tech | Windows only | AnimSequence → FBX |
|
||
| Local turnkey | **FreeMoCap v1.8** (`pip install freemocap`, official Mac support) + its Blender add-on | native | rigged armature → FBX/BVH. Best with 2–3 cameras |
|
||
| Hybrid best-quality | **GVHMR** on Colab/HF-Space or rented RTX (~$0.30/hr) → SMPL import via CEB/Meshcapade Blender add-ons | cloud solve, local everything else | world-grounded SMPL |
|
||
| Local previz | MediaPipe/BlendArMocap | native, realtime | rough |
|
||
| API | Move One $18/mo · DeepMotion $17/mo · Autodesk Flow Studio (ex-Wonder Studio) $10–45/mo · Rokoko Video (free ≤15s FBX) | — | FBX/BVH |
|
||
|
||
Not Mac-runnable locally (CUDA-locked): WHAM, TRAM, EasyMocap, ActionMesh, GEM-X/GENMO, DuoMo.
|
||
|
||
### Stage 5 — Rigging & retargeting (LOCAL, free)
|
||
- **Mesh2Motion** (MIT, runs locally via npm/Docker): template-skeleton rigging (human/quadruped/bird) + animation library, GLB in/out. *Semi-automatic* — you position the skeleton, it auto-skins.
|
||
- **Mixamo**: still free, still works from a browser, but frozen (bipeds only).
|
||
- **Blender 5.1.2** (installed, + BlenderMCP): Rigify baseline · **Rokoko plugin** (free retargeting) · **Retarget extension** (Expy Kit successor, Blender 5+) · keemap for exotic rigs · SMPL/SMPL-X add-ons for mocap import.
|
||
- AI auto-rig: **UniRig** (open weights, CUDA-only → cloud job) · **Tripo rig API** (25cr) · **Meshy rig API** (5cr, cheapest).
|
||
|
||
### Stage 6 — LLM brain (LOCAL, already proven)
|
||
- Ollama with **qwen3:235b already pulled**. Mid-2026 pattern: small fast MoE (Qwen3.6-35B-A3B class) for pipeline logic/tool-calling + the big model on demand. LM Studio/MLX when MLX outruns Ollama.
|
||
- Used inside the app for: workflow suggestion, prompt-writing for gen models, batch naming/tagging, "explain this failure".
|
||
|
||
---
|
||
|
||
## 2. The MODELBEAST app
|
||
|
||
### Architecture decisions (defaults chosen, all swappable)
|
||
| Concern | Choice | Why |
|
||
|---|---|---|
|
||
| Backend | **FastAPI** (Python 3.12 via `uv`) | All ML tooling is Python; async job streaming via WebSocket |
|
||
| Tool isolation | **each operator = subprocess in its own uv venv** | These repos have violently conflicting deps (torch pins, numpy, python versions). Never share an env. Homebrew Python 3.14 is too new for ML wheels — per-tool venvs pin 3.10–3.12 |
|
||
| DB | SQLite (WAL) | assets, jobs, workflow runs, provenance. Zero ops |
|
||
| Job model | DAG of operator invocations, asyncio worker pool, one GPU-heavy job at a time (Metal), N CPU jobs parallel | 256GB lets several stay resident |
|
||
| Frontend | React + Vite + Tailwind + three.js | drag/drop/paste ingest, GLB/splat preview, param forms |
|
||
| Param forms | **auto-generated from operator JSON-Schema manifests** | "FULL CONTROL" without hand-building UI per tool — every flag a tool exposes becomes a form field with sane defaults |
|
||
| Conversion | **Blender headless** (`blender -b -P convert.py`) as universal GLB↔FBX↔OBJ↔USD↔BLEND node + trimesh for light ops | FBX especially needs Blender |
|
||
| Storage | `data/{inbox,assets,jobs,outputs}` content-addressed; archive operator → **M4 Pro over Tailscale (100.69.21.128, 4TB)** via rsync | hot on local/external SSD, cold on M4 Pro |
|
||
| Access | bind 0.0.0.0, reach via Tailscale from any device | drive the beast from phone/laptop |
|
||
| API keys | `.env` + settings page (fal, Tripo, Meshy, Replicate) | cloud operators light up when keys exist |
|
||
|
||
### The operator concept (the heart of "full control")
|
||
Every capability — local or cloud — is one **operator**: a folder with `manifest.json` (name, input types, output types, JSON-Schema params, venv spec, resource class) and a `run.py`/shell entry. The UI reads manifests to render workflow choices and parameter forms; the DAG runner chains them by matching input/output types. Adding a new model = dropping in a new operator folder. Presets (saved DAGs + params) give one-click workflows:
|
||
|
||
- **"Video → Rigged Character"**: ffmpeg best-frame → trellis-mac (or Tripo API) → Mesh2Motion/Tripo-rig → retarget mocap → FBX
|
||
- **"Video → Splat"**: ffmpeg → COLMAP/GLOMAP → Brush → .ply/.spz
|
||
- **"Video → Scanned Mesh"**: ffmpeg → Object Capture → GLB/USDZ
|
||
- **"Video → Mocap"**: FreeMoCap (or UE MetaHuman externally) → BVH/FBX
|
||
- **"Image → Mesh ×N"**: fan the same image across trellis-mac + SF3D + Hunyuan + fal APIs, compare side-by-side in the viewer, keep the winner ← *the 256GB flex: run several models resident simultaneously*
|
||
|
||
### Format I/O matrix (via Blender headless + native tool outputs)
|
||
In: mp4/mov, png/jpg/webp/heic, glb/gltf, obj, fbx, blend, usd(z), ply/spz.
|
||
Out: glb, fbx, obj, usd(z), blend, ply/spz, bvh, abc, png sequences/turntable mp4 renders.
|
||
|
||
---
|
||
|
||
## 3. Build roadmap
|
||
|
||
- **Phase 0 — Skeleton (now)**: FastAPI + SQLite + asset ingest (upload/paste) + job queue + React UI with asset library, three.js GLB viewer, job dashboard. Operators: `ffprobe`, `ffmpeg-frames`, `blender-convert`. *Proves the whole loop: drop file → run op → preview → export.*
|
||
- **Phase 1 — First real generators**: `sf3d` (fast local), `fal-trellis` + `fal-trellis2` (API, trivial), `trellis-mac` (the local flagship — expect real install debugging). Side-by-side compare view.
|
||
- **Phase 2 — Scan track**: COLMAP+GLOMAP operator, Brush operator, Object Capture Swift CLI operator. Splat viewer in UI.
|
||
- **Phase 3 — Motion track**: FreeMoCap operator, UE MetaHuman handoff (watch folder → ingest FBX back), GVHMR cloud-solve helper, Blender retarget operator (Rokoko/Retarget).
|
||
- **Phase 4 — Character factory**: Tripo/Meshy full API operators (rig+animate), Mesh2Motion embed/bridge, Hunyuan3D-Omni posed-gen, LLM copilot pane (Ollama), archive-to-M4Pro operator, batch/watch-folder mode.
|
||
|
||
## 4. Open items
|
||
- ~~UE MetaHuman markerless Mac verdict~~ **RESOLVED: Windows-only** (see Stage 4). Experimental plugin — Epic's pattern (Creator→Mac in 5.7, Animator facial→Mac in 5.8) suggests body could follow in 5.9+, but no commitment exists. Watch the Fab listing. Filming workflow for now: film 60fps HD/4K on tripod, single performer, form-fitting clothing; face-solve on Mac, body-solve on rented Windows GPU VM (batch design = pay only for solve hours).
|
||
- M3 Ultra benchmarks for trellis-mac don't exist publicly yet (only M4 Pro ~3.5–5 min) — we'll be among the first; measure and record in this doc.
|
||
- Disk: ~400GB free locally; weights alone (Trellis 15GB, Hunyuan ~50GB, Qwen 142GB…) justify the planned external SSD; M4 Pro is the archive tier.
|
||
|
||
## 5. Key sources
|
||
trellis-mac github.com/shivampkumar/trellis-mac · TRELLIS.2 github.com/microsoft/TRELLIS.2 · SF3D github.com/Stability-AI/stable-fast-3d · Hunyuan3D 2.1 github.com/tencent-hunyuan/hunyuan3d-2.1 · Tripo pricing docs.tripo3d.ai/get-started/pricing.html · fal 3D fal.ai/3d-models · Brush github.com/ArthurBrussee/brush · OpenSplat github.com/pierotofy/OpenSplat · msplat github.com/rayanht/msplat · Object Capture developer.apple.com/documentation/RealityKit/creating-a-photogrammetry-command-line-app · FreeMoCap github.com/freemocap/freemocap · GVHMR github.com/zju3dv/GVHMR · Mesh2Motion github.com/Mesh2Motion/mesh2motion-app · UniRig github.com/VAST-AI-Research/UniRig · CEB_4d_Humans github.com/carlosedubarreto/CEB_4d_Humans · Meshcapade SMPL addon github.com/Meshcapade/SMPL_blender_addon
|