modelbeast/PLAN.md

MODELBEAST — Master Flow & Build Plan

Written 2026-07-12. Every tool/claim below was verified against primary sources (repos, docs, pricing pages) by research agents on this date — including corrections to the Gemini advice that seeded this project.

Mission

A local-first web app running on the M3 Ultra (reachable over Tailscale from any device) that turns videos, images, and 3D files into meshes, splats, mocap, and rigged animated characters. Local Apple Silicon tools do the heavy lifting; cloud APIs (fal.ai, Tripo, Meshy) are optional quality/convenience tiers — not necessities. Full parameter control at every stage.

                        ┌─────────────────────────────────────────────┐
                        │        MODELBEAST WEB UI  (React + three.js)│
                        │  drag/drop/paste: mp4 png jpg glb obj fbx   │
                        │  blend usdz ply · workflow picker · param   │
                        │  forms · job dashboard · 3D preview · export│
                        └──────────────────┬──────────────────────────┘
                                           │ REST/WebSocket
                        ┌──────────────────▼──────────────────────────┐
                        │   FastAPI ORCHESTRATOR  (asset DB + job DAG)│
                        └──┬──────┬──────┬──────┬──────┬──────┬───────┘
                           │      │      │      │      │      │
                 ┌─────────▼─┐ ┌──▼───┐ ┌▼─────┐ ┌────▼───┐ ┌▼──────────┐
                 │ INGEST    │ │MESH  │ │SCAN  │ │ MOCAP  │ │ RIG/ANIM  │
                 │ ffmpeg    │ │GEN   │ │3DGS+ │ │        │ │           │
                 │ frames,   │ │      │ │photo-│ │        │ │           │
                 │ scenes,   │ │      │ │gram  │ │        │ │           │
                 │ sharpness │ │      │ │      │ │        │ │           │
                 └───────────┘ └──────┘ └──────┘ └────────┘ └───────────┘
                    each stage = swappable "operators", local or API
                                           │
                        ┌──────────────────▼──────────────────────────┐
                        │  CONVERT/EXPORT: Blender headless (universal│
                        │  GLB/FBX/OBJ/USD/BLEND) · archive → M4 Pro  │
                        └─────────────────────────────────────────────┘

---

1. Verified capability matrix (July 2026)

Stage 1 — Video ingest & preprocessing (LOCAL, trivial)

• ffmpeg 8.1.2 (installed): frame extraction (fps=2..3 for orbits), mpdecimate to drop near-dupes, scdet/PySceneDetect for scene splits, I-frame selection.
• Sharpness culling: Laplacian-variance scoring per chunk; a few blurry frames can poison COLMAP matching. Target 150–300 frames/scene.

Stage 2 — Mesh generation from image(s)

Gemini said "local is impractical, use fal.ai" — REFUTED since early 2026.

Tier	Tool	Runs	Output	Notes
Local SOTA	trellis-mac (TRELLIS.2 MPS port) github.com/shivampkumar/trellis-mac	M3 Ultra, ~18GB peak, est. ~3–5 min/gen	GLB + baked PBR, ~400K verts	The headline local capability. MIT. ~15GB weights
Local fast draft	SF3D / SPAR3D (Stability, official MPS + Metal kernels)	seconds-fast	GLB, UV textures	Draft/iteration tier
Local 2nd opinion	Hunyuan3D 2.1 Mac forks (Brainkeys, SimonTingle ComfyUI-Mac)	shape solid on MPS; texture stage shaky w/o CUDA	GLB/OBJ	Best open textures when it works
API cheap	fal fal-ai/trellis	$0.02/gen	textured GLB	2024-tier quality
API strong	fal trellis-2 $0.25–0.35 · fal hunyuan3d/v2 $0.16–0.48 · Hunyuan 3.1 Pro $0.375+	—	GLB/OBJ, PBR	Zero-setup SOTA
API character pipeline	Tripo v3 API — the only gen→T-pose→quad retopo→auto-rig→animate chain	~$0.55–0.75/rigged char	FBX, quads, rigged	The "finished game asset" button
API hero assets	Rodin Gen-2 via fal hyper3d/rodin $0.40 (HighPack $1.20)	—	GLB/USDZ/FBX, quad, 4K	Film-grade fidelity

Also open + interesting: Hunyuan3D-Omni (pose-conditioned generation — make the character already posed), Hunyuan3D-Part / PartCrafter (part decomposition), TripoSG (open shape-only). TripoSR runs on Mac but is obsolete quality.

Stage 3 — Scanning: video → splat / mesh (LOCAL, strong)

Gemini's "Nerfstudio has strong Mac support" is wrong for splats — gsplat/splatfacto is CUDA-only on Mac.

• Poses: COLMAP via Homebrew with the GLOMAP global mapper (now merged into COLMAP — much faster than incremental). Optional: colmap-metal fork for Metal SIFT.
• Splat training: Brush (native Metal, active, best UX, splat .ply out) · OpenSplat (-DGPU_RUNTIME=MPS, the proven fallback) · msplat (new pure-Metal trainer, reportedly 5–6× gsplat speed — watch it).
• Meshes from reality: Apple Object Capture (PhotogrammetrySession, GUI in Reality Composer Pro, or a tiny Swift CLI we compile) — native, fast, free, and the correct answer since all 3DGS→mesh research tools (SuGaR/2DGS) are CUDA-locked. macOS 26 shipped an improved reconstruction algorithm.
• Capture front-ends: Scaniverse (free splat PLY/SPZ + mesh export), Polycam (paywalled formats).

Stage 4 — Motion capture from video

Gemini's claims here were mostly wrong: WHAM is CUDA-locked (name also garbled — it's "World-grounded Humans with Accurate Motion"); EasyMocap has zero Mac support; ActionMesh is real (Meta CVPR 2026) but CUDA-only and not mocap (rig-free animated mesh, no skeleton).

Tier	Tool	Runs	Output
Filming (face)	UE 5.8 MetaHuman Animator on Mac — facial solve (offline + realtime from iPhone/webcam) is officially supported on macOS in 5.8; M3 Ultra exceeds Epic's recommended M2 Ultra tier	UE 5.8 (installed)	MetaHuman facial anim → AnimSequence/FBX
Filming (body) — VERDICT	UE MetaHuman Markerless Mocap plugin is WINDOWS-ONLY (Epic docs: "available only on Windows for body or face and body capture"; Fab listing gates to Windows). Fully local, no API key, DX12-class GPU (AMD ok, no CUDA), ~1 min solve per second of 1080p60 footage, hands/fingers included, single performer, tripod required. Mac workaround: rent a Windows GPU VM (AWS g5/Paperspace/Shadow, DX12 + 8GB VRAM) only for solves, export FBX ("Performer Skeleton" option = retarget to non-MetaHuman rigs), pull back to Mac. Meshcapade's own cloud (meshcapade.me) was SHUT DOWN Apr 2026 post-acquisition — this plugin is the only outlet for the tech	Windows only	AnimSequence → FBX
Local turnkey	FreeMoCap v1.8 (pip install freemocap, official Mac support) + its Blender add-on	native	rigged armature → FBX/BVH. Best with 2–3 cameras
Hybrid best-quality	GVHMR on Colab/HF-Space or rented RTX (~$0.30/hr) → SMPL import via CEB/Meshcapade Blender add-ons	cloud solve, local everything else	world-grounded SMPL
Local previz	MediaPipe/BlendArMocap	native, realtime	rough
API	Move One $18/mo · DeepMotion $17/mo · Autodesk Flow Studio (ex-Wonder Studio) $10–45/mo · Rokoko Video (free ≤15s FBX)	—	FBX/BVH

Not Mac-runnable locally (CUDA-locked): WHAM, TRAM, EasyMocap, ActionMesh, GEM-X/GENMO, DuoMo.

Stage 5 — Rigging & retargeting (LOCAL, free)

• Mesh2Motion (MIT, runs locally via npm/Docker): template-skeleton rigging (human/quadruped/bird) + animation library, GLB in/out. Semi-automatic — you position the skeleton, it auto-skins.
• Mixamo: still free, still works from a browser, but frozen (bipeds only).
• Blender 5.1.2 (installed, + BlenderMCP): Rigify baseline · Rokoko plugin (free retargeting) · Retarget extension (Expy Kit successor, Blender 5+) · keemap for exotic rigs · SMPL/SMPL-X add-ons for mocap import.
• AI auto-rig: UniRig (open weights, CUDA-only → cloud job) · Tripo rig API (25cr) · Meshy rig API (5cr, cheapest).

Stage 6 — LLM brain (LOCAL, already proven)

• Ollama with qwen3:235b already pulled. Mid-2026 pattern: small fast MoE (Qwen3.6-35B-A3B class) for pipeline logic/tool-calling + the big model on demand. LM Studio/MLX when MLX outruns Ollama.
• Used inside the app for: workflow suggestion, prompt-writing for gen models, batch naming/tagging, "explain this failure".

---

2. The MODELBEAST app

Architecture decisions (defaults chosen, all swappable)

Concern	Choice	Why
Backend	FastAPI (Python 3.12 via uv)	All ML tooling is Python; async job streaming via WebSocket
Tool isolation	each operator = subprocess in its own uv venv	These repos have violently conflicting deps (torch pins, numpy, python versions). Never share an env. Homebrew Python 3.14 is too new for ML wheels — per-tool venvs pin 3.10–3.12
DB	SQLite (WAL)	assets, jobs, workflow runs, provenance. Zero ops
Job model	DAG of operator invocations, asyncio worker pool, one GPU-heavy job at a time (Metal), N CPU jobs parallel	256GB lets several stay resident
Frontend	React + Vite + Tailwind + three.js	drag/drop/paste ingest, GLB/splat preview, param forms
Param forms	auto-generated from operator JSON-Schema manifests	"FULL CONTROL" without hand-building UI per tool — every flag a tool exposes becomes a form field with sane defaults
Conversion	Blender headless (blender -b -P convert.py) as universal GLB↔FBX↔OBJ↔USD↔BLEND node + trimesh for light ops	FBX especially needs Blender
Storage	data/{inbox,assets,jobs,outputs} content-addressed; archive operator → M4 Pro over Tailscale (100.69.21.128, 4TB) via rsync	hot on local/external SSD, cold on M4 Pro
Access	bind 0.0.0.0, reach via Tailscale from any device	drive the beast from phone/laptop
API keys	.env + settings page (fal, Tripo, Meshy, Replicate)	cloud operators light up when keys exist

The operator concept (the heart of "full control")

Every capability — local or cloud — is one operator: a folder with manifest.json (name, input types, output types, JSON-Schema params, venv spec, resource class) and a run.py/shell entry. The UI reads manifests to render workflow choices and parameter forms; the DAG runner chains them by matching input/output types. Adding a new model = dropping in a new operator folder. Presets (saved DAGs + params) give one-click workflows:

• "Video → Rigged Character": ffmpeg best-frame → trellis-mac (or Tripo API) → Mesh2Motion/Tripo-rig → retarget mocap → FBX
• "Video → Splat": ffmpeg → COLMAP/GLOMAP → Brush → .ply/.spz
• "Video → Scanned Mesh": ffmpeg → Object Capture → GLB/USDZ
• "Video → Mocap": FreeMoCap (or UE MetaHuman externally) → BVH/FBX
• "Image → Mesh ×N": fan the same image across trellis-mac + SF3D + Hunyuan + fal APIs, compare side-by-side in the viewer, keep the winner ← the 256GB flex: run several models resident simultaneously

Format I/O matrix (via Blender headless + native tool outputs)

In: mp4/mov, png/jpg/webp/heic, glb/gltf, obj, fbx, blend, usd(z), ply/spz. Out: glb, fbx, obj, usd(z), blend, ply/spz, bvh, abc, png sequences/turntable mp4 renders.

---

3. Build roadmap

• Phase 0 — Skeleton (now): FastAPI + SQLite + asset ingest (upload/paste) + job queue + React UI with asset library, three.js GLB viewer, job dashboard. Operators: ffprobe, ffmpeg-frames, blender-convert. Proves the whole loop: drop file → run op → preview → export.
• Phase 1 — First real generators: sf3d (fast local), fal-trellis + fal-trellis2 (API, trivial), trellis-mac (the local flagship — expect real install debugging). Side-by-side compare view.
• Phase 2 — Scan track: COLMAP+GLOMAP operator, Brush operator, Object Capture Swift CLI operator. Splat viewer in UI.
• Phase 3 — Motion track: FreeMoCap operator, UE MetaHuman handoff (watch folder → ingest FBX back), GVHMR cloud-solve helper, Blender retarget operator (Rokoko/Retarget).
• Phase 4 — Character factory: Tripo/Meshy full API operators (rig+animate), Mesh2Motion embed/bridge, Hunyuan3D-Omni posed-gen, LLM copilot pane (Ollama), archive-to-M4Pro operator, batch/watch-folder mode.

4. Open items

• UE MetaHuman markerless Mac verdict RESOLVED: Windows-only (see Stage 4). Experimental plugin — Epic's pattern (Creator→Mac in 5.7, Animator facial→Mac in 5.8) suggests body could follow in 5.9+, but no commitment exists. Watch the Fab listing. Filming workflow for now: film 60fps HD/4K on tripod, single performer, form-fitting clothing; face-solve on Mac, body-solve on rented Windows GPU VM (batch design = pay only for solve hours).
• M3 Ultra benchmarks for trellis-mac don't exist publicly yet (only M4 Pro ~3.5–5 min) — we'll be among the first; measure and record in this doc.
• Disk: ~400GB free locally; weights alone (Trellis 15GB, Hunyuan ~50GB, Qwen 142GB…) justify the planned external SSD; M4 Pro is the archive tier.

5. Key sources

trellis-mac github.com/shivampkumar/trellis-mac · TRELLIS.2 github.com/microsoft/TRELLIS.2 · SF3D github.com/Stability-AI/stable-fast-3d · Hunyuan3D 2.1 github.com/tencent-hunyuan/hunyuan3d-2.1 · Tripo pricing docs.tripo3d.ai/get-started/pricing.html · fal 3D fal.ai/3d-models · Brush github.com/ArthurBrussee/brush · OpenSplat github.com/pierotofy/OpenSplat · msplat github.com/rayanht/msplat · Object Capture developer.apple.com/documentation/RealityKit/creating-a-photogrammetry-command-line-app · FreeMoCap github.com/freemocap/freemocap · GVHMR github.com/zju3dv/GVHMR · Mesh2Motion github.com/Mesh2Motion/mesh2motion-app · UniRig github.com/VAST-AI-Research/UniRig · CEB_4d_Humans github.com/carlosedubarreto/CEB_4d_Humans · Meshcapade SMPL addon github.com/Meshcapade/SMPL_blender_addon