211 lines
11 KiB
Markdown
211 lines
11 KiB
Markdown
# MOCAPGOD
|
||
|
||
**Your own private motion-capture studio.** Film yourself with a phone → get a game-ready
|
||
animation clip on your character rigs. One command, 100% local, 100% owned, zero cloud,
|
||
zero subscriptions. Built 2026-07-17 in seven Opus 4.8 sessions + two Fable review passes.
|
||
|
||
```
|
||
./mocap capture/inbox/my_move.mp4 --name dj_scratch_01
|
||
# → clips_out/dj_scratch_01.fbx + clips_out/dj_scratch_01_qc.mp4
|
||
```
|
||
|
||
The FBX is anim-only, Mixamo-"Without-Skin"-style, on the **mixamorig skeleton shared by
|
||
every GODVERSE character** — it drops into the same `merge_anims.py` → GLB clip-bank flow
|
||
that feeds 90sDJsim, THRIFTGOD, and Unreal (A_3D_STORE).
|
||
|
||
---
|
||
|
||
## Contents
|
||
1. [How it works](#how-it-works)
|
||
2. [Quickstart](#quickstart)
|
||
3. [The instruction manual](#the-instruction-manual)
|
||
4. [Test results — the evidence](#test-results--the-evidence)
|
||
5. [Known limits & open items](#known-limits--open-items)
|
||
6. [Legal](#legal)
|
||
7. [Development history](#development-history)
|
||
|
||
---
|
||
|
||
## How it works
|
||
|
||
```
|
||
you, filmed on a phone (docs/CAPTURE.md protocol)
|
||
│ AirDrop → capture/inbox/
|
||
▼
|
||
LANE A ingest.py probe → normalize 30fps h264 → queue/ → LEDGER.tsv
|
||
▼
|
||
LANE B pose_engine.py HSMR (ViT-H, CVPR'25) recovers the SKEL biomechanical
|
||
[.engine venv, skeleton per frame on Apple-Silicon MPS
|
||
~1.6× realtime] → out/<stem>/motion.bvh (SKEL-24, Y-up, metres)
|
||
→ motion.json sidecar (fps, root_space, joints…)
|
||
→ params.npz (raw poses — rerun cheaply with --from-params)
|
||
▼
|
||
LANE C retarget.py headless Blender 5: BVH → mixamorig (20-bone world-space
|
||
constraint map + visual-keying bake) → anim-only FBX
|
||
+ Workbench turntable QC mp4 (HOUSE RULE: eyeball it)
|
||
▼
|
||
LANE D mocap (CLI) the one command that chains A→C and enforces the naming
|
||
contract inbox → queue → out → clips_out
|
||
```
|
||
|
||
- **Engine:** [HSMR](https://github.com/IsshikiHugh/HSMR) (MIT code) predicting **SKEL**
|
||
(Max-Planck biomechanical body model, v1.1.1) — chosen over SMPL-family models
|
||
(GVHMR/WHAM/TRAM) after a license-verified bake-off. Full analysis:
|
||
`docs/B1_BAKEOFF.md`, `docs/DECISIONS.md`, `docs/LICENSES.md`.
|
||
- **All Apple Silicon:** the ViT-H recovery runs on MPS at 31 fps; detection runs CPU
|
||
mobilenet every-20th-frame with box reuse (single-subject assumption).
|
||
- **Formats treaty** (PLAN.md §Ground-rules): BVH@30fps + JSON sidecar between stages;
|
||
metres / Y-up / faces −Z; `root_space: "camera"|"world"` is a required sidecar field.
|
||
|
||
## Quickstart
|
||
|
||
```bash
|
||
# 0. One-time: engine env (if fresh clone) ~10 min + weights download
|
||
.engine/setup_hsmr.sh --patch # clone, venv, weights, apply ARM patches
|
||
# …then place the gated SKEL v1.1.1 files (docs/B_ENGINE_SETUP.md — John's login)
|
||
|
||
# 1. Film per docs/CAPTURE.md (landscape, FEET IN FRAME, T-pose bookends, one move/take)
|
||
# 2. AirDrop to capture/inbox/
|
||
# 3. One command:
|
||
./mocap capture/inbox/IMG_4242.mp4 --name head_nod_01
|
||
# 4. WATCH clips_out/head_nod_01_qc.mp4 ← house rule, every clip, before it enters a bank
|
||
```
|
||
|
||
Flags: `--character female|male` (female = proven path) · `--travel` (keep world root;
|
||
default in-place) · `--no-qc` · `--gender` (pose model override) · `--selfcheck`.
|
||
|
||
## The instruction manual
|
||
|
||
### Filming (docs/CAPTURE.md — the 20-second version)
|
||
Landscape, phone steady at 2–3m, your WHOLE body including feet in frame the entire take,
|
||
even light, fitted clothes, **1 second of T-pose at start and end**, one move per take,
|
||
5–30s. Break these and the skeleton quality drops — feet out of frame is the #1 killer.
|
||
|
||
### Running stages by hand (debugging / tuning)
|
||
```bash
|
||
python3 ingest.py capture/inbox/vid.mp4 --name test # Lane A alone
|
||
.engine/HSMR/.venv/bin/python pose_engine.py queue/<stem>.mp4 --gender female # Lane B
|
||
.engine/HSMR/.venv/bin/python pose_engine.py <stem> --from-params # re-emit BVH, skip HMR (fast)
|
||
/Applications/Blender.app/Contents/MacOS/Blender --background --python retarget.py -- \
|
||
--bvh out/<stem>/motion.bvh --name test # Lane C alone
|
||
```
|
||
|
||
### Where things live
|
||
| Path | What |
|
||
|---|---|
|
||
| `capture/inbox/` → `capture/done/` | drop zone → processed originals (ingest MOVES them) |
|
||
| `capture/LEDGER.tsv` | every ingest, accepted or rejected, with reason |
|
||
| `queue/` | normalized 30fps videos awaiting the pose engine |
|
||
| `out/<stem>/` | motion.bvh + motion.json + params.npz per clip |
|
||
| `clips_out/` | **the product**: `<name>.fbx` + `<name>_qc.mp4` |
|
||
| `.engine/HSMR/` | gitignored engine clone; rebuild any time via `setup_hsmr.sh --patch` |
|
||
| `engine_patches/` | our ARM/headless fixes (in git; survive re-clones) |
|
||
| `spec/` | skeleton ground truth: `mixamorig_bones.json`, `skel_to_mixamorig.json` |
|
||
|
||
### Into the games / Unreal
|
||
- **Clip banks:** merge FBX clips onto a character with
|
||
`character_kit/scripts/merge_anims.py` → multi-clip GLB (⚠️ needs the Blender-5
|
||
slot-bind fix — task spawned; see Known limits).
|
||
- **Unreal:** import the merged GLB via the A_3D_STORE deferred-import pattern
|
||
(slate post-tick, never a blocking import through the MCP bridge — see the
|
||
unreal-asset-convergence memory / A_3D_STORE experiments).
|
||
|
||
### Troubleshooting
|
||
| Symptom | Cause / fix |
|
||
|---|---|
|
||
| `✗ HSMR venv missing` | run `.engine/setup_hsmr.sh --patch` |
|
||
| smoke/engine fails at `skel_male.pkl` | gated SKEL v1.1.1 not placed — docs/B_ENGINE_SETUP.md |
|
||
| clip rejected at ingest | check `capture/LEDGER.tsv` note column (portrait? too long? corrupt?) |
|
||
| feet slide in QC | expected pre-foot-fix; worse on `--travel` (grounding pass not built yet) |
|
||
| QC motion is soup | filming protocol violated (feet cropped? multiple people in frame?) |
|
||
| everything re-detects slowly | detector runs every 20th frame; long multi-person videos will crawl — single subject only |
|
||
|
||
## Test results — the evidence
|
||
|
||
Every claim below was produced during the build and **re-verified independently in review**
|
||
(Fable re-ran tests rather than trusting the ledger). Raw logs: `out/*.log`, montages:
|
||
`out/qc_montage.png`, `out/mocap_qc_montage.png`.
|
||
|
||
### Detector unit test (session 4, re-run in review #1)
|
||
| Metric | Result |
|
||
|---|---|
|
||
| Demo images with person found | **7/7** |
|
||
| Human patches extracted @256×256 | 15 |
|
||
| detectron2 dependency | **eliminated** (torchvision drop-in, ARM-native) |
|
||
|
||
### Engine smoke test — first full forward pass (2026-07-17, gymnasts.mp4, 263 frames)
|
||
| Metric | Result | Bar |
|
||
|---|---|---|
|
||
| Device | **MPS** (Apple Silicon GPU) | must not be CPU |
|
||
| Patches recovered | 1,315 | — |
|
||
| Recovery throughput | **31.2 fps** | > realtime ✅ |
|
||
| NaNs in output | **0** | 0 ✅ |
|
||
| Finding | CPU detector was 80% of wall-clock → fixed in B2 (mobilenet + every-20) | |
|
||
|
||
### Lane B acceptance (session 5)
|
||
| Metric | Result | Bar |
|
||
|---|---|---|
|
||
| Steady-state pipeline speed | **~1.6× realtime** | <2× ✅ |
|
||
| BVH sanity (Blender import) | 24 bones, upright, **1.59m** figure | human-shaped ✅ |
|
||
| FK self-consistency | **0.0 mm** error | — |
|
||
| Unattended queue drain | 6 videos: 4 OK, **2 bad clips failed gracefully** | no crash ✅ |
|
||
| Axis correctness | Y-down OpenCV → Y-up/−Z via diag(1,−1,−1), verified upright | ✅ |
|
||
|
||
### Lane C acceptance (session 6)
|
||
| Check | Result |
|
||
|---|---|
|
||
| Bones constraint-mapped | 20 (+4 intentionally null: twist/heel; fingers = Lane E) |
|
||
| Retargeted FBX drives female_game.glb | ✅ verified — RightHand moved +121 units; playable 7.3MB GLB |
|
||
| QC montage eyeball | coherent gymnast motion, no soup, no candy-wrap arm twist |
|
||
| Blender 5 "Baklava" API breaks | found + solved (slot-bind, fcurves walk) — documented in docs/C_RETARGET.md |
|
||
|
||
### Lane D acceptance + independent review #2 (fresh end-to-end, 2026-07-17)
|
||
| Metric | Opus run (`gymnasts_d`) | Fable review run (`review_check`) |
|
||
|---|---|---|
|
||
| Full chain A→C one command | ✅ | ✅ (cold, renamed input) |
|
||
| Frames processed | 317 @ 30fps | 317 @ 30fps |
|
||
| FK error | 0.0 mm | 0.0 mm |
|
||
| Sidecar treaty fields | ✅ | ✅ (`root_space:"camera"`, 24 joints, fps 30) |
|
||
| Output FBX + QC | ✅ | ✅ 685KB FBX + QC mp4, frames eyeballed clean |
|
||
| Wall clock (10.6s clip) | — | ~35s engine total incl. one-time model build |
|
||
|
||
**Review #2 findings:** one real bug (FBX take-name is generic `Armature|Scene` on every
|
||
clip → bank-merge collisions; fix queued in retarget.py before first real clip banks) and
|
||
one cross-repo action (Blender-5 fix for character_kit merge_anims.py — task spawned).
|
||
|
||
## Known limits & open items
|
||
|
||
**Honest list — v1 is machinery-complete, not footage-verified.**
|
||
- 🎬 **Final acceptance awaits John's real footage** (walk / dance / pick-up-object).
|
||
Everything below the line is gated on it.
|
||
- 🐛 FBX take-name bug (see review #2) — MUST fix before clips enter banks.
|
||
- 🦶 Foot-contact fix is a stub — feet may slide, tuned against a real walk clip.
|
||
- 🚶 `--travel` (world root) is best-effort: HSMR is per-frame → camera-space root.
|
||
In-place clips (DJ moves!) are first-class; locomotion needs the grounding pass.
|
||
- ✋ Fingers are static (Lane E: hand pass).
|
||
- 🧍 Male rig path unverified until a male clip is run.
|
||
- 📧 Path-A license email drafted, unsent (docs/DECISIONS.md).
|
||
- Lanes E (MLX training: autoposer, inbetweening, centaur retarget) and F (drop-folder
|
||
daemon, dashboard) are designed but intentionally not started.
|
||
|
||
## Legal
|
||
|
||
Read `docs/LICENSES.md` before changing engines or training anything. The short version:
|
||
HSMR **code** is MIT; the **SKEL body model** (and BioAMASS dataset) are Max-Planck
|
||
research-licensed — commercial use is John's accepted Path-B judgment call, with the Path-A
|
||
commercial-license email drafted to retire the risk. **We ship only motion retargeted onto
|
||
our own rigs; no Max-Planck file is ever redistributed.** Mixamo clips: OK in games, never
|
||
as training data. Cascadeur: used as a licensed authoring tool, never reverse-engineered.
|
||
|
||
## Development history
|
||
|
||
Built 2026-07-17, PLAN.md → seven Opus sessions, reviewed twice by Fable (see PROGRESS.md
|
||
for the full ledger including the honest mistakes):
|
||
session 1 bake-off picked an animal-only model by accident → session 2 caught and reversed
|
||
it with a license-verified re-analysis → session 3 locked the HSMR/SKEL decision + MPS →
|
||
session 4 broke the detectron2 ARM wall with a torchvision drop-in → John supplied the
|
||
gated SKEL v1.1.1 → smoke test passed first try → session 5 shipped the pose engine (B2+B3)
|
||
+ six review orders → session 6 shipped the Blender retarget (and discovered the Blender-5
|
||
animation API changes the hard way) → session 7 shipped the `mocap` CLI → review #2
|
||
re-ran everything cold and it held.
|