# MOCAPGOD

**Your own private motion-capture studio.** Film yourself with a phone → get a game-ready
animation clip on your character rigs. One command, 100% local, 100% owned, zero cloud,
zero subscriptions. Built 2026-07-17 in seven Opus 4.8 sessions + two Fable review passes.

```
./mocap capture/inbox/my_move.mp4 --name dj_scratch_01
# → clips_out/dj_scratch_01.fbx  +  clips_out/dj_scratch_01_qc.mp4
```

The FBX is anim-only, Mixamo-"Without-Skin"-style, on the **mixamorig skeleton shared by
every GODVERSE character** — it drops into the same `merge_anims.py` → GLB clip-bank flow
that feeds 90sDJsim, THRIFTGOD, and Unreal (A_3D_STORE).

---

## Contents
1. [How it works](#how-it-works)
2. [Quickstart](#quickstart)
3. [The instruction manual](#the-instruction-manual)
4. [Test results — the evidence](#test-results--the-evidence)
5. [Known limits & open items](#known-limits--open-items)
6. [Legal](#legal)
7. [Development history](#development-history)

---

## How it works

```
you, filmed on a phone (docs/CAPTURE.md protocol)
      │  AirDrop → capture/inbox/
      ▼
LANE A  ingest.py          probe → normalize 30fps h264 → queue/  → LEDGER.tsv
      ▼
LANE B  pose_engine.py     HSMR (ViT-H, CVPR'25) recovers the SKEL biomechanical
        [.engine venv,      skeleton per frame on Apple-Silicon MPS
         ~1.6× realtime]    → out/<stem>/motion.bvh (SKEL-24, Y-up, metres)
                            → motion.json sidecar (fps, root_space, joints…)
                            → params.npz (raw poses — rerun cheaply with --from-params)
      ▼
LANE C  retarget.py        headless Blender 5: BVH → mixamorig (20-bone world-space
                            constraint map + visual-keying bake) → anim-only FBX
                            + Workbench turntable QC mp4 (HOUSE RULE: eyeball it)
      ▼
LANE D  mocap (CLI)        the one command that chains A→C and enforces the naming
                            contract inbox → queue → out → clips_out
```

- **Engine:** [HSMR](https://github.com/IsshikiHugh/HSMR) (MIT code) predicting **SKEL**
  (Max-Planck biomechanical body model, v1.1.1) — chosen over SMPL-family models
  (GVHMR/WHAM/TRAM) after a license-verified bake-off. Full analysis:
  `docs/B1_BAKEOFF.md`, `docs/DECISIONS.md`, `docs/LICENSES.md`.
- **All Apple Silicon:** the ViT-H recovery runs on MPS at 31 fps; detection runs CPU
  mobilenet every-20th-frame with box reuse (single-subject assumption).
- **Formats treaty** (PLAN.md §Ground-rules): BVH@30fps + JSON sidecar between stages;
  metres / Y-up / faces −Z; `root_space: "camera"|"world"` is a required sidecar field.

## Quickstart

```bash
# 0. One-time: engine env (if fresh clone)         ~10 min + weights download
.engine/setup_hsmr.sh --patch          # clone, venv, weights, apply ARM patches
#    …then place the gated SKEL v1.1.1 files (docs/B_ENGINE_SETUP.md — John's login)

# 1. Film per docs/CAPTURE.md  (landscape, FEET IN FRAME, T-pose bookends, one move/take)
# 2. AirDrop to capture/inbox/
# 3. One command:
./mocap capture/inbox/IMG_4242.mp4 --name head_nod_01
# 4. WATCH clips_out/head_nod_01_qc.mp4   ← house rule, every clip, before it enters a bank
```

Flags: `--character female|male` (female = proven path) · `--travel` (keep world root;
default in-place) · `--no-qc` · `--gender` (pose model override) · `--selfcheck`.

## The instruction manual

### Filming (docs/CAPTURE.md — the 20-second version)
Landscape, phone steady at 2–3m, your WHOLE body including feet in frame the entire take,
even light, fitted clothes, **1 second of T-pose at start and end**, one move per take,
5–30s. Break these and the skeleton quality drops — feet out of frame is the #1 killer.

### Running stages by hand (debugging / tuning)
```bash
python3 ingest.py capture/inbox/vid.mp4 --name test          # Lane A alone
.engine/HSMR/.venv/bin/python pose_engine.py queue/<stem>.mp4 --gender female   # Lane B
.engine/HSMR/.venv/bin/python pose_engine.py <stem> --from-params   # re-emit BVH, skip HMR (fast)
/Applications/Blender.app/Contents/MacOS/Blender --background --python retarget.py -- \
    --bvh out/<stem>/motion.bvh --name test                  # Lane C alone
```

### Where things live
| Path | What |
|---|---|
| `capture/inbox/` → `capture/done/` | drop zone → processed originals (ingest MOVES them) |
| `capture/LEDGER.tsv` | every ingest, accepted or rejected, with reason |
| `queue/` | normalized 30fps videos awaiting the pose engine |
| `out/<stem>/` | motion.bvh + motion.json + params.npz per clip |
| `clips_out/` | **the product**: `<name>.fbx` + `<name>_qc.mp4` |
| `.engine/HSMR/` | gitignored engine clone; rebuild any time via `setup_hsmr.sh --patch` |
| `engine_patches/` | our ARM/headless fixes (in git; survive re-clones) |
| `spec/` | skeleton ground truth: `mixamorig_bones.json`, `skel_to_mixamorig.json` |

### Into the games / Unreal
- **Clip banks:** merge FBX clips onto a character with
  `character_kit/scripts/merge_anims.py` → multi-clip GLB (⚠️ needs the Blender-5
  slot-bind fix — task spawned; see Known limits).
- **Unreal:** import the merged GLB via the A_3D_STORE deferred-import pattern
  (slate post-tick, never a blocking import through the MCP bridge — see the
  unreal-asset-convergence memory / A_3D_STORE experiments).

### Troubleshooting
| Symptom | Cause / fix |
|---|---|
| `✗ HSMR venv missing` | run `.engine/setup_hsmr.sh --patch` |
| smoke/engine fails at `skel_male.pkl` | gated SKEL v1.1.1 not placed — docs/B_ENGINE_SETUP.md |
| clip rejected at ingest | check `capture/LEDGER.tsv` note column (portrait? too long? corrupt?) |
| feet slide in QC | expected pre-foot-fix; worse on `--travel` (grounding pass not built yet) |
| QC motion is soup | filming protocol violated (feet cropped? multiple people in frame?) |
| everything re-detects slowly | detector runs every 20th frame; long multi-person videos will crawl — single subject only |

## Test results — the evidence

Every claim below was produced during the build and **re-verified independently in review**
(Fable re-ran tests rather than trusting the ledger). Raw logs: `out/*.log`, montages:
`out/qc_montage.png`, `out/mocap_qc_montage.png`.

### Detector unit test (session 4, re-run in review #1)
| Metric | Result |
|---|---|
| Demo images with person found | **7/7** |
| Human patches extracted @256×256 | 15 |
| detectron2 dependency | **eliminated** (torchvision drop-in, ARM-native) |

### Engine smoke test — first full forward pass (2026-07-17, gymnasts.mp4, 263 frames)
| Metric | Result | Bar |
|---|---|---|
| Device | **MPS** (Apple Silicon GPU) | must not be CPU |
| Patches recovered | 1,315 | — |
| Recovery throughput | **31.2 fps** | > realtime ✅ |
| NaNs in output | **0** | 0 ✅ |
| Finding | CPU detector was 80% of wall-clock → fixed in B2 (mobilenet + every-20) | |

### Lane B acceptance (session 5)
| Metric | Result | Bar |
|---|---|---|
| Steady-state pipeline speed | **~1.6× realtime** | <2× ✅ |
| BVH sanity (Blender import) | 24 bones, upright, **1.59m** figure | human-shaped ✅ |
| FK self-consistency | **0.0 mm** error | — |
| Unattended queue drain | 6 videos: 4 OK, **2 bad clips failed gracefully** | no crash ✅ |
| Axis correctness | Y-down OpenCV → Y-up/−Z via diag(1,−1,−1), verified upright | ✅ |

### Lane C acceptance (session 6)
| Check | Result |
|---|---|
| Bones constraint-mapped | 20 (+4 intentionally null: twist/heel; fingers = Lane E) |
| Retargeted FBX drives female_game.glb | ✅ verified — RightHand moved +121 units; playable 7.3MB GLB |
| QC montage eyeball | coherent gymnast motion, no soup, no candy-wrap arm twist |
| Blender 5 "Baklava" API breaks | found + solved (slot-bind, fcurves walk) — documented in docs/C_RETARGET.md |

### Lane D acceptance + independent review #2 (fresh end-to-end, 2026-07-17)
| Metric | Opus run (`gymnasts_d`) | Fable review run (`review_check`) |
|---|---|---|
| Full chain A→C one command | ✅ | ✅ (cold, renamed input) |
| Frames processed | 317 @ 30fps | 317 @ 30fps |
| FK error | 0.0 mm | 0.0 mm |
| Sidecar treaty fields | ✅ | ✅ (`root_space:"camera"`, 24 joints, fps 30) |
| Output FBX + QC | ✅ | ✅ 685KB FBX + QC mp4, frames eyeballed clean |
| Wall clock (10.6s clip) | — | ~35s engine total incl. one-time model build |

**Review #2 findings:** one real bug (FBX take-name is generic `Armature|Scene` on every
clip → bank-merge collisions; fix queued in retarget.py before first real clip banks) and
one cross-repo action (Blender-5 fix for character_kit merge_anims.py — task spawned).

## Known limits & open items

**Honest list — v1 is machinery-complete, not footage-verified.**
- 🎬 **Final acceptance awaits John's real footage** (walk / dance / pick-up-object).
  Everything below the line is gated on it.
- 🐛 FBX take-name bug (see review #2) — MUST fix before clips enter banks.
- 🦶 Foot-contact fix is a stub — feet may slide, tuned against a real walk clip.
- 🚶 `--travel` (world root) is best-effort: HSMR is per-frame → camera-space root.
  In-place clips (DJ moves!) are first-class; locomotion needs the grounding pass.
- ✋ Fingers are static (Lane E: hand pass).
- 🧍 Male rig path unverified until a male clip is run.
- 📧 Path-A license email drafted, unsent (docs/DECISIONS.md).
- Lanes E (MLX training: autoposer, inbetweening, centaur retarget) and F (drop-folder
  daemon, dashboard) are designed but intentionally not started.

## Legal

Read `docs/LICENSES.md` before changing engines or training anything. The short version:
HSMR **code** is MIT; the **SKEL body model** (and BioAMASS dataset) are Max-Planck
research-licensed — commercial use is John's accepted Path-B judgment call, with the Path-A
commercial-license email drafted to retire the risk. **We ship only motion retargeted onto
our own rigs; no Max-Planck file is ever redistributed.** Mixamo clips: OK in games, never
as training data. Cascadeur: used as a licensed authoring tool, never reverse-engineered.

## Development history

Built 2026-07-17, PLAN.md → seven Opus sessions, reviewed twice by Fable (see PROGRESS.md
for the full ledger including the honest mistakes):
session 1 bake-off picked an animal-only model by accident → session 2 caught and reversed
it with a license-verified re-analysis → session 3 locked the HSMR/SKEL decision + MPS →
session 4 broke the detectron2 ARM wall with a torchvision drop-in → John supplied the
gated SKEL v1.1.1 → smoke test passed first try → session 5 shipped the pose engine (B2+B3)
+ six review orders → session 6 shipped the Blender retarget (and discovered the Blender-5
animation API changes the hard way) → session 7 shipped the `mocap` CLI → review #2
re-ran everything cold and it held.