mocapgod/README.md

MOCAPGOD

Your own private motion-capture studio. Film yourself with a phone → get a game-ready animation clip on your character rigs. One command, 100% local, 100% owned, zero cloud, zero subscriptions. Built 2026-07-17 in seven Opus 4.8 sessions + two Fable review passes.

./mocap capture/inbox/my_move.mp4 --name dj_scratch_01
# → clips_out/dj_scratch_01.fbx  +  clips_out/dj_scratch_01_qc.mp4

The FBX is anim-only, Mixamo-"Without-Skin"-style, on the mixamorig skeleton shared by every GODVERSE character — it drops into the same merge_anims.py → GLB clip-bank flow that feeds 90sDJsim, THRIFTGOD, and Unreal (A_3D_STORE).

---

Contents

1. How it works
2. Quickstart
3. The instruction manual
4. Test results — the evidence
5. Known limits & open items
6. Legal
7. Development history

---

How it works

you, filmed on a phone (docs/CAPTURE.md protocol)
      │  AirDrop → capture/inbox/
      ▼
LANE A  ingest.py          probe → normalize 30fps h264 → queue/  → LEDGER.tsv
      ▼
LANE B  pose_engine.py     HSMR (ViT-H, CVPR'25) recovers the SKEL biomechanical
        [.engine venv,      skeleton per frame on Apple-Silicon MPS
         ~1.6× realtime]    → out/<stem>/motion.bvh (SKEL-24, Y-up, metres)
                            → motion.json sidecar (fps, root_space, joints…)
                            → params.npz (raw poses — rerun cheaply with --from-params)
      ▼
LANE C  retarget.py        headless Blender 5: BVH → mixamorig (20-bone world-space
                            constraint map + visual-keying bake) → anim-only FBX
                            + Workbench turntable QC mp4 (HOUSE RULE: eyeball it)
      ▼
LANE D  mocap (CLI)        the one command that chains A→C and enforces the naming
                            contract inbox → queue → out → clips_out

• Engine: HSMR (MIT code) predicting SKEL (Max-Planck biomechanical body model, v1.1.1) — chosen over SMPL-family models (GVHMR/WHAM/TRAM) after a license-verified bake-off. Full analysis: docs/B1_BAKEOFF.md, docs/DECISIONS.md, docs/LICENSES.md.
• All Apple Silicon: the ViT-H recovery runs on MPS at 31 fps; detection runs CPU mobilenet every-20th-frame with box reuse (single-subject assumption).
• Formats treaty (PLAN.md §Ground-rules): BVH@30fps + JSON sidecar between stages; metres / Y-up / faces −Z; root_space: "camera"|"world" is a required sidecar field.

Quickstart

# 0. One-time: engine env (if fresh clone)         ~10 min + weights download
.engine/setup_hsmr.sh --patch          # clone, venv, weights, apply ARM patches
#    …then place the gated SKEL v1.1.1 files (docs/B_ENGINE_SETUP.md — John's login)

# 1. Film per docs/CAPTURE.md  (landscape, FEET IN FRAME, T-pose bookends, one move/take)
# 2. AirDrop to capture/inbox/
# 3. One command:
./mocap capture/inbox/IMG_4242.mp4 --name head_nod_01
# 4. WATCH clips_out/head_nod_01_qc.mp4   ← house rule, every clip, before it enters a bank

Flags: --character female|male (female = proven path) · --travel (keep world root; default in-place) · --no-qc · --gender (pose model override) · --selfcheck.

The instruction manual

Filming (docs/CAPTURE.md — the 20-second version)

Landscape, phone steady at 2–3m, your WHOLE body including feet in frame the entire take, even light, fitted clothes, 1 second of T-pose at start and end, one move per take, 5–30s. Break these and the skeleton quality drops — feet out of frame is the #1 killer.

Running stages by hand (debugging / tuning)

python3 ingest.py capture/inbox/vid.mp4 --name test          # Lane A alone
.engine/HSMR/.venv/bin/python pose_engine.py queue/<stem>.mp4 --gender female   # Lane B
.engine/HSMR/.venv/bin/python pose_engine.py <stem> --from-params   # re-emit BVH, skip HMR (fast)
/Applications/Blender.app/Contents/MacOS/Blender --background --python retarget.py -- \
    --bvh out/<stem>/motion.bvh --name test                  # Lane C alone

Where things live

Path	What
capture/inbox/ → capture/done/	drop zone → processed originals (ingest MOVES them)
capture/LEDGER.tsv	every ingest, accepted or rejected, with reason
queue/	normalized 30fps videos awaiting the pose engine
out/<stem>/	motion.bvh + motion.json + params.npz per clip
clips_out/	the product: <name>.fbx + <name>_qc.mp4
.engine/HSMR/	gitignored engine clone; rebuild any time via setup_hsmr.sh --patch
engine_patches/	our ARM/headless fixes (in git; survive re-clones)
spec/	skeleton ground truth: mixamorig_bones.json, skel_to_mixamorig.json

Into the games / Unreal

• Clip banks: merge FBX clips onto a character with character_kit/scripts/merge_anims.py → multi-clip GLB (⚠️ needs the Blender-5 slot-bind fix — task spawned; see Known limits).
• Unreal: import the merged GLB via the A_3D_STORE deferred-import pattern (slate post-tick, never a blocking import through the MCP bridge — see the unreal-asset-convergence memory / A_3D_STORE experiments).

Troubleshooting

Symptom	Cause / fix
✗ HSMR venv missing	run .engine/setup_hsmr.sh --patch
smoke/engine fails at skel_male.pkl	gated SKEL v1.1.1 not placed — docs/B_ENGINE_SETUP.md
clip rejected at ingest	check capture/LEDGER.tsv note column (portrait? too long? corrupt?)
feet slide in QC	expected pre-foot-fix; worse on --travel (grounding pass not built yet)
QC motion is soup	filming protocol violated (feet cropped? multiple people in frame?)
everything re-detects slowly	detector runs every 20th frame; long multi-person videos will crawl — single subject only

Test results — the evidence

Every claim below was produced during the build and re-verified independently in review (Fable re-ran tests rather than trusting the ledger). Raw logs: out/*.log, montages: out/qc_montage.png, out/mocap_qc_montage.png.

Detector unit test (session 4, re-run in review #1)

Metric	Result
Demo images with person found	7/7
Human patches extracted @256×256	15
detectron2 dependency	eliminated (torchvision drop-in, ARM-native)

Engine smoke test — first full forward pass (2026-07-17, gymnasts.mp4, 263 frames)

Metric	Result	Bar
Device	MPS (Apple Silicon GPU)	must not be CPU
Patches recovered	1,315	—
Recovery throughput	31.2 fps	> realtime ✅
NaNs in output	0	0 ✅
Finding	CPU detector was 80% of wall-clock → fixed in B2 (mobilenet + every-20)

Lane B acceptance (session 5)

Metric	Result	Bar
Steady-state pipeline speed	~1.6× realtime	<2× ✅
BVH sanity (Blender import)	24 bones, upright, 1.59m figure	human-shaped ✅
FK self-consistency	0.0 mm error	—
Unattended queue drain	6 videos: 4 OK, 2 bad clips failed gracefully	no crash ✅
Axis correctness	Y-down OpenCV → Y-up/−Z via diag(1,−1,−1), verified upright	✅

Lane C acceptance (session 6)

Check	Result
Bones constraint-mapped	20 (+4 intentionally null: twist/heel; fingers = Lane E)
Retargeted FBX drives female_game.glb	✅ verified — RightHand moved +121 units; playable 7.3MB GLB
QC montage eyeball	coherent gymnast motion, no soup, no candy-wrap arm twist
Blender 5 "Baklava" API breaks	found + solved (slot-bind, fcurves walk) — documented in docs/C_RETARGET.md

Lane D acceptance + independent review #2 (fresh end-to-end, 2026-07-17)

Metric	Opus run (gymnasts_d)	Fable review run (review_check)
Full chain A→C one command	✅	✅ (cold, renamed input)
Frames processed	317 @ 30fps	317 @ 30fps
FK error	0.0 mm	0.0 mm
Sidecar treaty fields	✅	✅ (root_space:"camera", 24 joints, fps 30)
Output FBX + QC	✅	✅ 685KB FBX + QC mp4, frames eyeballed clean
Wall clock (10.6s clip)	—	~35s engine total incl. one-time model build

Review #2 findings: one real bug (FBX take-name is generic Armature|Scene on every clip → bank-merge collisions; fix queued in retarget.py before first real clip banks) and one cross-repo action (Blender-5 fix for character_kit merge_anims.py — task spawned).

Known limits & open items

Honest list — v1 is machinery-complete, not footage-verified.

• 🎬 Final acceptance awaits John's real footage (walk / dance / pick-up-object). Everything below the line is gated on it.
• 🐛 FBX take-name bug (see review #2) — MUST fix before clips enter banks.
• 🦶 Foot-contact fix is a stub — feet may slide, tuned against a real walk clip.
• 🚶 --travel (world root) is best-effort: HSMR is per-frame → camera-space root. In-place clips (DJ moves!) are first-class; locomotion needs the grounding pass.
• ✋ Fingers are static (Lane E: hand pass).
• 🧍 Male rig path unverified until a male clip is run.
• 📧 Path-A license email drafted, unsent (docs/DECISIONS.md).
• Lanes E (MLX training: autoposer, inbetweening, centaur retarget) and F (drop-folder daemon, dashboard) are designed but intentionally not started.

Legal

Read docs/LICENSES.md before changing engines or training anything. The short version: HSMR code is MIT; the SKEL body model (and BioAMASS dataset) are Max-Planck research-licensed — commercial use is John's accepted Path-B judgment call, with the Path-A commercial-license email drafted to retire the risk. We ship only motion retargeted onto our own rigs; no Max-Planck file is ever redistributed. Mixamo clips: OK in games, never as training data. Cascadeur: used as a licensed authoring tool, never reverse-engineered.

Development history

Built 2026-07-17, PLAN.md → seven Opus sessions, reviewed twice by Fable (see PROGRESS.md for the full ledger including the honest mistakes): session 1 bake-off picked an animal-only model by accident → session 2 caught and reversed it with a license-verified re-analysis → session 3 locked the HSMR/SKEL decision + MPS → session 4 broke the detectron2 ARM wall with a torchvision drop-in → John supplied the gated SKEL v1.1.1 → smoke test passed first try → session 5 shipped the pose engine (B2+B3)

• six review orders → session 6 shipped the Blender retarget (and discovered the Blender-5 animation API changes the hard way) → session 7 shipped the mocap CLI → review #2 re-ran everything cold and it held.