diff --git a/README.md b/README.md new file mode 100644 index 0000000..c49ffca --- /dev/null +++ b/README.md @@ -0,0 +1,210 @@ +# MOCAPGOD + +**Your own private motion-capture studio.** Film yourself with a phone → get a game-ready +animation clip on your character rigs. One command, 100% local, 100% owned, zero cloud, +zero subscriptions. Built 2026-07-17 in seven Opus 4.8 sessions + two Fable review passes. + +``` +./mocap capture/inbox/my_move.mp4 --name dj_scratch_01 +# → clips_out/dj_scratch_01.fbx + clips_out/dj_scratch_01_qc.mp4 +``` + +The FBX is anim-only, Mixamo-"Without-Skin"-style, on the **mixamorig skeleton shared by +every GODVERSE character** — it drops into the same `merge_anims.py` → GLB clip-bank flow +that feeds 90sDJsim, THRIFTGOD, and Unreal (A_3D_STORE). + +--- + +## Contents +1. [How it works](#how-it-works) +2. [Quickstart](#quickstart) +3. [The instruction manual](#the-instruction-manual) +4. [Test results — the evidence](#test-results--the-evidence) +5. [Known limits & open items](#known-limits--open-items) +6. [Legal](#legal) +7. [Development history](#development-history) + +--- + +## How it works + +``` +you, filmed on a phone (docs/CAPTURE.md protocol) + │ AirDrop → capture/inbox/ + ▼ +LANE A ingest.py probe → normalize 30fps h264 → queue/ → LEDGER.tsv + ▼ +LANE B pose_engine.py HSMR (ViT-H, CVPR'25) recovers the SKEL biomechanical + [.engine venv, skeleton per frame on Apple-Silicon MPS + ~1.6× realtime] → out//motion.bvh (SKEL-24, Y-up, metres) + → motion.json sidecar (fps, root_space, joints…) + → params.npz (raw poses — rerun cheaply with --from-params) + ▼ +LANE C retarget.py headless Blender 5: BVH → mixamorig (20-bone world-space + constraint map + visual-keying bake) → anim-only FBX + + Workbench turntable QC mp4 (HOUSE RULE: eyeball it) + ▼ +LANE D mocap (CLI) the one command that chains A→C and enforces the naming + contract inbox → queue → out → clips_out +``` + +- **Engine:** [HSMR](https://github.com/IsshikiHugh/HSMR) (MIT code) predicting **SKEL** + (Max-Planck biomechanical body model, v1.1.1) — chosen over SMPL-family models + (GVHMR/WHAM/TRAM) after a license-verified bake-off. Full analysis: + `docs/B1_BAKEOFF.md`, `docs/DECISIONS.md`, `docs/LICENSES.md`. +- **All Apple Silicon:** the ViT-H recovery runs on MPS at 31 fps; detection runs CPU + mobilenet every-20th-frame with box reuse (single-subject assumption). +- **Formats treaty** (PLAN.md §Ground-rules): BVH@30fps + JSON sidecar between stages; + metres / Y-up / faces −Z; `root_space: "camera"|"world"` is a required sidecar field. + +## Quickstart + +```bash +# 0. One-time: engine env (if fresh clone) ~10 min + weights download +.engine/setup_hsmr.sh --patch # clone, venv, weights, apply ARM patches +# …then place the gated SKEL v1.1.1 files (docs/B_ENGINE_SETUP.md — John's login) + +# 1. Film per docs/CAPTURE.md (landscape, FEET IN FRAME, T-pose bookends, one move/take) +# 2. AirDrop to capture/inbox/ +# 3. One command: +./mocap capture/inbox/IMG_4242.mp4 --name head_nod_01 +# 4. WATCH clips_out/head_nod_01_qc.mp4 ← house rule, every clip, before it enters a bank +``` + +Flags: `--character female|male` (female = proven path) · `--travel` (keep world root; +default in-place) · `--no-qc` · `--gender` (pose model override) · `--selfcheck`. + +## The instruction manual + +### Filming (docs/CAPTURE.md — the 20-second version) +Landscape, phone steady at 2–3m, your WHOLE body including feet in frame the entire take, +even light, fitted clothes, **1 second of T-pose at start and end**, one move per take, +5–30s. Break these and the skeleton quality drops — feet out of frame is the #1 killer. + +### Running stages by hand (debugging / tuning) +```bash +python3 ingest.py capture/inbox/vid.mp4 --name test # Lane A alone +.engine/HSMR/.venv/bin/python pose_engine.py queue/.mp4 --gender female # Lane B +.engine/HSMR/.venv/bin/python pose_engine.py --from-params # re-emit BVH, skip HMR (fast) +/Applications/Blender.app/Contents/MacOS/Blender --background --python retarget.py -- \ + --bvh out//motion.bvh --name test # Lane C alone +``` + +### Where things live +| Path | What | +|---|---| +| `capture/inbox/` → `capture/done/` | drop zone → processed originals (ingest MOVES them) | +| `capture/LEDGER.tsv` | every ingest, accepted or rejected, with reason | +| `queue/` | normalized 30fps videos awaiting the pose engine | +| `out//` | motion.bvh + motion.json + params.npz per clip | +| `clips_out/` | **the product**: `.fbx` + `_qc.mp4` | +| `.engine/HSMR/` | gitignored engine clone; rebuild any time via `setup_hsmr.sh --patch` | +| `engine_patches/` | our ARM/headless fixes (in git; survive re-clones) | +| `spec/` | skeleton ground truth: `mixamorig_bones.json`, `skel_to_mixamorig.json` | + +### Into the games / Unreal +- **Clip banks:** merge FBX clips onto a character with + `character_kit/scripts/merge_anims.py` → multi-clip GLB (⚠️ needs the Blender-5 + slot-bind fix — task spawned; see Known limits). +- **Unreal:** import the merged GLB via the A_3D_STORE deferred-import pattern + (slate post-tick, never a blocking import through the MCP bridge — see the + unreal-asset-convergence memory / A_3D_STORE experiments). + +### Troubleshooting +| Symptom | Cause / fix | +|---|---| +| `✗ HSMR venv missing` | run `.engine/setup_hsmr.sh --patch` | +| smoke/engine fails at `skel_male.pkl` | gated SKEL v1.1.1 not placed — docs/B_ENGINE_SETUP.md | +| clip rejected at ingest | check `capture/LEDGER.tsv` note column (portrait? too long? corrupt?) | +| feet slide in QC | expected pre-foot-fix; worse on `--travel` (grounding pass not built yet) | +| QC motion is soup | filming protocol violated (feet cropped? multiple people in frame?) | +| everything re-detects slowly | detector runs every 20th frame; long multi-person videos will crawl — single subject only | + +## Test results — the evidence + +Every claim below was produced during the build and **re-verified independently in review** +(Fable re-ran tests rather than trusting the ledger). Raw logs: `out/*.log`, montages: +`out/qc_montage.png`, `out/mocap_qc_montage.png`. + +### Detector unit test (session 4, re-run in review #1) +| Metric | Result | +|---|---| +| Demo images with person found | **7/7** | +| Human patches extracted @256×256 | 15 | +| detectron2 dependency | **eliminated** (torchvision drop-in, ARM-native) | + +### Engine smoke test — first full forward pass (2026-07-17, gymnasts.mp4, 263 frames) +| Metric | Result | Bar | +|---|---|---| +| Device | **MPS** (Apple Silicon GPU) | must not be CPU | +| Patches recovered | 1,315 | — | +| Recovery throughput | **31.2 fps** | > realtime ✅ | +| NaNs in output | **0** | 0 ✅ | +| Finding | CPU detector was 80% of wall-clock → fixed in B2 (mobilenet + every-20) | | + +### Lane B acceptance (session 5) +| Metric | Result | Bar | +|---|---|---| +| Steady-state pipeline speed | **~1.6× realtime** | <2× ✅ | +| BVH sanity (Blender import) | 24 bones, upright, **1.59m** figure | human-shaped ✅ | +| FK self-consistency | **0.0 mm** error | — | +| Unattended queue drain | 6 videos: 4 OK, **2 bad clips failed gracefully** | no crash ✅ | +| Axis correctness | Y-down OpenCV → Y-up/−Z via diag(1,−1,−1), verified upright | ✅ | + +### Lane C acceptance (session 6) +| Check | Result | +|---|---| +| Bones constraint-mapped | 20 (+4 intentionally null: twist/heel; fingers = Lane E) | +| Retargeted FBX drives female_game.glb | ✅ verified — RightHand moved +121 units; playable 7.3MB GLB | +| QC montage eyeball | coherent gymnast motion, no soup, no candy-wrap arm twist | +| Blender 5 "Baklava" API breaks | found + solved (slot-bind, fcurves walk) — documented in docs/C_RETARGET.md | + +### Lane D acceptance + independent review #2 (fresh end-to-end, 2026-07-17) +| Metric | Opus run (`gymnasts_d`) | Fable review run (`review_check`) | +|---|---|---| +| Full chain A→C one command | ✅ | ✅ (cold, renamed input) | +| Frames processed | 317 @ 30fps | 317 @ 30fps | +| FK error | 0.0 mm | 0.0 mm | +| Sidecar treaty fields | ✅ | ✅ (`root_space:"camera"`, 24 joints, fps 30) | +| Output FBX + QC | ✅ | ✅ 685KB FBX + QC mp4, frames eyeballed clean | +| Wall clock (10.6s clip) | — | ~35s engine total incl. one-time model build | + +**Review #2 findings:** one real bug (FBX take-name is generic `Armature|Scene` on every +clip → bank-merge collisions; fix queued in retarget.py before first real clip banks) and +one cross-repo action (Blender-5 fix for character_kit merge_anims.py — task spawned). + +## Known limits & open items + +**Honest list — v1 is machinery-complete, not footage-verified.** +- 🎬 **Final acceptance awaits John's real footage** (walk / dance / pick-up-object). + Everything below the line is gated on it. +- 🐛 FBX take-name bug (see review #2) — MUST fix before clips enter banks. +- 🦶 Foot-contact fix is a stub — feet may slide, tuned against a real walk clip. +- 🚶 `--travel` (world root) is best-effort: HSMR is per-frame → camera-space root. + In-place clips (DJ moves!) are first-class; locomotion needs the grounding pass. +- ✋ Fingers are static (Lane E: hand pass). +- 🧍 Male rig path unverified until a male clip is run. +- 📧 Path-A license email drafted, unsent (docs/DECISIONS.md). +- Lanes E (MLX training: autoposer, inbetweening, centaur retarget) and F (drop-folder + daemon, dashboard) are designed but intentionally not started. + +## Legal + +Read `docs/LICENSES.md` before changing engines or training anything. The short version: +HSMR **code** is MIT; the **SKEL body model** (and BioAMASS dataset) are Max-Planck +research-licensed — commercial use is John's accepted Path-B judgment call, with the Path-A +commercial-license email drafted to retire the risk. **We ship only motion retargeted onto +our own rigs; no Max-Planck file is ever redistributed.** Mixamo clips: OK in games, never +as training data. Cascadeur: used as a licensed authoring tool, never reverse-engineered. + +## Development history + +Built 2026-07-17, PLAN.md → seven Opus sessions, reviewed twice by Fable (see PROGRESS.md +for the full ledger including the honest mistakes): +session 1 bake-off picked an animal-only model by accident → session 2 caught and reversed +it with a license-verified re-analysis → session 3 locked the HSMR/SKEL decision + MPS → +session 4 broke the detectron2 ARM wall with a torchvision drop-in → John supplied the +gated SKEL v1.1.1 → smoke test passed first try → session 5 shipped the pose engine (B2+B3) ++ six review orders → session 6 shipped the Blender retarget (and discovered the Blender-5 +animation API changes the hard way) → session 7 shipped the `mocap` CLI → review #2 +re-ran everything cold and it held.