Compare commits

...

3 Commits

Author SHA1 Message Date
type-two
c918ac3dcd lane C+D: retarget + mocap CLI + docs; fix: FBX take carries the clip name (was Armature|Scene for every clip — bank collisions in take-addressed consumers)
First generated-footage clip proven through the full chain: Wan2.1 T2V (m3ultra, prompt-engineered to CAPTURE.md rules) -> ./mocap -> wan_walk_t1.fbx -> merge_anims bank (49 tracks). PROGRESS.md bug closed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 14:06:38 +10:00
type-two
4cf939b90b README: mega manual + verified test evidence
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 12:30:42 +10:00
type-two
782e8c8e20 Review #2: lanes B-D verified independently (E2E green); FBX take-name bug + Blender5 cross-repo debt logged
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 12:22:13 +10:00
6 changed files with 763 additions and 5 deletions

View File

@ -1,5 +1,29 @@
# MOCAPGOD progress ledger
## FABLE REVIEW #2 (2026-07-17, post lanes BD) — VERDICT: SHIPPED, one bug, one cross-repo action
Independently verified (not trusted): fresh end-to-end `./mocap` run on a renamed demo clip →
full chain green (317f, FK 0.0mm, treaty-conformant sidecar `root_space:"camera"` @30fps/24
joints, FBX 685KB + QC mp4; QC frames eyeballed = clean upright motion on female_game).
Detect-every-20 confirmed working (16 detector steps/317 frames). Docs honest; deferred items
real deferrals, not fakes. Path-A email draft is good — John should send it.
**BUG (fix before first real clip enters a bank): exported FBX take/action name is generic
(`Armature|Scene`), identical for every clip** (bake even transiently borrows an existing bank
clip's name). merge_anims.py keys NLA tracks off action names → our own clips will collide in
the bank. Fix in `retarget.py`: after bake, `action.name = <name>` and ensure the FBX take
exports as `<name>`. Acceptance: `strings clips_out/<n>.fbx | grep <n>` OR Blender import
lists action `<n>`.
**CROSS-REPO DEBT (out of MOCAPGOD scope, task spawned):** Blender 5 "Baklava" slot-bind —
`character_kit/scripts/merge_anims.py` is pre-5.0 and silently drives nothing on Blender 5
without `animation_data.action_slot = act.slots[0]` (session-6 discovery, verified). Affects
90sDJsim/OPSHOPGAME clip merging fleet-wide, not just MOCAPGOD.
**Remaining to v1 sign-off (unchanged, all John):** film walk/dance/pick-up per CAPTURE.md.
Then: quality verdict on real footage, tune foot-fix on the walk clip, verify male rig, wire
the deferred bank-merge/Unreal hooks. Send the MPI email.
One lane per session. Tick on completion. See PLAN.md for acceptance checks.
- [x] **Lane 0** — bootstrap. git + .gitignore, docs/LICENSES.md skeleton,
@ -25,8 +49,19 @@ One lane per session. Tick on completion. See PLAN.md for acceptance checks.
`.engine/HSMR/data_inputs/body_models/skel/skel_male.pkl` (+ `skel_female.pkl`) exist;
(2) run the forward-pass smoke test (MPS / timing <2× / NaN check); then B2 (service-ize
BVH) + B3 (jitter). Lanes C and D remain blocked on B.
- [ ] **Lane C** — retarget + cleanup (Blender). bvh → mixamorig FBX + QC turntable.
- [ ] **Lane D** — clip factory. `mocap` CLI end-to-end; games + Unreal integration.
- [x] **Lane C** — retarget + cleanup (Blender). **DONE (machinery) session 6 (2026-07-17).**
`retarget.py`: SKEL-24 BVH → mixamorig (spec/skel_to_mixamorig.json, world Copy-Rotation + bake)
→ anim-only FBX + Workbench turntable QC mp4. Verified A→C on the gymnasts demo: coherent human
motion (montage eyeballed, not soup, no arm twist); FBX **merges onto female_game.glb and drives
it** (RightHand +121u → playable GLB) after the Blender-5 slot-bind. Foot-contact fix deferred
(stub) until a real walk clip; final quality sign-off awaits John's footage. See `docs/C_RETARGET.md`.
- [x] **Lane D** — clip factory. **DONE (machinery) session 7 (2026-07-17).** `mocap` CLI wraps
ingest → pose_engine → retarget into ONE command; verified end-to-end on the gymnasts demo
(`./mocap capture/inbox/gymnasts_d.mp4 --name gymnasts_d` → 317f @ 30fps, FK 0.0mm →
`clips_out/gymnasts_d.fbx` + QC mp4, montage eyeballed = clean human motion). RUNBOOK.md
written. **Deferred (honest, not faked as dead flags):** clip-bank GLB merge, `--to-unreal`,
`--publish-3god` — all off-by-default per PLAN, gated on real footage + touch other systems;
how-to in RUNBOOK. **Final acceptance (clip in 90sDJsim + Unreal) still needs John's footage.**
- [ ] Lane E — ML upgrades (post-v1, gated).
- [ ] Lane F — UX (optional).
@ -100,10 +135,41 @@ is done; go straight to B2/B3).
the honest quality verdict + all of B's acceptance).
- Reboot m3ultra when convenient (applies staged 26.5.2 security update; housekeeping).
## LANE C — DONE (machinery), session 6 (2026-07-17)
`retarget.py` (headless Blender 5.0) is the deliverable; full writeup in `docs/C_RETARGET.md`.
- Proven A→C on the gymnasts demo: `pose_engine.py <clip>``out/<name>/motion.bvh`
`retarget.py --bvh …``clips_out/<name>.fbx` + `<name>_qc.mp4`. QC montage eyeballed = recognizable
human motion, no soup, no catastrophic arm twist (world Copy-Rotation held up — both rests are T-pose).
- **Merge verified**: `gymnasts.fbx` merges onto `female_game.glb` and drives it, exports a playable GLB —
BUT only after the **Blender-5 Baklava slot-bind** (`anim_data.action_slot = act.slots[0]`).
**HANDOFF TO LANE D (clip factory + integration):**
- The `mocap` CLI wraps ingest.py → pose_engine.py (local/ssh) → retarget.py → clips_out/. All three exist.
- **Merge to the clip-bank GLB MUST add the slot-bind line**`character_kit/scripts/merge_anims.py` is
pre-Baklava and will silently produce non-animating GLBs on Blender 5.0 without it.
- Also inherited by Blender-5 code: `Action.fcurves` gone (use layer/strip/channelbag walk); no FFMPEG
image output (render PNG frames + system ffmpeg); mixamo rig is ~156u tall, scale 0.01 — never scale it.
- Deferred from C (do when John's real footage lands): foot-contact IK (stub now — feet slide), `--travel`
root scaling, and the arms/hands quality pass (upgrade `map_and_bake` to rest-delta if DJ hands twist).
## LANE D — DONE (machinery), session 7 (2026-07-17)
`mocap` (repo root, executable) is the deliverable; full flow in `docs/RUNBOOK.md`.
- Wiring: `./mocap <video> --name <n> [--character female|male] [--travel] [--no-qc]`
ingest (moves source) → pose_engine (HSMR venv, MPS) → retarget (Blender) → `clips_out/<n>.fbx`
+ `<n>_qc.mp4`. `--name` flows to retarget so the final clip name stays clean despite ingest's
date-prefixed intermediates. `./mocap --selfcheck` guards the path-threading (no engine).
- **Proven A→D on gymnasts demo** (`out/mocap_qc_montage.png` eyeballed): one command, exit 0,
treaty-correct 30fps (real ingest normalizes; the earlier hand-run was 24.85fps).
- **v1 DONE for machinery.** The only thing between here and the PLAN's Lane D acceptance
("clip in 90sDJsim + Unreal") is (a) John's footage and (b) the three deferred publish hooks —
both intentionally not built blind. When footage lands: run `mocap` on it, eyeball QC, then
build the GLB-merge hook first (games load GLB) per RUNBOOK — that's the "lands in 90sDJsim" step.
## Notes for the next lane
- **Lane D can't be started until B + C ship + test footage exists** — it's the CLI that wraps
ingest → pose_engine.py → retarget.py, and the latter two don't exist yet. Session 3 (the
"Lane D" session) correctly built nothing; it locked Decision 1 and de-risked MPS instead.
- **UNCOMMITTED as of session 7:** Lane C (`retarget.py`, `docs/C_RETARGET.md`) was left
uncommitted by session 6, and Lane D (`mocap`, `docs/RUNBOOK.md`, this PROGRESS tick) sits on
top. Two clean per-lane commits are ready to make on John's word.
- Lane D built the CLI (was blocked two lanes deep at session 3; B shipped session 5, C session 6).
- **SKEL version trap:** HSMR needs **v1.1.1** exactly (`changelog_v1.1.1.txt`); older = silent
incompat. Source: `.engine/HSMR/docs/SETUP.md` (register skel.is.tue.mpg.de → download.php).
- **Engine wiring + ARM bypasses live in `docs/B_ENGINE_SETUP.md`** — all three ARM landmines

210
README.md Normal file
View File

@ -0,0 +1,210 @@
# MOCAPGOD
**Your own private motion-capture studio.** Film yourself with a phone → get a game-ready
animation clip on your character rigs. One command, 100% local, 100% owned, zero cloud,
zero subscriptions. Built 2026-07-17 in seven Opus 4.8 sessions + two Fable review passes.
```
./mocap capture/inbox/my_move.mp4 --name dj_scratch_01
# → clips_out/dj_scratch_01.fbx + clips_out/dj_scratch_01_qc.mp4
```
The FBX is anim-only, Mixamo-"Without-Skin"-style, on the **mixamorig skeleton shared by
every GODVERSE character** — it drops into the same `merge_anims.py` → GLB clip-bank flow
that feeds 90sDJsim, THRIFTGOD, and Unreal (A_3D_STORE).
---
## Contents
1. [How it works](#how-it-works)
2. [Quickstart](#quickstart)
3. [The instruction manual](#the-instruction-manual)
4. [Test results — the evidence](#test-results--the-evidence)
5. [Known limits & open items](#known-limits--open-items)
6. [Legal](#legal)
7. [Development history](#development-history)
---
## How it works
```
you, filmed on a phone (docs/CAPTURE.md protocol)
│ AirDrop → capture/inbox/
LANE A ingest.py probe → normalize 30fps h264 → queue/ → LEDGER.tsv
LANE B pose_engine.py HSMR (ViT-H, CVPR'25) recovers the SKEL biomechanical
[.engine venv, skeleton per frame on Apple-Silicon MPS
~1.6× realtime] → out/<stem>/motion.bvh (SKEL-24, Y-up, metres)
→ motion.json sidecar (fps, root_space, joints…)
→ params.npz (raw poses — rerun cheaply with --from-params)
LANE C retarget.py headless Blender 5: BVH → mixamorig (20-bone world-space
constraint map + visual-keying bake) → anim-only FBX
+ Workbench turntable QC mp4 (HOUSE RULE: eyeball it)
LANE D mocap (CLI) the one command that chains A→C and enforces the naming
contract inbox → queue → out → clips_out
```
- **Engine:** [HSMR](https://github.com/IsshikiHugh/HSMR) (MIT code) predicting **SKEL**
(Max-Planck biomechanical body model, v1.1.1) — chosen over SMPL-family models
(GVHMR/WHAM/TRAM) after a license-verified bake-off. Full analysis:
`docs/B1_BAKEOFF.md`, `docs/DECISIONS.md`, `docs/LICENSES.md`.
- **All Apple Silicon:** the ViT-H recovery runs on MPS at 31 fps; detection runs CPU
mobilenet every-20th-frame with box reuse (single-subject assumption).
- **Formats treaty** (PLAN.md §Ground-rules): BVH@30fps + JSON sidecar between stages;
metres / Y-up / faces Z; `root_space: "camera"|"world"` is a required sidecar field.
## Quickstart
```bash
# 0. One-time: engine env (if fresh clone) ~10 min + weights download
.engine/setup_hsmr.sh --patch # clone, venv, weights, apply ARM patches
# …then place the gated SKEL v1.1.1 files (docs/B_ENGINE_SETUP.md — John's login)
# 1. Film per docs/CAPTURE.md (landscape, FEET IN FRAME, T-pose bookends, one move/take)
# 2. AirDrop to capture/inbox/
# 3. One command:
./mocap capture/inbox/IMG_4242.mp4 --name head_nod_01
# 4. WATCH clips_out/head_nod_01_qc.mp4 ← house rule, every clip, before it enters a bank
```
Flags: `--character female|male` (female = proven path) · `--travel` (keep world root;
default in-place) · `--no-qc` · `--gender` (pose model override) · `--selfcheck`.
## The instruction manual
### Filming (docs/CAPTURE.md — the 20-second version)
Landscape, phone steady at 23m, your WHOLE body including feet in frame the entire take,
even light, fitted clothes, **1 second of T-pose at start and end**, one move per take,
530s. Break these and the skeleton quality drops — feet out of frame is the #1 killer.
### Running stages by hand (debugging / tuning)
```bash
python3 ingest.py capture/inbox/vid.mp4 --name test # Lane A alone
.engine/HSMR/.venv/bin/python pose_engine.py queue/<stem>.mp4 --gender female # Lane B
.engine/HSMR/.venv/bin/python pose_engine.py <stem> --from-params # re-emit BVH, skip HMR (fast)
/Applications/Blender.app/Contents/MacOS/Blender --background --python retarget.py -- \
--bvh out/<stem>/motion.bvh --name test # Lane C alone
```
### Where things live
| Path | What |
|---|---|
| `capture/inbox/``capture/done/` | drop zone → processed originals (ingest MOVES them) |
| `capture/LEDGER.tsv` | every ingest, accepted or rejected, with reason |
| `queue/` | normalized 30fps videos awaiting the pose engine |
| `out/<stem>/` | motion.bvh + motion.json + params.npz per clip |
| `clips_out/` | **the product**: `<name>.fbx` + `<name>_qc.mp4` |
| `.engine/HSMR/` | gitignored engine clone; rebuild any time via `setup_hsmr.sh --patch` |
| `engine_patches/` | our ARM/headless fixes (in git; survive re-clones) |
| `spec/` | skeleton ground truth: `mixamorig_bones.json`, `skel_to_mixamorig.json` |
### Into the games / Unreal
- **Clip banks:** merge FBX clips onto a character with
`character_kit/scripts/merge_anims.py` → multi-clip GLB (⚠️ needs the Blender-5
slot-bind fix — task spawned; see Known limits).
- **Unreal:** import the merged GLB via the A_3D_STORE deferred-import pattern
(slate post-tick, never a blocking import through the MCP bridge — see the
unreal-asset-convergence memory / A_3D_STORE experiments).
### Troubleshooting
| Symptom | Cause / fix |
|---|---|
| `✗ HSMR venv missing` | run `.engine/setup_hsmr.sh --patch` |
| smoke/engine fails at `skel_male.pkl` | gated SKEL v1.1.1 not placed — docs/B_ENGINE_SETUP.md |
| clip rejected at ingest | check `capture/LEDGER.tsv` note column (portrait? too long? corrupt?) |
| feet slide in QC | expected pre-foot-fix; worse on `--travel` (grounding pass not built yet) |
| QC motion is soup | filming protocol violated (feet cropped? multiple people in frame?) |
| everything re-detects slowly | detector runs every 20th frame; long multi-person videos will crawl — single subject only |
## Test results — the evidence
Every claim below was produced during the build and **re-verified independently in review**
(Fable re-ran tests rather than trusting the ledger). Raw logs: `out/*.log`, montages:
`out/qc_montage.png`, `out/mocap_qc_montage.png`.
### Detector unit test (session 4, re-run in review #1)
| Metric | Result |
|---|---|
| Demo images with person found | **7/7** |
| Human patches extracted @256×256 | 15 |
| detectron2 dependency | **eliminated** (torchvision drop-in, ARM-native) |
### Engine smoke test — first full forward pass (2026-07-17, gymnasts.mp4, 263 frames)
| Metric | Result | Bar |
|---|---|---|
| Device | **MPS** (Apple Silicon GPU) | must not be CPU |
| Patches recovered | 1,315 | — |
| Recovery throughput | **31.2 fps** | > realtime ✅ |
| NaNs in output | **0** | 0 ✅ |
| Finding | CPU detector was 80% of wall-clock → fixed in B2 (mobilenet + every-20) | |
### Lane B acceptance (session 5)
| Metric | Result | Bar |
|---|---|---|
| Steady-state pipeline speed | **~1.6× realtime** | <2× |
| BVH sanity (Blender import) | 24 bones, upright, **1.59m** figure | human-shaped ✅ |
| FK self-consistency | **0.0 mm** error | — |
| Unattended queue drain | 6 videos: 4 OK, **2 bad clips failed gracefully** | no crash ✅ |
| Axis correctness | Y-down OpenCV → Y-up/Z via diag(1,1,1), verified upright | ✅ |
### Lane C acceptance (session 6)
| Check | Result |
|---|---|
| Bones constraint-mapped | 20 (+4 intentionally null: twist/heel; fingers = Lane E) |
| Retargeted FBX drives female_game.glb | ✅ verified — RightHand moved +121 units; playable 7.3MB GLB |
| QC montage eyeball | coherent gymnast motion, no soup, no candy-wrap arm twist |
| Blender 5 "Baklava" API breaks | found + solved (slot-bind, fcurves walk) — documented in docs/C_RETARGET.md |
### Lane D acceptance + independent review #2 (fresh end-to-end, 2026-07-17)
| Metric | Opus run (`gymnasts_d`) | Fable review run (`review_check`) |
|---|---|---|
| Full chain A→C one command | ✅ | ✅ (cold, renamed input) |
| Frames processed | 317 @ 30fps | 317 @ 30fps |
| FK error | 0.0 mm | 0.0 mm |
| Sidecar treaty fields | ✅ | ✅ (`root_space:"camera"`, 24 joints, fps 30) |
| Output FBX + QC | ✅ | ✅ 685KB FBX + QC mp4, frames eyeballed clean |
| Wall clock (10.6s clip) | — | ~35s engine total incl. one-time model build |
**Review #2 findings:** one real bug (FBX take-name is generic `Armature|Scene` on every
clip → bank-merge collisions; fix queued in retarget.py before first real clip banks) and
one cross-repo action (Blender-5 fix for character_kit merge_anims.py — task spawned).
## Known limits & open items
**Honest list — v1 is machinery-complete, not footage-verified.**
- 🎬 **Final acceptance awaits John's real footage** (walk / dance / pick-up-object).
Everything below the line is gated on it.
- 🐛 FBX take-name bug (see review #2) — MUST fix before clips enter banks.
- 🦶 Foot-contact fix is a stub — feet may slide, tuned against a real walk clip.
- 🚶 `--travel` (world root) is best-effort: HSMR is per-frame → camera-space root.
In-place clips (DJ moves!) are first-class; locomotion needs the grounding pass.
- ✋ Fingers are static (Lane E: hand pass).
- 🧍 Male rig path unverified until a male clip is run.
- 📧 Path-A license email drafted, unsent (docs/DECISIONS.md).
- Lanes E (MLX training: autoposer, inbetweening, centaur retarget) and F (drop-folder
daemon, dashboard) are designed but intentionally not started.
## Legal
Read `docs/LICENSES.md` before changing engines or training anything. The short version:
HSMR **code** is MIT; the **SKEL body model** (and BioAMASS dataset) are Max-Planck
research-licensed — commercial use is John's accepted Path-B judgment call, with the Path-A
commercial-license email drafted to retire the risk. **We ship only motion retargeted onto
our own rigs; no Max-Planck file is ever redistributed.** Mixamo clips: OK in games, never
as training data. Cascadeur: used as a licensed authoring tool, never reverse-engineered.
## Development history
Built 2026-07-17, PLAN.md → seven Opus sessions, reviewed twice by Fable (see PROGRESS.md
for the full ledger including the honest mistakes):
session 1 bake-off picked an animal-only model by accident → session 2 caught and reversed
it with a license-verified re-analysis → session 3 locked the HSMR/SKEL decision + MPS →
session 4 broke the detectron2 ARM wall with a torchvision drop-in → John supplied the
gated SKEL v1.1.1 → smoke test passed first try → session 5 shipped the pose engine (B2+B3)
+ six review orders → session 6 shipped the Blender retarget (and discovered the Blender-5
animation API changes the hard way) → session 7 shipped the `mocap` CLI → review #2
re-ran everything cold and it held.

43
docs/C_RETARGET.md Normal file
View File

@ -0,0 +1,43 @@
# Lane C — retarget + cleanup (`retarget.py`)
**DONE (machinery) 2026-07-17 (session 6).** `retarget.py` takes a SKEL-24 `motion.bvh` (from
`pose_engine.py`) onto the character_kit mixamorig rig → baked anim-only FBX + QC turntable mp4.
Verified on the gymnasts demo: recognizable human motion (not soup), and the FBX merges onto
`female_game.glb` and drives it. Final quality sign-off awaits John's own walk/DJ footage.
## Run (headless Blender only)
```
/Applications/Blender.app/Contents/MacOS/Blender --background --python retarget.py -- \
--bvh out/<name>/motion.bvh [--rig <glb>] [--name <n>] [--travel] [--foot-fix] [--no-qc]
```
Defaults: rig = `character_kit/female/female_game.glb`, out = `clips_out/`, root in-place.
Outputs: `clips_out/<name>.fbx` (anim-only, Mixamo "Without Skin") + `clips_out/<name>_qc.mp4`.
Proven chain A→C: `pose_engine.py <clip>``out/<name>/motion.bvh``retarget.py --bvh …`.
## How it works
- Import mixamorig rig (glb) + SKEL BVH (`axis_forward=-Z, axis_up=Y`; both land Z-up in Blender).
- Per `spec/skel_to_mixamorig.json` (20 bones; 4 null twist/heel), add **world-space Copy Rotation**
(target mixamorig ← SKEL bone), then `nla.bake(visual_keying, clear_constraints)` → a clean
mixamorig action. `find_bone` tolerates the `mixamorig:` colon vs `mixamorig_` underscore trap.
- Root **in-place by default** (HSMR root is camera-space, not world-grounded); `--travel` copies it.
- Export anim-only FBX (`object_types={ARMATURE}`, `bake_anim`). QC = Workbench turntable, PNG frames
stitched to mp4 by system ffmpeg, camera auto-framed from the world bbox.
## Blender 5.0 gotchas hit + resolved (carry into Lane D)
- **Baklava actions**: `Action.fcurves` is gone → `action_fcurves()` walks layers/strips/channelbags.
- **Merge needs slot binding** (load-bearing): assigning an action is NOT enough — you must set
`animation_data.action_slot = act.slots[0]` or **nothing drives**. Verified: with the slot bound,
`gymnasts.fbx` drives `female_game.glb` (RightHand +121u), exports a playable 7.3 MB GLB.
**`character_kit/scripts/merge_anims.py` FIXED 2026-07-17** — binds `strip.action_slot` on each NLA strip (hasattr-guarded for 4.x); verified Lifting_Object drives female_game.glb (LeftHand delta 6.7u frame 1→138).
- **No FFMPEG image format** in Blender 5.0 → render PNG frames + system `ffmpeg` stitch.
- **Mixamo scale**: `female_game.glb` imports at armature-scale 0.01 but ≈156 units tall, centred z≈77.
QC auto-frames from the bbox; keep retarget **rotation-only** — never scale the rig (scaling a skinned
ancestor tears the mesh when it animates; the `merge_anims.py` units warning).
## Deferred (honest — not gold-plated)
- **Foot-contact fix is a stub** (`--foot-fix` logs planted frames only, no IK → feet may slide).
Deferred until a real **walk** clip exists to tune against (gymnasts are airborne). `# ponytail` in code.
- **`--travel`** (root translation + unit scaling) is wired but unverified; in-place is v1-first-class.
- **Arms/hands quality**: gymnasts looked clean (no catastrophic roll). If DJ hand work twists on John's
footage, upgrade the single `map_and_bake` to a rest-delta transfer (`Rt=Rs·Rs_rest⁻¹·Rt_rest`) — noted
in code; nothing else changes.

56
docs/RUNBOOK.md Normal file
View File

@ -0,0 +1,56 @@
# MOCAPGOD runbook — film → clip, one command
The whole v1 pipeline on one page. Everything runs on **ultra**.
## The one command
```
cd ~/Documents/MOCAPGOD
./mocap capture/inbox/<vid>.mp4 --name dj_scratch_01 # female rig, in-place
./mocap <vid>.mov --name spin --character male --travel # male rig, keep world travel
```
Options: `--character female|male` (default female), `--travel` (keep root translation; default
in-place), `--no-qc`, `--gender` (override pose gender), `--selfcheck`.
**Output:** `clips_out/<name>.fbx` (anim-only, Mixamo "Without Skin") + `clips_out/<name>_qc.mp4`.
**House rule:** watch the QC mp4 before the clip enters any bank. Not soup = good.
## What it chains (all proven; `mocap` is just the glue)
| Lane | Script | Does | Emits | Runs in |
|---|---|---|---|---|
| A | `ingest.py` | probe → 30fps h264 → rename → queue. **Moves** the source to `capture/done/`. | `queue/<date>_<name>.mp4` | system python3 |
| B | `pose_engine.py` | HSMR/SKEL monocular recovery + SKEL→BVH + jitter | `out/<stem>/motion.bvh` + `motion.json` + `params.npz` | `.engine/HSMR/.venv` (MPS) |
| C | `retarget.py` | BVH → mixamorig (constraint + bake) → FBX + turntable QC | `clips_out/<name>.fbx` + `_qc.mp4` | headless Blender 5 |
Timing: pose engine dominates (~1.6× realtime on ultra/MPS); a 10s clip ≈ a couple of minutes.
## Prerequisites (one-time)
- **SKEL v1.1.1** at `.engine/HSMR/data_inputs/body_models/skel/skel_male.pkl` (+ female). Gated
download — see `docs/B_ENGINE_SETUP.md`. This is the only hard blocker if the engine is fresh.
- HSMR env built: `.engine/setup_hsmr.sh --patch` (idempotent). Blender at `/Applications/Blender.app`.
- Filming protocol: `docs/CAPTURE.md` (landscape, **feet in frame**, T-pose bookends, one move/take).
## Known-good vs. rough edges (honest)
- **In-place clips are first-class.** HSMR is per-frame → root is camera-space, not world-grounded
(`motion.json: root_space:"camera"`). DJ moves / dancing-on-the-spot come out clean.
- **Travel (`--travel`) is best-effort** until a foot-contact grounding pass exists — walking clips
may drift/slide. Foot-fix is a logged stub today (`retarget.py foot_fix`); tune it against a real
walk clip (Lane C deferred).
- **Fingers are Lane E** — the body clip has static hands. Fine for v1 NPC/DJ body moves.
- Male rig (`--character male` → `hum_character.glb`) is **unverified** until a male clip is run;
female (`female_game.glb`) is the proven path.
## Deferred hooks — build these when real footage lands (not faked in the CLI)
Each is off by default and needs a real clip to verify, so none ship as dead flags today.
1. **Clip-bank GLB merge** (`--merge`, → 90sDJsim / three.js games). Games load GLB, so this is the
"lands in-game" step. Needs a small merge script first: `character_kit/scripts/merge_anims.py`
has **no CLI and is pre-Baklava** — it must add the Blender-5 slot-bind
(`animation_data.action_slot = act.slots[0]`, verified in Lane C) or the merged GLB won't animate.
Then merge `clips_out/<name>.fbx` onto the character's bank GLB and export.
2. **`--to-unreal`** — import via the A_3D_STORE deferred-import pattern (memory:
`unreal-asset-convergence` — slate post-tick, NOT a direct AssetImportTask through the bridge).
3. **`--publish-3god`** — depot pattern from procity `publish.py`.
## John's homework (blocks the real quality verdict)
Film **walk / dance / pick-up-object** per `docs/CAPTURE.md`, drop in `capture/inbox/`, run `mocap`.
Until then the pipeline is proven on the HSMR `gymnasts` demo only — machinery works; the honest
"is it good enough for my DJ NPCs" call needs footage of you.

119
mocap Executable file
View File

@ -0,0 +1,119 @@
#!/usr/bin/env python3
"""MOCAPGOD Lane D — `mocap`: one command, phone video → game-ready mocap clip.
./mocap capture/inbox/vid.mp4 --name dj_scratch_01
./mocap spin.mov --name spin --character male --travel
Chains the three proven stages, all on ultra:
Lane A ingest.py video → 30fps h264 → queue/<date>_<name>.mp4 (MOVES the source)
Lane B pose_engine.py video → HSMR/SKEL → out/<stem>/motion.bvh + sidecar [HSMR venv, MPS]
Lane C retarget.py BVH → mixamorig → clips_out/<name>.fbx + <name>_qc.mp4 [headless Blender]
Result: clips_out/<name>.fbx (anim-only, Mixamo "Without Skin") + <name>_qc.mp4 turntable.
House rule: eyeball the QC mp4 before the clip enters a bank.
Deferred hooks (off by default — built when John's real footage lands; see docs/RUNBOOK.md):
clip-bank GLB merge, --to-unreal, --publish-3god. merge_anims.py needs a CLI + the Blender-5
slot-bind first, and none of them are verifiable without a real clip, so they're not faked here.
"""
import argparse, datetime, re, subprocess, sys
from pathlib import Path
REPO = Path(__file__).resolve().parent
QUEUE = REPO / "queue"
OUT = REPO / "out"
CLIPS = REPO / "clips_out"
VENV_PY = REPO / ".engine/HSMR/.venv/bin/python" # Lane B lives in its own env
BLENDER = Path("/Applications/Blender.app/Contents/MacOS/Blender")
KIT = Path.home() / "Documents/character_kit"
RIGS = { # --character → (pose gender, retarget rig)
"female": ("female", KIT / "female/female_game.glb"), # v1 proven path
"male": ("male", KIT / "hum_character.glb"), # ponytail: male rig unverified until a male clip exists
}
def slug(s): # mirrors ingest.slug (kept standalone on purpose)
return re.sub(r"[^A-Za-z0-9_-]+", "_", s).strip("_") or "clip"
def paths_for(name):
"""The naming contract that threads a clip through all three stages. The one bit of real
logic here, so it gets the self-check below."""
stem = f"{datetime.date.today():%Y%m%d}_{name}" # ingest's queue naming
return stem, QUEUE / f"{stem}.mp4", OUT / stem / "motion.bvh", CLIPS / f"{name}.fbx"
def stage(title, cmd):
print(f"\n▶ {title}\n $ {' '.join(str(c) for c in cmd)}", flush=True)
if subprocess.run([str(c) for c in cmd]).returncode != 0:
sys.exit(f"✗ {title} failed — chain stopped.")
def selfcheck():
"""Path-threading only: no engine, no network, no Blender."""
stem, q, bvh, fbx = paths_for("dj_scratch_01")
assert stem.endswith("_dj_scratch_01"), stem
assert q == QUEUE / f"{stem}.mp4", q
assert bvh == OUT / stem / "motion.bvh", bvh
assert fbx == CLIPS / "dj_scratch_01.fbx", fbx # final name stays clean (retarget --name)
assert slug("DJ Scratch #1!") == "DJ_Scratch_1", slug("DJ Scratch #1!")
for c in RIGS.values():
assert c[0] in ("male", "female")
print("selfcheck OK — name threads inbox→queue→out→clips, final clip name stays clean")
return 0
def main():
ap = argparse.ArgumentParser(description="Lane D: one command, video → game-ready mocap clip.")
ap.add_argument("video", nargs="?", help="source clip (e.g. capture/inbox/move.mp4) — ingest MOVES it")
ap.add_argument("--name", help="clip name → clips_out/<name>.fbx")
ap.add_argument("--character", choices=list(RIGS), default="female")
ap.add_argument("--gender", choices=["male", "female"], help="override pose gender (default: from --character)")
ap.add_argument("--travel", action="store_true", help="keep world root translation (default: in-place)")
ap.add_argument("--no-qc", action="store_true", help="skip the QC turntable render")
ap.add_argument("--selfcheck", action="store_true", help="verify the path-threading logic and exit")
a = ap.parse_args()
if a.selfcheck:
return selfcheck()
if not a.video or not a.name:
ap.error("video and --name are required (or use --selfcheck)")
if not VENV_PY.exists():
sys.exit(f"✗ HSMR venv missing: {VENV_PY} — run .engine/setup_hsmr.sh (Lane B).")
if not BLENDER.exists():
sys.exit(f"✗ Blender not found: {BLENDER} (Lane C needs it).")
name = slug(a.name)
gender, rig = RIGS[a.character]
gender = a.gender or gender
if not rig.exists():
sys.exit(f"✗ rig for --character {a.character} not found: {rig}")
stem, queued, bvh, fbx = paths_for(name)
stage("Lane A · ingest", ["python3", REPO / "ingest.py", a.video, "--name", name])
if not queued.exists():
sys.exit(f"✗ no {queued} after ingest — clip rejected? check capture/LEDGER.tsv")
stage("Lane B · pose_engine (HSMR/SKEL, MPS — slow, ~1.6× realtime)",
[VENV_PY, REPO / "pose_engine.py", queued, "--gender", gender])
if not bvh.exists():
sys.exit(f"✗ no {bvh} after pose_engine.")
rt = [BLENDER, "--background", "--python", REPO / "retarget.py", "--",
"--bvh", bvh, "--rig", rig, "--name", name]
if a.travel: rt.append("--travel")
if a.no_qc: rt.append("--no-qc")
stage("Lane C · retarget + QC (Blender)", rt)
if not fbx.exists():
sys.exit(f"✗ no {fbx} after retarget.")
qc = CLIPS / f"{name}_qc.mp4"
print(f"\n✅ clip ready: {fbx}")
if qc.exists():
print(f" eyeball before it enters a bank: {qc}")
print(" merge-to-GLB / Unreal / 3god hooks are deferred — docs/RUNBOOK.md.")
return 0
if __name__ == "__main__":
sys.exit(main())

264
retarget.py Normal file
View File

@ -0,0 +1,264 @@
#!/usr/bin/env python3
"""Lane C — retarget a SKEL-24 BVH onto the character_kit mixamorig rig + QC render.
Headless Blender only:
/Applications/Blender.app/Contents/MacOS/Blender --background --python retarget.py -- \
--bvh out/<name>/motion.bvh [--rig <female_game.glb>] [--name <n>] [--travel] [--no-qc]
Pipeline: import mixamorig rig (female_game.glb) + the SKEL BVH Copy-Rotation-map SKEL bones
onto mixamorig (spec/skel_to_mixamorig.json), world-space bake to keyframes trim near-rest
bookends export anim-only FBX (Mixamo "Without Skin", merge_anims.py eats these) Workbench
turntable QC mp4. Root is in-place by default (HSMR root is camera-space; --travel copies it).
# ponytail: constraint-bake retarget (world Copy Rotation). If the QC shows arm-roll twist
# (SKEL vs mixamorig rest-roll differ), upgrade the ONE function `map_and_bake` to a rest-delta
# transfer (Rt_f = Rs_f · Rs_rest⁻¹ · Rt_rest); everything else stays.
"""
import bpy, json, sys, os
from pathlib import Path
from mathutils import Matrix
REPO = Path(__file__).resolve().parent
DEF_RIG = Path.home() / "Documents" / "character_kit" / "female" / "female_game.glb"
MAP = REPO / "spec" / "skel_to_mixamorig.json"
def args_after_ddash():
a = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else []
import argparse
p = argparse.ArgumentParser()
p.add_argument("--bvh", required=True)
p.add_argument("--rig", default=str(DEF_RIG))
p.add_argument("--name", default=None)
p.add_argument("--out", default=str(REPO / "clips_out"))
p.add_argument("--travel", action="store_true", help="copy root translation (default: in-place)")
p.add_argument("--foot-fix", action="store_true", help="clamp planted feet to the floor (heuristic)")
p.add_argument("--no-qc", action="store_true")
return p.parse_args(a)
def clean():
bpy.ops.wm.read_factory_settings(use_empty=True)
def imported(op, **kw):
before = set(bpy.data.objects)
op(**kw)
return [o for o in bpy.data.objects if o not in before]
def find_bone(arm, name):
"""Tolerate mixamorig: colon vs _ underscore variants (the PLAN's warned trap)."""
pbs = arm.pose.bones
if name in pbs:
return pbs[name]
alt = name.replace(":", "_") if ":" in name else name.replace("mixamorig_", "mixamorig:")
return pbs.get(alt)
def action_fcurves(action):
"""Blender 5.0 removed Action.fcurves for the layered ('Baklava') system — collect fcurves
from layers/strips/channelbags, with a fallback to the legacy flat list."""
if hasattr(action, "fcurves"):
return list(action.fcurves)
fcs = []
for layer in getattr(action, "layers", []):
for strip in layer.strips:
for cb in getattr(strip, "channelbags", []):
fcs.extend(cb.fcurves)
return fcs
def map_and_bake(target, source, name_map, f0, f1, travel):
"""Add world-space Copy Rotation constraints target←source per the map, then bake to keys."""
bpy.context.view_layer.objects.active = target
n = 0
for skel_bone, mixamo in name_map.items():
if not mixamo: # null = twist/heel, folded elsewhere
continue
pb = find_bone(target, mixamo)
if pb is None or skel_bone not in source.pose.bones:
print(f" skip {skel_bone}->{mixamo} (missing)")
continue
c = pb.constraints.new("COPY_ROTATION")
c.target, c.subtarget = source, skel_bone
c.target_space = c.owner_space = "WORLD"
c.mix_mode = "REPLACE"
n += 1
if travel: # root position (units differ; scale it)
hips = find_bone(target, name_map["pelvis"])
c = hips.constraints.new("COPY_LOCATION")
c.target, c.subtarget = source, "pelvis"
c.target_space = c.owner_space = "WORLD"
c.use_offset = True
print(f"mapped {n} bones; baking frames {f0}..{f1}")
bpy.ops.object.mode_set(mode="POSE")
bpy.ops.pose.select_all(action="SELECT")
bpy.ops.nla.bake(frame_start=f0, frame_end=f1, only_selected=True, visual_keying=True,
clear_constraints=True, use_current_action=True, bake_types={"POSE"})
bpy.ops.object.mode_set(mode="OBJECT")
def trim_bookends(arm, f0, f1, thresh_deg=6.0):
"""Trim leading/trailing frames that sit near the rest pose (T-pose bookends from CAPTURE.md).
Tolerant of absence (demo clips have none) returns the untouched range if nothing qualifies.
# ponytail: rest-deviation heuristic on the action's keyed rotations; good enough for trimming."""
import math
act = arm.animation_data.action if arm.animation_data else None
if not act:
return f0, f1
try:
# mean absolute keyed rotation magnitude per frame (quaternion channels), vs rest
rot_fcurves = [fc for fc in action_fcurves(act)
if fc.data_path.endswith("rotation_quaternion")]
except Exception as e:
print(f"trim: fcurve access failed ({e!r}); keeping full range")
return f0, f1
if not rot_fcurves:
return f0, f1
def near_rest(fr):
# quaternion identity is (1,0,0,0); deviation ~ how far w drifts from 1 across bones
dev = 0.0
for fc in rot_fcurves:
if fc.array_index == 0: # w component
dev += abs(1.0 - fc.evaluate(fr))
return math.degrees(dev) < thresh_deg
a, b = f0, f1
while a < b and near_rest(a):
a += 1
while b > a and near_rest(b):
b -= 1
if (a, b) != (f0, f1):
print(f"trimmed bookends: {f0}..{f1} -> {a}..{b}")
return a, b
def foot_fix(arm, f0, f1):
"""# ponytail: minimal stub — real IK foot-pinning is Lane-C-plus. Logs planted frames so the
upgrade has a target; does not yet re-solve. Enable heavy version when sliding annoys."""
print("foot-fix: heuristic stub (no IK yet) — feet may slide; upgrade path noted in PLAN")
def export_fbx(arm, path):
bpy.ops.object.select_all(action="DESELECT")
arm.select_set(True)
bpy.context.view_layer.objects.active = arm
bpy.ops.export_scene.fbx(
filepath=str(path), use_selection=True, object_types={"ARMATURE"},
add_leaf_bones=False, bake_anim=True, bake_anim_use_all_bones=True,
bake_anim_use_nla_strips=False, bake_anim_use_all_actions=False,
armature_nodetype="NULL", path_mode="COPY",
)
print(f"exported anim-only FBX -> {path} ({os.path.getsize(path)//1024}KB)")
def qc_render(target_arm, mesh_objs, f0, f1, path, fps):
"""Workbench turntable: camera orbits the character while the clip plays. Solid-shaded =
fast + reads the pose clearly (the QC eyeball the PLAN mandates before a clip enters the bank)."""
import math
from mathutils import Vector
# auto-frame from the character's world bbox (mixamo rigs import at odd scales — this one is
# ~156 units tall, centred at z≈77). Scale-agnostic so any rig frames correctly.
cs = []
for m in mesh_objs:
for c in m.bound_box:
cs.append(m.matrix_world @ Vector(c))
mn = Vector((min(c.x for c in cs), min(c.y for c in cs), min(c.z for c in cs)))
mx = Vector((max(c.x for c in cs), max(c.y for c in cs), max(c.z for c in cs)))
center = (mn + mx) / 2
size = max((mx - mn).x, (mx - mn).y, (mx - mn).z) or 1.0
dist = size * 1.9
piv = bpy.data.objects.new("qc_pivot", None)
piv.location = center
bpy.context.scene.collection.objects.link(piv)
cam_data = bpy.data.cameras.new("qc_cam")
cam_data.clip_start, cam_data.clip_end = size * 0.01, dist * 4
cam = bpy.data.objects.new("qc_cam", cam_data)
bpy.context.scene.collection.objects.link(cam)
cam.parent = piv
cam.matrix_parent_inverse = Matrix()
cam.location = (0, -dist, 0) # local to pivot → orbits at eye level
tc = cam.constraints.new("TRACK_TO")
tc.target = piv
bpy.context.scene.camera = cam
piv.rotation_euler = (0, 0, 0)
piv.keyframe_insert("rotation_euler", frame=f0)
piv.rotation_euler = (0, 0, math.radians(360))
piv.keyframe_insert("rotation_euler", frame=f1)
for fc in action_fcurves(piv.animation_data.action): # linear spin, no ease
for kp in fc.keyframe_points:
kp.interpolation = "LINEAR"
import subprocess, shutil, tempfile
sc = bpy.context.scene
sc.render.engine = "BLENDER_WORKBENCH"
sc.frame_start, sc.frame_end = f0, f1
sc.render.resolution_x = sc.render.resolution_y = 640
# Blender 5.0 dropped FFMPEG image output → render PNG frames, stitch with system ffmpeg.
frames = Path(tempfile.mkdtemp())
sc.render.image_settings.file_format = "PNG"
sc.render.filepath = str(frames / "f_")
bpy.ops.render.render(animation=True)
ff = shutil.which("ffmpeg") or "/opt/homebrew/bin/ffmpeg"
r = subprocess.run([ff, "-y", "-framerate", f"{fps:.3f}", "-start_number", str(f0),
"-i", str(frames / "f_%04d.png"), "-c:v", "libx264", "-pix_fmt", "yuv420p",
"-vf", "scale=trunc(iw/2)*2:trunc(ih/2)*2", str(path)],
capture_output=True, text=True)
if r.returncode or not Path(path).exists():
print("QC ffmpeg stitch FAILED:", r.stderr[-400:])
else:
print(f"QC render -> {path} ({os.path.getsize(path)//1024}KB)")
def main():
a = args_after_ddash()
bvh = Path(a.bvh).resolve()
name = a.name or bvh.parent.name
outdir = Path(a.out); outdir.mkdir(parents=True, exist_ok=True)
name_map = json.loads(MAP.read_text())["skel_to_mixamorig"]
clean()
rig_objs = imported(bpy.ops.import_scene.gltf, filepath=a.rig)
target = next(o for o in rig_objs if o.type == "ARMATURE")
meshes = [o for o in rig_objs if o.type == "MESH"]
print(f"rig: armature '{target.name}', {len(target.data.bones)} bones, {len(meshes)} mesh(es)")
src_objs = imported(bpy.ops.import_anim.bvh, filepath=str(bvh),
axis_forward="-Z", axis_up="Y", use_fps_scale=False, update_scene_fps=True)
source = next(o for o in src_objs if o.type == "ARMATURE")
act = source.animation_data.action
f0, f1 = (int(round(x)) for x in act.frame_range)
fps = bpy.context.scene.render.fps / max(bpy.context.scene.render.fps_base, 1e-9)
print(f"bvh: source '{source.name}', frames {f0}..{f1} @ ~{fps:.2f}fps")
map_and_bake(target, source, name_map, f0, f1, a.travel)
bpy.data.objects.remove(source, do_unlink=True) # drop the SKEL rig, keep baked mixamorig
if a.foot_fix:
foot_fix(target, f0, f1)
f0, f1 = trim_bookends(target, f0, f1)
fbx = outdir / f"{name}.fbx"
# the FBX take inherits the action name — un-named it exports as 'Armature|Scene' for EVERY clip,
# colliding in any take-addressed bank (Unreal/3GOD). merge_anims renames from the filename, but
# the take itself must be right (PROGRESS.md bug, fixed 2026-07-17).
target.animation_data.action.name = name
export_fbx(target, fbx)
# sanity: the FBX action must carry keys on the driven bones
assert target.animation_data and target.animation_data.action, "no baked action on target"
nkeys = len(action_fcurves(target.animation_data.action))
print(f"baked action '{target.animation_data.action.name}': {nkeys} fcurves")
assert nkeys > 20, f"too few baked channels ({nkeys}) — retarget likely failed"
if not a.no_qc:
qc_render(target, meshes, f0, f1, outdir / f"{name}_qc.mp4", fps)
print(f"LANE-C DONE: {name} -> {fbx.name}" + ("" if a.no_qc else f" + {name}_qc.mp4"))
if __name__ == "__main__":
main()