mocapgod/PROGRESS.md

# MOCAPGOD progress ledger

## FABLE REVIEW #2 (2026-07-17, post lanes B–D) — VERDICT: SHIPPED, one bug, one cross-repo action

Independently verified (not trusted): fresh end-to-end `./mocap` run on a renamed demo clip →
full chain green (317f, FK 0.0mm, treaty-conformant sidecar `root_space:"camera"` @30fps/24
joints, FBX 685KB + QC mp4; QC frames eyeballed = clean upright motion on female_game).
Detect-every-20 confirmed working (16 detector steps/317 frames). Docs honest; deferred items
real deferrals, not fakes. Path-A email draft is good — John should send it.

**BUG (fix before first real clip enters a bank): exported FBX take/action name is generic
(`Armature|Scene`), identical for every clip** (bake even transiently borrows an existing bank
clip's name). merge_anims.py keys NLA tracks off action names → our own clips will collide in
the bank. Fix in `retarget.py`: after bake, `action.name = <name>` and ensure the FBX take
exports as `<name>`. Acceptance: `strings clips_out/<n>.fbx | grep <n>` OR Blender import
lists action `<n>`.

**CROSS-REPO DEBT (out of MOCAPGOD scope, task spawned):** Blender 5 "Baklava" slot-bind —
`character_kit/scripts/merge_anims.py` is pre-5.0 and silently drives nothing on Blender 5
without `animation_data.action_slot = act.slots[0]` (session-6 discovery, verified). Affects
90sDJsim/OPSHOPGAME clip merging fleet-wide, not just MOCAPGOD.

**Remaining to v1 sign-off (unchanged, all John):** film walk/dance/pick-up per CAPTURE.md.
Then: quality verdict on real footage, tune foot-fix on the walk clip, verify male rig, wire
the deferred bank-merge/Unreal hooks. Send the MPI email.

One lane per session. Tick on completion. See PLAN.md for acceptance checks.

- [x] **Lane 0** — bootstrap. git + .gitignore, docs/LICENSES.md skeleton,
      spec/mixamorig_bones.json (65 bones, `mixamorig:` colon-prefixed). — 2026-07-17
- [x] **Lane A** — capture + ingest. docs/CAPTURE.md protocol, ingest.py
      (probe → 30fps normalize → rename → rsync to m3ultra queue → LEDGER.tsv). — 2026-07-17
- [x] **Lane B** — pose engine. **DONE session 5 (2026-07-17).** `pose_engine.py` ships:
      queue video → HSMR/SKEL → SKEL-24 BVH@fps + `motion.json` sidecar (`root_space:"camera"`),
      `spec/skel_to_mixamorig.json` name-map for Lane C. Verified on the demo clip: Blender import
      = 24 bones, **upright** (1.59 m), motion present, FK self-consistency **0.0 mm**; queue of 6
      drained unattended (2 bad clips failed gracefully, 4 valid BVH, heartbeat per clip); runtime
      **~1.6× realtime** steady-state (under the <2× bar). mobilenet detector + detect-every-20 +
      single-subject box reuse retired the CPU-detector wall-clock finding. Axis fix: HSMR emits
      Y-down (OpenCV) → `diag(1,-1,-1)` → Y-up/faces-−Z. **History (sessions 2–4):**
      **Decision made session 3: Path B** — engine =
      **HSMR → SKEL**, run local, ship rig-motion only, John accepts NC-licence risk
      (`docs/DECISIONS.md` Decision 1 RESOLVED). Env up in `.engine/HSMR` (uv venv, torch 2.13
      **MPS green**, ungated weights pulled). **Detector RESOLVED (session 4):** torchvision
      Faster R-CNN drop-in replaces the unbuildable detectron2; `test_detector.py` is green today
      (7/7 demo imgs → 15 patches, **no SKEL needed**). `smoke_test.py` now runs real detection +
      patch-crop and reaches the **exact SKEL gate**; chumpy cleared as a non-blocker. **Only open
      before B ships:** (1) John downloads gated **SKEL v1.1.1** so
      `.engine/HSMR/data_inputs/body_models/skel/skel_male.pkl` (+ `skel_female.pkl`) exist;
      (2) run the forward-pass smoke test (MPS / timing <2× / NaN check); then B2 (service-ize →
      BVH) + B3 (jitter). Lanes C and D remain blocked on B.
- [x] **Lane C** — retarget + cleanup (Blender). **DONE (machinery) session 6 (2026-07-17).**
      `retarget.py`: SKEL-24 BVH → mixamorig (spec/skel_to_mixamorig.json, world Copy-Rotation + bake)
      → anim-only FBX + Workbench turntable QC mp4. Verified A→C on the gymnasts demo: coherent human
      motion (montage eyeballed, not soup, no arm twist); FBX **merges onto female_game.glb and drives
      it** (RightHand +121u → playable GLB) after the Blender-5 slot-bind. Foot-contact fix deferred
      (stub) until a real walk clip; final quality sign-off awaits John's footage. See `docs/C_RETARGET.md`.
- [x] **Lane D** — clip factory. **DONE (machinery) session 7 (2026-07-17).** `mocap` CLI wraps
      ingest → pose_engine → retarget into ONE command; verified end-to-end on the gymnasts demo
      (`./mocap capture/inbox/gymnasts_d.mp4 --name gymnasts_d` → 317f @ 30fps, FK 0.0mm →
      `clips_out/gymnasts_d.fbx` + QC mp4, montage eyeballed = clean human motion). RUNBOOK.md
      written. **Deferred (honest, not faked as dead flags):** clip-bank GLB merge, `--to-unreal`,
      `--publish-3god` — all off-by-default per PLAN, gated on real footage + touch other systems;
      how-to in RUNBOOK. **Final acceptance (clip in 90sDJsim + Unreal) still needs John's footage.**
- [ ] Lane E — ML upgrades (post-v1, gated).
- [ ] Lane F — UX (optional).

## SESSION 5 — DONE (2026-07-17). All six orders executed; Lane B ships.
1. ✅ **Durable-ized** the ARM fixes → `engine_patches/` + `setup_hsmr.sh --patch`. Verified: fresh
   clone + `--patch` → `test_detector.py` green, no hand edits.
2. ✅ **Path-A email** drafted in `docs/DECISIONS.md` (John sends to ps-license@tue.mpg.de).
3. ✅ **Queue reconciled**: `ingest.py` defaults to local `queue/` on ultra; `--remote-queue` keeps
   the m3ultra path. PLAN Ground-rule-1 deviation noted; CAPTURE.md updated.
4. ✅ **Treaty amended**: `root_space` (camera|world) is a REQUIRED sidecar field; Lane C reads it.
5. ✅ **B2 + B3**: `pose_engine.py` (SKEL-24 BVH + sidecar + jitter) + `spec/skel_to_mixamorig.json`.
   Detector swapped to mobilenet; detect-every-20 + box reuse; build-once + per-video isolation.
   Ran the smoke test's SKEL forward on real params — no re-verify needed (smoke already PASSED).

**HANDOFF TO LANE C (retarget + cleanup, Blender, on ultra):**
- Input is `out/<name>/motion.bvh` (SKEL-24 skeleton, **Y-up, metres, faces −Z**) + `motion.json`.
  Import in Blender with `import_anim.bvh(axis_forward='-Z', axis_up='Y')` (verified upright).
- Constraint-map SKEL→mixamorig via **`spec/skel_to_mixamorig.json`** (20 mapped, 4 null=twist/heel;
  Spine1 interpolated, Head follows Neck, fingers are Lane E). mixamorig canon: `mixamorig:` colon.
- **`root_space` is `"camera"`** → root is NOT world-grounded (HSMR is per-frame). In-place clips are
  first-class; for traveling clips derive a world root from foot contacts (PLAN Lane C task).
- Rest pose is a clean T-pose at zero rotation; T-pose bookends only exist if John filmed the protocol
  (demo clips don't have them) — Lane C's trim must tolerate their absence.
- Re-run the SKEL→BVH stage cheaply with `pose_engine.py <clip> --from-params` (skips HMR) when
  tuning; raw params are saved at `out/<name>/params.npz`.

## SESSION 5 ORDERS — from Fable's review (2026-07-17, supersede older notes where they conflict)

Review verdict: Lanes 0/A solid; session 2's reversal of the B1 animal-model blunder was the
right call, well executed; ARM engineering verified independently (test_detector.py re-run
green 7/7 by reviewer). Proceed as follows, in order:

1. **Durable-ize the gitignored engine fixes (do first, ~30 min).** The torchvision detector,
   `_headless.py`, `test_detector.py`, `smoke_test.py` exist ONLY on ultra's disk inside the
   gitignored `.engine/`. Move canonical copies into git at `engine_patches/` and make
   `setup_hsmr.sh` apply them after clone. Acceptance: fresh clone in a temp dir + script →
   `test_detector.py` green with no hand edits.
2. **Draft the Path-A email (parallel risk-retirement, ~15 min).** Write the SKEL indie
   commercial-license request to ps-license@tue.mpg.de into `docs/DECISIONS.md` (project
   description, solo Australian dev, indie revenue, SKEL-in-pipeline-not-redistributed usage).
   John sends it himself. Path B continues meanwhile; a granted license converts B→A with zero
   rework. Do not wait on the reply.
3. **Reconcile the split-brain queue (~30 min).** Lane A's ingest.py rsyncs to m3ultra's queue,
   but the engine (and MPS verification) is on ultra — nothing consumes that queue. For v1:
   default ingest to LOCAL `queue/` on ultra (`--remote-queue` flag preserves the m3ultra path),
   note the deviation from PLAN Ground-rule 1 in PLAN.md with a `# ponytail: v1 inference on
   ultra; migrate to m3ultra when a real throughput need appears` marker. Update CAPTURE.md.
4. **Treaty amendment (small, load-bearing).** `motion.json` sidecar gains REQUIRED field
   `root_space: "camera" | "world"`. HSMR emits `"camera"`. Lane C must read it and, for
   locomotion clips, derive world root from foot contacts (already in notes). v1 quality bar:
   in-place clips are first-class; traveling clips best-effort until a grounding pass exists.
**SMOKE TEST RAN 2026-07-17 (Fable, after John's SKEL v1.1.1 download landed — installed at
the expected path): PASSED with findings.** device=mps, 263 frames → 1315 patches, recovery
31.2 fps, **nan=False**, `data_outputs/smoke/params.npz` written. Findings for B2: (1) the
CPU Faster R-CNN detector is the wall-clock hog (~1 fps, 4min of the 5min run) — B2 MUST add
detect-every-Nth-frame + box reuse/IoU tracking (single-subject footage makes this easy) or
the mobilenet swap; HMR itself is faster than realtime so the <2× acceptance is reachable.
(2) `prepare_mesh` MPS→numpy crash fixed in smoke_test.py (`.detach().cpu()`) — carry the
same pattern into pose_engine.py. Item 5 below is now UNBLOCKED (skip re-verifying: smoke
is done; go straight to B2/B3).

5. **If (and only if) SKEL v1.1.1 files are in place** (`data_inputs/body_models/skel/
   skel_male.pkl`): run smoke_test.py — MPS forward pass, runtime <2× realtime on demo video,
   NaN check — then proceed straight into B2 (params.npz → `spec/skel_to_mixamorig.json` map →
   BVH@30fps + sidecar per treaty) and B3 (One-Euro/Sav-Gol jitter pass, root smoothed
   separately). If SKEL absent: finish items 1–4, verify Decision-3 footage status, stop clean.

**John's homework (all three block different things):**
- SKEL v1.1.1 download (the ONLY hard blocker; instructions in `docs/B_ENGINE_SETUP.md`).
- Film walk / dance / pick-up-object per `docs/CAPTURE.md`, drop in `capture/inbox/` (blocks
  the honest quality verdict + all of B's acceptance).
- Reboot m3ultra when convenient (applies staged 26.5.2 security update; housekeeping).

## LANE C — DONE (machinery), session 6 (2026-07-17)
`retarget.py` (headless Blender 5.0) is the deliverable; full writeup in `docs/C_RETARGET.md`.
- Proven A→C on the gymnasts demo: `pose_engine.py <clip>` → `out/<name>/motion.bvh` →
  `retarget.py --bvh …` → `clips_out/<name>.fbx` + `<name>_qc.mp4`. QC montage eyeballed = recognizable
  human motion, no soup, no catastrophic arm twist (world Copy-Rotation held up — both rests are T-pose).
- **Merge verified**: `gymnasts.fbx` merges onto `female_game.glb` and drives it, exports a playable GLB —
  BUT only after the **Blender-5 Baklava slot-bind** (`anim_data.action_slot = act.slots[0]`).

**HANDOFF TO LANE D (clip factory + integration):**
- The `mocap` CLI wraps ingest.py → pose_engine.py (local/ssh) → retarget.py → clips_out/. All three exist.
- **Merge to the clip-bank GLB MUST add the slot-bind line** — `character_kit/scripts/merge_anims.py` is
  pre-Baklava and will silently produce non-animating GLBs on Blender 5.0 without it.
- Also inherited by Blender-5 code: `Action.fcurves` gone (use layer/strip/channelbag walk); no FFMPEG
  image output (render PNG frames + system ffmpeg); mixamo rig is ~156u tall, scale 0.01 — never scale it.
- Deferred from C (do when John's real footage lands): foot-contact IK (stub now — feet slide), `--travel`
  root scaling, and the arms/hands quality pass (upgrade `map_and_bake` to rest-delta if DJ hands twist).

## LANE D — DONE (machinery), session 7 (2026-07-17)
`mocap` (repo root, executable) is the deliverable; full flow in `docs/RUNBOOK.md`.
- Wiring: `./mocap <video> --name <n> [--character female|male] [--travel] [--no-qc]` →
  ingest (moves source) → pose_engine (HSMR venv, MPS) → retarget (Blender) → `clips_out/<n>.fbx`
  + `<n>_qc.mp4`. `--name` flows to retarget so the final clip name stays clean despite ingest's
  date-prefixed intermediates. `./mocap --selfcheck` guards the path-threading (no engine).
- **Proven A→D on gymnasts demo** (`out/mocap_qc_montage.png` eyeballed): one command, exit 0,
  treaty-correct 30fps (real ingest normalizes; the earlier hand-run was 24.85fps).
- **v1 DONE for machinery.** The only thing between here and the PLAN's Lane D acceptance
  ("clip in 90sDJsim + Unreal") is (a) John's footage and (b) the three deferred publish hooks —
  both intentionally not built blind. When footage lands: run `mocap` on it, eyeball QC, then
  build the GLB-merge hook first (games load GLB) per RUNBOOK — that's the "lands in 90sDJsim" step.

## Notes for the next lane
- **UNCOMMITTED as of session 7:** Lane C (`retarget.py`, `docs/C_RETARGET.md`) was left
  uncommitted by session 6, and Lane D (`mocap`, `docs/RUNBOOK.md`, this PROGRESS tick) sits on
  top. Two clean per-lane commits are ready to make on John's word.
- Lane D built the CLI (was blocked two lanes deep at session 3; B shipped session 5, C session 6).
- **SKEL version trap:** HSMR needs **v1.1.1** exactly (`changelog_v1.1.1.txt`); older = silent
  incompat. Source: `.engine/HSMR/docs/SETUP.md` (register skel.is.tue.mpg.de → download.php).
- **Engine wiring + ARM bypasses live in `docs/B_ENGINE_SETUP.md`** — all three ARM landmines
  are now cleared (session 4): detectron2 → **torchvision drop-in** (real detection, not a
  full-frame stub), the pyrender/EGL fake is centralized in `.engine/HSMR/_headless.py`, and
  chumpy is confirmed a non-blocker. `test_detector.py` proves the detector path green, SKEL-free.
- **World-grounded gap (important):** HSMR is per-frame → root is camera-space `cam_t`, NOT the
  treaty's world-grounded trajectory. Fine for in-place DJ moves; walking/locomotion needs Lane C
  to derive root from foot-contact (or a grounding pass). Carry into Lane C design.
- **HSMR is installed on ultra, not m3ultra** (PLAN Ground-Rule 1 puts the pose engine on
  m3ultra). Likely because m3ultra's disk reclaim waits on the reboot (Decision 2). Next B
  session decides: move it to m3ultra, or consciously host inference on ultra.
- **VOID** (was: "MoCapAnything is BVH-direct, skip the retarget rig"). Reversed — the model
  is animal-only and not our engine. Whatever human model we pick emits **SMPL or SKEL, not
  mixamorig**, so **Lane C's constraint-retarget IS required** and
  `spec/{smpl,skel}_to_mixamorig.json` is back on the menu. HSMR's SKEL output (a biomechanical
  skeleton) is the likely source format if John goes SKEL.
- **Check MPS + global-trajectory FIRST** for whichever human model John's decision unlocks
  (HSMR has an HF Space + demo to smoke-test before John's footage). These two risks are
  unchanged by the engine swap.
- Ground-truth skeleton naming is **`mixamorig:`** (colon), armature object named `Armature`.
  Lane C's name-map must handle colon vs underscore variants (PLAN warned about this).
- Blender headless entrypoint: `/Applications/Blender.app/Contents/MacOS/Blender --background --python <script>`.
- House style for Blender scripts: `~/Documents/character_kit/scripts/merge_anims.py`
  (note its units warning: mixamorig head sits at ~0.0505 native units — don't rescale via parent).