mocapgod/PROGRESS.md

MOCAPGOD progress ledger

FABLE REVIEW #2 (2026-07-17, post lanes B–D) — VERDICT: SHIPPED, one bug, one cross-repo action

Independently verified (not trusted): fresh end-to-end ./mocap run on a renamed demo clip → full chain green (317f, FK 0.0mm, treaty-conformant sidecar root_space:"camera" @30fps/24 joints, FBX 685KB + QC mp4; QC frames eyeballed = clean upright motion on female_game). Detect-every-20 confirmed working (16 detector steps/317 frames). Docs honest; deferred items real deferrals, not fakes. Path-A email draft is good — John should send it.

BUG (fix before first real clip enters a bank): exported FBX take/action name is generic (Armature|Scene), identical for every clip (bake even transiently borrows an existing bank clip's name). merge_anims.py keys NLA tracks off action names → our own clips will collide in the bank. Fix in retarget.py: after bake, action.name = <name> and ensure the FBX take exports as <name>. Acceptance: strings clips_out/<n>.fbx | grep <n> OR Blender import lists action <n>.

CROSS-REPO DEBT (out of MOCAPGOD scope, task spawned): Blender 5 "Baklava" slot-bind — character_kit/scripts/merge_anims.py is pre-5.0 and silently drives nothing on Blender 5 without animation_data.action_slot = act.slots[0] (session-6 discovery, verified). Affects 90sDJsim/OPSHOPGAME clip merging fleet-wide, not just MOCAPGOD.

Remaining to v1 sign-off (unchanged, all John): film walk/dance/pick-up per CAPTURE.md. Then: quality verdict on real footage, tune foot-fix on the walk clip, verify male rig, wire the deferred bank-merge/Unreal hooks. Send the MPI email.

One lane per session. Tick on completion. See PLAN.md for acceptance checks.

• Lane 0 — bootstrap. git + .gitignore, docs/LICENSES.md skeleton, spec/mixamorig_bones.json (65 bones, mixamorig: colon-prefixed). — 2026-07-17
• Lane A — capture + ingest. docs/CAPTURE.md protocol, ingest.py (probe → 30fps normalize → rename → rsync to m3ultra queue → LEDGER.tsv). — 2026-07-17
• Lane B — pose engine. DONE session 5 (2026-07-17). pose_engine.py ships: queue video → HSMR/SKEL → SKEL-24 BVH@fps + motion.json sidecar (root_space:"camera"), spec/skel_to_mixamorig.json name-map for Lane C. Verified on the demo clip: Blender import = 24 bones, upright (1.59 m), motion present, FK self-consistency 0.0 mm; queue of 6 drained unattended (2 bad clips failed gracefully, 4 valid BVH, heartbeat per clip); runtime ~1.6× realtime steady-state (under the <2× bar). mobilenet detector + detect-every-20 + single-subject box reuse retired the CPU-detector wall-clock finding. Axis fix: HSMR emits Y-down (OpenCV) → diag(1,-1,-1) → Y-up/faces-−Z. History (sessions 2–4): Decision made session 3: Path B — engine = HSMR → SKEL, run local, ship rig-motion only, John accepts NC-licence risk (docs/DECISIONS.md Decision 1 RESOLVED). Env up in .engine/HSMR (uv venv, torch 2.13 MPS green, ungated weights pulled). Detector RESOLVED (session 4): torchvision Faster R-CNN drop-in replaces the unbuildable detectron2; test_detector.py is green today (7/7 demo imgs → 15 patches, no SKEL needed). smoke_test.py now runs real detection + patch-crop and reaches the exact SKEL gate; chumpy cleared as a non-blocker. Only open before B ships: (1) John downloads gated SKEL v1.1.1 so .engine/HSMR/data_inputs/body_models/skel/skel_male.pkl (+ skel_female.pkl) exist; (2) run the forward-pass smoke test (MPS / timing <2× / NaN check); then B2 (service-ize → BVH) + B3 (jitter). Lanes C and D remain blocked on B.
• Lane C — retarget + cleanup (Blender). DONE (machinery) session 6 (2026-07-17). retarget.py: SKEL-24 BVH → mixamorig (spec/skel_to_mixamorig.json, world Copy-Rotation + bake) → anim-only FBX + Workbench turntable QC mp4. Verified A→C on the gymnasts demo: coherent human motion (montage eyeballed, not soup, no arm twist); FBX merges onto female_game.glb and drives it (RightHand +121u → playable GLB) after the Blender-5 slot-bind. Foot-contact fix deferred (stub) until a real walk clip; final quality sign-off awaits John's footage. See docs/C_RETARGET.md.
• Lane D — clip factory. DONE (machinery) session 7 (2026-07-17). mocap CLI wraps ingest → pose_engine → retarget into ONE command; verified end-to-end on the gymnasts demo (./mocap capture/inbox/gymnasts_d.mp4 --name gymnasts_d → 317f @ 30fps, FK 0.0mm → clips_out/gymnasts_d.fbx + QC mp4, montage eyeballed = clean human motion). RUNBOOK.md written. Deferred (honest, not faked as dead flags): clip-bank GLB merge, --to-unreal, --publish-3god — all off-by-default per PLAN, gated on real footage + touch other systems; how-to in RUNBOOK. Final acceptance (clip in 90sDJsim + Unreal) still needs John's footage.
• Lane E — ML upgrades (post-v1, gated).
• Lane F — UX (optional).

SESSION 5 — DONE (2026-07-17). All six orders executed; Lane B ships.

1. ✅ Durable-ized the ARM fixes → engine_patches/ + setup_hsmr.sh --patch. Verified: fresh clone + --patch → test_detector.py green, no hand edits.
2. ✅ Path-A email drafted in docs/DECISIONS.md (John sends to ps-license@tue.mpg.de).
3. ✅ Queue reconciled: ingest.py defaults to local queue/ on ultra; --remote-queue keeps the m3ultra path. PLAN Ground-rule-1 deviation noted; CAPTURE.md updated.
4. ✅ Treaty amended: root_space (camera|world) is a REQUIRED sidecar field; Lane C reads it.
5. ✅ B2 + B3: pose_engine.py (SKEL-24 BVH + sidecar + jitter) + spec/skel_to_mixamorig.json. Detector swapped to mobilenet; detect-every-20 + box reuse; build-once + per-video isolation. Ran the smoke test's SKEL forward on real params — no re-verify needed (smoke already PASSED).

HANDOFF TO LANE C (retarget + cleanup, Blender, on ultra):

• Input is out/<name>/motion.bvh (SKEL-24 skeleton, Y-up, metres, faces −Z) + motion.json. Import in Blender with import_anim.bvh(axis_forward='-Z', axis_up='Y') (verified upright).
• Constraint-map SKEL→mixamorig via spec/skel_to_mixamorig.json (20 mapped, 4 null=twist/heel; Spine1 interpolated, Head follows Neck, fingers are Lane E). mixamorig canon: mixamorig: colon.
• root_space is "camera" → root is NOT world-grounded (HSMR is per-frame). In-place clips are first-class; for traveling clips derive a world root from foot contacts (PLAN Lane C task).
• Rest pose is a clean T-pose at zero rotation; T-pose bookends only exist if John filmed the protocol (demo clips don't have them) — Lane C's trim must tolerate their absence.
• Re-run the SKEL→BVH stage cheaply with pose_engine.py <clip> --from-params (skips HMR) when tuning; raw params are saved at out/<name>/params.npz.

SESSION 5 ORDERS — from Fable's review (2026-07-17, supersede older notes where they conflict)

Review verdict: Lanes 0/A solid; session 2's reversal of the B1 animal-model blunder was the right call, well executed; ARM engineering verified independently (test_detector.py re-run green 7/7 by reviewer). Proceed as follows, in order:

1. Durable-ize the gitignored engine fixes (do first, ~30 min). The torchvision detector, _headless.py, test_detector.py, smoke_test.py exist ONLY on ultra's disk inside the gitignored .engine/. Move canonical copies into git at engine_patches/ and make setup_hsmr.sh apply them after clone. Acceptance: fresh clone in a temp dir + script → test_detector.py green with no hand edits.

2. Draft the Path-A email (parallel risk-retirement, ~15 min). Write the SKEL indie commercial-license request to ps-license@tue.mpg.de into docs/DECISIONS.md (project description, solo Australian dev, indie revenue, SKEL-in-pipeline-not-redistributed usage). John sends it himself. Path B continues meanwhile; a granted license converts B→A with zero rework. Do not wait on the reply.

3. Reconcile the split-brain queue (~30 min). Lane A's ingest.py rsyncs to m3ultra's queue, but the engine (and MPS verification) is on ultra — nothing consumes that queue. For v1: default ingest to LOCAL queue/ on ultra (--remote-queue flag preserves the m3ultra path), note the deviation from PLAN Ground-rule 1 in PLAN.md with a # ponytail: v1 inference on ultra; migrate to m3ultra when a real throughput need appears marker. Update CAPTURE.md.

4. Treaty amendment (small, load-bearing). motion.json sidecar gains REQUIRED field root_space: "camera" | "world". HSMR emits "camera". Lane C must read it and, for locomotion clips, derive world root from foot contacts (already in notes). v1 quality bar: in-place clips are first-class; traveling clips best-effort until a grounding pass exists. SMOKE TEST RAN 2026-07-17 (Fable, after John's SKEL v1.1.1 download landed — installed at the expected path): PASSED with findings. device=mps, 263 frames → 1315 patches, recovery 31.2 fps, nan=False, data_outputs/smoke/params.npz written. Findings for B2: (1) the CPU Faster R-CNN detector is the wall-clock hog (~1 fps, 4min of the 5min run) — B2 MUST add detect-every-Nth-frame + box reuse/IoU tracking (single-subject footage makes this easy) or the mobilenet swap; HMR itself is faster than realtime so the <2× acceptance is reachable. (2) prepare_mesh MPS→numpy crash fixed in smoke_test.py (.detach().cpu()) — carry the same pattern into pose_engine.py. Item 5 below is now UNBLOCKED (skip re-verifying: smoke is done; go straight to B2/B3).

5. If (and only if) SKEL v1.1.1 files are in place (data_inputs/body_models/skel/ skel_male.pkl): run smoke_test.py — MPS forward pass, runtime <2× realtime on demo video, NaN check — then proceed straight into B2 (params.npz → spec/skel_to_mixamorig.json map → BVH@30fps + sidecar per treaty) and B3 (One-Euro/Sav-Gol jitter pass, root smoothed separately). If SKEL absent: finish items 1–4, verify Decision-3 footage status, stop clean.

John's homework (all three block different things):

• SKEL v1.1.1 download (the ONLY hard blocker; instructions in docs/B_ENGINE_SETUP.md).
• Film walk / dance / pick-up-object per docs/CAPTURE.md, drop in capture/inbox/ (blocks the honest quality verdict + all of B's acceptance).
• Reboot m3ultra when convenient (applies staged 26.5.2 security update; housekeeping).

LANE C — DONE (machinery), session 6 (2026-07-17)

retarget.py (headless Blender 5.0) is the deliverable; full writeup in docs/C_RETARGET.md.

• Proven A→C on the gymnasts demo: pose_engine.py <clip> → out/<name>/motion.bvh → retarget.py --bvh … → clips_out/<name>.fbx + <name>_qc.mp4. QC montage eyeballed = recognizable human motion, no soup, no catastrophic arm twist (world Copy-Rotation held up — both rests are T-pose).
• Merge verified: gymnasts.fbx merges onto female_game.glb and drives it, exports a playable GLB — BUT only after the Blender-5 Baklava slot-bind (anim_data.action_slot = act.slots[0]).

HANDOFF TO LANE D (clip factory + integration):

• The mocap CLI wraps ingest.py → pose_engine.py (local/ssh) → retarget.py → clips_out/. All three exist.
• Merge to the clip-bank GLB MUST add the slot-bind line — character_kit/scripts/merge_anims.py is pre-Baklava and will silently produce non-animating GLBs on Blender 5.0 without it.
• Also inherited by Blender-5 code: Action.fcurves gone (use layer/strip/channelbag walk); no FFMPEG image output (render PNG frames + system ffmpeg); mixamo rig is ~156u tall, scale 0.01 — never scale it.
• Deferred from C (do when John's real footage lands): foot-contact IK (stub now — feet slide), --travel root scaling, and the arms/hands quality pass (upgrade map_and_bake to rest-delta if DJ hands twist).

LANE D — DONE (machinery), session 7 (2026-07-17)

mocap (repo root, executable) is the deliverable; full flow in docs/RUNBOOK.md.

• Wiring: ./mocap <video> --name <n> [--character female|male] [--travel] [--no-qc] → ingest (moves source) → pose_engine (HSMR venv, MPS) → retarget (Blender) → clips_out/<n>.fbx
  • <n>_qc.mp4. --name flows to retarget so the final clip name stays clean despite ingest's date-prefixed intermediates. ./mocap --selfcheck guards the path-threading (no engine).
• Proven A→D on gymnasts demo (out/mocap_qc_montage.png eyeballed): one command, exit 0, treaty-correct 30fps (real ingest normalizes; the earlier hand-run was 24.85fps).
• v1 DONE for machinery. The only thing between here and the PLAN's Lane D acceptance ("clip in 90sDJsim + Unreal") is (a) John's footage and (b) the three deferred publish hooks — both intentionally not built blind. When footage lands: run mocap on it, eyeball QC, then build the GLB-merge hook first (games load GLB) per RUNBOOK — that's the "lands in 90sDJsim" step.

Notes for the next lane

• UNCOMMITTED as of session 7: Lane C (retarget.py, docs/C_RETARGET.md) was left uncommitted by session 6, and Lane D (mocap, docs/RUNBOOK.md, this PROGRESS tick) sits on top. Two clean per-lane commits are ready to make on John's word.
• Lane D built the CLI (was blocked two lanes deep at session 3; B shipped session 5, C session 6).
• SKEL version trap: HSMR needs v1.1.1 exactly (changelog_v1.1.1.txt); older = silent incompat. Source: .engine/HSMR/docs/SETUP.md (register skel.is.tue.mpg.de → download.php).
• Engine wiring + ARM bypasses live in docs/B_ENGINE_SETUP.md — all three ARM landmines are now cleared (session 4): detectron2 → torchvision drop-in (real detection, not a full-frame stub), the pyrender/EGL fake is centralized in .engine/HSMR/_headless.py, and chumpy is confirmed a non-blocker. test_detector.py proves the detector path green, SKEL-free.
• World-grounded gap (important): HSMR is per-frame → root is camera-space cam_t, NOT the treaty's world-grounded trajectory. Fine for in-place DJ moves; walking/locomotion needs Lane C to derive root from foot-contact (or a grounding pass). Carry into Lane C design.
• HSMR is installed on ultra, not m3ultra (PLAN Ground-Rule 1 puts the pose engine on m3ultra). Likely because m3ultra's disk reclaim waits on the reboot (Decision 2). Next B session decides: move it to m3ultra, or consciously host inference on ultra.
• VOID (was: "MoCapAnything is BVH-direct, skip the retarget rig"). Reversed — the model is animal-only and not our engine. Whatever human model we pick emits SMPL or SKEL, not mixamorig, so Lane C's constraint-retarget IS required and spec/{smpl,skel}_to_mixamorig.json is back on the menu. HSMR's SKEL output (a biomechanical skeleton) is the likely source format if John goes SKEL.
• Check MPS + global-trajectory FIRST for whichever human model John's decision unlocks (HSMR has an HF Space + demo to smoke-test before John's footage). These two risks are unchanged by the engine swap.
• Ground-truth skeleton naming is mixamorig: (colon), armature object named Armature. Lane C's name-map must handle colon vs underscore variants (PLAN warned about this).
• Blender headless entrypoint: /Applications/Blender.app/Contents/MacOS/Blender --background --python <script>.
• House style for Blender scripts: ~/Documents/character_kit/scripts/merge_anims.py (note its units warning: mixamorig head sits at ~0.0505 native units — don't rescale via parent).