mocapgod/PROGRESS.md

MOCAPGOD progress ledger

One lane per session. Tick on completion. See PLAN.md for acceptance checks.

• Lane 0 — bootstrap. git + .gitignore, docs/LICENSES.md skeleton, spec/mixamorig_bones.json (65 bones, mixamorig: colon-prefixed). — 2026-07-17
• Lane A — capture + ingest. docs/CAPTURE.md protocol, ingest.py (probe → 30fps normalize → rename → rsync to m3ultra queue → LEDGER.tsv). — 2026-07-17
• Lane B — pose engine. DONE session 5 (2026-07-17). pose_engine.py ships: queue video → HSMR/SKEL → SKEL-24 BVH@fps + motion.json sidecar (root_space:"camera"), spec/skel_to_mixamorig.json name-map for Lane C. Verified on the demo clip: Blender import = 24 bones, upright (1.59 m), motion present, FK self-consistency 0.0 mm; queue of 6 drained unattended (2 bad clips failed gracefully, 4 valid BVH, heartbeat per clip); runtime ~1.6× realtime steady-state (under the <2× bar). mobilenet detector + detect-every-20 + single-subject box reuse retired the CPU-detector wall-clock finding. Axis fix: HSMR emits Y-down (OpenCV) → diag(1,-1,-1) → Y-up/faces-−Z. History (sessions 2–4): Decision made session 3: Path B — engine = HSMR → SKEL, run local, ship rig-motion only, John accepts NC-licence risk (docs/DECISIONS.md Decision 1 RESOLVED). Env up in .engine/HSMR (uv venv, torch 2.13 MPS green, ungated weights pulled). Detector RESOLVED (session 4): torchvision Faster R-CNN drop-in replaces the unbuildable detectron2; test_detector.py is green today (7/7 demo imgs → 15 patches, no SKEL needed). smoke_test.py now runs real detection + patch-crop and reaches the exact SKEL gate; chumpy cleared as a non-blocker. Only open before B ships: (1) John downloads gated SKEL v1.1.1 so .engine/HSMR/data_inputs/body_models/skel/skel_male.pkl (+ skel_female.pkl) exist; (2) run the forward-pass smoke test (MPS / timing <2× / NaN check); then B2 (service-ize → BVH) + B3 (jitter). Lanes C and D remain blocked on B.
• Lane C — retarget + cleanup (Blender). bvh → mixamorig FBX + QC turntable.
• Lane D — clip factory. mocap CLI end-to-end; games + Unreal integration.
• Lane E — ML upgrades (post-v1, gated).
• Lane F — UX (optional).

SESSION 5 — DONE (2026-07-17). All six orders executed; Lane B ships.

1. ✅ Durable-ized the ARM fixes → engine_patches/ + setup_hsmr.sh --patch. Verified: fresh clone + --patch → test_detector.py green, no hand edits.
2. ✅ Path-A email drafted in docs/DECISIONS.md (John sends to ps-license@tue.mpg.de).
3. ✅ Queue reconciled: ingest.py defaults to local queue/ on ultra; --remote-queue keeps the m3ultra path. PLAN Ground-rule-1 deviation noted; CAPTURE.md updated.
4. ✅ Treaty amended: root_space (camera|world) is a REQUIRED sidecar field; Lane C reads it.
5. ✅ B2 + B3: pose_engine.py (SKEL-24 BVH + sidecar + jitter) + spec/skel_to_mixamorig.json. Detector swapped to mobilenet; detect-every-20 + box reuse; build-once + per-video isolation. Ran the smoke test's SKEL forward on real params — no re-verify needed (smoke already PASSED).

HANDOFF TO LANE C (retarget + cleanup, Blender, on ultra):

• Input is out/<name>/motion.bvh (SKEL-24 skeleton, Y-up, metres, faces −Z) + motion.json. Import in Blender with import_anim.bvh(axis_forward='-Z', axis_up='Y') (verified upright).
• Constraint-map SKEL→mixamorig via spec/skel_to_mixamorig.json (20 mapped, 4 null=twist/heel; Spine1 interpolated, Head follows Neck, fingers are Lane E). mixamorig canon: mixamorig: colon.
• root_space is "camera" → root is NOT world-grounded (HSMR is per-frame). In-place clips are first-class; for traveling clips derive a world root from foot contacts (PLAN Lane C task).
• Rest pose is a clean T-pose at zero rotation; T-pose bookends only exist if John filmed the protocol (demo clips don't have them) — Lane C's trim must tolerate their absence.
• Re-run the SKEL→BVH stage cheaply with pose_engine.py <clip> --from-params (skips HMR) when tuning; raw params are saved at out/<name>/params.npz.

SESSION 5 ORDERS — from Fable's review (2026-07-17, supersede older notes where they conflict)

Review verdict: Lanes 0/A solid; session 2's reversal of the B1 animal-model blunder was the right call, well executed; ARM engineering verified independently (test_detector.py re-run green 7/7 by reviewer). Proceed as follows, in order:

1. Durable-ize the gitignored engine fixes (do first, ~30 min). The torchvision detector, _headless.py, test_detector.py, smoke_test.py exist ONLY on ultra's disk inside the gitignored .engine/. Move canonical copies into git at engine_patches/ and make setup_hsmr.sh apply them after clone. Acceptance: fresh clone in a temp dir + script → test_detector.py green with no hand edits.

2. Draft the Path-A email (parallel risk-retirement, ~15 min). Write the SKEL indie commercial-license request to ps-license@tue.mpg.de into docs/DECISIONS.md (project description, solo Australian dev, indie revenue, SKEL-in-pipeline-not-redistributed usage). John sends it himself. Path B continues meanwhile; a granted license converts B→A with zero rework. Do not wait on the reply.

3. Reconcile the split-brain queue (~30 min). Lane A's ingest.py rsyncs to m3ultra's queue, but the engine (and MPS verification) is on ultra — nothing consumes that queue. For v1: default ingest to LOCAL queue/ on ultra (--remote-queue flag preserves the m3ultra path), note the deviation from PLAN Ground-rule 1 in PLAN.md with a # ponytail: v1 inference on ultra; migrate to m3ultra when a real throughput need appears marker. Update CAPTURE.md.

4. Treaty amendment (small, load-bearing). motion.json sidecar gains REQUIRED field root_space: "camera" | "world". HSMR emits "camera". Lane C must read it and, for locomotion clips, derive world root from foot contacts (already in notes). v1 quality bar: in-place clips are first-class; traveling clips best-effort until a grounding pass exists. SMOKE TEST RAN 2026-07-17 (Fable, after John's SKEL v1.1.1 download landed — installed at the expected path): PASSED with findings. device=mps, 263 frames → 1315 patches, recovery 31.2 fps, nan=False, data_outputs/smoke/params.npz written. Findings for B2: (1) the CPU Faster R-CNN detector is the wall-clock hog (~1 fps, 4min of the 5min run) — B2 MUST add detect-every-Nth-frame + box reuse/IoU tracking (single-subject footage makes this easy) or the mobilenet swap; HMR itself is faster than realtime so the <2× acceptance is reachable. (2) prepare_mesh MPS→numpy crash fixed in smoke_test.py (.detach().cpu()) — carry the same pattern into pose_engine.py. Item 5 below is now UNBLOCKED (skip re-verifying: smoke is done; go straight to B2/B3).

5. If (and only if) SKEL v1.1.1 files are in place (data_inputs/body_models/skel/ skel_male.pkl): run smoke_test.py — MPS forward pass, runtime <2× realtime on demo video, NaN check — then proceed straight into B2 (params.npz → spec/skel_to_mixamorig.json map → BVH@30fps + sidecar per treaty) and B3 (One-Euro/Sav-Gol jitter pass, root smoothed separately). If SKEL absent: finish items 1–4, verify Decision-3 footage status, stop clean.

John's homework (all three block different things):

• SKEL v1.1.1 download (the ONLY hard blocker; instructions in docs/B_ENGINE_SETUP.md).
• Film walk / dance / pick-up-object per docs/CAPTURE.md, drop in capture/inbox/ (blocks the honest quality verdict + all of B's acceptance).
• Reboot m3ultra when convenient (applies staged 26.5.2 security update; housekeeping).

Notes for the next lane

• Lane D can't be started until B + C ship + test footage exists — it's the CLI that wraps ingest → pose_engine.py → retarget.py, and the latter two don't exist yet. Session 3 (the "Lane D" session) correctly built nothing; it locked Decision 1 and de-risked MPS instead.
• SKEL version trap: HSMR needs v1.1.1 exactly (changelog_v1.1.1.txt); older = silent incompat. Source: .engine/HSMR/docs/SETUP.md (register skel.is.tue.mpg.de → download.php).
• Engine wiring + ARM bypasses live in docs/B_ENGINE_SETUP.md — all three ARM landmines are now cleared (session 4): detectron2 → torchvision drop-in (real detection, not a full-frame stub), the pyrender/EGL fake is centralized in .engine/HSMR/_headless.py, and chumpy is confirmed a non-blocker. test_detector.py proves the detector path green, SKEL-free.
• World-grounded gap (important): HSMR is per-frame → root is camera-space cam_t, NOT the treaty's world-grounded trajectory. Fine for in-place DJ moves; walking/locomotion needs Lane C to derive root from foot-contact (or a grounding pass). Carry into Lane C design.
• HSMR is installed on ultra, not m3ultra (PLAN Ground-Rule 1 puts the pose engine on m3ultra). Likely because m3ultra's disk reclaim waits on the reboot (Decision 2). Next B session decides: move it to m3ultra, or consciously host inference on ultra.
• VOID (was: "MoCapAnything is BVH-direct, skip the retarget rig"). Reversed — the model is animal-only and not our engine. Whatever human model we pick emits SMPL or SKEL, not mixamorig, so Lane C's constraint-retarget IS required and spec/{smpl,skel}_to_mixamorig.json is back on the menu. HSMR's SKEL output (a biomechanical skeleton) is the likely source format if John goes SKEL.
• Check MPS + global-trajectory FIRST for whichever human model John's decision unlocks (HSMR has an HF Space + demo to smoke-test before John's footage). These two risks are unchanged by the engine swap.
• Ground-truth skeleton naming is mixamorig: (colon), armature object named Armature. Lane C's name-map must handle colon vs underscore variants (PLAN warned about this).
• Blender headless entrypoint: /Applications/Blender.app/Contents/MacOS/Blender --background --python <script>.
• House style for Blender scripts: ~/Documents/character_kit/scripts/merge_anims.py (note its units warning: mixamorig head sits at ~0.0505 native units — don't rescale via parent).