diff --git a/.gitignore b/.gitignore index 39d6033..87f10ed 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ # weights + runtime data stay out of git weights/ +.engine/ capture/ queue/ out/ diff --git a/PROGRESS.md b/PROGRESS.md index b144725..9cf4cb5 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -6,23 +6,42 @@ One lane per session. Tick on completion. See PLAN.md for acceptance checks. spec/mixamorig_bones.json (65 bones, `mixamorig:` colon-prefixed). — 2026-07-17 - [x] **Lane A** — capture + ingest. docs/CAPTURE.md protocol, ingest.py (probe → 30fps normalize → rename → rsync to m3ultra queue → LEDGER.tsv). — 2026-07-17 -- [~] **Lane B** — pose engine (m3ultra). B1 research + selection DONE (2026-07-17): - picked **MoCapAnything V2** (MIT/MIT, no SMPL, BVH-direct onto mixamorig) — see - `docs/B1_BAKEOFF.md`. **Stopped at the John-decision gate** (`docs/DECISIONS.md`): - confirm engine + clear m3ultra disk (97% full) + supply test footage. Inference - eyeball, B2 (service-ize), B3 (jitter) remain. +- [ ] **Lane B** — pose engine. **Decision made 2026-07-17 (session 3): Path B** — engine = + **HSMR → SKEL**, run local, ship rig-motion only, John accepts NC-licence risk + (`docs/DECISIONS.md` Decision 1 RESOLVED). Env is 90% up in `.engine/HSMR` (uv venv, + torch 2.13 **MPS verified green**, ungated weights pulled). **Still open before B ships:** + (1) John downloads gated **SKEL v1.1.1** → `.engine/HSMR/data_inputs/body_models/skel/`; + (2) fix the person-detector (detectron2 + chumpy failed to build — see gotchas); + (3) real forward-pass smoke test on an HSMR demo image; then B2 (service-ize → BVH) + B3 + (jitter). Lanes C and D remain blocked on B. - [ ] **Lane C** — retarget + cleanup (Blender). bvh → mixamorig FBX + QC turntable. - [ ] **Lane D** — clip factory. `mocap` CLI end-to-end; games + Unreal integration. - [ ] Lane E — ML upgrades (post-v1, gated). - [ ] Lane F — UX (optional). ## Notes for the next lane -- **MoCapAnything is BVH-direct onto an arbitrary rig.** If it hits mixamorig directly, do - NOT build `spec/smpl_to_mixamorig.json` (B2 task 2) or Lane C's constraint-retarget rig — - both are obsoleted. Verify the direct-to-mixamorig path first; only build a mapper if it - can't. (`# ponytail: don't write the retarget machinery a BVH-direct model deletes.`) -- **Check MPS + global-trajectory FIRST next session** (the two unproven risks in B1_BAKEOFF). - MoCapAnything ships a ~160MB sample + HF demo — smoke-test without waiting on John's footage. +- **Lane D can't be started until B + C ship + test footage exists** — it's the CLI that wraps + ingest → pose_engine.py → retarget.py, and the latter two don't exist yet. Session 3 (the + "Lane D" session) correctly built nothing; it locked Decision 1 and de-risked MPS instead. +- **SKEL version trap:** HSMR needs **v1.1.1** exactly (`changelog_v1.1.1.txt`); older = silent + incompat. Source: `.engine/HSMR/docs/SETUP.md` (register skel.is.tue.mpg.de → download.php). +- **Engine wiring + ARM bypasses live in `docs/B_ENGINE_SETUP.md`** — `smoke_test.py` already + stubs the un-buildable detectron2 (full-frame bbox) and headless pyrender (QC → Blender), so + the detector swap + real QC are B2 polish, **not** blockers for a centered demo clip. +- **World-grounded gap (important):** HSMR is per-frame → root is camera-space `cam_t`, NOT the + treaty's world-grounded trajectory. Fine for in-place DJ moves; walking/locomotion needs Lane C + to derive root from foot-contact (or a grounding pass). Carry into Lane C design. +- **HSMR is installed on ultra, not m3ultra** (PLAN Ground-Rule 1 puts the pose engine on + m3ultra). Likely because m3ultra's disk reclaim waits on the reboot (Decision 2). Next B + session decides: move it to m3ultra, or consciously host inference on ultra. +- **VOID** (was: "MoCapAnything is BVH-direct, skip the retarget rig"). Reversed — the model + is animal-only and not our engine. Whatever human model we pick emits **SMPL or SKEL, not + mixamorig**, so **Lane C's constraint-retarget IS required** and + `spec/{smpl,skel}_to_mixamorig.json` is back on the menu. HSMR's SKEL output (a biomechanical + skeleton) is the likely source format if John goes SKEL. +- **Check MPS + global-trajectory FIRST** for whichever human model John's decision unlocks + (HSMR has an HF Space + demo to smoke-test before John's footage). These two risks are + unchanged by the engine swap. - Ground-truth skeleton naming is **`mixamorig:`** (colon), armature object named `Armature`. Lane C's name-map must handle colon vs underscore variants (PLAN warned about this). - Blender headless entrypoint: `/Applications/Blender.app/Contents/MacOS/Blender --background --python