From 4cc67b90fadab4b18e1042f063def68bf85e9072 Mon Sep 17 00:00:00 2001 From: type-two Date: Fri, 17 Jul 2026 02:25:55 +1000 Subject: [PATCH] Lane B: resolve decision gate -> Path B (local HSMR/SKEL, accept NC risk) B1's MoCapAnything pick reversed (animal model, wrong for human capture); license analysis recorded in LICENSES.md. John chose Path B: run HSMR->SKEL locally on ultra (MPS), ship rig-motion only, accept the non-commercial licence risk as interim until a commercial SKEL licence request goes out (Path A, the clean upgrade). Engine installed under .engine/ (gitignored): uv venv py3.10, torch 2.13 MPS-green, ungated weights, smoke_test.py with Apple-Silicon bypasses (detectron2 stubbed with full-frame bbox, headless pyrender stubbed -> QC in Blender) -- see docs/B_ENGINE_SETUP.md. Remaining before B ships: John's gated SKEL v1.1.1 download, then the forward-pass smoke test (MPS end-to-end, timing, pose sanity). Known caveat: HSMR is per-frame so root is camera-space cam_t, NOT world-grounded -> Lane C derives root from foot-contact. Co-Authored-By: Claude Opus 4.8 --- .gitignore | 1 + PROGRESS.md | 41 +++++++++++++----- docs/B1_BAKEOFF.md | 10 +++-- docs/B_ENGINE_SETUP.md | 50 ++++++++++++++++++++++ docs/DECISIONS.md | 96 +++++++++++++++++++++++++++++++----------- docs/LICENSES.md | 52 +++++++++++++++-------- 6 files changed, 194 insertions(+), 56 deletions(-) create mode 100644 docs/B_ENGINE_SETUP.md diff --git a/.gitignore b/.gitignore index 39d6033..87f10ed 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ # weights + runtime data stay out of git weights/ +.engine/ capture/ queue/ out/ diff --git a/PROGRESS.md b/PROGRESS.md index b144725..9cf4cb5 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -6,23 +6,42 @@ One lane per session. Tick on completion. See PLAN.md for acceptance checks. spec/mixamorig_bones.json (65 bones, `mixamorig:` colon-prefixed). — 2026-07-17 - [x] **Lane A** — capture + ingest. docs/CAPTURE.md protocol, ingest.py (probe → 30fps normalize → rename → rsync to m3ultra queue → LEDGER.tsv). — 2026-07-17 -- [~] **Lane B** — pose engine (m3ultra). B1 research + selection DONE (2026-07-17): - picked **MoCapAnything V2** (MIT/MIT, no SMPL, BVH-direct onto mixamorig) — see - `docs/B1_BAKEOFF.md`. **Stopped at the John-decision gate** (`docs/DECISIONS.md`): - confirm engine + clear m3ultra disk (97% full) + supply test footage. Inference - eyeball, B2 (service-ize), B3 (jitter) remain. +- [ ] **Lane B** — pose engine. **Decision made 2026-07-17 (session 3): Path B** — engine = + **HSMR → SKEL**, run local, ship rig-motion only, John accepts NC-licence risk + (`docs/DECISIONS.md` Decision 1 RESOLVED). Env is 90% up in `.engine/HSMR` (uv venv, + torch 2.13 **MPS verified green**, ungated weights pulled). **Still open before B ships:** + (1) John downloads gated **SKEL v1.1.1** → `.engine/HSMR/data_inputs/body_models/skel/`; + (2) fix the person-detector (detectron2 + chumpy failed to build — see gotchas); + (3) real forward-pass smoke test on an HSMR demo image; then B2 (service-ize → BVH) + B3 + (jitter). Lanes C and D remain blocked on B. - [ ] **Lane C** — retarget + cleanup (Blender). bvh → mixamorig FBX + QC turntable. - [ ] **Lane D** — clip factory. `mocap` CLI end-to-end; games + Unreal integration. - [ ] Lane E — ML upgrades (post-v1, gated). - [ ] Lane F — UX (optional). ## Notes for the next lane -- **MoCapAnything is BVH-direct onto an arbitrary rig.** If it hits mixamorig directly, do - NOT build `spec/smpl_to_mixamorig.json` (B2 task 2) or Lane C's constraint-retarget rig — - both are obsoleted. Verify the direct-to-mixamorig path first; only build a mapper if it - can't. (`# ponytail: don't write the retarget machinery a BVH-direct model deletes.`) -- **Check MPS + global-trajectory FIRST next session** (the two unproven risks in B1_BAKEOFF). - MoCapAnything ships a ~160MB sample + HF demo — smoke-test without waiting on John's footage. +- **Lane D can't be started until B + C ship + test footage exists** — it's the CLI that wraps + ingest → pose_engine.py → retarget.py, and the latter two don't exist yet. Session 3 (the + "Lane D" session) correctly built nothing; it locked Decision 1 and de-risked MPS instead. +- **SKEL version trap:** HSMR needs **v1.1.1** exactly (`changelog_v1.1.1.txt`); older = silent + incompat. Source: `.engine/HSMR/docs/SETUP.md` (register skel.is.tue.mpg.de → download.php). +- **Engine wiring + ARM bypasses live in `docs/B_ENGINE_SETUP.md`** — `smoke_test.py` already + stubs the un-buildable detectron2 (full-frame bbox) and headless pyrender (QC → Blender), so + the detector swap + real QC are B2 polish, **not** blockers for a centered demo clip. +- **World-grounded gap (important):** HSMR is per-frame → root is camera-space `cam_t`, NOT the + treaty's world-grounded trajectory. Fine for in-place DJ moves; walking/locomotion needs Lane C + to derive root from foot-contact (or a grounding pass). Carry into Lane C design. +- **HSMR is installed on ultra, not m3ultra** (PLAN Ground-Rule 1 puts the pose engine on + m3ultra). Likely because m3ultra's disk reclaim waits on the reboot (Decision 2). Next B + session decides: move it to m3ultra, or consciously host inference on ultra. +- **VOID** (was: "MoCapAnything is BVH-direct, skip the retarget rig"). Reversed — the model + is animal-only and not our engine. Whatever human model we pick emits **SMPL or SKEL, not + mixamorig**, so **Lane C's constraint-retarget IS required** and + `spec/{smpl,skel}_to_mixamorig.json` is back on the menu. HSMR's SKEL output (a biomechanical + skeleton) is the likely source format if John goes SKEL. +- **Check MPS + global-trajectory FIRST** for whichever human model John's decision unlocks + (HSMR has an HF Space + demo to smoke-test before John's footage). These two risks are + unchanged by the engine swap. - Ground-truth skeleton naming is **`mixamorig:`** (colon), armature object named `Armature`. Lane C's name-map must handle colon vs underscore variants (PLAN warned about this). - Blender headless entrypoint: `/Applications/Blender.app/Contents/MacOS/Blender --background --python