diff --git a/PROGRESS.md b/PROGRESS.md index 377eef5..cef5445 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -6,7 +6,15 @@ One lane per session. Tick on completion. See PLAN.md for acceptance checks. spec/mixamorig_bones.json (65 bones, `mixamorig:` colon-prefixed). — 2026-07-17 - [x] **Lane A** — capture + ingest. docs/CAPTURE.md protocol, ingest.py (probe → 30fps normalize → rename → rsync to m3ultra queue → LEDGER.tsv). — 2026-07-17 -- [ ] **Lane B** — pose engine. **Decision made 2026-07-17 (session 3): Path B** — engine = +- [x] **Lane B** — pose engine. **DONE session 5 (2026-07-17).** `pose_engine.py` ships: + queue video → HSMR/SKEL → SKEL-24 BVH@fps + `motion.json` sidecar (`root_space:"camera"`), + `spec/skel_to_mixamorig.json` name-map for Lane C. Verified on the demo clip: Blender import + = 24 bones, **upright** (1.59 m), motion present, FK self-consistency **0.0 mm**; queue of 6 + drained unattended (2 bad clips failed gracefully, 4 valid BVH, heartbeat per clip); runtime + **~1.6× realtime** steady-state (under the <2× bar). mobilenet detector + detect-every-20 + + single-subject box reuse retired the CPU-detector wall-clock finding. Axis fix: HSMR emits + Y-down (OpenCV) → `diag(1,-1,-1)` → Y-up/faces-−Z. **History (sessions 2–4):** + **Decision made session 3: Path B** — engine = **HSMR → SKEL**, run local, ship rig-motion only, John accepts NC-licence risk (`docs/DECISIONS.md` Decision 1 RESOLVED). Env up in `.engine/HSMR` (uv venv, torch 2.13 **MPS green**, ungated weights pulled). **Detector RESOLVED (session 4):** torchvision @@ -22,6 +30,29 @@ One lane per session. Tick on completion. See PLAN.md for acceptance checks. - [ ] Lane E — ML upgrades (post-v1, gated). - [ ] Lane F — UX (optional). +## SESSION 5 — DONE (2026-07-17). All six orders executed; Lane B ships. +1. ✅ **Durable-ized** the ARM fixes → `engine_patches/` + `setup_hsmr.sh --patch`. Verified: fresh + clone + `--patch` → `test_detector.py` green, no hand edits. +2. ✅ **Path-A email** drafted in `docs/DECISIONS.md` (John sends to ps-license@tue.mpg.de). +3. ✅ **Queue reconciled**: `ingest.py` defaults to local `queue/` on ultra; `--remote-queue` keeps + the m3ultra path. PLAN Ground-rule-1 deviation noted; CAPTURE.md updated. +4. ✅ **Treaty amended**: `root_space` (camera|world) is a REQUIRED sidecar field; Lane C reads it. +5. ✅ **B2 + B3**: `pose_engine.py` (SKEL-24 BVH + sidecar + jitter) + `spec/skel_to_mixamorig.json`. + Detector swapped to mobilenet; detect-every-20 + box reuse; build-once + per-video isolation. + Ran the smoke test's SKEL forward on real params — no re-verify needed (smoke already PASSED). + +**HANDOFF TO LANE C (retarget + cleanup, Blender, on ultra):** +- Input is `out//motion.bvh` (SKEL-24 skeleton, **Y-up, metres, faces −Z**) + `motion.json`. + Import in Blender with `import_anim.bvh(axis_forward='-Z', axis_up='Y')` (verified upright). +- Constraint-map SKEL→mixamorig via **`spec/skel_to_mixamorig.json`** (20 mapped, 4 null=twist/heel; + Spine1 interpolated, Head follows Neck, fingers are Lane E). mixamorig canon: `mixamorig:` colon. +- **`root_space` is `"camera"`** → root is NOT world-grounded (HSMR is per-frame). In-place clips are + first-class; for traveling clips derive a world root from foot contacts (PLAN Lane C task). +- Rest pose is a clean T-pose at zero rotation; T-pose bookends only exist if John filmed the protocol + (demo clips don't have them) — Lane C's trim must tolerate their absence. +- Re-run the SKEL→BVH stage cheaply with `pose_engine.py --from-params` (skips HMR) when + tuning; raw params are saved at `out//params.npz`. + ## SESSION 5 ORDERS — from Fable's review (2026-07-17, supersede older notes where they conflict) Review verdict: Lanes 0/A solid; session 2's reversal of the B1 animal-model blunder was the diff --git a/docs/B_ENGINE_SETUP.md b/docs/B_ENGINE_SETUP.md index 62a41d1..de94d40 100644 --- a/docs/B_ENGINE_SETUP.md +++ b/docs/B_ENGINE_SETUP.md @@ -60,3 +60,28 @@ realtime)? pose sane?** ⚠️ HSMR is a **per-frame** recoverer → root is **camera-space `cam_t`, not world-grounded**. Fine for in-place DJ moves; locomotion (walk) needs Lane C's foot-contact root derivation or a grounding pass. This is the treaty's global-trajectory caveat, carried into Path B. + +## Run the pose engine (B2/B3, session 5) — the real service +`pose_engine.py` (repo root, tracked) wraps the engine into the pipeline. Run under the engine venv: +``` +cd ~/Documents/MOCAPGOD +.engine/HSMR/.venv/bin/python pose_engine.py # drain queue/ → out// +.engine/HSMR/.venv/bin/python pose_engine.py queue/clip.mp4 # one clip +.engine/HSMR/.venv/bin/python pose_engine.py --from-params # re-run SKEL→BVH, skip HMR +.engine/HSMR/.venv/bin/python pose_engine.py --selfcheck # BVH-math self-test, no engine +``` +Per clip → `out//`: `motion.bvh` (SKEL-24, **Y-up, metres, faces −Z**), `motion.json` +(sidecar, `root_space:"camera"`), `params.npz` (raw poses/betas/cam_t for QC / `--from-params`). +Heartbeat: `~/.jobs/pose_engine`. Models build **once** per invocation (queue amortizes); a bad +clip is logged and skipped, not fatal. + +Key engine facts learned this session (for whoever touches B again): +- **SKEL emits `poses` = (F, 46)** biomechanical q, NOT SMPL's 72 axis-angles. Don't decode q by + hand — run `SKEL.forward()` and consume `joints_ori` (F,24,3,3 global orientations) + `joints_tpose`. +- BVH is rigid-bone: rest-aware FK (`offset=rest_J[j]-rest_J[parent]`, `W=Rg@rest_R^T`, + `local=W[parent]^T@W[j]`) reconstructs SKEL joints to ~4 cm; the residual is SKEL's coupled + knee/ankle translations, which a rotation-only BVH cannot carry (accept it). +- **HSMR camera frame is Y-DOWN** (OpenCV; verified head.y < pelvis.y). `pose_engine` applies + `diag(1,-1,-1)` to reach the treaty's Y-up / faces-−Z. Undo/adjust there if a future model differs. +- Detector is **mobilenet** now (was R50) — CPU detection was the wall-clock hog. It self-caps input + to ~640 px internally, so pre-downscaling does nothing; `--detect-every` is the real throughput knob.