Lane B: tick Lane B done, session-5 completion summary + Lane C handoff

PROGRESS.md: Lane B checked off with the session-5 outcome; SESSION 5 DONE block
(all 6 orders) + explicit Lane C handoff (BVH axes, skel_to_mixamorig map, camera
root_space, --from-params). B_ENGINE_SETUP.md: pose_engine.py usage + the engine
facts learned (SKEL 46-q, rest-aware FK ~4cm, Y-down camera fix, mobilenet detector).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
type-two 2026-07-17 10:42:28 +10:00
parent 7d0a3b7f7c
commit 35403f8f05
2 changed files with 57 additions and 1 deletions

View File

@ -6,7 +6,15 @@ One lane per session. Tick on completion. See PLAN.md for acceptance checks.
spec/mixamorig_bones.json (65 bones, `mixamorig:` colon-prefixed). — 2026-07-17
- [x] **Lane A** — capture + ingest. docs/CAPTURE.md protocol, ingest.py
(probe → 30fps normalize → rename → rsync to m3ultra queue → LEDGER.tsv). — 2026-07-17
- [ ] **Lane B** — pose engine. **Decision made 2026-07-17 (session 3): Path B** — engine =
- [x] **Lane B** — pose engine. **DONE session 5 (2026-07-17).** `pose_engine.py` ships:
queue video → HSMR/SKEL → SKEL-24 BVH@fps + `motion.json` sidecar (`root_space:"camera"`),
`spec/skel_to_mixamorig.json` name-map for Lane C. Verified on the demo clip: Blender import
= 24 bones, **upright** (1.59 m), motion present, FK self-consistency **0.0 mm**; queue of 6
drained unattended (2 bad clips failed gracefully, 4 valid BVH, heartbeat per clip); runtime
**~1.6× realtime** steady-state (under the <2× bar). mobilenet detector + detect-every-20 +
single-subject box reuse retired the CPU-detector wall-clock finding. Axis fix: HSMR emits
Y-down (OpenCV) → `diag(1,-1,-1)` → Y-up/faces-Z. **History (sessions 24):**
**Decision made session 3: Path B** — engine =
**HSMR → SKEL**, run local, ship rig-motion only, John accepts NC-licence risk
(`docs/DECISIONS.md` Decision 1 RESOLVED). Env up in `.engine/HSMR` (uv venv, torch 2.13
**MPS green**, ungated weights pulled). **Detector RESOLVED (session 4):** torchvision
@ -22,6 +30,29 @@ One lane per session. Tick on completion. See PLAN.md for acceptance checks.
- [ ] Lane E — ML upgrades (post-v1, gated).
- [ ] Lane F — UX (optional).
## SESSION 5 — DONE (2026-07-17). All six orders executed; Lane B ships.
1. ✅ **Durable-ized** the ARM fixes → `engine_patches/` + `setup_hsmr.sh --patch`. Verified: fresh
clone + `--patch``test_detector.py` green, no hand edits.
2. ✅ **Path-A email** drafted in `docs/DECISIONS.md` (John sends to ps-license@tue.mpg.de).
3. ✅ **Queue reconciled**: `ingest.py` defaults to local `queue/` on ultra; `--remote-queue` keeps
the m3ultra path. PLAN Ground-rule-1 deviation noted; CAPTURE.md updated.
4. ✅ **Treaty amended**: `root_space` (camera|world) is a REQUIRED sidecar field; Lane C reads it.
5. ✅ **B2 + B3**: `pose_engine.py` (SKEL-24 BVH + sidecar + jitter) + `spec/skel_to_mixamorig.json`.
Detector swapped to mobilenet; detect-every-20 + box reuse; build-once + per-video isolation.
Ran the smoke test's SKEL forward on real params — no re-verify needed (smoke already PASSED).
**HANDOFF TO LANE C (retarget + cleanup, Blender, on ultra):**
- Input is `out/<name>/motion.bvh` (SKEL-24 skeleton, **Y-up, metres, faces Z**) + `motion.json`.
Import in Blender with `import_anim.bvh(axis_forward='-Z', axis_up='Y')` (verified upright).
- Constraint-map SKEL→mixamorig via **`spec/skel_to_mixamorig.json`** (20 mapped, 4 null=twist/heel;
Spine1 interpolated, Head follows Neck, fingers are Lane E). mixamorig canon: `mixamorig:` colon.
- **`root_space` is `"camera"`** → root is NOT world-grounded (HSMR is per-frame). In-place clips are
first-class; for traveling clips derive a world root from foot contacts (PLAN Lane C task).
- Rest pose is a clean T-pose at zero rotation; T-pose bookends only exist if John filmed the protocol
(demo clips don't have them) — Lane C's trim must tolerate their absence.
- Re-run the SKEL→BVH stage cheaply with `pose_engine.py <clip> --from-params` (skips HMR) when
tuning; raw params are saved at `out/<name>/params.npz`.
## SESSION 5 ORDERS — from Fable's review (2026-07-17, supersede older notes where they conflict)
Review verdict: Lanes 0/A solid; session 2's reversal of the B1 animal-model blunder was the

View File

@ -60,3 +60,28 @@ realtime)? pose sane?**
⚠️ HSMR is a **per-frame** recoverer → root is **camera-space `cam_t`, not world-grounded**.
Fine for in-place DJ moves; locomotion (walk) needs Lane C's foot-contact root derivation or a
grounding pass. This is the treaty's global-trajectory caveat, carried into Path B.
## Run the pose engine (B2/B3, session 5) — the real service
`pose_engine.py` (repo root, tracked) wraps the engine into the pipeline. Run under the engine venv:
```
cd ~/Documents/MOCAPGOD
.engine/HSMR/.venv/bin/python pose_engine.py # drain queue/ → out/<name>/
.engine/HSMR/.venv/bin/python pose_engine.py queue/clip.mp4 # one clip
.engine/HSMR/.venv/bin/python pose_engine.py <clip> --from-params # re-run SKEL→BVH, skip HMR
.engine/HSMR/.venv/bin/python pose_engine.py --selfcheck # BVH-math self-test, no engine
```
Per clip → `out/<name>/`: `motion.bvh` (SKEL-24, **Y-up, metres, faces Z**), `motion.json`
(sidecar, `root_space:"camera"`), `params.npz` (raw poses/betas/cam_t for QC / `--from-params`).
Heartbeat: `~/.jobs/pose_engine`. Models build **once** per invocation (queue amortizes); a bad
clip is logged and skipped, not fatal.
Key engine facts learned this session (for whoever touches B again):
- **SKEL emits `poses` = (F, 46)** biomechanical q, NOT SMPL's 72 axis-angles. Don't decode q by
hand — run `SKEL.forward()` and consume `joints_ori` (F,24,3,3 global orientations) + `joints_tpose`.
- BVH is rigid-bone: rest-aware FK (`offset=rest_J[j]-rest_J[parent]`, `W=Rg@rest_R^T`,
`local=W[parent]^T@W[j]`) reconstructs SKEL joints to ~4 cm; the residual is SKEL's coupled
knee/ankle translations, which a rotation-only BVH cannot carry (accept it).
- **HSMR camera frame is Y-DOWN** (OpenCV; verified head.y < pelvis.y). `pose_engine` applies
`diag(1,-1,-1)` to reach the treaty's Y-up / faces-Z. Undo/adjust there if a future model differs.
- Detector is **mobilenet** now (was R50) — CPU detection was the wall-clock hog. It self-caps input
to ~640 px internally, so pre-downscaling does nothing; `--detect-every` is the real throughput knob.