Lane B: tick Lane B done, session-5 completion summary + Lane C handoff
PROGRESS.md: Lane B checked off with the session-5 outcome; SESSION 5 DONE block (all 6 orders) + explicit Lane C handoff (BVH axes, skel_to_mixamorig map, camera root_space, --from-params). B_ENGINE_SETUP.md: pose_engine.py usage + the engine facts learned (SKEL 46-q, rest-aware FK ~4cm, Y-down camera fix, mobilenet detector). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
7d0a3b7f7c
commit
35403f8f05
33
PROGRESS.md
33
PROGRESS.md
@ -6,7 +6,15 @@ One lane per session. Tick on completion. See PLAN.md for acceptance checks.
|
||||
spec/mixamorig_bones.json (65 bones, `mixamorig:` colon-prefixed). — 2026-07-17
|
||||
- [x] **Lane A** — capture + ingest. docs/CAPTURE.md protocol, ingest.py
|
||||
(probe → 30fps normalize → rename → rsync to m3ultra queue → LEDGER.tsv). — 2026-07-17
|
||||
- [ ] **Lane B** — pose engine. **Decision made 2026-07-17 (session 3): Path B** — engine =
|
||||
- [x] **Lane B** — pose engine. **DONE session 5 (2026-07-17).** `pose_engine.py` ships:
|
||||
queue video → HSMR/SKEL → SKEL-24 BVH@fps + `motion.json` sidecar (`root_space:"camera"`),
|
||||
`spec/skel_to_mixamorig.json` name-map for Lane C. Verified on the demo clip: Blender import
|
||||
= 24 bones, **upright** (1.59 m), motion present, FK self-consistency **0.0 mm**; queue of 6
|
||||
drained unattended (2 bad clips failed gracefully, 4 valid BVH, heartbeat per clip); runtime
|
||||
**~1.6× realtime** steady-state (under the <2× bar). mobilenet detector + detect-every-20 +
|
||||
single-subject box reuse retired the CPU-detector wall-clock finding. Axis fix: HSMR emits
|
||||
Y-down (OpenCV) → `diag(1,-1,-1)` → Y-up/faces-−Z. **History (sessions 2–4):**
|
||||
**Decision made session 3: Path B** — engine =
|
||||
**HSMR → SKEL**, run local, ship rig-motion only, John accepts NC-licence risk
|
||||
(`docs/DECISIONS.md` Decision 1 RESOLVED). Env up in `.engine/HSMR` (uv venv, torch 2.13
|
||||
**MPS green**, ungated weights pulled). **Detector RESOLVED (session 4):** torchvision
|
||||
@ -22,6 +30,29 @@ One lane per session. Tick on completion. See PLAN.md for acceptance checks.
|
||||
- [ ] Lane E — ML upgrades (post-v1, gated).
|
||||
- [ ] Lane F — UX (optional).
|
||||
|
||||
## SESSION 5 — DONE (2026-07-17). All six orders executed; Lane B ships.
|
||||
1. ✅ **Durable-ized** the ARM fixes → `engine_patches/` + `setup_hsmr.sh --patch`. Verified: fresh
|
||||
clone + `--patch` → `test_detector.py` green, no hand edits.
|
||||
2. ✅ **Path-A email** drafted in `docs/DECISIONS.md` (John sends to ps-license@tue.mpg.de).
|
||||
3. ✅ **Queue reconciled**: `ingest.py` defaults to local `queue/` on ultra; `--remote-queue` keeps
|
||||
the m3ultra path. PLAN Ground-rule-1 deviation noted; CAPTURE.md updated.
|
||||
4. ✅ **Treaty amended**: `root_space` (camera|world) is a REQUIRED sidecar field; Lane C reads it.
|
||||
5. ✅ **B2 + B3**: `pose_engine.py` (SKEL-24 BVH + sidecar + jitter) + `spec/skel_to_mixamorig.json`.
|
||||
Detector swapped to mobilenet; detect-every-20 + box reuse; build-once + per-video isolation.
|
||||
Ran the smoke test's SKEL forward on real params — no re-verify needed (smoke already PASSED).
|
||||
|
||||
**HANDOFF TO LANE C (retarget + cleanup, Blender, on ultra):**
|
||||
- Input is `out/<name>/motion.bvh` (SKEL-24 skeleton, **Y-up, metres, faces −Z**) + `motion.json`.
|
||||
Import in Blender with `import_anim.bvh(axis_forward='-Z', axis_up='Y')` (verified upright).
|
||||
- Constraint-map SKEL→mixamorig via **`spec/skel_to_mixamorig.json`** (20 mapped, 4 null=twist/heel;
|
||||
Spine1 interpolated, Head follows Neck, fingers are Lane E). mixamorig canon: `mixamorig:` colon.
|
||||
- **`root_space` is `"camera"`** → root is NOT world-grounded (HSMR is per-frame). In-place clips are
|
||||
first-class; for traveling clips derive a world root from foot contacts (PLAN Lane C task).
|
||||
- Rest pose is a clean T-pose at zero rotation; T-pose bookends only exist if John filmed the protocol
|
||||
(demo clips don't have them) — Lane C's trim must tolerate their absence.
|
||||
- Re-run the SKEL→BVH stage cheaply with `pose_engine.py <clip> --from-params` (skips HMR) when
|
||||
tuning; raw params are saved at `out/<name>/params.npz`.
|
||||
|
||||
## SESSION 5 ORDERS — from Fable's review (2026-07-17, supersede older notes where they conflict)
|
||||
|
||||
Review verdict: Lanes 0/A solid; session 2's reversal of the B1 animal-model blunder was the
|
||||
|
||||
@ -60,3 +60,28 @@ realtime)? pose sane?**
|
||||
⚠️ HSMR is a **per-frame** recoverer → root is **camera-space `cam_t`, not world-grounded**.
|
||||
Fine for in-place DJ moves; locomotion (walk) needs Lane C's foot-contact root derivation or a
|
||||
grounding pass. This is the treaty's global-trajectory caveat, carried into Path B.
|
||||
|
||||
## Run the pose engine (B2/B3, session 5) — the real service
|
||||
`pose_engine.py` (repo root, tracked) wraps the engine into the pipeline. Run under the engine venv:
|
||||
```
|
||||
cd ~/Documents/MOCAPGOD
|
||||
.engine/HSMR/.venv/bin/python pose_engine.py # drain queue/ → out/<name>/
|
||||
.engine/HSMR/.venv/bin/python pose_engine.py queue/clip.mp4 # one clip
|
||||
.engine/HSMR/.venv/bin/python pose_engine.py <clip> --from-params # re-run SKEL→BVH, skip HMR
|
||||
.engine/HSMR/.venv/bin/python pose_engine.py --selfcheck # BVH-math self-test, no engine
|
||||
```
|
||||
Per clip → `out/<name>/`: `motion.bvh` (SKEL-24, **Y-up, metres, faces −Z**), `motion.json`
|
||||
(sidecar, `root_space:"camera"`), `params.npz` (raw poses/betas/cam_t for QC / `--from-params`).
|
||||
Heartbeat: `~/.jobs/pose_engine`. Models build **once** per invocation (queue amortizes); a bad
|
||||
clip is logged and skipped, not fatal.
|
||||
|
||||
Key engine facts learned this session (for whoever touches B again):
|
||||
- **SKEL emits `poses` = (F, 46)** biomechanical q, NOT SMPL's 72 axis-angles. Don't decode q by
|
||||
hand — run `SKEL.forward()` and consume `joints_ori` (F,24,3,3 global orientations) + `joints_tpose`.
|
||||
- BVH is rigid-bone: rest-aware FK (`offset=rest_J[j]-rest_J[parent]`, `W=Rg@rest_R^T`,
|
||||
`local=W[parent]^T@W[j]`) reconstructs SKEL joints to ~4 cm; the residual is SKEL's coupled
|
||||
knee/ankle translations, which a rotation-only BVH cannot carry (accept it).
|
||||
- **HSMR camera frame is Y-DOWN** (OpenCV; verified head.y < pelvis.y). `pose_engine` applies
|
||||
`diag(1,-1,-1)` to reach the treaty's Y-up / faces-−Z. Undo/adjust there if a future model differs.
|
||||
- Detector is **mobilenet** now (was R50) — CPU detection was the wall-clock hog. It self-caps input
|
||||
to ~640 px internally, so pre-downscaling does nothing; `--detect-every` is the real throughput knob.
|
||||
|
||||
Loading…
Reference in New Issue
Block a user