4cc67b90fa
B1's MoCapAnything pick reversed (animal model, wrong for human capture); license analysis recorded in LICENSES.md. John chose Path B: run HSMR->SKEL locally on ultra (MPS), ship rig-motion only, accept the non-commercial licence risk as interim until a commercial SKEL licence request goes out (Path A, the clean upgrade). Engine installed under .engine/ (gitignored): uv venv py3.10, torch 2.13 MPS-green, ungated weights, smoke_test.py with Apple-Silicon bypasses (detectron2 stubbed with full-frame bbox, headless pyrender stubbed -> QC in Blender) -- see docs/B_ENGINE_SETUP.md. Remaining before B ships: John's gated SKEL v1.1.1 download, then the forward-pass smoke test (MPS end-to-end, timing, pose sanity). Known caveat: HSMR is per-frame so root is camera-space cam_t, NOT world-grounded -> Lane C derives root from foot-contact. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
51 lines
3.1 KiB
Markdown
51 lines
3.1 KiB
Markdown
# Lane B engine — HSMR install & smoke-test state (session 3, 2026-07-17)
|
||
|
||
Engine per Decision 1 = **Path B: HSMR → SKEL**, run local on **ultra**, MPS. Installed under
|
||
`.engine/HSMR` (gitignored). This doc = how it's wired + how to run once SKEL lands.
|
||
|
||
## State
|
||
- ✅ `uv venv` py3.10 at `.engine/HSMR/.venv`; torch **2.13.0**, `mps.is_available() == True`.
|
||
- ✅ Ungated weights in `.engine/HSMR/data_inputs/`: HSMR-ViTH ckpt
|
||
(`released_models/HSMR-ViTH-r1d1/checkpoints/hsmr.ckpt`), ViTPose backbone, 3 regressors.
|
||
- ✅ Smoke-test runner `.engine/HSMR/smoke_test.py` — **validated to the exact SKEL gate**:
|
||
imports + detector stub + patch-crop all run; it fails only on the missing `skel_male.pkl`.
|
||
- 🚧 **Blocked on the gated SKEL download (John's manual step, below).**
|
||
- Install script (re-runnable): `.engine/setup_hsmr.sh`.
|
||
|
||
## Two Apple-Silicon bypasses (non-obvious — next session, inherit these)
|
||
- **detectron2 will NOT build on ARM Mac** (uv build-isolation hides the installed torch from
|
||
its `setup.py`; a `uv pip install --no-build-isolation` retry may fix it if a real detector is
|
||
wanted). The runner **sidesteps it**: stubs the `vitdet` import and injects a **full-frame
|
||
bbox** (fine for centered single-subject demo clips). *B2 decision:* swap detectron2 for a
|
||
light torchvision/YOLO person-cropper rather than fight the from-source build.
|
||
- **pyrender is dead headless on macOS** — `py_renderer/__init__.py` forces
|
||
`PYOPENGL_PLATFORM=egl` and there is no EGL on Mac. The runner permissively stubs
|
||
`pyrender`/`OpenGL` and does **visual QC in Blender** (fleet renderer + Lane C's tool). It
|
||
dumps `mesh_qc.npz` (skin/skel verts+faces for Blender) and `params.npz` (SKEL
|
||
poses/betas/cam_t — the Path-B payload we keep).
|
||
- **chumpy** also failed to install. Likely irrelevant for SKEL (its `.pkl` is modern, not
|
||
chumpy-pickled); only matters if the SMPL model (eval-only) is ever loaded. If SKEL load
|
||
throws on a chumpy import, `uv pip install --no-build-isolation chumpy` + a numpy-alias patch.
|
||
|
||
## John's one step — drop the SKEL files (login + licence, so it's yours to do)
|
||
Register **skel.is.tue.mpg.de** → Download → **"SKEL and BSM models"** → `skel_models_v1.1.zip`
|
||
(**must be v1.1.1** — check `changelog_v1.1.1.txt` inside; older = silent incompatibility).
|
||
Unzip, then:
|
||
```
|
||
mv /path/to/skel_models_v1.1 ~/Documents/MOCAPGOD/.engine/HSMR/data_inputs/body_models/skel
|
||
```
|
||
so that `…/data_inputs/body_models/skel/skel_male.pkl` exists.
|
||
|
||
## Run the smoke test (once SKEL is in place)
|
||
```
|
||
cd ~/Documents/MOCAPGOD/.engine/HSMR && source .venv/bin/activate
|
||
PYTHONPATH=. python smoke_test.py -i data_inputs/demo/example_videos/gymnasts.mp4 # or .../example_imgs
|
||
```
|
||
Outputs → `data_outputs/smoke/`: `params.npz` (rotations), `mesh_qc.npz` (Blender QC), and
|
||
console device/fps/NaN. Answers the Lane B smoke questions: **MPS forward OK? timing (<2×
|
||
realtime)? pose sane?**
|
||
|
||
⚠️ HSMR is a **per-frame** recoverer → root is **camera-space `cam_t`, not world-grounded**.
|
||
Fine for in-place DJ moves; locomotion (walk) needs Lane C's foot-contact root derivation or a
|
||
grounding pass. This is the treaty's global-trajectory caveat, carried into Path B.
|