61 lines
4.1 KiB
Markdown
61 lines
4.1 KiB
Markdown
# Lane B engine — HSMR install & smoke-test state (session 3, 2026-07-17)
|
||
|
||
Engine per Decision 1 = **Path B: HSMR → SKEL**, run local on **ultra**, MPS. Installed under
|
||
`.engine/HSMR` (gitignored). This doc = how it's wired + how to run once SKEL lands.
|
||
|
||
## State
|
||
- ✅ `uv venv` py3.10 at `.engine/HSMR/.venv`; torch **2.13.0**, `mps.is_available() == True`.
|
||
- ✅ Ungated weights in `.engine/HSMR/data_inputs/`: HSMR-ViTH ckpt
|
||
(`released_models/HSMR-ViTH-r1d1/checkpoints/hsmr.ckpt`), ViTPose backbone, 3 regressors.
|
||
- ✅ **Real person detector** (torchvision Faster R-CNN, no detectron2) — checked by
|
||
`.engine/HSMR/test_detector.py`, green **today with no SKEL**: 7/7 demo images → 15 patches
|
||
@256×256 through HSMR's own cropper.
|
||
- ✅ Smoke-test runner `.engine/HSMR/smoke_test.py` — **validated to the exact SKEL gate**:
|
||
imports + real detection + patch-crop all run; it fails only on the missing `skel_male.pkl`.
|
||
- 🚧 **Blocked on the gated SKEL download (John's manual step, below) — the last thing standing.**
|
||
- Install script (re-runnable): `.engine/setup_hsmr.sh`.
|
||
- ⚠️ **The engine lives under `.engine/` (gitignored) — the detector fix is on ultra's disk, not
|
||
in git.** Durable copies: `vitdet/__init__.py`, `_headless.py`, `test_detector.py`, and the
|
||
`smoke_test.py` edits. A full re-clone of HSMR drops them; re-apply from this doc if that happens.
|
||
|
||
## Apple-Silicon blockers — all three resolved (session 4, "Lane B detector")
|
||
- **detectron2 won't build on ARM Mac** → **RESOLVED.** Replaced its whole ViTDet detector with
|
||
a torchvision Faster R-CNN drop-in in `lib/modeling/pipelines/vitdet/__init__.py` — same
|
||
`build_detector(...) -> (dets, downsample_ratios)` contract, torchvision's person class (1)
|
||
remapped to HSMR's human id (0), boxes as lurb, ratio 1.0. This also killed a **load-time
|
||
landmine**: the old `import detectron2` there exploded anything importing `lib.kits.hsmr_demo`
|
||
(why session 3 had to stub it). Detector pinned to **CPU** — cheap next to the ViT-H pass and
|
||
it dodges MPS RoIAlign/NMS gaps; an in-file `# ponytail` note marks the mobilenet swap if
|
||
per-frame detection ever dominates on long videos.
|
||
- **pyrender is dead headless on macOS** (`py_renderer` forces `PYOPENGL_PLATFORM=egl`, no EGL on
|
||
Mac; `pyrender` is even referenced in a type annotation, so a plain import-guard won't do).
|
||
Centralized the permissive `pyrender`/`OpenGL` fake into **`.engine/HSMR/_headless.py`** — every
|
||
entrypoint does `import _headless` before any `lib.*` import. QC still happens in Blender (fleet
|
||
renderer + Lane C's tool); the runner dumps `params.npz` (SKEL poses/betas/cam_t — the Path-B
|
||
payload) and `mesh_qc.npz` (verts/faces for Blender).
|
||
- **chumpy failed to build** → **NOT a blocker (verified).** The runtime `SKELWrapper` loads its
|
||
pkls with plain `pickle.load(encoding='latin1')` — no chumpy import. chumpy only appears in
|
||
SKEL's offline model-*build* script (`merge_smplh_mano.py`), which we never run. Ignore the WARN.
|
||
|
||
## John's one step — drop the SKEL files (login + licence, so it's yours to do)
|
||
Register **skel.is.tue.mpg.de** → Download → **"SKEL and BSM models"** → `skel_models_v1.1.zip`
|
||
(**must be v1.1.1** — check `changelog_v1.1.1.txt` inside; older = silent incompatibility).
|
||
Unzip, then:
|
||
```
|
||
mv /path/to/skel_models_v1.1 ~/Documents/MOCAPGOD/.engine/HSMR/data_inputs/body_models/skel
|
||
```
|
||
so that `…/data_inputs/body_models/skel/skel_male.pkl` exists.
|
||
|
||
## Run the smoke test (once SKEL is in place)
|
||
```
|
||
cd ~/Documents/MOCAPGOD/.engine/HSMR && source .venv/bin/activate
|
||
PYTHONPATH=. python smoke_test.py -i data_inputs/demo/example_videos/gymnasts.mp4 # or .../example_imgs
|
||
```
|
||
Outputs → `data_outputs/smoke/`: `params.npz` (rotations), `mesh_qc.npz` (Blender QC), and
|
||
console device/fps/NaN. Answers the Lane B smoke questions: **MPS forward OK? timing (<2×
|
||
realtime)? pose sane?**
|
||
|
||
⚠️ HSMR is a **per-frame** recoverer → root is **camera-space `cam_t`, not world-grounded**.
|
||
Fine for in-place DJ moves; locomotion (walk) needs Lane C's foot-contact root derivation or a
|
||
grounding pass. This is the treaty's global-trajectory caveat, carried into Path B.
|