mocapgod/docs/B_ENGINE_SETUP.md

# Lane B engine — HSMR install & smoke-test state (session 3, 2026-07-17)

Engine per Decision 1 = **Path B: HSMR → SKEL**, run local on **ultra**, MPS. Installed under
`.engine/HSMR` (gitignored). This doc = how it's wired + how to run once SKEL lands.

## State
- ✅ `uv venv` py3.10 at `.engine/HSMR/.venv`; torch **2.13.0**, `mps.is_available() == True`.
- ✅ Ungated weights in `.engine/HSMR/data_inputs/`: HSMR-ViTH ckpt
  (`released_models/HSMR-ViTH-r1d1/checkpoints/hsmr.ckpt`), ViTPose backbone, 3 regressors.
- ✅ **Real person detector** (torchvision Faster R-CNN, no detectron2) — checked by
  `.engine/HSMR/test_detector.py`, green **today with no SKEL**: 7/7 demo images → 15 patches
  @256×256 through HSMR's own cropper.
- ✅ Smoke-test runner `.engine/HSMR/smoke_test.py` — **PASSED 2026-07-17** (session 5): SKEL
  v1.1.1 in place, device=mps, 263 frames → 1315 patches, recovery 31.2 fps, **nan=False**,
  `data_outputs/smoke/params.npz` written (poses (N,46), betas (N,10), cam_t (N,3)).
- ✅ Install script (re-runnable): **`./setup_hsmr.sh`** at the repo root (tracked in git).
  `./setup_hsmr.sh --patch` re-applies the ARM/headless fixes onto an existing/re-cloned HSMR.
- ✅ **The ARM fixes are now durable in git** (session 5, was: "on ultra's disk only"). Canonical
  copies live in **`engine_patches/`** (`_headless.py`, `test_detector.py`, `smoke_test.py`,
  `lib/modeling/pipelines/vitdet/__init__.py`); `setup_hsmr.sh` overlays them onto the gitignored
  `.engine/HSMR` clone. Verified: fresh clone + `--patch` → `test_detector.py` green, no hand edits.

## Apple-Silicon blockers — all three resolved (session 4, "Lane B detector")
- **detectron2 won't build on ARM Mac** → **RESOLVED.** Replaced its whole ViTDet detector with
  a torchvision Faster R-CNN drop-in in `lib/modeling/pipelines/vitdet/__init__.py` — same
  `build_detector(...) -> (dets, downsample_ratios)` contract, torchvision's person class (1)
  remapped to HSMR's human id (0), boxes as lurb, ratio 1.0. This also killed a **load-time
  landmine**: the old `import detectron2` there exploded anything importing `lib.kits.hsmr_demo`
  (why session 3 had to stub it). Detector pinned to **CPU** — cheap next to the ViT-H pass and
  it dodges MPS RoIAlign/NMS gaps; an in-file `# ponytail` note marks the mobilenet swap if
  per-frame detection ever dominates on long videos.
- **pyrender is dead headless on macOS** (`py_renderer` forces `PYOPENGL_PLATFORM=egl`, no EGL on
  Mac; `pyrender` is even referenced in a type annotation, so a plain import-guard won't do).
  Centralized the permissive `pyrender`/`OpenGL` fake into **`.engine/HSMR/_headless.py`** — every
  entrypoint does `import _headless` before any `lib.*` import. QC still happens in Blender (fleet
  renderer + Lane C's tool); the runner dumps `params.npz` (SKEL poses/betas/cam_t — the Path-B
  payload) and `mesh_qc.npz` (verts/faces for Blender).
- **chumpy failed to build** → **NOT a blocker (verified).** The runtime `SKELWrapper` loads its
  pkls with plain `pickle.load(encoding='latin1')` — no chumpy import. chumpy only appears in
  SKEL's offline model-*build* script (`merge_smplh_mano.py`), which we never run. Ignore the WARN.

## John's one step — drop the SKEL files (login + licence, so it's yours to do)
Register **skel.is.tue.mpg.de** → Download → **"SKEL and BSM models"** → `skel_models_v1.1.zip`
(**must be v1.1.1** — check `changelog_v1.1.1.txt` inside; older = silent incompatibility).
Unzip, then:
```
mv /path/to/skel_models_v1.1 ~/Documents/MOCAPGOD/.engine/HSMR/data_inputs/body_models/skel
```
so that `…/data_inputs/body_models/skel/skel_male.pkl` exists.

## Run the smoke test (once SKEL is in place)
```
cd ~/Documents/MOCAPGOD/.engine/HSMR && source .venv/bin/activate
PYTHONPATH=. python smoke_test.py -i data_inputs/demo/example_videos/gymnasts.mp4   # or .../example_imgs
```
Outputs → `data_outputs/smoke/`: `params.npz` (rotations), `mesh_qc.npz` (Blender QC), and
console device/fps/NaN. Answers the Lane B smoke questions: **MPS forward OK? timing (<2×
realtime)? pose sane?**

⚠️ HSMR is a **per-frame** recoverer → root is **camera-space `cam_t`, not world-grounded**.
Fine for in-place DJ moves; locomotion (walk) needs Lane C's foot-contact root derivation or a
grounding pass. This is the treaty's global-trajectory caveat, carried into Path B.

## Run the pose engine (B2/B3, session 5) — the real service
`pose_engine.py` (repo root, tracked) wraps the engine into the pipeline. Run under the engine venv:
```
cd ~/Documents/MOCAPGOD
.engine/HSMR/.venv/bin/python pose_engine.py                    # drain queue/ → out/<name>/
.engine/HSMR/.venv/bin/python pose_engine.py queue/clip.mp4     # one clip
.engine/HSMR/.venv/bin/python pose_engine.py <clip> --from-params  # re-run SKEL→BVH, skip HMR
.engine/HSMR/.venv/bin/python pose_engine.py --selfcheck        # BVH-math self-test, no engine
```
Per clip → `out/<name>/`: `motion.bvh` (SKEL-24, **Y-up, metres, faces −Z**), `motion.json`
(sidecar, `root_space:"camera"`), `params.npz` (raw poses/betas/cam_t for QC / `--from-params`).
Heartbeat: `~/.jobs/pose_engine`. Models build **once** per invocation (queue amortizes); a bad
clip is logged and skipped, not fatal.

Key engine facts learned this session (for whoever touches B again):
- **SKEL emits `poses` = (F, 46)** biomechanical q, NOT SMPL's 72 axis-angles. Don't decode q by
  hand — run `SKEL.forward()` and consume `joints_ori` (F,24,3,3 global orientations) + `joints_tpose`.
- BVH is rigid-bone: rest-aware FK (`offset=rest_J[j]-rest_J[parent]`, `W=Rg@rest_R^T`,
  `local=W[parent]^T@W[j]`) reconstructs SKEL joints to ~4 cm; the residual is SKEL's coupled
  knee/ankle translations, which a rotation-only BVH cannot carry (accept it).
- **HSMR camera frame is Y-DOWN** (OpenCV; verified head.y < pelvis.y). `pose_engine` applies
  `diag(1,-1,-1)` to reach the treaty's Y-up / faces-−Z. Undo/adjust there if a future model differs.
- Detector is **mobilenet** now (was R50) — CPU detection was the wall-clock hog. It self-caps input
  to ~640 px internally, so pre-downscaling does nothing; `--detect-every` is the real throughput knob.