mocapgod/docs/B_ENGINE_SETUP.md

Lane B engine — HSMR install & smoke-test state (session 3, 2026-07-17)

Engine per Decision 1 = Path B: HSMR → SKEL, run local on ultra, MPS. Installed under .engine/HSMR (gitignored). This doc = how it's wired + how to run once SKEL lands.

State

• ✅ uv venv py3.10 at .engine/HSMR/.venv; torch 2.13.0, mps.is_available() == True.
• ✅ Ungated weights in .engine/HSMR/data_inputs/: HSMR-ViTH ckpt (released_models/HSMR-ViTH-r1d1/checkpoints/hsmr.ckpt), ViTPose backbone, 3 regressors.
• ✅ Real person detector (torchvision Faster R-CNN, no detectron2) — checked by .engine/HSMR/test_detector.py, green today with no SKEL: 7/7 demo images → 15 patches @256×256 through HSMR's own cropper.
• ✅ Smoke-test runner .engine/HSMR/smoke_test.py — PASSED 2026-07-17 (session 5): SKEL v1.1.1 in place, device=mps, 263 frames → 1315 patches, recovery 31.2 fps, nan=False, data_outputs/smoke/params.npz written (poses (N,46), betas (N,10), cam_t (N,3)).
• ✅ Install script (re-runnable): ./setup_hsmr.sh at the repo root (tracked in git). ./setup_hsmr.sh --patch re-applies the ARM/headless fixes onto an existing/re-cloned HSMR.
• ✅ The ARM fixes are now durable in git (session 5, was: "on ultra's disk only"). Canonical copies live in engine_patches/ (_headless.py, test_detector.py, smoke_test.py, lib/modeling/pipelines/vitdet/__init__.py); setup_hsmr.sh overlays them onto the gitignored .engine/HSMR clone. Verified: fresh clone + --patch → test_detector.py green, no hand edits.

Apple-Silicon blockers — all three resolved (session 4, "Lane B detector")

• detectron2 won't build on ARM Mac → RESOLVED. Replaced its whole ViTDet detector with a torchvision Faster R-CNN drop-in in lib/modeling/pipelines/vitdet/__init__.py — same build_detector(...) -> (dets, downsample_ratios) contract, torchvision's person class (1) remapped to HSMR's human id (0), boxes as lurb, ratio 1.0. This also killed a load-time landmine: the old import detectron2 there exploded anything importing lib.kits.hsmr_demo (why session 3 had to stub it). Detector pinned to CPU — cheap next to the ViT-H pass and it dodges MPS RoIAlign/NMS gaps; an in-file # ponytail note marks the mobilenet swap if per-frame detection ever dominates on long videos.
• pyrender is dead headless on macOS (py_renderer forces PYOPENGL_PLATFORM=egl, no EGL on Mac; pyrender is even referenced in a type annotation, so a plain import-guard won't do). Centralized the permissive pyrender/OpenGL fake into .engine/HSMR/_headless.py — every entrypoint does import _headless before any lib.* import. QC still happens in Blender (fleet renderer + Lane C's tool); the runner dumps params.npz (SKEL poses/betas/cam_t — the Path-B payload) and mesh_qc.npz (verts/faces for Blender).
• chumpy failed to build → NOT a blocker (verified). The runtime SKELWrapper loads its pkls with plain pickle.load(encoding='latin1') — no chumpy import. chumpy only appears in SKEL's offline model-build script (merge_smplh_mano.py), which we never run. Ignore the WARN.

John's one step — drop the SKEL files (login + licence, so it's yours to do)

Register skel.is.tue.mpg.de → Download → "SKEL and BSM models" → skel_models_v1.1.zip (must be v1.1.1 — check changelog_v1.1.1.txt inside; older = silent incompatibility). Unzip, then:

mv /path/to/skel_models_v1.1 ~/Documents/MOCAPGOD/.engine/HSMR/data_inputs/body_models/skel

so that …/data_inputs/body_models/skel/skel_male.pkl exists.

Run the smoke test (once SKEL is in place)

cd ~/Documents/MOCAPGOD/.engine/HSMR && source .venv/bin/activate
PYTHONPATH=. python smoke_test.py -i data_inputs/demo/example_videos/gymnasts.mp4   # or .../example_imgs

Outputs → data_outputs/smoke/: params.npz (rotations), mesh_qc.npz (Blender QC), and console device/fps/NaN. Answers the Lane B smoke questions: MPS forward OK? timing (<2× realtime)? pose sane?

⚠️ HSMR is a per-frame recoverer → root is camera-space cam_t, not world-grounded. Fine for in-place DJ moves; locomotion (walk) needs Lane C's foot-contact root derivation or a grounding pass. This is the treaty's global-trajectory caveat, carried into Path B.

Run the pose engine (B2/B3, session 5) — the real service

pose_engine.py (repo root, tracked) wraps the engine into the pipeline. Run under the engine venv:

cd ~/Documents/MOCAPGOD
.engine/HSMR/.venv/bin/python pose_engine.py                    # drain queue/ → out/<name>/
.engine/HSMR/.venv/bin/python pose_engine.py queue/clip.mp4     # one clip
.engine/HSMR/.venv/bin/python pose_engine.py <clip> --from-params  # re-run SKEL→BVH, skip HMR
.engine/HSMR/.venv/bin/python pose_engine.py --selfcheck        # BVH-math self-test, no engine

Per clip → out/<name>/: motion.bvh (SKEL-24, Y-up, metres, faces −Z), motion.json (sidecar, root_space:"camera"), params.npz (raw poses/betas/cam_t for QC / --from-params). Heartbeat: ~/.jobs/pose_engine. Models build once per invocation (queue amortizes); a bad clip is logged and skipped, not fatal.

Key engine facts learned this session (for whoever touches B again):

• SKEL emits poses = (F, 46) biomechanical q, NOT SMPL's 72 axis-angles. Don't decode q by hand — run SKEL.forward() and consume joints_ori (F,24,3,3 global orientations) + joints_tpose.
• BVH is rigid-bone: rest-aware FK (offset=rest_J[j]-rest_J[parent], W=Rg@rest_R^T, local=W[parent]^T@W[j]) reconstructs SKEL joints to ~4 cm; the residual is SKEL's coupled knee/ankle translations, which a rotation-only BVH cannot carry (accept it).
• HSMR camera frame is Y-DOWN (OpenCV; verified head.y < pelvis.y). pose_engine applies diag(1,-1,-1) to reach the treaty's Y-up / faces-−Z. Undo/adjust there if a future model differs.
• Detector is mobilenet now (was R50) — CPU detection was the wall-clock hog. It self-caps input to ~640 px internally, so pre-downscaling does nothing; --detect-every is the real throughput knob.