monster/mocapgod
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
4cc67b90fa
mocapgod/docs/B_ENGINE_SETUP.md
type-two 4cc67b90fa Lane B: resolve decision gate -> Path B (local HSMR/SKEL, accept NC risk) 
B1's MoCapAnything pick reversed (animal model, wrong for human capture);
license analysis recorded in LICENSES.md. John chose Path B: run HSMR->SKEL
locally on ultra (MPS), ship rig-motion only, accept the non-commercial
licence risk as interim until a commercial SKEL licence request goes out
(Path A, the clean upgrade).

Engine installed under .engine/ (gitignored): uv venv py3.10, torch 2.13
MPS-green, ungated weights, smoke_test.py with Apple-Silicon bypasses
(detectron2 stubbed with full-frame bbox, headless pyrender stubbed -> QC in
Blender) -- see docs/B_ENGINE_SETUP.md.

Remaining before B ships: John's gated SKEL v1.1.1 download, then the
forward-pass smoke test (MPS end-to-end, timing, pose sanity). Known caveat:
HSMR is per-frame so root is camera-space cam_t, NOT world-grounded -> Lane C
derives root from foot-contact.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 02:25:55 +10:00

3.1 KiB
Raw Blame History

Lane B engine — HSMR install & smoke-test state (session 3, 2026-07-17)

Engine per Decision 1 = Path B: HSMR → SKEL, run local on ultra, MPS. Installed under .engine/HSMR (gitignored). This doc = how it's wired + how to run once SKEL lands.

State

  • ✅ uv venv py3.10 at .engine/HSMR/.venv; torch 2.13.0, mps.is_available() == True.
  • ✅ Ungated weights in .engine/HSMR/data_inputs/: HSMR-ViTH ckpt (released_models/HSMR-ViTH-r1d1/checkpoints/hsmr.ckpt), ViTPose backbone, 3 regressors.
  • ✅ Smoke-test runner .engine/HSMR/smoke_test.py — validated to the exact SKEL gate: imports + detector stub + patch-crop all run; it fails only on the missing skel_male.pkl.
  • 🚧 Blocked on the gated SKEL download (John's manual step, below).
  • Install script (re-runnable): .engine/setup_hsmr.sh.

Two Apple-Silicon bypasses (non-obvious — next session, inherit these)

  • detectron2 will NOT build on ARM Mac (uv build-isolation hides the installed torch from its setup.py; a uv pip install --no-build-isolation retry may fix it if a real detector is wanted). The runner sidesteps it: stubs the vitdet import and injects a full-frame bbox (fine for centered single-subject demo clips). B2 decision: swap detectron2 for a light torchvision/YOLO person-cropper rather than fight the from-source build.
  • pyrender is dead headless on macOS — py_renderer/__init__.py forces PYOPENGL_PLATFORM=egl and there is no EGL on Mac. The runner permissively stubs pyrender/OpenGL and does visual QC in Blender (fleet renderer + Lane C's tool). It dumps mesh_qc.npz (skin/skel verts+faces for Blender) and params.npz (SKEL poses/betas/cam_t — the Path-B payload we keep).
  • chumpy also failed to install. Likely irrelevant for SKEL (its .pkl is modern, not chumpy-pickled); only matters if the SMPL model (eval-only) is ever loaded. If SKEL load throws on a chumpy import, uv pip install --no-build-isolation chumpy + a numpy-alias patch.

John's one step — drop the SKEL files (login + licence, so it's yours to do)

Register skel.is.tue.mpg.de → Download → "SKEL and BSM models" → skel_models_v1.1.zip (must be v1.1.1 — check changelog_v1.1.1.txt inside; older = silent incompatibility). Unzip, then:

mv /path/to/skel_models_v1.1 ~/Documents/MOCAPGOD/.engine/HSMR/data_inputs/body_models/skel

so that …/data_inputs/body_models/skel/skel_male.pkl exists.

Run the smoke test (once SKEL is in place)

cd ~/Documents/MOCAPGOD/.engine/HSMR && source .venv/bin/activate
PYTHONPATH=. python smoke_test.py -i data_inputs/demo/example_videos/gymnasts.mp4   # or .../example_imgs

Outputs → data_outputs/smoke/: params.npz (rotations), mesh_qc.npz (Blender QC), and console device/fps/NaN. Answers the Lane B smoke questions: MPS forward OK? timing (<2× realtime)? pose sane?

⚠️ HSMR is a per-frame recoverer → root is camera-space cam_t, not world-grounded. Fine for in-place DJ moves; locomotion (walk) needs Lane C's foot-contact root derivation or a grounding pass. This is the treaty's global-trajectory caveat, carried into Path B.