Lane B / session-5 item 1: durable-ize the ARM engine fixes into git

The torchvision detector drop-in, headless pyrender/OpenGL shims, and the
detector+smoke tests existed only on ultra's disk inside gitignored .engine/.
Captured byte-identical copies into engine_patches/ (mirrors the HSMR subtree)
and moved setup_hsmr.sh to the repo root (tracked, path-relative) with a
--patch step that overlays them onto the clone.

Acceptance PASSED: fresh shallow clone of HSMR in a temp dir + './setup_hsmr.sh
--patch' + validated venv -> test_detector.py green (7/7 demo imgs, 15 patches),
no hand edits. A re-clone can no longer silently drop the ARM fixes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs/B_ENGINE_SETUP.md

@@ -10,13 +10,15 @@ Engine per Decision 1 = **Path B: HSMR → SKEL**, run local on **ultra**, MPS.
 - ✅ **Real person detector** (torchvision Faster R-CNN, no detectron2) — checked by
   `.engine/HSMR/test_detector.py`, green **today with no SKEL**: 7/7 demo images → 15 patches
   @256×256 through HSMR's own cropper.
-- ✅ Smoke-test runner `.engine/HSMR/smoke_test.py` — **validated to the exact SKEL gate**:
-  imports + real detection + patch-crop all run; it fails only on the missing `skel_male.pkl`.
-- 🚧 **Blocked on the gated SKEL download (John's manual step, below) — the last thing standing.**
-- Install script (re-runnable): `.engine/setup_hsmr.sh`.
-- ⚠️ **The engine lives under `.engine/` (gitignored) — the detector fix is on ultra's disk, not
-  in git.** Durable copies: `vitdet/__init__.py`, `_headless.py`, `test_detector.py`, and the
-  `smoke_test.py` edits. A full re-clone of HSMR drops them; re-apply from this doc if that happens.
+- ✅ Smoke-test runner `.engine/HSMR/smoke_test.py` — **PASSED 2026-07-17** (session 5): SKEL
+  v1.1.1 in place, device=mps, 263 frames → 1315 patches, recovery 31.2 fps, **nan=False**,
+  `data_outputs/smoke/params.npz` written (poses (N,46), betas (N,10), cam_t (N,3)).
+- ✅ Install script (re-runnable): **`./setup_hsmr.sh`** at the repo root (tracked in git).
+  `./setup_hsmr.sh --patch` re-applies the ARM/headless fixes onto an existing/re-cloned HSMR.
+- ✅ **The ARM fixes are now durable in git** (session 5, was: "on ultra's disk only"). Canonical
+  copies live in **`engine_patches/`** (`_headless.py`, `test_detector.py`, `smoke_test.py`,
+  `lib/modeling/pipelines/vitdet/__init__.py`); `setup_hsmr.sh` overlays them onto the gitignored
+  `.engine/HSMR` clone. Verified: fresh clone + `--patch` → `test_detector.py` green, no hand edits.
 
 ## Apple-Silicon blockers — all three resolved (session 4, "Lane B detector")
 - **detectron2 won't build on ARM Mac** → **RESOLVED.** Replaced its whole ViTDet detector with

engine_patches/_headless.py

@@ -0,0 +1,34 @@
+"""Headless-macOS import shims for the HSMR engine (our fleet is all Apple Silicon).
+
+pyrender -> PyOpenGL -> EGL cannot load without a GL context, yet several vendored modules
+`import pyrender` at load time — some even in type annotations (`List[pyrender.Node]`) — so
+the import chain explodes before any of our code runs. We never render in-process; Blender
+is the fleet renderer (Lane C). So we drop permissive fakes for the pyrender + OpenGL stack
+into sys.modules, letting the chain import while keeping everything else (ColorPalette,
+Wis3D, the models) real.
+
+Import this BEFORE the first `lib.*` import:  `import _headless  # noqa: F401`
+
+# ponytail: fakes, not a real headless GL backend (osmesa/egl). We don't render here, so a
+# working GL context would be dead weight. Wire a real one only if in-process render is ever
+# needed off-fleet.
+"""
+import sys
+import types
+
+
+class _Any:
+    def __call__(self, *a, **k): return self
+    def __getattr__(self, n): return self
+
+
+class _AnyModule(types.ModuleType):
+    def __getattr__(self, n):
+        if n.startswith('__') and n.endswith('__'):
+            raise AttributeError(n)
+        return _Any()
+
+
+# setdefault: never clobber a real module that a caller already imported on purpose.
+for _n in ('pyrender', 'OpenGL', 'OpenGL.GL', 'OpenGL.error', 'OpenGL.platform'):
+    sys.modules.setdefault(_n, _AnyModule(_n))

engine_patches/lib/modeling/pipelines/vitdet/__init__.py

@@ -0,0 +1,75 @@
+"""
+Person detector — torchvision drop-in replacement for HSMR's detectron2/ViTDet.
+
+Why: detectron2 + chumpy won't build from source on Apple Silicon (the Lane B blocker),
+and this module's `import detectron2` was a load-time landmine for everything that touches
+`lib.kits.hsmr_demo` (smoke_test.py, the future pose_engine.py). torchvision ships a
+COCO-pretrained Faster R-CNN with ungated weights and no build step.
+
+Same public contract as the original, so run_demo.py / smoke_test.py / pose_engine.py
+import it unchanged:
+
+    build_detector(...) -> callable(raw_imgs) -> (dets, downsample_ratios)
+
+    dets[i]           : dict of CPU tensors (empty tensors if no person) —
+                          pred_classes : all 0  (== hsmr_demo CLASS_HUMAN_ID)
+                          scores       : float
+                          pred_boxes   : (N, 4) left-upper-right-bottom pixels
+    downsample_ratios : one float per image; boxes are in native-resolution pixels -> 1.0
+
+Contract is checked by test_detector.py (runs today, no SKEL needed).
+
+# ponytail: ~35 lines replaces the whole detectron2 stack. The ViT-H HMR forward pass is
+# the runtime cost, not detection, so we don't chase detector throughput here.
+"""
+import numpy as np
+import torch
+from tqdm import tqdm
+from torchvision.models.detection import (
+    fasterrcnn_resnet50_fpn_v2,
+    FasterRCNN_ResNet50_FPN_V2_Weights,
+)
+
+_COCO_PERSON = 1  # torchvision COCO label id for "person" (0 == background)
+
+
+class _PersonDetector:
+    """Callable over a list of RGB HxWx3 images (what HSMR's load_inputs yields).
+    uint8 or float, [0,255] or [0,1] are all handled."""
+
+    def __init__(self, device='cpu', score_thresh=0.25):
+        # score_thresh 0.25 mirrors the old detectron2 test_score_thresh; hsmr_demo's
+        # _img_det2patches re-filters at 0.5, so this only widens recall a little.
+        weights = FasterRCNN_ResNet50_FPN_V2_Weights.DEFAULT
+        self.model = fasterrcnn_resnet50_fpn_v2(
+            weights=weights, box_score_thresh=score_thresh,
+        ).eval().to(device)
+        self.device = device
+
+    @torch.no_grad()
+    def __call__(self, raw_imgs):
+        dets, ratios = [], []
+        for img in tqdm(raw_imgs, desc='Detecting'):
+            t = torch.as_tensor(np.ascontiguousarray(img), device=self.device).float()
+            if float(t.max()) > 1.5:          # [0,255] -> [0,1]
+                t = t / 255.0
+            t = t.permute(2, 0, 1)            # HWC -> CHW
+            out = self.model([t])[0]
+            keep = out['labels'] == _COCO_PERSON
+            dets.append({
+                'pred_classes': torch.zeros(int(keep.sum()), dtype=torch.long),  # human -> 0
+                'scores'      : out['scores'][keep].cpu(),
+                'pred_boxes'  : out['boxes'][keep].cpu(),   # xyxy == left-upper-right-bottom
+            })
+            ratios.append(1.0)               # detected at native resolution
+        return dets, ratios
+
+
+def build_detector(batch_size=1, max_img_size=512, device='cpu'):
+    """Signature-compatible with the old detectron2 builder. batch_size / max_img_size are
+    accepted for parity but unused — detection runs per-frame at native resolution.
+
+    # ponytail: detector pinned to CPU regardless of `device`. It's cheap next to the
+    # ViT-H HMR pass, and CPU sidesteps the MPS coverage gaps in RoIAlign/NMS. Flip to
+    # `device` only if per-frame detection ever dominates on long videos."""
+    return _PersonDetector(device='cpu')

engine_patches/smoke_test.py

@@ -0,0 +1,100 @@
+#!/usr/bin/env python
+"""
+HSMR smoke test on Apple Silicon (MPS) — Path B engine verification.
+
+Answers the Lane B smoke-test questions on HSMR's bundled demo clips (no John footage,
+no SKEL license needed beyond the model files John downloaded):
+  1. Does the HSMR forward run on MPS?  (device + CPU-fallback)
+  2. Timing (target < 2x realtime).
+  3. Pose sanity (shape, NaNs) + an overlay render to eyeball quality.
+
+Detection uses the torchvision person detector (lib.modeling.pipelines.vitdet, the
+detectron2-free drop-in). The one remaining gate is SKEL: build_inference_pipeline
+instantiates the SKEL body model, so this completes only once John's SKEL files are in place.
+
+Run (once SKEL files are in data_inputs/body_models/skel/):
+  cd ~/Documents/MOCAPGOD/.engine/HSMR && source .venv/bin/activate
+  PYTHONPATH=. python smoke_test.py -i data_inputs/demo/example_videos/gymnasts.mp4
+  PYTHONPATH=. python smoke_test.py -i data_inputs/demo/example_imgs   # image folder
+"""
+import os, time, argparse
+os.environ.setdefault('PYTORCH_ENABLE_MPS_FALLBACK', '1')  # some ops lack MPS kernels
+
+import _headless  # noqa: F401  fake pyrender/OpenGL for headless macOS — MUST precede lib.* imports
+
+# One import line: every helper is an attribute of hsmr_demo (via its star-imports).
+from lib.kits.hsmr_demo import (
+    load_inputs, imgs_det2patches, build_inference_pipeline, prepare_mesh,
+    visualize_full_img, IMG_MEAN_255, IMG_STD_255, asb, assemble_dict,
+    get_logger, save_video, save_img, np, torch, Path, DEFAULT_HSMR_ROOT,
+    build_detector,
+)
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument('-i', '--input_path', required=True)
+    ap.add_argument('-o', '--output_path', default='data_outputs/smoke')
+    ap.add_argument('-d', '--device', default='mps')
+    ap.add_argument('-m', '--model_root', default=None)
+    ap.add_argument('--rec_bs', type=int, default=64)
+    ap.add_argument('--ignore_skel', action='store_true')
+    args = ap.parse_args()
+
+    log = get_logger(brief=True)
+    model_root = Path(args.model_root) if args.model_root else DEFAULT_HSMR_ROOT
+    out = Path(args.output_path); out.mkdir(parents=True, exist_ok=True)
+
+    # Inputs (reuse the demo loader; it only reads .input_path / .input_type).
+    fake = argparse.Namespace(input_path=args.input_path, input_type='auto')
+    raw_imgs, meta = load_inputs(fake)
+    log.info(f'device={args.device}  type={meta["type"]}  frames={len(raw_imgs)}')
+
+    # Detection -> patches (real torchvision person detector, on CPU).
+    dets, ratios = build_detector(device=args.device)(raw_imgs)
+    patches, det_meta = imgs_det2patches(raw_imgs, dets, ratios, 5)
+    log.info(f'patches={len(patches)}')
+
+    # Build pipeline — instantiates SKEL, so this is where a missing SKEL file fails.
+    pipe = build_inference_pipeline(model_root=model_root, device=args.device)
+
+    # Recovery loop (mirrors run_demo), timed.
+    t0 = time.time()
+    pd_params, pd_cam_t = [], []
+    for bw in asb(total=len(patches), bs_scope=args.rec_bs, enable_tqdm=True):
+        p = patches[bw.sid:bw.eid]
+        pn = ((p - IMG_MEAN_255) / IMG_STD_255).transpose(0, 3, 1, 2)
+        with torch.no_grad():
+            o = pipe(pn)
+        pd_params.append({k: v.detach().cpu().clone() for k, v in o['pd_params'].items()})
+        pd_cam_t.append(o['pd_cam_t'].detach().cpu().clone())
+    dt = time.time() - t0
+    pd_params = assemble_dict(pd_params, expand_dim=False)
+    pd_cam_t = torch.cat(pd_cam_t, dim=0)
+
+    poses = pd_params['poses']
+    fps = len(patches) / dt if dt else 0
+    log.info(f'RECOVERY ok: {len(patches)} patches in {dt:.2f}s ({fps:.2f} fps)  '
+             f'poses={tuple(poses.shape)}  nan={bool(torch.isnan(poses).any())}')
+
+    # Dump raw params regardless (this is the Path-B payload: SKEL rotations).
+    np.savez(out / 'params.npz', poses=poses.numpy(),
+             betas=pd_params['betas'].numpy(), cam_t=pd_cam_t.numpy())
+    log.info(f'params -> {out/"params.npz"}')
+
+    # Mesh geometry for visual QC in Blender (pyrender/EGL is dead headless on macOS, and
+    # Blender is the fleet renderer + Lane C's tool). prepare_mesh uses skel_model, no GL.
+    try:
+        m_skin, m_skel = prepare_mesh(pipe, pd_params)
+        np.savez(out / 'mesh_qc.npz',
+                 skin_v=m_skin['v'].detach().cpu().numpy(), skin_f=np.asarray(m_skin['f']),
+                 skel_v=m_skel['v'].detach().cpu().numpy(), skel_f=np.asarray(m_skel['f']))
+        log.info(f'mesh geometry -> {out/"mesh_qc.npz"} (import in Blender to eyeball)')
+    except Exception as e:
+        log.warning(f'prepare_mesh failed (params still saved): {e!r}')
+
+    log.info('smoke test complete.')
+
+
+if __name__ == '__main__':
+    main()

engine_patches/test_detector.py

@@ -0,0 +1,71 @@
+#!/usr/bin/env python
+"""Detector smoke check — runs TODAY, no SKEL, no HMR forward pass.
+
+Proves the torchvision person detector (lib/modeling/pipelines/vitdet) imports without
+detectron2 and honors the contract lib.kits.hsmr_demo._img_det2patches depends on.
+
+    cd ~/Documents/MOCAPGOD/.engine/HSMR && source .venv/bin/activate
+    PYTHONPATH=. python test_detector.py
+"""
+from pathlib import Path
+import cv2
+import torch
+
+import _headless  # noqa: F401  fake pyrender/OpenGL for headless macOS — MUST precede lib.* imports
+from lib.modeling.pipelines.vitdet import build_detector  # must import without detectron2
+from lib.kits.hsmr_demo import imgs_det2patches            # the real consumer of detector output
+
+DEMO = Path('data_inputs/demo/example_imgs')
+DET_THRESHOLD_SCORE = 0.5  # matches hsmr_demo._img_det2patches
+
+
+def main():
+    fns = sorted(p for p in DEMO.glob('*')
+                 if p.suffix.lower() in {'.jpg', '.jpeg', '.png', '.webp'})
+    assert fns, f'no demo images under {DEMO}'
+
+    imgs = []
+    for fn in fns:
+        bgr = cv2.imread(str(fn))
+        assert bgr is not None, f'cv2 failed to read {fn}'
+        imgs.append(cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB))  # HSMR feeds RGB
+
+    dets, ratios = build_detector(device='cpu')(imgs)
+    assert len(dets) == len(imgs) == len(ratios), 'per-image list length mismatch'
+
+    n_person_imgs = 0
+    for fn, img, d, r in zip(fns, imgs, dets, ratios):
+        H, W = img.shape[:2]
+        assert set(d) == {'pred_classes', 'scores', 'pred_boxes'}, f'bad keys {set(d)}'
+        assert d['pred_classes'].dtype == torch.long
+        assert (d['pred_classes'] == 0).all(), 'human class must be 0 (CLASS_HUMAN_ID)'
+        assert d['pred_boxes'].ndim == 2 and d['pred_boxes'].shape[1] == 4, 'boxes must be (N,4)'
+        assert r == 1.0, 'native-resolution detection -> ratio 1.0'
+
+        strong = d['scores'] > DET_THRESHOLD_SCORE
+        if strong.any():
+            n_person_imgs += 1
+            x1, y1, x2, y2 = d['pred_boxes'][strong].unbind(1)
+            assert (x2 > x1).all() and (y2 > y1).all(), 'boxes must be lurb (x2>x1, y2>y1)'
+            assert (x1 >= -1).all() and (y1 >= -1).all() \
+                and (x2 <= W + 1).all() and (y2 <= H + 1).all(), 'box out of frame'
+            print(f'  ✓ {fn.name}: {int(strong.sum())} person(s), '
+                  f'top score {float(d["scores"][strong].max()):.2f}')
+        else:
+            print(f'  · {fn.name}: no confident person')
+
+    # Every demo image has an obvious human; allow one miss for robustness slack.
+    assert n_person_imgs >= len(imgs) - 1, \
+        f'expected a person in ~all demo imgs, got {n_person_imgs}/{len(imgs)}'
+    print(f'DETECTOR OK: person found in {n_person_imgs}/{len(imgs)} demo images.')
+
+    # Close the loop: feed detector output through HSMR's real cropper (SKEL-free) and
+    # confirm it yields the 256x256 patches the HMR forward pass expects.
+    patches, det_meta = imgs_det2patches(imgs, dets, ratios, max_instances_per_img=5)
+    assert patches.ndim == 4 and patches.shape[1:] == (256, 256, 3), f'bad patch shape {patches.shape}'
+    assert patches.shape[0] == sum(det_meta['n_patch_per_img']), 'patch count / meta mismatch'
+    print(f'PATCHES OK: {patches.shape[0]} human patches @ 256x256 — ready for the HMR pass.')
+
+
+if __name__ == '__main__':
+    main()

setup_hsmr.sh

@@ -0,0 +1,87 @@
+#!/bin/bash
+# HSMR pose-engine install on Apple Silicon (MPS) — Path B (local HSMR/SKEL).
+# Clone -> uv venv(3.10) -> deps -> ungated weights -> apply engine_patches/ -> STOP at SKEL gate.
+#
+#   ./setup_hsmr.sh            full install (re-runnable; STOPS at the SKEL download gate)
+#   ./setup_hsmr.sh --patch    re-apply engine_patches/ onto an existing clone and exit
+#
+# engine_patches/ holds our ARM/headless fixes (torchvision detector drop-in, headless
+# pyrender/OpenGL shims, smoke + detector tests). HSMR itself is gitignored under .engine/,
+# so a re-clone would drop those fixes — this script restores them. Override the clone
+# location with HSMR_DIR=... (used by the durability test against a temp clone).
+#
+# ponytail: no `set -e` — optional deps (detectron2/chumpy/render) WARN and we still want the
+# env + weights down so the smoke test can run and any failure is diagnosable.
+set -uo pipefail
+REPO="$(cd "$(dirname "$0")" && pwd)"
+ENGINE="$REPO/.engine"
+HSMR="${HSMR_DIR:-$ENGINE/HSMR}"
+PATCHES="$REPO/engine_patches"
+LOG="$ENGINE/setup.log"
+JOB="$HOME/.jobs/hsmr_setup"
+
+apply_patches(){
+  [ -d "$HSMR" ] || { echo "no HSMR clone at $HSMR — clone first"; exit 1; }
+  [ -d "$PATCHES" ] || { echo "no engine_patches/ at $PATCHES"; exit 1; }
+  ( cd "$PATCHES" && find . -type f ! -name '.DS_Store' ) | sed 's|^\./||' | while read -r f; do
+    mkdir -p "$HSMR/$(dirname "$f")"
+    cp "$PATCHES/$f" "$HSMR/$f"
+    echo "  patched $f"
+  done
+}
+
+# Standalone re-apply (after a manual HSMR re-clone).
+if [ "${1:-}" = "--patch" ]; then
+  echo ">>> applying engine_patches/ -> $HSMR"
+  apply_patches
+  echo "patches applied."
+  exit 0
+fi
+
+mkdir -p "$ENGINE" "$HOME/.jobs"
+exec > >(tee -a "$LOG") 2>&1
+hb(){ echo "$(date +%s) $*" > "$JOB"; echo ">>> [$(date +%H:%M:%S)] $*"; }
+
+hb "clone HSMR + submodules (SKEL)"
+if [ ! -d "$HSMR/.git" ]; then
+  git clone --recurse-submodules https://github.com/IsshikiHugh/HSMR "$HSMR" || { hb "FAIL clone"; exit 1; }
+else
+  git -C "$HSMR" submodule update --init || true
+fi
+cd "$HSMR" || { hb "FAIL cd"; exit 1; }
+
+hb "uv venv python 3.10"
+uv venv --python 3.10 .venv || { hb "FAIL venv"; exit 1; }
+source .venv/bin/activate
+
+hb "install torch + torchvision (MPS wheel)"
+uv pip install torch torchvision || { hb "FAIL torch"; exit 1; }
+
+hb "install requirements.txt"
+uv pip install -r requirements.txt || hb "WARN requirements had issues"
+
+hb "pin numpy<2 (chumpy/detectron2/smplx need 1.x)"
+uv pip install "numpy==1.26.4"
+
+# detectron2/chumpy are NOT needed — engine_patches replaces the detector and the SKEL runtime
+# loads pkls without chumpy (see docs/B_ENGINE_SETUP.md). Left out on purpose; they don't build on ARM.
+
+hb "pip install -e HSMR + thirdparty/SKEL"
+uv pip install -e . || hb "WARN pip -e . failed"
+uv pip install -e thirdparty/SKEL || hb "WARN SKEL editable install failed"
+
+hb "download UNGATED weights from HuggingFace (regressors + HSMR ckpt + ViTPose backbone)"
+hf download IsshikiHugh/HSMR-data_inputs \
+  --include "body_models/SMPL_to_J19.pkl" "body_models/J_regressor_SKEL_mix_MALE.pkl" "body_models/J_regressor_SMPL_MALE.pkl" "released_models/HSMR-ViTH-r1d1.tar.gz" "backbone/vitpose_backbone.pth" \
+  --local-dir data_inputs || hb "WARN hf download issue"
+if [ -f data_inputs/released_models/HSMR-ViTH-r1d1.tar.gz ]; then
+  hb "extract HSMR checkpoint"
+  tar -xzf data_inputs/released_models/HSMR-ViTH-r1d1.tar.gz -C data_inputs/released_models/ && rm -f data_inputs/released_models/HSMR-ViTH-r1d1.tar.gz
+fi
+mkdir -p data_inputs/body_models/skel
+
+hb "apply engine_patches/ (ARM detector, headless shims, tests)"
+apply_patches
+
+hb "DONE-ENV — SKEL gate: John drops skel_models_v1.1/ (v1.1.1) into data_inputs/body_models/skel/"
+echo "=== pip freeze (key) ===" ; uv pip freeze | grep -iE 'torch|numpy|smplx|pytorch-lightning|timm' || true