From f8037f7b8564a772ae06fb0e9e36795c7d43b717 Mon Sep 17 00:00:00 2001 From: type-two Date: Fri, 17 Jul 2026 09:34:39 +1000 Subject: [PATCH] Lane B / session-5 item 1: durable-ize the ARM engine fixes into git The torchvision detector drop-in, headless pyrender/OpenGL shims, and the detector+smoke tests existed only on ultra's disk inside gitignored .engine/. Captured byte-identical copies into engine_patches/ (mirrors the HSMR subtree) and moved setup_hsmr.sh to the repo root (tracked, path-relative) with a --patch step that overlays them onto the clone. Acceptance PASSED: fresh shallow clone of HSMR in a temp dir + './setup_hsmr.sh --patch' + validated venv -> test_detector.py green (7/7 demo imgs, 15 patches), no hand edits. A re-clone can no longer silently drop the ARM fixes. Co-Authored-By: Claude Opus 4.8 --- docs/B_ENGINE_SETUP.md | 16 +-- engine_patches/_headless.py | 34 ++++++ .../lib/modeling/pipelines/vitdet/__init__.py | 75 +++++++++++++ engine_patches/smoke_test.py | 100 ++++++++++++++++++ engine_patches/test_detector.py | 71 +++++++++++++ setup_hsmr.sh | 87 +++++++++++++++ 6 files changed, 376 insertions(+), 7 deletions(-) create mode 100644 engine_patches/_headless.py create mode 100644 engine_patches/lib/modeling/pipelines/vitdet/__init__.py create mode 100644 engine_patches/smoke_test.py create mode 100644 engine_patches/test_detector.py create mode 100755 setup_hsmr.sh diff --git a/docs/B_ENGINE_SETUP.md b/docs/B_ENGINE_SETUP.md index 41065f7..62a41d1 100644 --- a/docs/B_ENGINE_SETUP.md +++ b/docs/B_ENGINE_SETUP.md @@ -10,13 +10,15 @@ Engine per Decision 1 = **Path B: HSMR → SKEL**, run local on **ultra**, MPS. - ✅ **Real person detector** (torchvision Faster R-CNN, no detectron2) — checked by `.engine/HSMR/test_detector.py`, green **today with no SKEL**: 7/7 demo images → 15 patches @256×256 through HSMR's own cropper. -- ✅ Smoke-test runner `.engine/HSMR/smoke_test.py` — **validated to the exact SKEL gate**: - imports + real detection + patch-crop all run; it fails only on the missing `skel_male.pkl`. -- 🚧 **Blocked on the gated SKEL download (John's manual step, below) — the last thing standing.** -- Install script (re-runnable): `.engine/setup_hsmr.sh`. -- ⚠️ **The engine lives under `.engine/` (gitignored) — the detector fix is on ultra's disk, not - in git.** Durable copies: `vitdet/__init__.py`, `_headless.py`, `test_detector.py`, and the - `smoke_test.py` edits. A full re-clone of HSMR drops them; re-apply from this doc if that happens. +- ✅ Smoke-test runner `.engine/HSMR/smoke_test.py` — **PASSED 2026-07-17** (session 5): SKEL + v1.1.1 in place, device=mps, 263 frames → 1315 patches, recovery 31.2 fps, **nan=False**, + `data_outputs/smoke/params.npz` written (poses (N,46), betas (N,10), cam_t (N,3)). +- ✅ Install script (re-runnable): **`./setup_hsmr.sh`** at the repo root (tracked in git). + `./setup_hsmr.sh --patch` re-applies the ARM/headless fixes onto an existing/re-cloned HSMR. +- ✅ **The ARM fixes are now durable in git** (session 5, was: "on ultra's disk only"). Canonical + copies live in **`engine_patches/`** (`_headless.py`, `test_detector.py`, `smoke_test.py`, + `lib/modeling/pipelines/vitdet/__init__.py`); `setup_hsmr.sh` overlays them onto the gitignored + `.engine/HSMR` clone. Verified: fresh clone + `--patch` → `test_detector.py` green, no hand edits. ## Apple-Silicon blockers — all three resolved (session 4, "Lane B detector") - **detectron2 won't build on ARM Mac** → **RESOLVED.** Replaced its whole ViTDet detector with diff --git a/engine_patches/_headless.py b/engine_patches/_headless.py new file mode 100644 index 0000000..23c3faa --- /dev/null +++ b/engine_patches/_headless.py @@ -0,0 +1,34 @@ +"""Headless-macOS import shims for the HSMR engine (our fleet is all Apple Silicon). + +pyrender -> PyOpenGL -> EGL cannot load without a GL context, yet several vendored modules +`import pyrender` at load time — some even in type annotations (`List[pyrender.Node]`) — so +the import chain explodes before any of our code runs. We never render in-process; Blender +is the fleet renderer (Lane C). So we drop permissive fakes for the pyrender + OpenGL stack +into sys.modules, letting the chain import while keeping everything else (ColorPalette, +Wis3D, the models) real. + +Import this BEFORE the first `lib.*` import: `import _headless # noqa: F401` + +# ponytail: fakes, not a real headless GL backend (osmesa/egl). We don't render here, so a +# working GL context would be dead weight. Wire a real one only if in-process render is ever +# needed off-fleet. +""" +import sys +import types + + +class _Any: + def __call__(self, *a, **k): return self + def __getattr__(self, n): return self + + +class _AnyModule(types.ModuleType): + def __getattr__(self, n): + if n.startswith('__') and n.endswith('__'): + raise AttributeError(n) + return _Any() + + +# setdefault: never clobber a real module that a caller already imported on purpose. +for _n in ('pyrender', 'OpenGL', 'OpenGL.GL', 'OpenGL.error', 'OpenGL.platform'): + sys.modules.setdefault(_n, _AnyModule(_n)) diff --git a/engine_patches/lib/modeling/pipelines/vitdet/__init__.py b/engine_patches/lib/modeling/pipelines/vitdet/__init__.py new file mode 100644 index 0000000..68b76eb --- /dev/null +++ b/engine_patches/lib/modeling/pipelines/vitdet/__init__.py @@ -0,0 +1,75 @@ +""" +Person detector — torchvision drop-in replacement for HSMR's detectron2/ViTDet. + +Why: detectron2 + chumpy won't build from source on Apple Silicon (the Lane B blocker), +and this module's `import detectron2` was a load-time landmine for everything that touches +`lib.kits.hsmr_demo` (smoke_test.py, the future pose_engine.py). torchvision ships a +COCO-pretrained Faster R-CNN with ungated weights and no build step. + +Same public contract as the original, so run_demo.py / smoke_test.py / pose_engine.py +import it unchanged: + + build_detector(...) -> callable(raw_imgs) -> (dets, downsample_ratios) + + dets[i] : dict of CPU tensors (empty tensors if no person) — + pred_classes : all 0 (== hsmr_demo CLASS_HUMAN_ID) + scores : float + pred_boxes : (N, 4) left-upper-right-bottom pixels + downsample_ratios : one float per image; boxes are in native-resolution pixels -> 1.0 + +Contract is checked by test_detector.py (runs today, no SKEL needed). + +# ponytail: ~35 lines replaces the whole detectron2 stack. The ViT-H HMR forward pass is +# the runtime cost, not detection, so we don't chase detector throughput here. +""" +import numpy as np +import torch +from tqdm import tqdm +from torchvision.models.detection import ( + fasterrcnn_resnet50_fpn_v2, + FasterRCNN_ResNet50_FPN_V2_Weights, +) + +_COCO_PERSON = 1 # torchvision COCO label id for "person" (0 == background) + + +class _PersonDetector: + """Callable over a list of RGB HxWx3 images (what HSMR's load_inputs yields). + uint8 or float, [0,255] or [0,1] are all handled.""" + + def __init__(self, device='cpu', score_thresh=0.25): + # score_thresh 0.25 mirrors the old detectron2 test_score_thresh; hsmr_demo's + # _img_det2patches re-filters at 0.5, so this only widens recall a little. + weights = FasterRCNN_ResNet50_FPN_V2_Weights.DEFAULT + self.model = fasterrcnn_resnet50_fpn_v2( + weights=weights, box_score_thresh=score_thresh, + ).eval().to(device) + self.device = device + + @torch.no_grad() + def __call__(self, raw_imgs): + dets, ratios = [], [] + for img in tqdm(raw_imgs, desc='Detecting'): + t = torch.as_tensor(np.ascontiguousarray(img), device=self.device).float() + if float(t.max()) > 1.5: # [0,255] -> [0,1] + t = t / 255.0 + t = t.permute(2, 0, 1) # HWC -> CHW + out = self.model([t])[0] + keep = out['labels'] == _COCO_PERSON + dets.append({ + 'pred_classes': torch.zeros(int(keep.sum()), dtype=torch.long), # human -> 0 + 'scores' : out['scores'][keep].cpu(), + 'pred_boxes' : out['boxes'][keep].cpu(), # xyxy == left-upper-right-bottom + }) + ratios.append(1.0) # detected at native resolution + return dets, ratios + + +def build_detector(batch_size=1, max_img_size=512, device='cpu'): + """Signature-compatible with the old detectron2 builder. batch_size / max_img_size are + accepted for parity but unused — detection runs per-frame at native resolution. + + # ponytail: detector pinned to CPU regardless of `device`. It's cheap next to the + # ViT-H HMR pass, and CPU sidesteps the MPS coverage gaps in RoIAlign/NMS. Flip to + # `device` only if per-frame detection ever dominates on long videos.""" + return _PersonDetector(device='cpu') diff --git a/engine_patches/smoke_test.py b/engine_patches/smoke_test.py new file mode 100644 index 0000000..3483445 --- /dev/null +++ b/engine_patches/smoke_test.py @@ -0,0 +1,100 @@ +#!/usr/bin/env python +""" +HSMR smoke test on Apple Silicon (MPS) — Path B engine verification. + +Answers the Lane B smoke-test questions on HSMR's bundled demo clips (no John footage, +no SKEL license needed beyond the model files John downloaded): + 1. Does the HSMR forward run on MPS? (device + CPU-fallback) + 2. Timing (target < 2x realtime). + 3. Pose sanity (shape, NaNs) + an overlay render to eyeball quality. + +Detection uses the torchvision person detector (lib.modeling.pipelines.vitdet, the +detectron2-free drop-in). The one remaining gate is SKEL: build_inference_pipeline +instantiates the SKEL body model, so this completes only once John's SKEL files are in place. + +Run (once SKEL files are in data_inputs/body_models/skel/): + cd ~/Documents/MOCAPGOD/.engine/HSMR && source .venv/bin/activate + PYTHONPATH=. python smoke_test.py -i data_inputs/demo/example_videos/gymnasts.mp4 + PYTHONPATH=. python smoke_test.py -i data_inputs/demo/example_imgs # image folder +""" +import os, time, argparse +os.environ.setdefault('PYTORCH_ENABLE_MPS_FALLBACK', '1') # some ops lack MPS kernels + +import _headless # noqa: F401 fake pyrender/OpenGL for headless macOS — MUST precede lib.* imports + +# One import line: every helper is an attribute of hsmr_demo (via its star-imports). +from lib.kits.hsmr_demo import ( + load_inputs, imgs_det2patches, build_inference_pipeline, prepare_mesh, + visualize_full_img, IMG_MEAN_255, IMG_STD_255, asb, assemble_dict, + get_logger, save_video, save_img, np, torch, Path, DEFAULT_HSMR_ROOT, + build_detector, +) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument('-i', '--input_path', required=True) + ap.add_argument('-o', '--output_path', default='data_outputs/smoke') + ap.add_argument('-d', '--device', default='mps') + ap.add_argument('-m', '--model_root', default=None) + ap.add_argument('--rec_bs', type=int, default=64) + ap.add_argument('--ignore_skel', action='store_true') + args = ap.parse_args() + + log = get_logger(brief=True) + model_root = Path(args.model_root) if args.model_root else DEFAULT_HSMR_ROOT + out = Path(args.output_path); out.mkdir(parents=True, exist_ok=True) + + # Inputs (reuse the demo loader; it only reads .input_path / .input_type). + fake = argparse.Namespace(input_path=args.input_path, input_type='auto') + raw_imgs, meta = load_inputs(fake) + log.info(f'device={args.device} type={meta["type"]} frames={len(raw_imgs)}') + + # Detection -> patches (real torchvision person detector, on CPU). + dets, ratios = build_detector(device=args.device)(raw_imgs) + patches, det_meta = imgs_det2patches(raw_imgs, dets, ratios, 5) + log.info(f'patches={len(patches)}') + + # Build pipeline — instantiates SKEL, so this is where a missing SKEL file fails. + pipe = build_inference_pipeline(model_root=model_root, device=args.device) + + # Recovery loop (mirrors run_demo), timed. + t0 = time.time() + pd_params, pd_cam_t = [], [] + for bw in asb(total=len(patches), bs_scope=args.rec_bs, enable_tqdm=True): + p = patches[bw.sid:bw.eid] + pn = ((p - IMG_MEAN_255) / IMG_STD_255).transpose(0, 3, 1, 2) + with torch.no_grad(): + o = pipe(pn) + pd_params.append({k: v.detach().cpu().clone() for k, v in o['pd_params'].items()}) + pd_cam_t.append(o['pd_cam_t'].detach().cpu().clone()) + dt = time.time() - t0 + pd_params = assemble_dict(pd_params, expand_dim=False) + pd_cam_t = torch.cat(pd_cam_t, dim=0) + + poses = pd_params['poses'] + fps = len(patches) / dt if dt else 0 + log.info(f'RECOVERY ok: {len(patches)} patches in {dt:.2f}s ({fps:.2f} fps) ' + f'poses={tuple(poses.shape)} nan={bool(torch.isnan(poses).any())}') + + # Dump raw params regardless (this is the Path-B payload: SKEL rotations). + np.savez(out / 'params.npz', poses=poses.numpy(), + betas=pd_params['betas'].numpy(), cam_t=pd_cam_t.numpy()) + log.info(f'params -> {out/"params.npz"}') + + # Mesh geometry for visual QC in Blender (pyrender/EGL is dead headless on macOS, and + # Blender is the fleet renderer + Lane C's tool). prepare_mesh uses skel_model, no GL. + try: + m_skin, m_skel = prepare_mesh(pipe, pd_params) + np.savez(out / 'mesh_qc.npz', + skin_v=m_skin['v'].detach().cpu().numpy(), skin_f=np.asarray(m_skin['f']), + skel_v=m_skel['v'].detach().cpu().numpy(), skel_f=np.asarray(m_skel['f'])) + log.info(f'mesh geometry -> {out/"mesh_qc.npz"} (import in Blender to eyeball)') + except Exception as e: + log.warning(f'prepare_mesh failed (params still saved): {e!r}') + + log.info('smoke test complete.') + + +if __name__ == '__main__': + main() diff --git a/engine_patches/test_detector.py b/engine_patches/test_detector.py new file mode 100644 index 0000000..a3119c0 --- /dev/null +++ b/engine_patches/test_detector.py @@ -0,0 +1,71 @@ +#!/usr/bin/env python +"""Detector smoke check — runs TODAY, no SKEL, no HMR forward pass. + +Proves the torchvision person detector (lib/modeling/pipelines/vitdet) imports without +detectron2 and honors the contract lib.kits.hsmr_demo._img_det2patches depends on. + + cd ~/Documents/MOCAPGOD/.engine/HSMR && source .venv/bin/activate + PYTHONPATH=. python test_detector.py +""" +from pathlib import Path +import cv2 +import torch + +import _headless # noqa: F401 fake pyrender/OpenGL for headless macOS — MUST precede lib.* imports +from lib.modeling.pipelines.vitdet import build_detector # must import without detectron2 +from lib.kits.hsmr_demo import imgs_det2patches # the real consumer of detector output + +DEMO = Path('data_inputs/demo/example_imgs') +DET_THRESHOLD_SCORE = 0.5 # matches hsmr_demo._img_det2patches + + +def main(): + fns = sorted(p for p in DEMO.glob('*') + if p.suffix.lower() in {'.jpg', '.jpeg', '.png', '.webp'}) + assert fns, f'no demo images under {DEMO}' + + imgs = [] + for fn in fns: + bgr = cv2.imread(str(fn)) + assert bgr is not None, f'cv2 failed to read {fn}' + imgs.append(cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)) # HSMR feeds RGB + + dets, ratios = build_detector(device='cpu')(imgs) + assert len(dets) == len(imgs) == len(ratios), 'per-image list length mismatch' + + n_person_imgs = 0 + for fn, img, d, r in zip(fns, imgs, dets, ratios): + H, W = img.shape[:2] + assert set(d) == {'pred_classes', 'scores', 'pred_boxes'}, f'bad keys {set(d)}' + assert d['pred_classes'].dtype == torch.long + assert (d['pred_classes'] == 0).all(), 'human class must be 0 (CLASS_HUMAN_ID)' + assert d['pred_boxes'].ndim == 2 and d['pred_boxes'].shape[1] == 4, 'boxes must be (N,4)' + assert r == 1.0, 'native-resolution detection -> ratio 1.0' + + strong = d['scores'] > DET_THRESHOLD_SCORE + if strong.any(): + n_person_imgs += 1 + x1, y1, x2, y2 = d['pred_boxes'][strong].unbind(1) + assert (x2 > x1).all() and (y2 > y1).all(), 'boxes must be lurb (x2>x1, y2>y1)' + assert (x1 >= -1).all() and (y1 >= -1).all() \ + and (x2 <= W + 1).all() and (y2 <= H + 1).all(), 'box out of frame' + print(f' ✓ {fn.name}: {int(strong.sum())} person(s), ' + f'top score {float(d["scores"][strong].max()):.2f}') + else: + print(f' · {fn.name}: no confident person') + + # Every demo image has an obvious human; allow one miss for robustness slack. + assert n_person_imgs >= len(imgs) - 1, \ + f'expected a person in ~all demo imgs, got {n_person_imgs}/{len(imgs)}' + print(f'DETECTOR OK: person found in {n_person_imgs}/{len(imgs)} demo images.') + + # Close the loop: feed detector output through HSMR's real cropper (SKEL-free) and + # confirm it yields the 256x256 patches the HMR forward pass expects. + patches, det_meta = imgs_det2patches(imgs, dets, ratios, max_instances_per_img=5) + assert patches.ndim == 4 and patches.shape[1:] == (256, 256, 3), f'bad patch shape {patches.shape}' + assert patches.shape[0] == sum(det_meta['n_patch_per_img']), 'patch count / meta mismatch' + print(f'PATCHES OK: {patches.shape[0]} human patches @ 256x256 — ready for the HMR pass.') + + +if __name__ == '__main__': + main() diff --git a/setup_hsmr.sh b/setup_hsmr.sh new file mode 100755 index 0000000..ce779e3 --- /dev/null +++ b/setup_hsmr.sh @@ -0,0 +1,87 @@ +#!/bin/bash +# HSMR pose-engine install on Apple Silicon (MPS) — Path B (local HSMR/SKEL). +# Clone -> uv venv(3.10) -> deps -> ungated weights -> apply engine_patches/ -> STOP at SKEL gate. +# +# ./setup_hsmr.sh full install (re-runnable; STOPS at the SKEL download gate) +# ./setup_hsmr.sh --patch re-apply engine_patches/ onto an existing clone and exit +# +# engine_patches/ holds our ARM/headless fixes (torchvision detector drop-in, headless +# pyrender/OpenGL shims, smoke + detector tests). HSMR itself is gitignored under .engine/, +# so a re-clone would drop those fixes — this script restores them. Override the clone +# location with HSMR_DIR=... (used by the durability test against a temp clone). +# +# ponytail: no `set -e` — optional deps (detectron2/chumpy/render) WARN and we still want the +# env + weights down so the smoke test can run and any failure is diagnosable. +set -uo pipefail +REPO="$(cd "$(dirname "$0")" && pwd)" +ENGINE="$REPO/.engine" +HSMR="${HSMR_DIR:-$ENGINE/HSMR}" +PATCHES="$REPO/engine_patches" +LOG="$ENGINE/setup.log" +JOB="$HOME/.jobs/hsmr_setup" + +apply_patches(){ + [ -d "$HSMR" ] || { echo "no HSMR clone at $HSMR — clone first"; exit 1; } + [ -d "$PATCHES" ] || { echo "no engine_patches/ at $PATCHES"; exit 1; } + ( cd "$PATCHES" && find . -type f ! -name '.DS_Store' ) | sed 's|^\./||' | while read -r f; do + mkdir -p "$HSMR/$(dirname "$f")" + cp "$PATCHES/$f" "$HSMR/$f" + echo " patched $f" + done +} + +# Standalone re-apply (after a manual HSMR re-clone). +if [ "${1:-}" = "--patch" ]; then + echo ">>> applying engine_patches/ -> $HSMR" + apply_patches + echo "patches applied." + exit 0 +fi + +mkdir -p "$ENGINE" "$HOME/.jobs" +exec > >(tee -a "$LOG") 2>&1 +hb(){ echo "$(date +%s) $*" > "$JOB"; echo ">>> [$(date +%H:%M:%S)] $*"; } + +hb "clone HSMR + submodules (SKEL)" +if [ ! -d "$HSMR/.git" ]; then + git clone --recurse-submodules https://github.com/IsshikiHugh/HSMR "$HSMR" || { hb "FAIL clone"; exit 1; } +else + git -C "$HSMR" submodule update --init || true +fi +cd "$HSMR" || { hb "FAIL cd"; exit 1; } + +hb "uv venv python 3.10" +uv venv --python 3.10 .venv || { hb "FAIL venv"; exit 1; } +source .venv/bin/activate + +hb "install torch + torchvision (MPS wheel)" +uv pip install torch torchvision || { hb "FAIL torch"; exit 1; } + +hb "install requirements.txt" +uv pip install -r requirements.txt || hb "WARN requirements had issues" + +hb "pin numpy<2 (chumpy/detectron2/smplx need 1.x)" +uv pip install "numpy==1.26.4" + +# detectron2/chumpy are NOT needed — engine_patches replaces the detector and the SKEL runtime +# loads pkls without chumpy (see docs/B_ENGINE_SETUP.md). Left out on purpose; they don't build on ARM. + +hb "pip install -e HSMR + thirdparty/SKEL" +uv pip install -e . || hb "WARN pip -e . failed" +uv pip install -e thirdparty/SKEL || hb "WARN SKEL editable install failed" + +hb "download UNGATED weights from HuggingFace (regressors + HSMR ckpt + ViTPose backbone)" +hf download IsshikiHugh/HSMR-data_inputs \ + --include "body_models/SMPL_to_J19.pkl" "body_models/J_regressor_SKEL_mix_MALE.pkl" "body_models/J_regressor_SMPL_MALE.pkl" "released_models/HSMR-ViTH-r1d1.tar.gz" "backbone/vitpose_backbone.pth" \ + --local-dir data_inputs || hb "WARN hf download issue" +if [ -f data_inputs/released_models/HSMR-ViTH-r1d1.tar.gz ]; then + hb "extract HSMR checkpoint" + tar -xzf data_inputs/released_models/HSMR-ViTH-r1d1.tar.gz -C data_inputs/released_models/ && rm -f data_inputs/released_models/HSMR-ViTH-r1d1.tar.gz +fi +mkdir -p data_inputs/body_models/skel + +hb "apply engine_patches/ (ARM detector, headless shims, tests)" +apply_patches + +hb "DONE-ENV — SKEL gate: John drops skel_models_v1.1/ (v1.1.1) into data_inputs/body_models/skel/" +echo "=== pip freeze (key) ===" ; uv pip freeze | grep -iE 'torch|numpy|smplx|pytorch-lightning|timm' || true