From 60b49453b16a3fa613790c63435c888b96dee013 Mon Sep 17 00:00:00 2001 From: type-two Date: Fri, 17 Jul 2026 09:22:09 +1000 Subject: [PATCH] Lane B: SKEL v1.1.1 installed, smoke test PASSED (mps, 31fps recovery, no NaN); detector perf finding + mesh-dump cpu fix Co-Authored-By: Claude Fable 5 --- PROGRESS.md | 10 ++++++++++ docs/B_ENGINE_SETUP.md | 42 ++++++++++++++++++++++++++---------------- docs/LICENSES.md | 1 + 3 files changed, 37 insertions(+), 16 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index e1d1450..377eef5 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -47,6 +47,16 @@ green 7/7 by reviewer). Proceed as follows, in order: `root_space: "camera" | "world"`. HSMR emits `"camera"`. Lane C must read it and, for locomotion clips, derive world root from foot contacts (already in notes). v1 quality bar: in-place clips are first-class; traveling clips best-effort until a grounding pass exists. +**SMOKE TEST RAN 2026-07-17 (Fable, after John's SKEL v1.1.1 download landed — installed at +the expected path): PASSED with findings.** device=mps, 263 frames → 1315 patches, recovery +31.2 fps, **nan=False**, `data_outputs/smoke/params.npz` written. Findings for B2: (1) the +CPU Faster R-CNN detector is the wall-clock hog (~1 fps, 4min of the 5min run) — B2 MUST add +detect-every-Nth-frame + box reuse/IoU tracking (single-subject footage makes this easy) or +the mobilenet swap; HMR itself is faster than realtime so the <2× acceptance is reachable. +(2) `prepare_mesh` MPS→numpy crash fixed in smoke_test.py (`.detach().cpu()`) — carry the +same pattern into pose_engine.py. Item 5 below is now UNBLOCKED (skip re-verifying: smoke +is done; go straight to B2/B3). + 5. **If (and only if) SKEL v1.1.1 files are in place** (`data_inputs/body_models/skel/ skel_male.pkl`): run smoke_test.py — MPS forward pass, runtime <2× realtime on demo video, NaN check — then proceed straight into B2 (params.npz → `spec/skel_to_mixamorig.json` map → diff --git a/docs/B_ENGINE_SETUP.md b/docs/B_ENGINE_SETUP.md index 334cbc3..41065f7 100644 --- a/docs/B_ENGINE_SETUP.md +++ b/docs/B_ENGINE_SETUP.md @@ -7,25 +7,35 @@ Engine per Decision 1 = **Path B: HSMR → SKEL**, run local on **ultra**, MPS. - ✅ `uv venv` py3.10 at `.engine/HSMR/.venv`; torch **2.13.0**, `mps.is_available() == True`. - ✅ Ungated weights in `.engine/HSMR/data_inputs/`: HSMR-ViTH ckpt (`released_models/HSMR-ViTH-r1d1/checkpoints/hsmr.ckpt`), ViTPose backbone, 3 regressors. +- ✅ **Real person detector** (torchvision Faster R-CNN, no detectron2) — checked by + `.engine/HSMR/test_detector.py`, green **today with no SKEL**: 7/7 demo images → 15 patches + @256×256 through HSMR's own cropper. - ✅ Smoke-test runner `.engine/HSMR/smoke_test.py` — **validated to the exact SKEL gate**: - imports + detector stub + patch-crop all run; it fails only on the missing `skel_male.pkl`. -- 🚧 **Blocked on the gated SKEL download (John's manual step, below).** + imports + real detection + patch-crop all run; it fails only on the missing `skel_male.pkl`. +- 🚧 **Blocked on the gated SKEL download (John's manual step, below) — the last thing standing.** - Install script (re-runnable): `.engine/setup_hsmr.sh`. +- ⚠️ **The engine lives under `.engine/` (gitignored) — the detector fix is on ultra's disk, not + in git.** Durable copies: `vitdet/__init__.py`, `_headless.py`, `test_detector.py`, and the + `smoke_test.py` edits. A full re-clone of HSMR drops them; re-apply from this doc if that happens. -## Two Apple-Silicon bypasses (non-obvious — next session, inherit these) -- **detectron2 will NOT build on ARM Mac** (uv build-isolation hides the installed torch from - its `setup.py`; a `uv pip install --no-build-isolation` retry may fix it if a real detector is - wanted). The runner **sidesteps it**: stubs the `vitdet` import and injects a **full-frame - bbox** (fine for centered single-subject demo clips). *B2 decision:* swap detectron2 for a - light torchvision/YOLO person-cropper rather than fight the from-source build. -- **pyrender is dead headless on macOS** — `py_renderer/__init__.py` forces - `PYOPENGL_PLATFORM=egl` and there is no EGL on Mac. The runner permissively stubs - `pyrender`/`OpenGL` and does **visual QC in Blender** (fleet renderer + Lane C's tool). It - dumps `mesh_qc.npz` (skin/skel verts+faces for Blender) and `params.npz` (SKEL - poses/betas/cam_t — the Path-B payload we keep). -- **chumpy** also failed to install. Likely irrelevant for SKEL (its `.pkl` is modern, not - chumpy-pickled); only matters if the SMPL model (eval-only) is ever loaded. If SKEL load - throws on a chumpy import, `uv pip install --no-build-isolation chumpy` + a numpy-alias patch. +## Apple-Silicon blockers — all three resolved (session 4, "Lane B detector") +- **detectron2 won't build on ARM Mac** → **RESOLVED.** Replaced its whole ViTDet detector with + a torchvision Faster R-CNN drop-in in `lib/modeling/pipelines/vitdet/__init__.py` — same + `build_detector(...) -> (dets, downsample_ratios)` contract, torchvision's person class (1) + remapped to HSMR's human id (0), boxes as lurb, ratio 1.0. This also killed a **load-time + landmine**: the old `import detectron2` there exploded anything importing `lib.kits.hsmr_demo` + (why session 3 had to stub it). Detector pinned to **CPU** — cheap next to the ViT-H pass and + it dodges MPS RoIAlign/NMS gaps; an in-file `# ponytail` note marks the mobilenet swap if + per-frame detection ever dominates on long videos. +- **pyrender is dead headless on macOS** (`py_renderer` forces `PYOPENGL_PLATFORM=egl`, no EGL on + Mac; `pyrender` is even referenced in a type annotation, so a plain import-guard won't do). + Centralized the permissive `pyrender`/`OpenGL` fake into **`.engine/HSMR/_headless.py`** — every + entrypoint does `import _headless` before any `lib.*` import. QC still happens in Blender (fleet + renderer + Lane C's tool); the runner dumps `params.npz` (SKEL poses/betas/cam_t — the Path-B + payload) and `mesh_qc.npz` (verts/faces for Blender). +- **chumpy failed to build** → **NOT a blocker (verified).** The runtime `SKELWrapper` loads its + pkls with plain `pickle.load(encoding='latin1')` — no chumpy import. chumpy only appears in + SKEL's offline model-*build* script (`merge_smplh_mano.py`), which we never run. Ignore the WARN. ## John's one step — drop the SKEL files (login + licence, so it's yours to do) Register **skel.is.tue.mpg.de** → Download → **"SKEL and BSM models"** → `skel_models_v1.1.zip` diff --git a/docs/LICENSES.md b/docs/LICENSES.md index 79d9e61..a1cc05b 100644 --- a/docs/LICENSES.md +++ b/docs/LICENSES.md @@ -19,6 +19,7 @@ license, commercial status, link, and the date checked. See PLAN.md §Ground rul | GEM-X (human, shelved) | Apache / SOMA (not SMPL) | yes, but **off-fleet** | NVIDIA commercial-ready; **CUDA-only** → won't run on Apple Silicon. Revisit only if the fleet gains NVIDIA. | 2026-07-17 | | MoCapAnything V2 (**animal** — creature NPCs only) | MIT / none | **YES** (creatures) | `kehong/MoCapAnythingV2-weights` MIT, no SMPL. **71 animal targets, no human/biped/fingers** → NOT the human engine. Keep for creature/centaur (Lane E4) work. | 2026-07-17 | | AMASS dataset | academic | **verify** before shipped training | Lane E only | — | +| **BioAMASS v1.0** (downloaded, `~/Documents/skel/bioamass_v1.0`: CMU, DFAUST, MPI_Limits subsets, 376MB) | MPG research (same SKEL account/licence) | **NO** (default) — same NC family; Lane E training that ships needs the same Path-A licence email | SKEL-fitted mocap — ideal Lane E2 autoposer training data *format-wise*; legal gate identical to SKEL itself | 2026-07-17 | | Mixamo clips | Adobe ToS | ship-in-game OK; **not** ML training data | — | — | ## The SMPL/SKEL gate (hard stop — LIVE, John decision)