Compare commits

..

11 Commits

Author SHA1 Message Date
type-two
35403f8f05 Lane B: tick Lane B done, session-5 completion summary + Lane C handoff
PROGRESS.md: Lane B checked off with the session-5 outcome; SESSION 5 DONE block
(all 6 orders) + explicit Lane C handoff (BVH axes, skel_to_mixamorig map, camera
root_space, --from-params). B_ENGINE_SETUP.md: pose_engine.py usage + the engine
facts learned (SKEL 46-q, rest-aware FK ~4cm, Y-down camera fix, mobilenet detector).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 10:42:28 +10:00
type-two
7d0a3b7f7c Lane B / session-5 item 5: pose_engine.py — SKEL BVH + sidecar (B2) + jitter (B3)
pose_engine.py: queue video -> HSMR/SKEL -> BVH@fps + sidecar JSON per the treaty.
- detection: torchvision mobilenet (engine_patches), every-Nth frame + single best-box
  reuse (single-subject) -> ~1 pose/frame aligned timeline. mobilenet swap + detect-every
  address the CPU-detector wall-clock finding.
- recovery: HSMR ViT-H on MPS -> SKEL params (poses q46, betas, cam_t).
- SKEL forward -> per-joint global orientations; rigid-bone BVH via rest-aware FK
  (offset[j]=rest_J[j]-rest_J[parent]; W[j]=Rg[j]@rest_R[j]^T; local Q=W[parent]^T@W[j]).
  Verified: rest-aware FK reconstructs SKEL joints to ~4cm (residual = SKEL biomechanical
  coupled translations, which rigid BVH can't carry), in-BVH FK self-consistency 0.0mm.
- camera->world axis fix: HSMR emits Y-down (OpenCV, verified head.y<pelvis.y); diag(1,-1,-1)
  -> Y-up, faces -Z (treaty). Blender import: 24 bones, upright, 1.59m, motion present.
- B3 jitter: savgol on hemisphere-aligned quaternions (rotations) + root xyz separately.
- sidecar: root_space=camera (per-frame HSMR), joint_order, root_trajectory, model, fps.
- build models once + per-video try/except: queue of 6 drained unattended, 2 bad clips
  (no-person, garbage) failed gracefully, 4 produced valid BVH; heartbeat per clip.
- runtime steady-state ~1.6x realtime on ultra (detect+recover), under the <2x bar.
- --selfcheck (engine-free): BVH FK exact + Euler round-trip exact.

spec/skel_to_mixamorig.json: SKEL-24 -> mixamorig name map for Lane C's retarget
(20 mapped, 4 null twist/heel; validated against both skeletons).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 10:41:08 +10:00
type-two
0e2f7d0c12 Lane B / session-5 item 4: treaty amendment — required root_space sidecar field
motion.json sidecar gains REQUIRED root_space in {camera, world}. HSMR is
per-frame -> emits camera; Lane C reads it and derives a world root from foot
contacts for locomotion clips. PLAN treaty + Lane C task updated. v1 quality bar:
in-place clips first-class, traveling best-effort until a grounding pass exists.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 10:02:53 +10:00
type-two
0cf673aa82 Lane B / session-5 item 3: default ingest to local queue/ on ultra
The pose engine (HSMR) runs on ultra, so nothing was consuming the m3ultra queue
ingest.py shipped to. ingest.py now delivers to a local queue/ dir by default;
--remote-queue [user@host:/path] restores the m3ultra rsync path for when a real
throughput need appears (ponytail marker + PLAN Ground-rule-1 deviation noted).
CAPTURE.md updated. Verified: inbox drop -> local queue/, ledger row, selfcheck green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 10:02:16 +10:00
type-two
9db4fec988 Lane B / session-5 item 2: draft SKEL Path-A commercial-license email
Ready-to-send draft in docs/DECISIONS.md for John to send to ps-license@tue.mpg.de
(indie AU dev, in-pipeline use, SKEL files not redistributed — only retargeted
motion ships). Path B continues; a grant converts B->A with zero pipeline rework.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 09:35:39 +10:00
type-two
f8037f7b85 Lane B / session-5 item 1: durable-ize the ARM engine fixes into git
The torchvision detector drop-in, headless pyrender/OpenGL shims, and the
detector+smoke tests existed only on ultra's disk inside gitignored .engine/.
Captured byte-identical copies into engine_patches/ (mirrors the HSMR subtree)
and moved setup_hsmr.sh to the repo root (tracked, path-relative) with a
--patch step that overlays them onto the clone.

Acceptance PASSED: fresh shallow clone of HSMR in a temp dir + './setup_hsmr.sh
--patch' + validated venv -> test_detector.py green (7/7 demo imgs, 15 patches),
no hand edits. A re-clone can no longer silently drop the ARM fixes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 09:34:39 +10:00
type-two
60b49453b1 Lane B: SKEL v1.1.1 installed, smoke test PASSED (mps, 31fps recovery, no NaN); detector perf finding + mesh-dump cpu fix
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 09:22:09 +10:00
type-two
2dbd99579b Review: session-5 orders from Fable review of lanes 0-B
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 08:48:34 +10:00
type-two
4cc67b90fa Lane B: resolve decision gate -> Path B (local HSMR/SKEL, accept NC risk)
B1's MoCapAnything pick reversed (animal model, wrong for human capture);
license analysis recorded in LICENSES.md. John chose Path B: run HSMR->SKEL
locally on ultra (MPS), ship rig-motion only, accept the non-commercial
licence risk as interim until a commercial SKEL licence request goes out
(Path A, the clean upgrade).

Engine installed under .engine/ (gitignored): uv venv py3.10, torch 2.13
MPS-green, ungated weights, smoke_test.py with Apple-Silicon bypasses
(detectron2 stubbed with full-frame bbox, headless pyrender stubbed -> QC in
Blender) -- see docs/B_ENGINE_SETUP.md.

Remaining before B ships: John's gated SKEL v1.1.1 download, then the
forward-pass smoke test (MPS end-to-end, timing, pose sanity). Known caveat:
HSMR is per-frame so root is camera-space cam_t, NOT world-grounded -> Lane C
derives root from foot-contact.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 02:25:55 +10:00
type-two
fefb1f71ff Lane B: free m3ultra disk — deleted GLM-4.6-GGUF (147G) HF cache
Confirmed prune. Space pinned by OS-update snapshots (macOS 26.5.2
staged); realizes on next reboot. Updated DECISIONS.md Decision 2.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 01:29:08 +10:00
type-two
7819e12a5a Lane B / B1: pose-model bake-off + SMPL decision gate
Fresh SOTA survey (per PLAN — distrust the snapshot). Findings:
- MoCapAnything V2 (MIT code+weights, no SMPL, BVH-direct onto arbitrary
  rig) is the pick — sidesteps the SMPL commercial-license hard-stop and
  collapses the SMPL->mixamorig mapper + most of Lane C retarget.
- GVHMR: best accuracy but non-commercial code license -> reference only.
- GEM-X: commercial-clean + perfect output but CUDA-only -> off-fleet.

Stopped at the John-decision gate (docs/DECISIONS.md): confirm engine,
clear m3ultra disk (97% full), supply test footage. Inference eyeball,
B2 service-ize, B3 jitter remain.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 01:10:26 +10:00
16 changed files with 1369 additions and 43 deletions

1
.gitignore vendored
View File

@ -1,5 +1,6 @@
# weights + runtime data stay out of git # weights + runtime data stay out of git
weights/ weights/
.engine/
capture/ capture/
queue/ queue/
out/ out/

18
PLAN.md
View File

@ -18,6 +18,10 @@ critical path. Lanes EF are follow-ons. Do NOT start E/F before D ships.
training. Python via conda/uv there. MODELBEAST lives here; mocap becomes a sibling training. Python via conda/uv there. MODELBEAST lives here; mocap becomes a sibling
service, same patterns (job dirs, batch keys — see `MESHGOD/scripts/mb_*` for the service, same patterns (job dirs, batch keys — see `MESHGOD/scripts/mb_*` for the
client convention). client convention).
- **v1 deviation (session 5):** HSMR is installed and MPS-verified on **ultra**, not
m3ultra, so v1 runs inference on ultra and `ingest.py` defaults to a **local** `queue/`.
`# ponytail: v1 inference on ultra; migrate to m3ultra when a real throughput need appears`
(`ingest.py --remote-queue` already ships to m3ultra when that day comes).
- **ultra** (this Mac, M1 Ultra 128GB) — GLUE + DCC. Blender headless retarget/cleanup/ - **ultra** (this Mac, M1 Ultra 128GB) — GLUE + DCC. Blender headless retarget/cleanup/
export, orchestration, Unreal verification. Owns the canonical repo. export, orchestration, Unreal verification. Owns the canonical repo.
- **JING5** (M5 MBP) — capture companion + portable inference later. Not in v1 critical path. - **JING5** (M5 MBP) — capture companion + portable inference later. Not in v1 critical path.
@ -47,8 +51,11 @@ critical path. Lanes EF are follow-ons. Do NOT start E/F before D ships.
- Skeleton: **mixamorig 65-joint** as defined by `~/Documents/character_kit` - Skeleton: **mixamorig 65-joint** as defined by `~/Documents/character_kit`
(`female_game.glb` / male base). Canonical bone list generated in Lane 0. (`female_game.glb` / male base). Canonical bone list generated in Lane 0.
- Intermediate motion format: **BVH at 30fps** + sidecar JSON - Intermediate motion format: **BVH at 30fps** + sidecar JSON
(`{fps, joint_order, root_trajectory, source_video, model, version}`). (`{fps, joint_order, root_space, root_trajectory, source_video, model, version}`).
BVH because Blender imports it natively and it's human-diffable. **`root_space` ∈ {`"camera"`, `"world"`} is REQUIRED** (session-5 amendment): HSMR is a
per-frame recoverer so it emits `"camera"`; Lane C reads this and, for locomotion clips,
derives a world root from foot contacts. BVH because Blender imports it natively and it's
human-diffable.
- Units metres, Y-up, character faces Z (three.js convention; Unreal yaw-fix happens - Units metres, Y-up, character faces Z (three.js convention; Unreal yaw-fix happens
at UE import, as established in the A_3D_STORE experiments). at UE import, as established in the A_3D_STORE experiments).
- Final artifact: FBX (anim-only, Mixamo-style "Without Skin") dropped into - Final artifact: FBX (anim-only, Mixamo-style "Without Skin") dropped into
@ -153,8 +160,11 @@ Tasks:
`mixamorig_` variants — this bit us before), bake to keyframes. `mixamorig_` variants — this bit us before), bake to keyframes.
- Foot-contact fix: detect frames where foot velocity ≈ 0 & height < threshold IK - Foot-contact fix: detect frames where foot velocity ≈ 0 & height < threshold IK
pin. `# ponytail: threshold heuristic, upgrade to learned contact detection in Lane E if it annoys`. pin. `# ponytail: threshold heuristic, upgrade to learned contact detection in Lane E if it annoys`.
- Trim the T-pose bookends (Lane A protocol guarantees them), root-motion option - Trim the T-pose bookends (Lane A protocol guarantees them). Root motion is driven by the
(in-place vs traveling) as a flag. sidecar **`root_space`** (session-5 treaty): `"camera"` (HSMR) → in-place by default, or
derive a world root from foot contacts for traveling clips; a `--in-place`/`--traveling`
flag overrides. v1 quality bar: **in-place clips are first-class; traveling is best-effort**
until a proper grounding pass exists.
- Export anim-only FBX (Mixamo "Without Skin" style — merge_anims.py already eats these). - Export anim-only FBX (Mixamo "Without Skin" style — merge_anims.py already eats these).
2. QC render: 3-second turntable mp4 of the clip on the female_game body, written next 2. QC render: 3-second turntable mp4 of the clip on the female_game body, written next
to the FBX. **Verify with own eyes** (the rule) — every clip gets its mp4 looked at to the FBX. **Verify with own eyes** (the rule) — every clip gets its mp4 looked at

View File

@ -6,13 +6,124 @@ One lane per session. Tick on completion. See PLAN.md for acceptance checks.
spec/mixamorig_bones.json (65 bones, `mixamorig:` colon-prefixed). — 2026-07-17 spec/mixamorig_bones.json (65 bones, `mixamorig:` colon-prefixed). — 2026-07-17
- [x] **Lane A** — capture + ingest. docs/CAPTURE.md protocol, ingest.py - [x] **Lane A** — capture + ingest. docs/CAPTURE.md protocol, ingest.py
(probe → 30fps normalize → rename → rsync to m3ultra queue → LEDGER.tsv). — 2026-07-17 (probe → 30fps normalize → rename → rsync to m3ultra queue → LEDGER.tsv). — 2026-07-17
- [ ] **Lane B** — pose engine (m3ultra). Model bake-off, service-ize, jitter pass. - [x] **Lane B** — pose engine. **DONE session 5 (2026-07-17).** `pose_engine.py` ships:
queue video → HSMR/SKEL → SKEL-24 BVH@fps + `motion.json` sidecar (`root_space:"camera"`),
`spec/skel_to_mixamorig.json` name-map for Lane C. Verified on the demo clip: Blender import
= 24 bones, **upright** (1.59 m), motion present, FK self-consistency **0.0 mm**; queue of 6
drained unattended (2 bad clips failed gracefully, 4 valid BVH, heartbeat per clip); runtime
**~1.6× realtime** steady-state (under the <2× bar). mobilenet detector + detect-every-20 +
single-subject box reuse retired the CPU-detector wall-clock finding. Axis fix: HSMR emits
Y-down (OpenCV) → `diag(1,-1,-1)` → Y-up/faces-Z. **History (sessions 24):**
**Decision made session 3: Path B** — engine =
**HSMR → SKEL**, run local, ship rig-motion only, John accepts NC-licence risk
(`docs/DECISIONS.md` Decision 1 RESOLVED). Env up in `.engine/HSMR` (uv venv, torch 2.13
**MPS green**, ungated weights pulled). **Detector RESOLVED (session 4):** torchvision
Faster R-CNN drop-in replaces the unbuildable detectron2; `test_detector.py` is green today
(7/7 demo imgs → 15 patches, **no SKEL needed**). `smoke_test.py` now runs real detection +
patch-crop and reaches the **exact SKEL gate**; chumpy cleared as a non-blocker. **Only open
before B ships:** (1) John downloads gated **SKEL v1.1.1** so
`.engine/HSMR/data_inputs/body_models/skel/skel_male.pkl` (+ `skel_female.pkl`) exist;
(2) run the forward-pass smoke test (MPS / timing <2× / NaN check); then B2 (service-ize
BVH) + B3 (jitter). Lanes C and D remain blocked on B.
- [ ] **Lane C** — retarget + cleanup (Blender). bvh → mixamorig FBX + QC turntable. - [ ] **Lane C** — retarget + cleanup (Blender). bvh → mixamorig FBX + QC turntable.
- [ ] **Lane D** — clip factory. `mocap` CLI end-to-end; games + Unreal integration. - [ ] **Lane D** — clip factory. `mocap` CLI end-to-end; games + Unreal integration.
- [ ] Lane E — ML upgrades (post-v1, gated). - [ ] Lane E — ML upgrades (post-v1, gated).
- [ ] Lane F — UX (optional). - [ ] Lane F — UX (optional).
## SESSION 5 — DONE (2026-07-17). All six orders executed; Lane B ships.
1. ✅ **Durable-ized** the ARM fixes → `engine_patches/` + `setup_hsmr.sh --patch`. Verified: fresh
clone + `--patch``test_detector.py` green, no hand edits.
2. ✅ **Path-A email** drafted in `docs/DECISIONS.md` (John sends to ps-license@tue.mpg.de).
3. ✅ **Queue reconciled**: `ingest.py` defaults to local `queue/` on ultra; `--remote-queue` keeps
the m3ultra path. PLAN Ground-rule-1 deviation noted; CAPTURE.md updated.
4. ✅ **Treaty amended**: `root_space` (camera|world) is a REQUIRED sidecar field; Lane C reads it.
5. ✅ **B2 + B3**: `pose_engine.py` (SKEL-24 BVH + sidecar + jitter) + `spec/skel_to_mixamorig.json`.
Detector swapped to mobilenet; detect-every-20 + box reuse; build-once + per-video isolation.
Ran the smoke test's SKEL forward on real params — no re-verify needed (smoke already PASSED).
**HANDOFF TO LANE C (retarget + cleanup, Blender, on ultra):**
- Input is `out/<name>/motion.bvh` (SKEL-24 skeleton, **Y-up, metres, faces Z**) + `motion.json`.
Import in Blender with `import_anim.bvh(axis_forward='-Z', axis_up='Y')` (verified upright).
- Constraint-map SKEL→mixamorig via **`spec/skel_to_mixamorig.json`** (20 mapped, 4 null=twist/heel;
Spine1 interpolated, Head follows Neck, fingers are Lane E). mixamorig canon: `mixamorig:` colon.
- **`root_space` is `"camera"`** → root is NOT world-grounded (HSMR is per-frame). In-place clips are
first-class; for traveling clips derive a world root from foot contacts (PLAN Lane C task).
- Rest pose is a clean T-pose at zero rotation; T-pose bookends only exist if John filmed the protocol
(demo clips don't have them) — Lane C's trim must tolerate their absence.
- Re-run the SKEL→BVH stage cheaply with `pose_engine.py <clip> --from-params` (skips HMR) when
tuning; raw params are saved at `out/<name>/params.npz`.
## SESSION 5 ORDERS — from Fable's review (2026-07-17, supersede older notes where they conflict)
Review verdict: Lanes 0/A solid; session 2's reversal of the B1 animal-model blunder was the
right call, well executed; ARM engineering verified independently (test_detector.py re-run
green 7/7 by reviewer). Proceed as follows, in order:
1. **Durable-ize the gitignored engine fixes (do first, ~30 min).** The torchvision detector,
`_headless.py`, `test_detector.py`, `smoke_test.py` exist ONLY on ultra's disk inside the
gitignored `.engine/`. Move canonical copies into git at `engine_patches/` and make
`setup_hsmr.sh` apply them after clone. Acceptance: fresh clone in a temp dir + script →
`test_detector.py` green with no hand edits.
2. **Draft the Path-A email (parallel risk-retirement, ~15 min).** Write the SKEL indie
commercial-license request to ps-license@tue.mpg.de into `docs/DECISIONS.md` (project
description, solo Australian dev, indie revenue, SKEL-in-pipeline-not-redistributed usage).
John sends it himself. Path B continues meanwhile; a granted license converts B→A with zero
rework. Do not wait on the reply.
3. **Reconcile the split-brain queue (~30 min).** Lane A's ingest.py rsyncs to m3ultra's queue,
but the engine (and MPS verification) is on ultra — nothing consumes that queue. For v1:
default ingest to LOCAL `queue/` on ultra (`--remote-queue` flag preserves the m3ultra path),
note the deviation from PLAN Ground-rule 1 in PLAN.md with a `# ponytail: v1 inference on
ultra; migrate to m3ultra when a real throughput need appears` marker. Update CAPTURE.md.
4. **Treaty amendment (small, load-bearing).** `motion.json` sidecar gains REQUIRED field
`root_space: "camera" | "world"`. HSMR emits `"camera"`. Lane C must read it and, for
locomotion clips, derive world root from foot contacts (already in notes). v1 quality bar:
in-place clips are first-class; traveling clips best-effort until a grounding pass exists.
**SMOKE TEST RAN 2026-07-17 (Fable, after John's SKEL v1.1.1 download landed — installed at
the expected path): PASSED with findings.** device=mps, 263 frames → 1315 patches, recovery
31.2 fps, **nan=False**, `data_outputs/smoke/params.npz` written. Findings for B2: (1) the
CPU Faster R-CNN detector is the wall-clock hog (~1 fps, 4min of the 5min run) — B2 MUST add
detect-every-Nth-frame + box reuse/IoU tracking (single-subject footage makes this easy) or
the mobilenet swap; HMR itself is faster than realtime so the <2× acceptance is reachable.
(2) `prepare_mesh` MPS→numpy crash fixed in smoke_test.py (`.detach().cpu()`) — carry the
same pattern into pose_engine.py. Item 5 below is now UNBLOCKED (skip re-verifying: smoke
is done; go straight to B2/B3).
5. **If (and only if) SKEL v1.1.1 files are in place** (`data_inputs/body_models/skel/
skel_male.pkl`): run smoke_test.py — MPS forward pass, runtime <2× realtime on demo video,
NaN check — then proceed straight into B2 (params.npz → `spec/skel_to_mixamorig.json` map →
BVH@30fps + sidecar per treaty) and B3 (One-Euro/Sav-Gol jitter pass, root smoothed
separately). If SKEL absent: finish items 14, verify Decision-3 footage status, stop clean.
**John's homework (all three block different things):**
- SKEL v1.1.1 download (the ONLY hard blocker; instructions in `docs/B_ENGINE_SETUP.md`).
- Film walk / dance / pick-up-object per `docs/CAPTURE.md`, drop in `capture/inbox/` (blocks
the honest quality verdict + all of B's acceptance).
- Reboot m3ultra when convenient (applies staged 26.5.2 security update; housekeeping).
## Notes for the next lane ## Notes for the next lane
- **Lane D can't be started until B + C ship + test footage exists** — it's the CLI that wraps
ingest → pose_engine.py → retarget.py, and the latter two don't exist yet. Session 3 (the
"Lane D" session) correctly built nothing; it locked Decision 1 and de-risked MPS instead.
- **SKEL version trap:** HSMR needs **v1.1.1** exactly (`changelog_v1.1.1.txt`); older = silent
incompat. Source: `.engine/HSMR/docs/SETUP.md` (register skel.is.tue.mpg.de → download.php).
- **Engine wiring + ARM bypasses live in `docs/B_ENGINE_SETUP.md`** — all three ARM landmines
are now cleared (session 4): detectron2 → **torchvision drop-in** (real detection, not a
full-frame stub), the pyrender/EGL fake is centralized in `.engine/HSMR/_headless.py`, and
chumpy is confirmed a non-blocker. `test_detector.py` proves the detector path green, SKEL-free.
- **World-grounded gap (important):** HSMR is per-frame → root is camera-space `cam_t`, NOT the
treaty's world-grounded trajectory. Fine for in-place DJ moves; walking/locomotion needs Lane C
to derive root from foot-contact (or a grounding pass). Carry into Lane C design.
- **HSMR is installed on ultra, not m3ultra** (PLAN Ground-Rule 1 puts the pose engine on
m3ultra). Likely because m3ultra's disk reclaim waits on the reboot (Decision 2). Next B
session decides: move it to m3ultra, or consciously host inference on ultra.
- **VOID** (was: "MoCapAnything is BVH-direct, skip the retarget rig"). Reversed — the model
is animal-only and not our engine. Whatever human model we pick emits **SMPL or SKEL, not
mixamorig**, so **Lane C's constraint-retarget IS required** and
`spec/{smpl,skel}_to_mixamorig.json` is back on the menu. HSMR's SKEL output (a biomechanical
skeleton) is the likely source format if John goes SKEL.
- **Check MPS + global-trajectory FIRST** for whichever human model John's decision unlocks
(HSMR has an HF Space + demo to smoke-test before John's footage). These two risks are
unchanged by the engine swap.
- Ground-truth skeleton naming is **`mixamorig:`** (colon), armature object named `Armature`. - Ground-truth skeleton naming is **`mixamorig:`** (colon), armature object named `Armature`.
Lane C's name-map must handle colon vs underscore variants (PLAN warned about this). Lane C's name-map must handle colon vs underscore variants (PLAN warned about this).
- Blender headless entrypoint: `/Applications/Blender.app/Contents/MacOS/Blender --background --python <script>`. - Blender headless entrypoint: `/Applications/Blender.app/Contents/MacOS/Blender --background --python <script>`.

107
docs/B1_BAKEOFF.md Normal file
View File

@ -0,0 +1,107 @@
# B1 — pose-model bake-off
**Status: SUPERSEDED 2026-07-17 (session 2).** The selection below (MoCapAnything V2) was a
**category error** — MoCapAnything is an **animal** mocap model (71 animal targets, no
human/biped/mixamorig, no fingers), so it cannot be MOCAPGOD's human v1 engine. Its "SMPL gate
dissolved" conclusion is therefore wrong; the gate is **LIVE**. See `docs/DECISIONS.md` for the
corrected, license-verified decision. The candidate research below is kept as history — its
*human*-model findings (GVHMR non-commercial, GEM-X CUDA-only) still stand; its recommendation
does not.
The PLAN told this lane to distrust the July-2024 snapshot and search fresh. It paid off:
two models that post-date the snapshot change the whole calculus — one of them **removes
the SMPL licensing hard-stop the plan was built around.**
## Selection criteria (PLAN order)
1. permissive code license (must ship in commercial games)
2. world-grounded global trajectory (not pose-in-place)
3. runs on Apple Silicon (m3ultra — MPS or acceptable CPU; no NVIDIA on the fleet)
4. SMPL-dependency status (the legal rail)
## Candidate matrix
| Model | World-grounded | Code lic. | Weights lic. | SMPL? | Apple Silicon | Output | Commercial-ship |
|---|---|---|---|---|---|---|---|
| **MoCapAnything V2** ⭐ | trajectories (global vs camera **unverified**) | MIT | **MIT** | **No** | standard PyTorch, no custom CUDA ops → MPS-portable *with effort* | **BVH rotations onto an arbitrary target rig** (feed it mixamorig) + hands | **YES (MIT/MIT)** |
| GVHMR | Yes — SOTA, beats WHAM 1828% WA-MPJPE | **research-only** | needs SMPL | Yes | plain PyTorch, MPS-portable w/ effort | SMPL params (rot) + global traj (.pt) | **NO** — code is non-commercial *and* SMPL-gated |
| GEM-X (NVIDIA) | Yes | Apache-2.0 | NVIDIA Open Model Lic. (NVIDIA states commercial-ready) | **No** (SOMA body model) | **CUDA 12.1+ only, NVIDIA arch** — will not run on the fleet | 77-joint axis-angle + world root (m), incl. hands | Yes, but **off-fleet** |
| WHAM | Yes (autoregressive; GVHMR's predecessor) | research/NC (not re-verified) | needs SMPL | Yes | PyTorch, MPS-portable | SMPL + traj | NO (gate) — dominated by GVHMR |
| TRAM | Yes (SLAM-based camera+human) | mixed; +DROID-SLAM | needs SMPL | Yes | heavier (SLAM stage) | SMPL + traj | NO (gate) — heaviest, not pursued |
Sources verified this session: GVHMR repo LICENSE (custom research-only, "prohibited for
commercial use", contact xwzhou@zju.edu.cn); GEM-X model card (Apache code / NVIDIA Open
Model License weights / SOMA / CUDA-only); MoCapAnything V2 weights card
(`kehong/MoCapAnythingV2-weights`, **License: mit**, author Kehong Gong = paper author) and
code repo `github.com/phongdaot/MocapAnything` (MIT). WHAM/TRAM licenses not re-verified —
both are SMPL-gated and dominated by GVHMR on the world-grounded metric, so not installed.
## The two findings that matter
**1. GVHMR (the accuracy leader) is double-blocked for us.** Its own code license is
non-commercial — that's a wall *independent of* SMPL, and the joints-only Path-B from the
PLAN can't get around a non-commercial *code* license. John's clips ship in commercial
games. GVHMR is therefore a research/accuracy *reference* only, unless John negotiates a
paid ZJU commercial license. It is **not** a v1 shipping engine.
**2. MoCapAnything V2 dissolves the SMPL gate.** MIT code + MIT weights, commercial use
permitted, **no SMPL files anywhere** in the pipeline or dependency list (torch/opencv/
trimesh only). It is category-agnostic: you give it a monocular video **plus the target
rig as a prompt**, and it emits a rotation-based BVH that drives *that rig* directly, via
constraint-aware IK. Cross-species too (it retargets across heterogeneous rigs — a free
preview of the Lane E4 centaur work).
### Why #2 is bigger than "gate cleared"
If we prompt MoCapAnything with the **mixamorig** skeleton (`spec/mixamorig_bones.json` /
`female_game.glb`), it emits **mixamorig BVH directly**. That means:
- `spec/smpl_to_mixamorig.json` (B2 task 2) is **unnecessary** — there is no SMPL
intermediate to map.
- Lane C's core constraint-retarget (bvh→mixamorig) **largely collapses** — the BVH is
already on-skeleton. Lane C shrinks to: import → foot-contact fix → jitter → trim →
export FBX.
The fresh search didn't just clear the legal hold-up; it deleted a spec file and most of a
lane. `# ponytail win: don't build the SMPL→mixamorig mapper or the Lane C retarget rig
until we've confirmed MoCapAnything can't hit mixamorig directly.`
**Caveat (honest):** getting it to emit *exactly* the treaty's mixamorig (joint order,
Y-up, faces Z, bone-roll) will still need a rig-prompt config and a convention
reconciliation pass. "Collapses" = "greatly simplifies," not "free."
## Recommendation
**Primary: MoCapAnything V2.** Only candidate that is commercial-clean AND on-fleet AND
BVH-direct. Accept two risks, both verifiable cheaply at inference (below).
**Reference (optional): GVHMR** under its research license, purely to benchmark how much
accuracy we're leaving on the table — informs whether a paid ZJU license is ever worth it.
**Shelved: GEM-X** — revisit only if the fleet ever gains an NVIDIA GPU (it's the cleanest
option the day CUDA is available; zero-cloud rules out renting it).
## Must-verify at the inference eyeball (the other half of B1)
These are why the lane isn't ticked. Cheap to settle — MoCapAnything ships a ~160 MB sample
dataset and an HF demo Space, so we don't even need John's footage to answer them:
1. **Global trajectory?** Treaty requires world-space root motion. MoCapAnything says
"temporally coherent trajectories" — confirm the character *travels through the world*,
not runs-in-place. If camera-space only, we need a grounding pass (or fall back to GVHMR
for locomotion clips). **This is the #1 risk.**
2. **MPS reality.** Standard PyTorch predicts yes, but the PLAN's rule is "check MPS early —
it decides everything." Expect some ops to need CPU fallback; time it (<2× realtime is
the bar).
3. **Quality.** The linked code is a re-implementation ("use as reference, not
reproduction"). Eyeball walk / dance / pick-up-object in Blender before trusting it.
## Preconditions before the install/inference session
- **m3ultra disk: 29 GB free of 926 GB (97% full).** Marginal for one torch env + weights;
too tight for two. Clear space first (see `DECISIONS.md`).
- **Test footage:** inbox is empty. Sample data covers the smoke test; John's walk / dance /
pick-up-object clips are needed for the real quality call and B2/B3/acceptance.
- **Engine decision:** confirm MoCapAnything-primary (vs. paying for GVHMR-grade). See
`DECISIONS.md`.
## Sources
- [GVHMR](https://github.com/zju3dv/GVHMR) · [GVHMR paper](https://arxiv.org/html/2409.06662v1)
- [NVIDIA GEM-X](https://huggingface.co/nvidia/GEM-X)
- [MoCapAnything project](https://animotionlab.github.io/MoCapAnything/) ·
[V2 weights (MIT)](https://huggingface.co/kehong/MoCapAnythingV2-weights) ·
[code](https://github.com/phongdaot/MocapAnything) ·
[paper](https://arxiv.org/abs/2512.10881)
- [TRAM](https://github.com/yufu-wang/tram)

87
docs/B_ENGINE_SETUP.md Normal file
View File

@ -0,0 +1,87 @@
# Lane B engine — HSMR install & smoke-test state (session 3, 2026-07-17)
Engine per Decision 1 = **Path B: HSMR → SKEL**, run local on **ultra**, MPS. Installed under
`.engine/HSMR` (gitignored). This doc = how it's wired + how to run once SKEL lands.
## State
- ✅ `uv venv` py3.10 at `.engine/HSMR/.venv`; torch **2.13.0**, `mps.is_available() == True`.
- ✅ Ungated weights in `.engine/HSMR/data_inputs/`: HSMR-ViTH ckpt
(`released_models/HSMR-ViTH-r1d1/checkpoints/hsmr.ckpt`), ViTPose backbone, 3 regressors.
- ✅ **Real person detector** (torchvision Faster R-CNN, no detectron2) — checked by
`.engine/HSMR/test_detector.py`, green **today with no SKEL**: 7/7 demo images → 15 patches
@256×256 through HSMR's own cropper.
- ✅ Smoke-test runner `.engine/HSMR/smoke_test.py`**PASSED 2026-07-17** (session 5): SKEL
v1.1.1 in place, device=mps, 263 frames → 1315 patches, recovery 31.2 fps, **nan=False**,
`data_outputs/smoke/params.npz` written (poses (N,46), betas (N,10), cam_t (N,3)).
- ✅ Install script (re-runnable): **`./setup_hsmr.sh`** at the repo root (tracked in git).
`./setup_hsmr.sh --patch` re-applies the ARM/headless fixes onto an existing/re-cloned HSMR.
- ✅ **The ARM fixes are now durable in git** (session 5, was: "on ultra's disk only"). Canonical
copies live in **`engine_patches/`** (`_headless.py`, `test_detector.py`, `smoke_test.py`,
`lib/modeling/pipelines/vitdet/__init__.py`); `setup_hsmr.sh` overlays them onto the gitignored
`.engine/HSMR` clone. Verified: fresh clone + `--patch``test_detector.py` green, no hand edits.
## Apple-Silicon blockers — all three resolved (session 4, "Lane B detector")
- **detectron2 won't build on ARM Mac****RESOLVED.** Replaced its whole ViTDet detector with
a torchvision Faster R-CNN drop-in in `lib/modeling/pipelines/vitdet/__init__.py` — same
`build_detector(...) -> (dets, downsample_ratios)` contract, torchvision's person class (1)
remapped to HSMR's human id (0), boxes as lurb, ratio 1.0. This also killed a **load-time
landmine**: the old `import detectron2` there exploded anything importing `lib.kits.hsmr_demo`
(why session 3 had to stub it). Detector pinned to **CPU** — cheap next to the ViT-H pass and
it dodges MPS RoIAlign/NMS gaps; an in-file `# ponytail` note marks the mobilenet swap if
per-frame detection ever dominates on long videos.
- **pyrender is dead headless on macOS** (`py_renderer` forces `PYOPENGL_PLATFORM=egl`, no EGL on
Mac; `pyrender` is even referenced in a type annotation, so a plain import-guard won't do).
Centralized the permissive `pyrender`/`OpenGL` fake into **`.engine/HSMR/_headless.py`** — every
entrypoint does `import _headless` before any `lib.*` import. QC still happens in Blender (fleet
renderer + Lane C's tool); the runner dumps `params.npz` (SKEL poses/betas/cam_t — the Path-B
payload) and `mesh_qc.npz` (verts/faces for Blender).
- **chumpy failed to build****NOT a blocker (verified).** The runtime `SKELWrapper` loads its
pkls with plain `pickle.load(encoding='latin1')` — no chumpy import. chumpy only appears in
SKEL's offline model-*build* script (`merge_smplh_mano.py`), which we never run. Ignore the WARN.
## John's one step — drop the SKEL files (login + licence, so it's yours to do)
Register **skel.is.tue.mpg.de** → Download → **"SKEL and BSM models"** → `skel_models_v1.1.zip`
(**must be v1.1.1** — check `changelog_v1.1.1.txt` inside; older = silent incompatibility).
Unzip, then:
```
mv /path/to/skel_models_v1.1 ~/Documents/MOCAPGOD/.engine/HSMR/data_inputs/body_models/skel
```
so that `…/data_inputs/body_models/skel/skel_male.pkl` exists.
## Run the smoke test (once SKEL is in place)
```
cd ~/Documents/MOCAPGOD/.engine/HSMR && source .venv/bin/activate
PYTHONPATH=. python smoke_test.py -i data_inputs/demo/example_videos/gymnasts.mp4 # or .../example_imgs
```
Outputs → `data_outputs/smoke/`: `params.npz` (rotations), `mesh_qc.npz` (Blender QC), and
console device/fps/NaN. Answers the Lane B smoke questions: **MPS forward OK? timing (<2×
realtime)? pose sane?**
⚠️ HSMR is a **per-frame** recoverer → root is **camera-space `cam_t`, not world-grounded**.
Fine for in-place DJ moves; locomotion (walk) needs Lane C's foot-contact root derivation or a
grounding pass. This is the treaty's global-trajectory caveat, carried into Path B.
## Run the pose engine (B2/B3, session 5) — the real service
`pose_engine.py` (repo root, tracked) wraps the engine into the pipeline. Run under the engine venv:
```
cd ~/Documents/MOCAPGOD
.engine/HSMR/.venv/bin/python pose_engine.py # drain queue/ → out/<name>/
.engine/HSMR/.venv/bin/python pose_engine.py queue/clip.mp4 # one clip
.engine/HSMR/.venv/bin/python pose_engine.py <clip> --from-params # re-run SKEL→BVH, skip HMR
.engine/HSMR/.venv/bin/python pose_engine.py --selfcheck # BVH-math self-test, no engine
```
Per clip → `out/<name>/`: `motion.bvh` (SKEL-24, **Y-up, metres, faces Z**), `motion.json`
(sidecar, `root_space:"camera"`), `params.npz` (raw poses/betas/cam_t for QC / `--from-params`).
Heartbeat: `~/.jobs/pose_engine`. Models build **once** per invocation (queue amortizes); a bad
clip is logged and skipped, not fatal.
Key engine facts learned this session (for whoever touches B again):
- **SKEL emits `poses` = (F, 46)** biomechanical q, NOT SMPL's 72 axis-angles. Don't decode q by
hand — run `SKEL.forward()` and consume `joints_ori` (F,24,3,3 global orientations) + `joints_tpose`.
- BVH is rigid-bone: rest-aware FK (`offset=rest_J[j]-rest_J[parent]`, `W=Rg@rest_R^T`,
`local=W[parent]^T@W[j]`) reconstructs SKEL joints to ~4 cm; the residual is SKEL's coupled
knee/ankle translations, which a rotation-only BVH cannot carry (accept it).
- **HSMR camera frame is Y-DOWN** (OpenCV; verified head.y < pelvis.y). `pose_engine` applies
`diag(1,-1,-1)` to reach the treaty's Y-up / faces-Z. Undo/adjust there if a future model differs.
- Detector is **mobilenet** now (was R50) — CPU detection was the wall-clock hog. It self-caps input
to ~640 px internally, so pre-downscaling does nothing; `--detect-every` is the real throughput knob.

View File

@ -25,8 +25,9 @@ uses to auto-trim the clip. No bookends → manual trimming.
- **AirDrop** the clip from the phone to **ultra**, save into - **AirDrop** the clip from the phone to **ultra**, save into
`~/Documents/MOCAPGOD/capture/inbox/`. `~/Documents/MOCAPGOD/capture/inbox/`.
- Then run **`python3 ingest.py`** (or let the Lane D `mocap` CLI do it). Ingest validates, - Then run **`python3 ingest.py`** (or let the Lane D `mocap` CLI do it). Ingest validates,
normalizes to 30fps, and ships it to the m3ultra queue. Rejects (portrait, no feet, garbage) normalizes to 30fps, and puts it in the local pose queue (`queue/` on ultra, where the engine
land in `capture/rejected/` with a reason in `capture/LEDGER.tsv` — check there if a clip vanishes. runs). Rejects (portrait, no feet, garbage) land in `capture/rejected/` with a reason in
`capture/LEDGER.tsv` — check there if a clip vanishes. (`--remote-queue` ships to m3ultra instead.)
## Common rejections (and the fix) ## Common rejections (and the fix)
| Symptom | Fix | | Symptom | Fix |

139
docs/DECISIONS.md Normal file
View File

@ -0,0 +1,139 @@
# Decisions needing John
**Status 2026-07-17 (session 2): Lane B1's engine pick is REVERSED** (Decision 1). The disk
work (Decision 2) from the concurrent session is preserved intact. Corrected, license-verified
analysis below + `LICENSES.md`. Prior "SMPL hard-stop avoided / go MoCapAnything" memo
superseded (git @ 7819e12).
## Decision 1 — pose engine: B1 picked an ANIMAL model for a HUMAN job; the gate is LIVE
> **RESOLVED 2026-07-17 — John chose Path B (run local, accept NC risk).** Engine = **HSMR
> → SKEL** on the fleet. Ship only the retargeted mixamorig motion (no Max-Planck file
> redistributed); John explicitly accepts the judgment call that authoring game clips from an
> NC-licensed model is his risk. Path A (email MPI for a commercial licence) stays on the shelf
> as the clean upgrade if that risk ever needs retiring. **MPS verified green on ultra this
> session** (torch 2.13.0, `mps_available=True`, `mps_built=True`). **Sole remaining blocker to
> a first forward pass: the gated SKEL download — John's manual step** (register at
> skel.is.tue.mpg.de → download.php → `skel_models_v1.1.zip`, **must be v1.1.1**, unzip into
> `.engine/HSMR/data_inputs/body_models/skel/` per HSMR `docs/SETUP.md`). Behind a login +
> NC-licence agreement, so it is John's to do — I can't create the account or accept the terms.
> Analysis below is retained for the record.
B1 chose **MoCapAnything V2** for being MIT/MIT with no SMPL, and concluded "the SMPL hard-stop
is avoided." Reading the actual repo this session shows why that's wrong: MoCapAnything is an
**animal** motion-capture system — **71 animal retarget targets** (Alligator…Tyranno), trained
on `zoo1030` (animals) + `obj1k` (objects), **no human/biped/mixamorig reference and no
fingers**. MOCAPGOD's v1 job is capturing **John (a human)** onto mixamorig (PLAN line 1; the
Lane B brief says "monocular **human** motion recovery"). MoCapAnything didn't dissolve the SMPL
gate — it swapped in a task the gate doesn't cover. **Wrong tool for the human path.** (It's a
clean, strong tool for *creature* NPCs — option C.)
**Verified from license text this session (links/dates in `LICENSES.md`):**
| Human-capable option | Code license | Body model | Body-model license | On the fleet? |
|---|---|---|---|---|
| GVHMR / WHAM / TRAM / OnlineHMR (accuracy leaders) | research/NC (GVHMR NC confirmed) | **SMPL** | NC — commercial via Meshcapade (quote) | PyTorch→MPS ✓ |
| **HSMR** (CVPR25 oral) / SKEL-CF (Nov 2025) | **MIT** (HSMR) | **SKEL** | NC — commercial via MPI `ps-license@tue.mpg.de` | PyTorch→MPS ✓ (unverified) |
| GEM-X (NVIDIA) | Apache | SOMA (not SMPL) | commercial-ready | **CUDA-only — OFF fleet** ✗ |
| MoCapAnything V2 | MIT | none | — (clean) | ✓ but **animal-only** ✗ for humans |
**No free-lunch, commercial-clean, on-fleet, world-grounded *human* model exists.** SMPL and
SKEL are both Max-Planck research licenses with an "email for commercial" path; the one
clean-licensed human model (GEM-X) needs NVIDIA hardware the fleet lacks. Pick a direction:
**A. License a body model, use a proven human model — RECOMMENDED.** Email for an indie
commercial license. Primary: **SKEL → HSMR** (code is MIT, outputs a biomechanical **skeleton**
ideal for the mixamorig retarget) via `ps-license@tue.mpg.de`. Fallback: **SMPL** via
`sales@meshcapade.com` (unlocks WHAM/TRAM, though their *code* licenses also need clearing).
One email + likely a modest fee + a few days. Clean and shippable. *I can draft the email; I
can't send it.*
**B. SMPL-free-outputs path (PLAN Path B) — run locally, ship only your rig's motion.** HSMR's
code is MIT: run on the fleet, extract joint **rotations**, retarget onto YOUR mixamorig, ship
only that — distribute no Max-Planck file. Zero cost, unblocks today. **Risk:** the NC licenses
forbid "production of artefacts for commercial purposes"; whether authoring game clips counts is
a legal judgment call. The clean mitigation (email MPI to ask) collapses into A. John-accepts-risk.
**C. Reconsider scope — creatures now, defer the human license.** MOCAPGOD's downstream is
stylized NPCs + a centaur RPG; some targets aren't human. If the near-term need is creature
NPCs, **MoCapAnything (MIT, clean, on-fleet) does that TODAY**; defer the human-capture license
until you need your own DJ clips.
**Recommendation A.** Honest, cheap, converts B's grey into a clear yes. In parallel I keep
MoCapAnything as the sanctioned *creature* engine (free upside, blocks nothing) and verify
HSMR's MPS-runnability so we're ready when a license lands. I will **not** build the human v1
pipeline on an NC model without your explicit go (that's B).
## Path A — SKEL commercial-licence email (READY TO SEND — John sends it himself)
Parallel risk-retirement (session-5 order 2). Path B ships meanwhile; if MPI grants a licence
the grey area in Decision 1 disappears with **zero pipeline rework** (same HSMR/SKEL engine).
**Do not wait on the reply.** John: fill the two `[brackets]`, send from your address to
`ps-license@tue.mpg.de`. I can't send email or create the account — this is yours.
> **To:** ps-license@tue.mpg.de
> **Subject:** Commercial licence enquiry — SKEL body model (indie game developer)
>
> Dear Perceiving Systems licensing team,
>
> I'm a solo independent game developer based in Australia. I'd like to enquire about a
> commercial licence for the **SKEL body model** (v1.1.1, from skel.is.tue.mpg.de).
>
> **Use case.** I'm building a small in-house tool that recovers 3D human motion from ordinary
> single-camera video of myself, to author animation clips for stylised characters in my own
> indie games. SKEL is used inside the pipeline at inference time (via the HSMR pose-recovery
> model) to estimate body pose.
>
> **Redistribution: none.** The SKEL/BSM model files stay on my own machine — I do not
> redistribute them or any Max-Planck-derived asset. What ships in the games is only the
> resulting joint **motion**, retargeted onto my own separately-authored character skeletons
> (a Mixamo-style rig). No SKEL mesh, shape space, or model file is included or exposed in any
> shipped product.
>
> **Scale.** Small and indie: modest commercial revenue, a single developer, no external funding.
>
> Could you let me know whether an **indie / small-developer commercial licence** is available
> for this kind of in-pipeline, non-redistributed use, and the terms and any fee? I'm happy to
> provide more detail on the pipeline or sign whatever agreement you need.
>
> Thank you,
> [your name]
> monsterrobot.party · [contact email]
If MPI replies with terms, record the outcome here and flip Decision 1 from Path B to Path A
(nothing in the pipeline changes — same engine, the licence just retires the NC risk).
## Decision 2 — clear m3ultra disk (RESOLVED pending reboot, 2026-07-17)
Was 98% full (20 GB free). Deleted `unsloth/GLM-4.6-GGUF` (147 GB, re-downloadable, 0 code
refs) from the HF cache with John's OK. **But `df` still shows 20 GB** — the freed blocks are
pinned by macOS **OS-update snapshots**: a **macOS Tahoe 26.5.2 update is staged** (recommended
/ security, "Action: restart") and the box has 18 days uptime. Space realizes on **next
reboot**, which also applies the pending update. No `~/.jobs` heartbeat job is running (load is
just the Claude desktop app), so a reboot kills nothing tracked.
**John: reboot m3ultra when convenient** → applies 26.5.2 + reclaims 147 GB (→ ~167 GB free,
ample for Lane B). Not doing this myself — it's your workstation. Kept the FLUX variants
(you deselected them); another ~200 GB is prunable there later if wanted.
## Decision 3 — test footage (blocker for the real quality call)
`capture/inbox/` is empty. Whichever human engine Decision 1 unlocks, the honest
walk/dance/pick-up-object quality verdict — and all of B2/B3 + the acceptance check — needs your
clips. (HSMR has an HF Space + demo images for the MPS/trajectory smoke-test before your
footage.) Film the PLAN's three per `docs/CAPTURE.md` (T-pose bookends, feet in frame), drop
them in the inbox, then `python3 ingest.py`.
→ **John: film + ingest walk / dance / pick-up-object when convenient.**
## Blast radius of the Decision-1 reversal (next session must know)
- **Lane B1 is NOT done** — wrong engine selected. Re-opened; PROGRESS un-ticked.
- **VOID the old note "MoCapAnything is BVH-direct, skip the retarget rig."** Any human model
emits **SMPL or SKEL, not mixamorig****Lane C's constraint-retarget IS required again**,
and `spec/{smpl,skel}_to_mixamorig.json` is back on the menu.
- Preconditions: disk resolves on the m3ultra reboot (Decision 2); the MPS/global-trajectory
smoke-test still stands for whichever engine John's pick unlocks.
---
Once Decision 1 is picked and 23 clear, the next session installs the chosen human engine,
proves MPS + global trajectory, and proceeds to B2 (service-ize) / B3 (jitter).

View File

@ -1,24 +1,43 @@
# Licenses & legal findings # Licenses & legal findings
Every third-party model/dataset/tool that touches the pipeline gets a row here with Every third-party model/dataset/tool that touches the pipeline gets a row here with its
its license, commercial status, link, and the date checked. Skeleton now; Lane B fills license, commercial status, link, and the date checked. See PLAN.md §Ground rules.
the pose-model rows, Lane E fills any training-dataset rows. See PLAN.md §Ground rules.
> **2026-07-17 correction (session 2):** B1's original row billed MoCapAnything as the human
> engine — it is an **animal** model (see `DECISIONS.md`). The SMPL/SKEL gate is **LIVE**, not
> avoided. Rows below corrected and license-verified.
## Components ## Components
| Component | Version | License | Commercial OK? | Notes | Checked | | Component | License (code / body model) | Commercial OK? | Notes | Checked |
|---|---|---|---|---|---| |---|---|---|---|---|
| _pose model (Lane B1 winner)_ | — | — | — | fill after bake-off | — | | **HSMR** (human — candidate engine) | code **MIT** / body **SKEL** | code yes / **model NO** | `IsshikiHugh/HSMR`, CVPR25 oral. Predicts **SKEL** params (biomechanical skeleton, great for retarget). Needs SKEL files (NC) at inference. MPS-runnability unverified. | 2026-07-17 |
| SMPL / SMPL-X body model | — | research (Meshcapade) | **NO** by default | commercial needs Meshcapade license; or use joints-only path (PLAN §5) | — | | SKEL-CF (human — candidate) | code (verify) / body **SKEL** | **model NO** | Coarse-to-fine skeleton+mesh, Nov 2025. Same SKEL gate. | 2026-07-17 |
| AMASS dataset | — | academic | **verify** before any shipped training | Lane E only | — | | **SKEL body model** | MPG research | **NO** (default) | "any use for commercial… purposes is prohibited"; commercial via **ps-license@tue.mpg.de**. Same family as SMPL. | 2026-07-17 |
| Mixamo clips | — | Adobe ToS | ship-in-game OK; **not** ML training data | — | — | | **SMPL / SMPL-X body model** | MPG research (Meshcapade sublicenses) | **NO** (default) | Commercial via **sales@meshcapade.com** (quote-based; incl. SMPL+H/-X/STAR). Gate for all SMPL-based human models. | 2026-07-17 |
| GVHMR / WHAM / TRAM / OnlineHMR (human, SMPL) | research/NC (GVHMR NC confirmed) / SMPL | **NO** | Accuracy leaders but **double gate**: NC *code* AND SMPL. GVHMR commercial via xwzhou@zju.edu.cn. Reference/yardstick unless licensed. | 2026-07-17 |
| GEM-X (human, shelved) | Apache / SOMA (not SMPL) | yes, but **off-fleet** | NVIDIA commercial-ready; **CUDA-only** → won't run on Apple Silicon. Revisit only if the fleet gains NVIDIA. | 2026-07-17 |
| MoCapAnything V2 (**animal** — creature NPCs only) | MIT / none | **YES** (creatures) | `kehong/MoCapAnythingV2-weights` MIT, no SMPL. **71 animal targets, no human/biped/fingers** → NOT the human engine. Keep for creature/centaur (Lane E4) work. | 2026-07-17 |
| AMASS dataset | academic | **verify** before shipped training | Lane E only | — |
| **BioAMASS v1.0** (downloaded, `~/Documents/skel/bioamass_v1.0`: CMU, DFAUST, MPI_Limits subsets, 376MB) | MPG research (same SKEL account/licence) | **NO** (default) — same NC family; Lane E training that ships needs the same Path-A licence email | SKEL-fitted mocap — ideal Lane E2 autoposer training data *format-wise*; legal gate identical to SKEL itself | 2026-07-17 |
| Mixamo clips | Adobe ToS | ship-in-game OK; **not** ML training data | — | — |
## The SMPL gate (hard stop — John decision) ## The SMPL/SKEL gate (hard stop — LIVE, John decision)
Most SOTA monocular mocap models emit SMPL parameters. Before any commercial use: Every accurate world-grounded **human** model depends on a Max-Planck body model — **SMPL**
- **Path A:** buy a Meshcapade commercial license (cheap tiers exist). (GVHMR/WHAM/TRAM/OnlineHMR) or **SKEL** (HSMR/SKEL-CF) — both research-licensed. Two paths:
- **Path B (SMPL-free):** use only the model's joint positions/rotations, never the SMPL - **Path A — license it.** Email MPI (SKEL, `ps-license@tue.mpg.de`) or Meshcapade (SMPL,
mesh/shape files, AND confirm the inference repo's own code license permits commercial `sales@meshcapade.com`) for an indie commercial license. Clean; quote-based.
use of outputs. - **Path B — SMPL-free outputs.** Run an MIT-code model (HSMR) locally, keep only joint
rotations, retarget onto our own mixamorig, distribute no Max-Planck file. Grey area: the NC
license bars "production of artefacts for commercial purposes." John-accepts-risk.
Lane B writes the decision to `docs/DECISIONS.md` and stops there if unresolved. Decision lives in `docs/DECISIONS.md`. The human path does not proceed until John picks.
## Sources (session 2, 2026-07-17)
- SKEL license (commercial prohibited): https://skel.is.tue.mpg.de/license.html
- HSMR (MIT code, SKEL-based): https://github.com/IsshikiHugh/HSMR ·
https://isshikihugh.github.io/HSMR/ · paper https://arxiv.org/html/2503.21751v1
- SKEL-CF: https://arxiv.org/html/2511.20157
- SMPL commercial licensing (Meshcapade): https://meshcapade.com/infopages/licensing.html
- GVHMR (NC): https://github.com/zju3dv/GVHMR · OnlineHMR: https://github.com/Tsukasane/Video-OnlineHMR

View File

@ -0,0 +1,34 @@
"""Headless-macOS import shims for the HSMR engine (our fleet is all Apple Silicon).
pyrender -> PyOpenGL -> EGL cannot load without a GL context, yet several vendored modules
`import pyrender` at load time some even in type annotations (`List[pyrender.Node]`) so
the import chain explodes before any of our code runs. We never render in-process; Blender
is the fleet renderer (Lane C). So we drop permissive fakes for the pyrender + OpenGL stack
into sys.modules, letting the chain import while keeping everything else (ColorPalette,
Wis3D, the models) real.
Import this BEFORE the first `lib.*` import: `import _headless # noqa: F401`
# ponytail: fakes, not a real headless GL backend (osmesa/egl). We don't render here, so a
# working GL context would be dead weight. Wire a real one only if in-process render is ever
# needed off-fleet.
"""
import sys
import types
class _Any:
def __call__(self, *a, **k): return self
def __getattr__(self, n): return self
class _AnyModule(types.ModuleType):
def __getattr__(self, n):
if n.startswith('__') and n.endswith('__'):
raise AttributeError(n)
return _Any()
# setdefault: never clobber a real module that a caller already imported on purpose.
for _n in ('pyrender', 'OpenGL', 'OpenGL.GL', 'OpenGL.error', 'OpenGL.platform'):
sys.modules.setdefault(_n, _AnyModule(_n))

View File

@ -0,0 +1,77 @@
"""
Person detector torchvision drop-in replacement for HSMR's detectron2/ViTDet.
Why: detectron2 + chumpy won't build from source on Apple Silicon (the Lane B blocker),
and this module's `import detectron2` was a load-time landmine for everything that touches
`lib.kits.hsmr_demo` (smoke_test.py, the future pose_engine.py). torchvision ships a
COCO-pretrained Faster R-CNN with ungated weights and no build step.
Same public contract as the original, so run_demo.py / smoke_test.py / pose_engine.py
import it unchanged:
build_detector(...) -> callable(raw_imgs) -> (dets, downsample_ratios)
dets[i] : dict of CPU tensors (empty tensors if no person)
pred_classes : all 0 (== hsmr_demo CLASS_HUMAN_ID)
scores : float
pred_boxes : (N, 4) left-upper-right-bottom pixels
downsample_ratios : one float per image; boxes are in native-resolution pixels -> 1.0
Contract is checked by test_detector.py (runs today, no SKEL needed).
# ponytail: ~35 lines replaces the whole detectron2 stack. Detection on CPU is the pose-engine
# wall-clock bottleneck, so this uses the fast mobilenet backbone and callers detect every-Nth frame.
"""
import numpy as np
import torch
from tqdm import tqdm
from torchvision.models.detection import (
fasterrcnn_mobilenet_v3_large_fpn,
FasterRCNN_MobileNet_V3_Large_FPN_Weights,
)
_COCO_PERSON = 1 # torchvision COCO label id for "person" (0 == background)
class _PersonDetector:
"""Callable over a list of RGB HxWx3 images (what HSMR's load_inputs yields).
uint8 or float, [0,255] or [0,1] are all handled."""
def __init__(self, device='cpu', score_thresh=0.25):
# score_thresh 0.25 mirrors the old detectron2 test_score_thresh; hsmr_demo's
# _img_det2patches re-filters at 0.5, so this only widens recall a little.
# mobilenet_v3 (not resnet50) — ~5-10x faster on CPU, which is the pose-engine wall-clock
# bottleneck (Lane B finding). Plenty accurate for one prominent, in-frame subject.
weights = FasterRCNN_MobileNet_V3_Large_FPN_Weights.DEFAULT
self.model = fasterrcnn_mobilenet_v3_large_fpn(
weights=weights, box_score_thresh=score_thresh,
).eval().to(device)
self.device = device
@torch.no_grad()
def __call__(self, raw_imgs):
dets, ratios = [], []
for img in tqdm(raw_imgs, desc='Detecting'):
t = torch.as_tensor(np.ascontiguousarray(img), device=self.device).float()
if float(t.max()) > 1.5: # [0,255] -> [0,1]
t = t / 255.0
t = t.permute(2, 0, 1) # HWC -> CHW (the model's own transform caps size)
out = self.model([t])[0]
keep = out['labels'] == _COCO_PERSON
dets.append({
'pred_classes': torch.zeros(int(keep.sum()), dtype=torch.long), # human -> 0
'scores' : out['scores'][keep].cpu(),
'pred_boxes' : out['boxes'][keep].cpu(), # xyxy == left-upper-right-bottom
})
ratios.append(1.0) # detected at native resolution
return dets, ratios
def build_detector(batch_size=1, max_img_size=512, device='cpu'):
"""Signature-compatible with the old detectron2 builder. batch_size / max_img_size are
accepted for parity but unused detection runs per-frame at native resolution.
# ponytail: detector pinned to CPU regardless of `device`. It's cheap next to the
# ViT-H HMR pass, and CPU sidesteps the MPS coverage gaps in RoIAlign/NMS. Flip to
# `device` only if per-frame detection ever dominates on long videos."""
return _PersonDetector(device='cpu')

View File

@ -0,0 +1,100 @@
#!/usr/bin/env python
"""
HSMR smoke test on Apple Silicon (MPS) Path B engine verification.
Answers the Lane B smoke-test questions on HSMR's bundled demo clips (no John footage,
no SKEL license needed beyond the model files John downloaded):
1. Does the HSMR forward run on MPS? (device + CPU-fallback)
2. Timing (target < 2x realtime).
3. Pose sanity (shape, NaNs) + an overlay render to eyeball quality.
Detection uses the torchvision person detector (lib.modeling.pipelines.vitdet, the
detectron2-free drop-in). The one remaining gate is SKEL: build_inference_pipeline
instantiates the SKEL body model, so this completes only once John's SKEL files are in place.
Run (once SKEL files are in data_inputs/body_models/skel/):
cd ~/Documents/MOCAPGOD/.engine/HSMR && source .venv/bin/activate
PYTHONPATH=. python smoke_test.py -i data_inputs/demo/example_videos/gymnasts.mp4
PYTHONPATH=. python smoke_test.py -i data_inputs/demo/example_imgs # image folder
"""
import os, time, argparse
os.environ.setdefault('PYTORCH_ENABLE_MPS_FALLBACK', '1') # some ops lack MPS kernels
import _headless # noqa: F401 fake pyrender/OpenGL for headless macOS — MUST precede lib.* imports
# One import line: every helper is an attribute of hsmr_demo (via its star-imports).
from lib.kits.hsmr_demo import (
load_inputs, imgs_det2patches, build_inference_pipeline, prepare_mesh,
visualize_full_img, IMG_MEAN_255, IMG_STD_255, asb, assemble_dict,
get_logger, save_video, save_img, np, torch, Path, DEFAULT_HSMR_ROOT,
build_detector,
)
def main():
ap = argparse.ArgumentParser()
ap.add_argument('-i', '--input_path', required=True)
ap.add_argument('-o', '--output_path', default='data_outputs/smoke')
ap.add_argument('-d', '--device', default='mps')
ap.add_argument('-m', '--model_root', default=None)
ap.add_argument('--rec_bs', type=int, default=64)
ap.add_argument('--ignore_skel', action='store_true')
args = ap.parse_args()
log = get_logger(brief=True)
model_root = Path(args.model_root) if args.model_root else DEFAULT_HSMR_ROOT
out = Path(args.output_path); out.mkdir(parents=True, exist_ok=True)
# Inputs (reuse the demo loader; it only reads .input_path / .input_type).
fake = argparse.Namespace(input_path=args.input_path, input_type='auto')
raw_imgs, meta = load_inputs(fake)
log.info(f'device={args.device} type={meta["type"]} frames={len(raw_imgs)}')
# Detection -> patches (real torchvision person detector, on CPU).
dets, ratios = build_detector(device=args.device)(raw_imgs)
patches, det_meta = imgs_det2patches(raw_imgs, dets, ratios, 5)
log.info(f'patches={len(patches)}')
# Build pipeline — instantiates SKEL, so this is where a missing SKEL file fails.
pipe = build_inference_pipeline(model_root=model_root, device=args.device)
# Recovery loop (mirrors run_demo), timed.
t0 = time.time()
pd_params, pd_cam_t = [], []
for bw in asb(total=len(patches), bs_scope=args.rec_bs, enable_tqdm=True):
p = patches[bw.sid:bw.eid]
pn = ((p - IMG_MEAN_255) / IMG_STD_255).transpose(0, 3, 1, 2)
with torch.no_grad():
o = pipe(pn)
pd_params.append({k: v.detach().cpu().clone() for k, v in o['pd_params'].items()})
pd_cam_t.append(o['pd_cam_t'].detach().cpu().clone())
dt = time.time() - t0
pd_params = assemble_dict(pd_params, expand_dim=False)
pd_cam_t = torch.cat(pd_cam_t, dim=0)
poses = pd_params['poses']
fps = len(patches) / dt if dt else 0
log.info(f'RECOVERY ok: {len(patches)} patches in {dt:.2f}s ({fps:.2f} fps) '
f'poses={tuple(poses.shape)} nan={bool(torch.isnan(poses).any())}')
# Dump raw params regardless (this is the Path-B payload: SKEL rotations).
np.savez(out / 'params.npz', poses=poses.numpy(),
betas=pd_params['betas'].numpy(), cam_t=pd_cam_t.numpy())
log.info(f'params -> {out/"params.npz"}')
# Mesh geometry for visual QC in Blender (pyrender/EGL is dead headless on macOS, and
# Blender is the fleet renderer + Lane C's tool). prepare_mesh uses skel_model, no GL.
try:
m_skin, m_skel = prepare_mesh(pipe, pd_params)
np.savez(out / 'mesh_qc.npz',
skin_v=m_skin['v'].detach().cpu().numpy(), skin_f=np.asarray(m_skin['f']),
skel_v=m_skel['v'].detach().cpu().numpy(), skel_f=np.asarray(m_skel['f']))
log.info(f'mesh geometry -> {out/"mesh_qc.npz"} (import in Blender to eyeball)')
except Exception as e:
log.warning(f'prepare_mesh failed (params still saved): {e!r}')
log.info('smoke test complete.')
if __name__ == '__main__':
main()

View File

@ -0,0 +1,71 @@
#!/usr/bin/env python
"""Detector smoke check — runs TODAY, no SKEL, no HMR forward pass.
Proves the torchvision person detector (lib/modeling/pipelines/vitdet) imports without
detectron2 and honors the contract lib.kits.hsmr_demo._img_det2patches depends on.
cd ~/Documents/MOCAPGOD/.engine/HSMR && source .venv/bin/activate
PYTHONPATH=. python test_detector.py
"""
from pathlib import Path
import cv2
import torch
import _headless # noqa: F401 fake pyrender/OpenGL for headless macOS — MUST precede lib.* imports
from lib.modeling.pipelines.vitdet import build_detector # must import without detectron2
from lib.kits.hsmr_demo import imgs_det2patches # the real consumer of detector output
DEMO = Path('data_inputs/demo/example_imgs')
DET_THRESHOLD_SCORE = 0.5 # matches hsmr_demo._img_det2patches
def main():
fns = sorted(p for p in DEMO.glob('*')
if p.suffix.lower() in {'.jpg', '.jpeg', '.png', '.webp'})
assert fns, f'no demo images under {DEMO}'
imgs = []
for fn in fns:
bgr = cv2.imread(str(fn))
assert bgr is not None, f'cv2 failed to read {fn}'
imgs.append(cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)) # HSMR feeds RGB
dets, ratios = build_detector(device='cpu')(imgs)
assert len(dets) == len(imgs) == len(ratios), 'per-image list length mismatch'
n_person_imgs = 0
for fn, img, d, r in zip(fns, imgs, dets, ratios):
H, W = img.shape[:2]
assert set(d) == {'pred_classes', 'scores', 'pred_boxes'}, f'bad keys {set(d)}'
assert d['pred_classes'].dtype == torch.long
assert (d['pred_classes'] == 0).all(), 'human class must be 0 (CLASS_HUMAN_ID)'
assert d['pred_boxes'].ndim == 2 and d['pred_boxes'].shape[1] == 4, 'boxes must be (N,4)'
assert r == 1.0, 'native-resolution detection -> ratio 1.0'
strong = d['scores'] > DET_THRESHOLD_SCORE
if strong.any():
n_person_imgs += 1
x1, y1, x2, y2 = d['pred_boxes'][strong].unbind(1)
assert (x2 > x1).all() and (y2 > y1).all(), 'boxes must be lurb (x2>x1, y2>y1)'
assert (x1 >= -1).all() and (y1 >= -1).all() \
and (x2 <= W + 1).all() and (y2 <= H + 1).all(), 'box out of frame'
print(f'{fn.name}: {int(strong.sum())} person(s), '
f'top score {float(d["scores"][strong].max()):.2f}')
else:
print(f' · {fn.name}: no confident person')
# Every demo image has an obvious human; allow one miss for robustness slack.
assert n_person_imgs >= len(imgs) - 1, \
f'expected a person in ~all demo imgs, got {n_person_imgs}/{len(imgs)}'
print(f'DETECTOR OK: person found in {n_person_imgs}/{len(imgs)} demo images.')
# Close the loop: feed detector output through HSMR's real cropper (SKEL-free) and
# confirm it yields the 256x256 patches the HMR forward pass expects.
patches, det_meta = imgs_det2patches(imgs, dets, ratios, max_instances_per_img=5)
assert patches.ndim == 4 and patches.shape[1:] == (256, 256, 3), f'bad patch shape {patches.shape}'
assert patches.shape[0] == sum(det_meta['n_patch_per_img']), 'patch count / meta mismatch'
print(f'PATCHES OK: {patches.shape[0]} human patches @ 256x256 — ready for the HMR pass.')
if __name__ == '__main__':
main()

View File

@ -1,15 +1,17 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Lane A ingest — capture/inbox/*.{mp4,mov} → validate → 30fps h264 → rename → rsync to """Lane A ingest — capture/inbox/*.{mp4,mov} → validate → 30fps h264 → rename → pose queue
the m3ultra pose queue capture/LEDGER.tsv. capture/LEDGER.tsv.
python3 ingest.py # scan capture/inbox/ and ship everything python3 ingest.py # scan capture/inbox/ → local queue/ (the v1 default)
python3 ingest.py clip.mov --name dj_scratch_01 # one file, explicit clip name python3 ingest.py clip.mov --name dj_scratch_01 # one file, explicit clip name
python3 ingest.py --remote-queue # ship to the m3ultra queue instead (rsync)
python3 ingest.py --selfcheck # run built-in probe/validate self-test, no network python3 ingest.py --selfcheck # run built-in probe/validate self-test, no network
The pose engine (HSMR) currently runs on ultra, so ingest defaults to the LOCAL queue/ dir.
--remote-queue [user@host:/path] rsyncs to m3ultra instead (default target: MOCAP_REMOTE).
Rejects (portrait / malformed) move to capture/rejected/ with a reason in the ledger. Rejects (portrait / malformed) move to capture/rejected/ with a reason in the ledger.
Successful originals move to capture/done/; on rsync failure the normalized file is kept Successful originals move to capture/done/; on a failed remote ship the normalized file is
in capture/staged/ so a re-run retries the ship without re-encoding. kept in capture/staged/ so a re-run retries without re-encoding.
Remote defaults to m3ultra; override with MOCAP_REMOTE=user@host:/path.
""" """
import argparse, datetime, json, os, re, shutil, subprocess, sys, tempfile import argparse, datetime, json, os, re, shutil, subprocess, sys, tempfile
from pathlib import Path from pathlib import Path
@ -20,7 +22,9 @@ STAGED = ROOT / "capture" / "staged"
DONE = ROOT / "capture" / "done" DONE = ROOT / "capture" / "done"
REJECTED = ROOT / "capture" / "rejected" REJECTED = ROOT / "capture" / "rejected"
LEDGER = ROOT / "capture" / "LEDGER.tsv" LEDGER = ROOT / "capture" / "LEDGER.tsv"
REMOTE = os.environ.get("MOCAP_REMOTE", "m3ultra@100.89.131.57:~/Documents/MOCAPGOD/queue") QUEUE = ROOT / "queue" # v1 default: pose engine (HSMR) runs on ultra, next to the repo
# ponytail: v1 inference on ultra; migrate to m3ultra (--remote-queue) when a real throughput need appears
REMOTE_DEFAULT = os.environ.get("MOCAP_REMOTE", "m3ultra@100.89.131.57:~/Documents/MOCAPGOD/queue")
VIDEO_EXT = {".mp4", ".mov", ".m4v"} VIDEO_EXT = {".mp4", ".mov", ".m4v"}
HEADERS = ["utc", "original", "output", "fps_in", "dur_s", "resolution", "audio", "status", "note"] HEADERS = ["utc", "original", "output", "fps_in", "dur_s", "resolution", "audio", "status", "note"]
@ -90,13 +94,21 @@ def normalize(src, dst):
raise ValueError("ffmpeg normalize failed: " + tail) raise ValueError("ffmpeg normalize failed: " + tail)
def ship(staged): def deliver(staged, remote):
host, _, rpath = REMOTE.partition(":") """Put the normalized clip in the pose queue. Local queue/ dir by default; rsync to a
run(["ssh", host, f"mkdir -p {rpath}"]) # queue dir may not exist yet remote (user@host:/path) when --remote-queue is used. Returns the destination string."""
r = run(["rsync", "-az", str(staged), REMOTE + "/"]) if remote is None:
QUEUE.mkdir(parents=True, exist_ok=True)
dst = QUEUE / staged.name
shutil.move(str(staged), dst) # same-fs move; local delivery can't half-fail
return str(dst)
host, _, rpath = remote.partition(":")
run(["ssh", host, f"mkdir -p {rpath}"]) # remote queue dir may not exist yet
r = run(["rsync", "-az", str(staged), remote + "/"])
if r.returncode != 0: if r.returncode != 0:
tail = (r.stderr.strip().splitlines() or ["?"])[-1] tail = (r.stderr.strip().splitlines() or ["?"])[-1]
raise ConnectionError(f"rsync to {REMOTE} failed: {tail}") raise ConnectionError(f"rsync to {remote} failed: {tail}")
return f"{remote}/{staged.name}"
def ledger(row): def ledger(row):
@ -112,7 +124,7 @@ def slug(s):
return re.sub(r"[^A-Za-z0-9_-]+", "_", s).strip("_") or "clip" return re.sub(r"[^A-Za-z0-9_-]+", "_", s).strip("_") or "clip"
def process_one(src, name=None): def process_one(src, name=None, remote=None):
src = Path(src) src = Path(src)
out_name = f"{datetime.date.today():%Y%m%d}_{slug(name or src.stem)}.mp4" out_name = f"{datetime.date.today():%Y%m%d}_{slug(name or src.stem)}.mp4"
ts = datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") ts = datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
@ -139,9 +151,9 @@ def process_one(src, name=None):
res, audio = f"{info['w']}x{info['h']}", "yes" if info["audio"] else "no" res, audio = f"{info['w']}x{info['h']}", "yes" if info["audio"] else "no"
staged = STAGED / out_name staged = STAGED / out_name
try: try:
if not staged.exists(): # skip re-encode when retrying a failed ship if not staged.exists(): # skip re-encode when retrying a failed remote ship
normalize(src, staged) normalize(src, staged)
ship(staged) dest = deliver(staged, remote)
except (ValueError, ConnectionError) as e: except (ValueError, ConnectionError) as e:
ledger([ts, src.name, out_name, info["fps"], f"{info['dur']:.1f}", res, audio, "FAIL", str(e)]) ledger([ts, src.name, out_name, info["fps"], f"{info['dur']:.1f}", res, audio, "FAIL", str(e)])
print(f" ! {src.name}: {e} (left in inbox; staged kept for retry)") print(f" ! {src.name}: {e} (left in inbox; staged kept for retry)")
@ -152,7 +164,7 @@ def process_one(src, name=None):
staged.unlink(missing_ok=True) staged.unlink(missing_ok=True)
ledger([ts, src.name, out_name, info["fps"], f"{info['dur']:.1f}", res, audio, ledger([ts, src.name, out_name, info["fps"], f"{info['dur']:.1f}", res, audio,
"OK", "" if info["audio"] else "audio stripped (none present)"]) "OK", "" if info["audio"] else "audio stripped (none present)"])
print(f"{src.name}{REMOTE}/{out_name} ({res} {info['fps']}fps {info['dur']:.1f}s)") print(f"{src.name}{dest} ({res} {info['fps']}fps {info['dur']:.1f}s)")
return True return True
@ -174,24 +186,29 @@ def selfcheck():
def main(): def main():
ap = argparse.ArgumentParser(description="Lane A: ingest phone videos into the m3ultra pose queue.") ap = argparse.ArgumentParser(description="Lane A: ingest phone videos into the pose queue.")
ap.add_argument("video", nargs="?", help="single file to ingest (default: scan capture/inbox/)") ap.add_argument("video", nargs="?", help="single file to ingest (default: scan capture/inbox/)")
ap.add_argument("--name", help="clip name for the output (default: source filename)") ap.add_argument("--name", help="clip name for the output (default: source filename)")
ap.add_argument("--remote-queue", nargs="?", const=REMOTE_DEFAULT, default=None,
metavar="USER@HOST:PATH",
help=f"rsync to a remote pose queue instead of the local queue/ dir "
f"(default target: {REMOTE_DEFAULT})")
ap.add_argument("--selfcheck", action="store_true", help="run the built-in self-test and exit") ap.add_argument("--selfcheck", action="store_true", help="run the built-in self-test and exit")
args = ap.parse_args() args = ap.parse_args()
if args.selfcheck: if args.selfcheck:
return selfcheck() return selfcheck()
dest_label = args.remote_queue if args.remote_queue else str(QUEUE)
if args.video: if args.video:
return 0 if process_one(args.video, args.name) else 1 return 0 if process_one(args.video, args.name, args.remote_queue) else 1
INBOX.mkdir(parents=True, exist_ok=True) INBOX.mkdir(parents=True, exist_ok=True)
vids = sorted(p for p in INBOX.iterdir() if p.suffix.lower() in VIDEO_EXT) vids = sorted(p for p in INBOX.iterdir() if p.suffix.lower() in VIDEO_EXT)
if not vids: if not vids:
print(f"inbox empty: {INBOX}") print(f"inbox empty: {INBOX}")
return 0 return 0
print(f"ingesting {len(vids)} file(s) from {INBOX}{REMOTE}") print(f"ingesting {len(vids)} file(s) from {INBOX}{dest_label}")
n_ok = sum(process_one(p) for p in vids) n_ok = sum(process_one(p, remote=args.remote_queue) for p in vids)
print(f"done: {n_ok}/{len(vids)} ok") print(f"done: {n_ok}/{len(vids)} ok")
return 0 if n_ok == len(vids) else 1 return 0 if n_ok == len(vids) else 1

427
pose_engine.py Normal file
View File

@ -0,0 +1,427 @@
#!/usr/bin/env python3
"""Lane B (B2 + B3) pose engine — queue video → HSMR/SKEL → BVH@30fps + sidecar JSON.
# runs under the HSMR engine venv (torch/torchvision/skel live there):
.engine/HSMR/.venv/bin/python pose_engine.py # drain the queue/ dir
.engine/HSMR/.venv/bin/python pose_engine.py queue/clip.mp4 # one clip
python3 pose_engine.py --selfcheck # BVH math self-test, no engine
Pipeline per clip: person detection (torchvision, every-Nth-frame + box reuse single-subject
footage, this is the wall-clock win) single best box per frame HSMR ViT-H recovery SKEL
params (poses q46, betas, cam_t) SKEL forward for per-joint global orientations rigid-bone
BVH on SKEL's 24-joint skeleton + sidecar. Root is camera-space (root_space="camera"); Lane C
retargets onto mixamorig via spec/skel_to_mixamorig.json and grounds locomotion from foot contacts.
Output: out/<name>/motion.bvh + motion.json. Heartbeat: ~/.jobs/pose_engine.
"""
import argparse, datetime, json, os, subprocess, sys, time
from pathlib import Path
import numpy as np
from scipy.signal import savgol_filter
from scipy.spatial.transform import Rotation
REPO = Path(__file__).resolve().parent
ENGINE = REPO / ".engine" / "HSMR"
SKELDIR = ENGINE / "data_inputs" / "body_models" / "skel"
QUEUE = REPO / "queue"
OUT = REPO / "out"
JOB = Path.home() / ".jobs" / "pose_engine"
MODEL_TAG = "HSMR-ViTH-SKEL"
VERSION = "1.0"
EULER = "ZYX" # scipy intrinsic; BVH channels declared Z,Y,X to match
VIDEO_EXT = {".mp4", ".mov", ".m4v"}
def hb(msg):
JOB.parent.mkdir(parents=True, exist_ok=True)
JOB.write_text(f"{int(time.time())} {msg}\n")
print(f">>> [{datetime.datetime.now():%H:%M:%S}] {msg}", flush=True)
# ---------------------------------------------------------------- BVH math (engine-free) ----
def probe_fps(video, default=30.0):
r = subprocess.run(["ffprobe", "-v", "error", "-select_streams", "v:0",
"-show_entries", "stream=avg_frame_rate", "-of", "csv=p=0", str(video)],
capture_output=True, text=True)
try:
n, _, d = r.stdout.strip().partition("/")
f = float(n) / float(d or 1)
return round(f, 3) if f > 0 else default
except (ValueError, ZeroDivisionError):
return default
def smooth_rotations(W, win, poly):
"""SavitzkyGolay low-pass on a (F,3,3) global-rotation track, filtered as hemisphere-
aligned quaternions so it doesn't fight Euler wrap or quaternion double-cover."""
F = len(W)
q = Rotation.from_matrix(W).as_quat() # (F,4) xyzw
for i in range(1, F):
if float(q[i] @ q[i - 1]) < 0:
q[i] = -q[i]
w = _odd_window(win, F)
if w and w > poly:
q = savgol_filter(q, w, poly, axis=0)
q /= np.linalg.norm(q, axis=1, keepdims=True)
return Rotation.from_quat(q).as_matrix()
def smooth_xyz(P, win, poly):
w = _odd_window(win, len(P))
return savgol_filter(P, w, poly, axis=0) if (w and w > poly) else P
def _odd_window(win, n):
w = min(win, n if n % 2 else n - 1) # ≤ n, odd
return w if w >= 3 else 0 # too short to filter → skip
def locals_from_globals(W, parents):
"""BVH local rotations Q[j] = W[parent]^T · W[j] (root local == world)."""
Q = np.empty_like(W)
for j, p in enumerate(parents):
Q[j] = W[j] if p < 0 else W[p].T @ W[j]
return Q
def write_bvh(path, names, parents, offsets, root_pos, local_rot, fps):
"""SKEL-skeleton BVH. local_rot: (F,J,3,3) local rotations; root_pos: (F,3); offsets: (J,3)."""
F, J = local_rot.shape[:2]
children = {j: [c for c, p in enumerate(parents) if p == j] for j in range(J)}
ROT = "Zrotation Yrotation Xrotation" # order matches EULER='ZYX'
lines = []
def rec(j, depth):
pad = "\t" * depth
tag = "ROOT" if parents[j] < 0 else "JOINT"
lines.append(f"{pad}{tag} {names[j]}")
lines.append(f"{pad}{{")
ox, oy, oz = offsets[j]
lines.append(f"{pad}\tOFFSET {ox:.6f} {oy:.6f} {oz:.6f}")
if parents[j] < 0:
lines.append(f"{pad}\tCHANNELS 6 Xposition Yposition Zposition {ROT}")
else:
lines.append(f"{pad}\tCHANNELS 3 {ROT}")
for c in children[j]:
rec(c, depth + 1)
if not children[j]: # BVH needs an End Site for leaves
lines.append(f"{pad}\tEnd Site")
lines.append(f"{pad}\t{{")
lines.append(f"{pad}\t\tOFFSET 0.000000 0.100000 0.000000")
lines.append(f"{pad}\t}}")
lines.append(f"{pad}}}")
root = next(j for j, p in enumerate(parents) if p < 0)
lines.append("HIERARCHY")
rec(root, 0)
lines.append("MOTION")
lines.append(f"Frames: {F}")
lines.append(f"Frame Time: {1.0/fps:.6f}")
order_zyx = [0, 1, 2] # as_euler('ZYX') already returns Z,Y,X
dfs = [] # channel joint order = hierarchy DFS
def dfs_order(j):
dfs.append(j)
for c in children[j]:
dfs_order(c)
dfs_order(root)
for f in range(F):
row = []
for j in dfs:
zyx = Rotation.from_matrix(local_rot[f, j]).as_euler(EULER, degrees=True)[order_zyx]
if j == root:
px, py, pz = root_pos[f]
row += [f"{px:.6f}", f"{py:.6f}", f"{pz:.6f}"]
row += [f"{zyx[0]:.6f}", f"{zyx[1]:.6f}", f"{zyx[2]:.6f}"]
lines.append(" ".join(row))
Path(path).write_text("\n".join(lines) + "\n")
return dfs
def bvh_fk_positions(names, parents, offsets, root_pos, local_rot):
"""Reconstruct world joint positions from BVH data (verification / self-check)."""
F, J = local_rot.shape[:2]
pos = np.zeros((F, J, 3)); Wg = np.zeros((F, J, 3, 3))
for j, p in enumerate(parents):
if p < 0:
Wg[:, j] = local_rot[:, j]
pos[:, j] = root_pos + np.einsum("fab,b->fa", Wg[:, j], offsets[j])
else:
Wg[:, j] = Wg[:, p] @ local_rot[:, j]
pos[:, j] = pos[:, p] + np.einsum("fab,b->fa", Wg[:, p], offsets[j])
return pos
# ------------------------------------------------------------------ HSMR/SKEL (engine) ------
def build_engine(device, gender):
"""Load the engine once (imports, headless shims, SKEL body model). Detector + ViT-H pipeline
are built lazily on first recovery, so a --from-params re-run and the self-test skip them and
a queue amortizes the load across clips."""
os.environ.setdefault("PYTORCH_ENABLE_MPS_FALLBACK", "1")
sys.path.insert(0, str(ENGINE))
sys.path.insert(0, str(ENGINE / "thirdparty" / "SKEL"))
import _headless # noqa: F401 headless pyrender/OpenGL shims — before any lib.* import
from lib.kits.hsmr_demo import (
load_inputs, imgs_det2patches, build_inference_pipeline, build_detector,
asb, assemble_dict, IMG_MEAN_255, IMG_STD_255, DEFAULT_HSMR_ROOT,
)
import torch
from skel.skel_model import SKEL
from skel.kin_skel import skel_joints_name
os.chdir(ENGINE) # HSMR resolves model/cfg paths relative to its own root
skel = SKEL(gender=gender, model_path=str(SKELDIR)).eval()
return dict(load_inputs=load_inputs, imgs_det2patches=imgs_det2patches, asb=asb,
assemble_dict=assemble_dict, IMG_MEAN_255=IMG_MEAN_255, IMG_STD_255=IMG_STD_255,
torch=torch, device=device, skel=skel, skel_names=skel_joints_name,
_build_detector=build_detector, _build_pipe=build_inference_pipeline,
_hsmr_root=DEFAULT_HSMR_ROOT, det=None, pipe=None)
def _ensure_hmr(E):
if E["pipe"] is None: # build detector + ViT-H pipeline once, on first recovery
hb("build detector + HSMR ViT-H pipeline (once)")
E["det"] = E["_build_detector"](device=E["device"])
E["pipe"] = E["_build_pipe"](model_root=E["_hsmr_root"], device=E["device"])
def _single_subject_dets(det, frames, every, torch):
"""Detect on every-Nth frame, keep one best (largest, confident) box per frame, forward/
backward-fill gaps so patches align 1:1 with frames. Single-subject footage assumption."""
n = len(frames)
key = list(range(0, n, every))
kdets, _ = det([frames[i] for i in key])
best = []
for d in kdets:
s, b = d["scores"], d["pred_boxes"]
m = s > 0.5
if bool(m.any()):
bb = b[m]
area = (bb[:, 2] - bb[:, 0]) * (bb[:, 3] - bb[:, 1])
k = int(area.argmax())
best.append({"pred_classes": torch.zeros(1, dtype=torch.long),
"scores": bb.new_tensor([float(s[m][k])]), "pred_boxes": bb[k:k + 1]})
else:
best.append(None)
valid = [i for i, x in enumerate(best) if x is not None]
if not valid:
raise RuntimeError("no confident person detected in any sampled frame")
dets = []
for i in range(n):
ki = i // every
if best[ki] is None:
ki = min(valid, key=lambda v: abs(v - ki))
dets.append(best[ki])
return dets, [1.0] * n
def run_recovery(E, video, detect_every):
"""video → (poses (F,46), betas (F,10), cam_t (F,3)) with one pose per frame."""
torch = E["torch"]
_ensure_hmr(E)
fake = argparse.Namespace(input_path=str(video), input_type="auto")
frames, meta = E["load_inputs"](fake)
hb(f"detect (every {detect_every}f) over {len(frames)} frames")
dets, ratios = _single_subject_dets(E["det"], frames, detect_every, torch)
patches, det_meta = E["imgs_det2patches"](frames, dets, ratios, 1)
assert patches.shape[0] == len(frames), f"patch/frame misalign {patches.shape[0]}!={len(frames)}"
hb(f"recover {len(patches)} patches (HSMR ViT-H)")
params, cams = [], []
for bw in E["asb"](total=len(patches), bs_scope=64, enable_tqdm=True):
p = patches[bw.sid:bw.eid]
pn = ((p - E["IMG_MEAN_255"]) / E["IMG_STD_255"]).transpose(0, 3, 1, 2)
with torch.no_grad():
o = E["pipe"](pn)
params.append({k: v.detach().cpu().clone() for k, v in o["pd_params"].items()})
cams.append(o["pd_cam_t"].detach().cpu().clone())
params = E["assemble_dict"](params, expand_dim=False)
return params["poses"].numpy(), params["betas"].numpy(), torch.cat(cams, 0).numpy()
def skel_kinematics(E, poses, betas, cam_t):
"""SKEL forward on recovered params → posed joints/orientations + rest skeleton.
Returns names, parents(len J, root=-1), rest_offsets(J,3), root_pos(F,3), W(F,J,3,3)."""
torch = E["torch"]
skel = E["skel"]; skel_joints_name = E["skel_names"]
parents = np.concatenate([[-1], skel.parent.cpu().numpy()]).astype(int) # (J,), root=-1
P = torch.tensor(poses).float(); B = torch.tensor(betas).float(); T = torch.tensor(cam_t).float()
Rg, Jp = [], []
for s in range(0, len(P), 256): # batch to bound memory
with torch.no_grad():
o = skel(poses=P[s:s+256], betas=B[s:s+256], trans=T[s:s+256], skelmesh=False)
Rg.append(o.joints_ori.cpu().numpy()); Jp.append(o.joints.cpu().numpy())
Rg = np.concatenate(Rg); Jp = np.concatenate(Jp) # (F,J,3,3),(F,J,3)
with torch.no_grad(): # one rest skeleton from mean shape
r = skel(poses=torch.zeros(1, 46), betas=B.mean(0, keepdim=True),
trans=torch.zeros(1, 3), skelmesh=False)
rest_J = r.joints[0].cpu().numpy(); rest_R = r.joints_ori[0].cpu().numpy()
W = np.einsum("fjab,jcb->fjac", Rg, rest_R) # W[f,j] = Rg[f,j] @ rest_R[j]^T (camera frame)
root_pos = Jp[:, 0]
# HSMR emits camera space (X-right, Y-DOWN, Z-fwd — OpenCV image convention, verified: head.y <
# pelvis.y). Rotate 180° about X -> Y-up, character faces -Z, matching the treaty world axes.
R_CW = np.diag([1.0, -1.0, -1.0])
W = np.einsum("ab,fjbc->fjac", R_CW, W)
root_pos = root_pos @ R_CW.T
offsets = np.zeros((len(parents), 3))
for j, p in enumerate(parents):
offsets[j] = rest_J[j] if p < 0 else rest_J[j] - rest_J[p]
offsets[next(j for j, p in enumerate(parents) if p < 0)] = 0.0 # root at origin; abs pos in channels
return skel_joints_name, parents, offsets, root_pos, W
# --------------------------------------------------------------------- orchestration --------
def process(E, video, gender="male", detect_every=10,
filt_win=11, filt_poly=3, no_filter=False, from_params=False):
video = Path(video).resolve() # engine work chdir's into .engine/HSMR; keep paths absolute
name = video.stem
outdir = OUT / name; outdir.mkdir(parents=True, exist_ok=True)
pfile = outdir / "params.npz"
hb(f"START {name}")
if from_params and pfile.exists(): # re-run SKEL→BVH without the HMR pass (filter/axis tweaks)
z = np.load(pfile); poses, betas, cam_t = z["poses"], z["betas"], z["cam_t"]
hb(f"reuse params ({len(poses)} frames)")
else:
poses, betas, cam_t = run_recovery(E, video, detect_every)
np.savez(pfile, poses=poses, betas=betas, cam_t=cam_t) # raw payload for QC / re-processing
hb(f"skel forward ({len(poses)} frames)")
names, parents, offsets, root_pos, W = skel_kinematics(E, poses, betas, cam_t)
if not no_filter:
hb("jitter pass (savgol on rotations + root)")
W = np.stack([smooth_rotations(W[:, j], filt_win, filt_poly) for j in range(W.shape[1])], axis=1)
root_pos = smooth_xyz(root_pos, filt_win, filt_poly)
local_rot = np.stack([locals_from_globals(W[f], parents) for f in range(len(W))])
fps = probe_fps(video)
write_bvh(outdir / "motion.bvh", names, parents, offsets, root_pos, local_rot, fps)
# verify the BVH numerically encodes the motion we intended (rigid-bone tolerance)
recon = bvh_fk_positions(names, parents, offsets, root_pos, local_rot)
intended = _intended_positions(offsets, root_pos, W, parents)
err = float(np.abs(recon - intended).max())
assert err < 1e-3, f"BVH FK self-inconsistency {err:.4f}m (Euler/channel-order bug)"
sidecar = {
"fps": fps, "n_frames": int(len(local_rot)), "model": MODEL_TAG, "version": VERSION,
"gender": gender, "skeleton": "SKEL-24", "root_space": "camera", # HSMR is per-frame
"joint_order": list(names), "source_video": video.name,
"root_trajectory": np.round(root_pos, 5).tolist(),
"bvh_units": "metres, Y-up", "created": datetime.datetime.now().strftime("%Y-%m-%dT%H:%M:%S"),
"note": "camera-space root; Lane C grounds locomotion from foot contacts and retargets "
"to mixamorig via spec/skel_to_mixamorig.json",
}
(outdir / "motion.json").write_text(json.dumps(sidecar, indent=1))
hb(f"DONE {name}: {len(local_rot)} frames @ {fps}fps → {outdir} (FK err {err*1000:.1f}mm)")
return outdir
def _intended_positions(offsets, root_pos, W, parents):
"""World joint positions implied by the (unfiltered-or-filtered) global rotations W —
the ground truth the BVH must reproduce."""
F, J = W.shape[:2]
pos = np.zeros((F, J, 3))
root = next(j for j, p in enumerate(parents) if p < 0)
pos[:, root] = root_pos
order = _topo(parents)
for j in order:
p = parents[j]
if p >= 0:
pos[:, j] = pos[:, p] + np.einsum("fab,b->fa", W[:, p], offsets[j])
return pos
def _topo(parents):
order, seen = [], set()
def visit(j):
if j in seen:
return
if parents[j] >= 0:
visit(parents[j])
seen.add(j); order.append(j)
for j in range(len(parents)):
visit(j)
return order
def selfcheck():
"""Engine-free: a small tree + random global rotations → write BVH → parse-independent FK
reconstruction must match the intended positions. Proves Euler order channel declaration
FK are mutually consistent (the geometric crux)."""
import tempfile
rng = np.random.default_rng(0)
names = ["root", "spine", "head", "arm_l", "arm_r"]
parents = np.array([-1, 0, 1, 1, 1])
offsets = np.array([[0, 0, 0], [0, .5, 0], [0, .3, 0], [-.4, .1, 0], [.4, .1, 0]], float)
F = 20
W = np.stack([Rotation.random(len(names), random_state=rng).as_matrix() for _ in range(F)])
root_pos = np.cumsum(rng.normal(0, .01, (F, 3)), axis=0)
local = np.stack([locals_from_globals(W[f], parents) for f in range(F)])
d = Path(tempfile.mkdtemp())
dfs = write_bvh(d / "t.bvh", names, parents, offsets, root_pos, local, 30.0)
recon = bvh_fk_positions(names, parents, offsets, root_pos, local)
intended = _intended_positions(offsets, root_pos, W, parents)
err = float(np.abs(recon - intended).max())
assert err < 1e-9, f"BVH FK mismatch {err}"
assert (d / "t.bvh").read_text().count("JOINT") == 4 and dfs[0] == 0
# rotation round-trip in the declared order
R = Rotation.random(50, random_state=rng).as_matrix()
e = Rotation.from_matrix(R).as_euler(EULER, degrees=True)
back = Rotation.from_euler(EULER, e, degrees=True).as_matrix()
assert np.abs(back - R).max() < 1e-9
print(f"selfcheck OK — BVH FK exact ({err:.1e}m), Euler round-trip exact, hierarchy sane")
return 0
def main():
ap = argparse.ArgumentParser(description="Lane B pose engine: video → SKEL BVH + sidecar.")
ap.add_argument("video", nargs="?", help="single clip (default: drain queue/)")
ap.add_argument("--gender", default="male", choices=["male", "female"])
ap.add_argument("--device", default="mps")
ap.add_argument("--detect-every", type=int, default=20,
help="detect person every Nth frame; box reused between (single-subject footage)")
ap.add_argument("--filt-win", type=int, default=11, help="savgol window (odd); jitter pass")
ap.add_argument("--filt-poly", type=int, default=3, help="savgol polyorder")
ap.add_argument("--no-filter", action="store_true", help="skip the B3 jitter pass")
ap.add_argument("--from-params", action="store_true",
help="reuse out/<name>/params.npz, skip the HMR pass (filter/axis tweaks)")
ap.add_argument("--selfcheck", action="store_true", help="run the BVH-math self-test, no engine")
a = ap.parse_args()
if a.selfcheck:
return selfcheck()
kw = dict(gender=a.gender, detect_every=a.detect_every, filt_win=a.filt_win,
filt_poly=a.filt_poly, no_filter=a.no_filter, from_params=a.from_params)
video = Path(a.video).resolve() if a.video else None # resolve before build_engine chdir's away
E = build_engine(a.device, a.gender) # load models once; queue reuses them
if video:
process(E, video, **kw)
return 0
QUEUE.mkdir(parents=True, exist_ok=True)
vids = sorted(p for p in QUEUE.iterdir() if p.suffix.lower() in VIDEO_EXT)
if not vids:
print(f"queue empty: {QUEUE}")
return 0
hb(f"draining {len(vids)} clip(s) from {QUEUE}")
fails = 0
for v in vids:
try: # one bad clip must not sink the queue
process(E, v, **kw)
except Exception as e:
fails += 1
hb(f"FAIL {v.name}: {e}")
import traceback; traceback.print_exc()
hb(f"queue drained: {len(vids) - fails}/{len(vids)} clip(s) ok")
return 1 if fails else 0
if __name__ == "__main__":
sys.exit(main())

87
setup_hsmr.sh Executable file
View File

@ -0,0 +1,87 @@
#!/bin/bash
# HSMR pose-engine install on Apple Silicon (MPS) — Path B (local HSMR/SKEL).
# Clone -> uv venv(3.10) -> deps -> ungated weights -> apply engine_patches/ -> STOP at SKEL gate.
#
# ./setup_hsmr.sh full install (re-runnable; STOPS at the SKEL download gate)
# ./setup_hsmr.sh --patch re-apply engine_patches/ onto an existing clone and exit
#
# engine_patches/ holds our ARM/headless fixes (torchvision detector drop-in, headless
# pyrender/OpenGL shims, smoke + detector tests). HSMR itself is gitignored under .engine/,
# so a re-clone would drop those fixes — this script restores them. Override the clone
# location with HSMR_DIR=... (used by the durability test against a temp clone).
#
# ponytail: no `set -e` — optional deps (detectron2/chumpy/render) WARN and we still want the
# env + weights down so the smoke test can run and any failure is diagnosable.
set -uo pipefail
REPO="$(cd "$(dirname "$0")" && pwd)"
ENGINE="$REPO/.engine"
HSMR="${HSMR_DIR:-$ENGINE/HSMR}"
PATCHES="$REPO/engine_patches"
LOG="$ENGINE/setup.log"
JOB="$HOME/.jobs/hsmr_setup"
apply_patches(){
[ -d "$HSMR" ] || { echo "no HSMR clone at $HSMR — clone first"; exit 1; }
[ -d "$PATCHES" ] || { echo "no engine_patches/ at $PATCHES"; exit 1; }
( cd "$PATCHES" && find . -type f ! -name '.DS_Store' ) | sed 's|^\./||' | while read -r f; do
mkdir -p "$HSMR/$(dirname "$f")"
cp "$PATCHES/$f" "$HSMR/$f"
echo " patched $f"
done
}
# Standalone re-apply (after a manual HSMR re-clone).
if [ "${1:-}" = "--patch" ]; then
echo ">>> applying engine_patches/ -> $HSMR"
apply_patches
echo "patches applied."
exit 0
fi
mkdir -p "$ENGINE" "$HOME/.jobs"
exec > >(tee -a "$LOG") 2>&1
hb(){ echo "$(date +%s) $*" > "$JOB"; echo ">>> [$(date +%H:%M:%S)] $*"; }
hb "clone HSMR + submodules (SKEL)"
if [ ! -d "$HSMR/.git" ]; then
git clone --recurse-submodules https://github.com/IsshikiHugh/HSMR "$HSMR" || { hb "FAIL clone"; exit 1; }
else
git -C "$HSMR" submodule update --init || true
fi
cd "$HSMR" || { hb "FAIL cd"; exit 1; }
hb "uv venv python 3.10"
uv venv --python 3.10 .venv || { hb "FAIL venv"; exit 1; }
source .venv/bin/activate
hb "install torch + torchvision (MPS wheel)"
uv pip install torch torchvision || { hb "FAIL torch"; exit 1; }
hb "install requirements.txt"
uv pip install -r requirements.txt || hb "WARN requirements had issues"
hb "pin numpy<2 (chumpy/detectron2/smplx need 1.x)"
uv pip install "numpy==1.26.4"
# detectron2/chumpy are NOT needed — engine_patches replaces the detector and the SKEL runtime
# loads pkls without chumpy (see docs/B_ENGINE_SETUP.md). Left out on purpose; they don't build on ARM.
hb "pip install -e HSMR + thirdparty/SKEL"
uv pip install -e . || hb "WARN pip -e . failed"
uv pip install -e thirdparty/SKEL || hb "WARN SKEL editable install failed"
hb "download UNGATED weights from HuggingFace (regressors + HSMR ckpt + ViTPose backbone)"
hf download IsshikiHugh/HSMR-data_inputs \
--include "body_models/SMPL_to_J19.pkl" "body_models/J_regressor_SKEL_mix_MALE.pkl" "body_models/J_regressor_SMPL_MALE.pkl" "released_models/HSMR-ViTH-r1d1.tar.gz" "backbone/vitpose_backbone.pth" \
--local-dir data_inputs || hb "WARN hf download issue"
if [ -f data_inputs/released_models/HSMR-ViTH-r1d1.tar.gz ]; then
hb "extract HSMR checkpoint"
tar -xzf data_inputs/released_models/HSMR-ViTH-r1d1.tar.gz -C data_inputs/released_models/ && rm -f data_inputs/released_models/HSMR-ViTH-r1d1.tar.gz
fi
mkdir -p data_inputs/body_models/skel
hb "apply engine_patches/ (ARM detector, headless shims, tests)"
apply_patches
hb "DONE-ENV — SKEL gate: John drops skel_models_v1.1/ (v1.1.1) into data_inputs/body_models/skel/"
echo "=== pip freeze (key) ===" ; uv pip freeze | grep -iE 'torch|numpy|smplx|pytorch-lightning|timm' || true

View File

@ -0,0 +1,38 @@
{
"comment": "SKEL 24-joint -> mixamorig name map for Lane C's constraint retarget (pose_engine.py emits a SKEL-native BVH; Lane C maps it onto the character_kit mixamorig rig). null = a SKEL joint with no direct mixamorig target (twist/heel DoF folded elsewhere). mixamorig bones absent here (Spine1, Head, fingers, *_End) are not driven by SKEL — see notes.",
"source_skeleton": "SKEL-24 (skel/kin_skel.py order)",
"target_skeleton": "mixamorig (spec/mixamorig_bones.json, colon-prefixed, 65 joints)",
"skel_to_mixamorig": {
"pelvis": "mixamorig:Hips",
"lumbar_body": "mixamorig:Spine",
"thorax": "mixamorig:Spine2",
"head": "mixamorig:Neck",
"scapula_r": "mixamorig:RightShoulder",
"humerus_r": "mixamorig:RightArm",
"ulna_r": "mixamorig:RightForeArm",
"radius_r": null,
"hand_r": "mixamorig:RightHand",
"scapula_l": "mixamorig:LeftShoulder",
"humerus_l": "mixamorig:LeftArm",
"ulna_l": "mixamorig:LeftForeArm",
"radius_l": null,
"hand_l": "mixamorig:LeftHand",
"femur_r": "mixamorig:RightUpLeg",
"tibia_r": "mixamorig:RightLeg",
"talus_r": "mixamorig:RightFoot",
"calcn_r": null,
"toes_r": "mixamorig:RightToeBase",
"femur_l": "mixamorig:LeftUpLeg",
"tibia_l": "mixamorig:LeftLeg",
"talus_l": "mixamorig:LeftFoot",
"calcn_l": null,
"toes_l": "mixamorig:LeftToeBase"
},
"notes": {
"spine": "SKEL has 2 trunk joints (lumbar_body, thorax); mixamorig has 3 (Spine, Spine1, Spine2). lumbar_body->Spine, thorax->Spine2; Lane C interpolates Spine1 (or leaves identity).",
"neck_head": "SKEL 'head' is a single thorax-relative joint -> mapped to mixamorig:Neck. mixamorig:Head is left to Lane C (identity, or a small follow of Neck).",
"radius_calcn": "radius_* (forearm pronation/supination twist) and calcn_* (heel) have no direct mixamorig bone; their DoF fold into ForeArm/Foot in mixamorig. null = skip in the constraint map.",
"fingers": "SKEL has no fingers -> mixamorig hand/finger bones stay undriven; fingers arrive in Lane E1.",
"toe_end_headtop": "mixamorig *_End tips (HeadTop_End, LeftToe_End, RightToe_End) are leaf markers, not driven."
}
}