Compare commits

...

11 Commits

Author SHA1 Message Date
type-two
35403f8f05 Lane B: tick Lane B done, session-5 completion summary + Lane C handoff
PROGRESS.md: Lane B checked off with the session-5 outcome; SESSION 5 DONE block
(all 6 orders) + explicit Lane C handoff (BVH axes, skel_to_mixamorig map, camera
root_space, --from-params). B_ENGINE_SETUP.md: pose_engine.py usage + the engine
facts learned (SKEL 46-q, rest-aware FK ~4cm, Y-down camera fix, mobilenet detector).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 10:42:28 +10:00
type-two
7d0a3b7f7c Lane B / session-5 item 5: pose_engine.py — SKEL BVH + sidecar (B2) + jitter (B3)
pose_engine.py: queue video -> HSMR/SKEL -> BVH@fps + sidecar JSON per the treaty.
- detection: torchvision mobilenet (engine_patches), every-Nth frame + single best-box
  reuse (single-subject) -> ~1 pose/frame aligned timeline. mobilenet swap + detect-every
  address the CPU-detector wall-clock finding.
- recovery: HSMR ViT-H on MPS -> SKEL params (poses q46, betas, cam_t).
- SKEL forward -> per-joint global orientations; rigid-bone BVH via rest-aware FK
  (offset[j]=rest_J[j]-rest_J[parent]; W[j]=Rg[j]@rest_R[j]^T; local Q=W[parent]^T@W[j]).
  Verified: rest-aware FK reconstructs SKEL joints to ~4cm (residual = SKEL biomechanical
  coupled translations, which rigid BVH can't carry), in-BVH FK self-consistency 0.0mm.
- camera->world axis fix: HSMR emits Y-down (OpenCV, verified head.y<pelvis.y); diag(1,-1,-1)
  -> Y-up, faces -Z (treaty). Blender import: 24 bones, upright, 1.59m, motion present.
- B3 jitter: savgol on hemisphere-aligned quaternions (rotations) + root xyz separately.
- sidecar: root_space=camera (per-frame HSMR), joint_order, root_trajectory, model, fps.
- build models once + per-video try/except: queue of 6 drained unattended, 2 bad clips
  (no-person, garbage) failed gracefully, 4 produced valid BVH; heartbeat per clip.
- runtime steady-state ~1.6x realtime on ultra (detect+recover), under the <2x bar.
- --selfcheck (engine-free): BVH FK exact + Euler round-trip exact.

spec/skel_to_mixamorig.json: SKEL-24 -> mixamorig name map for Lane C's retarget
(20 mapped, 4 null twist/heel; validated against both skeletons).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 10:41:08 +10:00
type-two
0e2f7d0c12 Lane B / session-5 item 4: treaty amendment — required root_space sidecar field
motion.json sidecar gains REQUIRED root_space in {camera, world}. HSMR is
per-frame -> emits camera; Lane C reads it and derives a world root from foot
contacts for locomotion clips. PLAN treaty + Lane C task updated. v1 quality bar:
in-place clips first-class, traveling best-effort until a grounding pass exists.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 10:02:53 +10:00
type-two
0cf673aa82 Lane B / session-5 item 3: default ingest to local queue/ on ultra
The pose engine (HSMR) runs on ultra, so nothing was consuming the m3ultra queue
ingest.py shipped to. ingest.py now delivers to a local queue/ dir by default;
--remote-queue [user@host:/path] restores the m3ultra rsync path for when a real
throughput need appears (ponytail marker + PLAN Ground-rule-1 deviation noted).
CAPTURE.md updated. Verified: inbox drop -> local queue/, ledger row, selfcheck green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 10:02:16 +10:00
type-two
9db4fec988 Lane B / session-5 item 2: draft SKEL Path-A commercial-license email
Ready-to-send draft in docs/DECISIONS.md for John to send to ps-license@tue.mpg.de
(indie AU dev, in-pipeline use, SKEL files not redistributed — only retargeted
motion ships). Path B continues; a grant converts B->A with zero pipeline rework.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 09:35:39 +10:00
type-two
f8037f7b85 Lane B / session-5 item 1: durable-ize the ARM engine fixes into git
The torchvision detector drop-in, headless pyrender/OpenGL shims, and the
detector+smoke tests existed only on ultra's disk inside gitignored .engine/.
Captured byte-identical copies into engine_patches/ (mirrors the HSMR subtree)
and moved setup_hsmr.sh to the repo root (tracked, path-relative) with a
--patch step that overlays them onto the clone.

Acceptance PASSED: fresh shallow clone of HSMR in a temp dir + './setup_hsmr.sh
--patch' + validated venv -> test_detector.py green (7/7 demo imgs, 15 patches),
no hand edits. A re-clone can no longer silently drop the ARM fixes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 09:34:39 +10:00
type-two
60b49453b1 Lane B: SKEL v1.1.1 installed, smoke test PASSED (mps, 31fps recovery, no NaN); detector perf finding + mesh-dump cpu fix
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 09:22:09 +10:00
type-two
2dbd99579b Review: session-5 orders from Fable review of lanes 0-B
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 08:48:34 +10:00
type-two
4cc67b90fa Lane B: resolve decision gate -> Path B (local HSMR/SKEL, accept NC risk)
B1's MoCapAnything pick reversed (animal model, wrong for human capture);
license analysis recorded in LICENSES.md. John chose Path B: run HSMR->SKEL
locally on ultra (MPS), ship rig-motion only, accept the non-commercial
licence risk as interim until a commercial SKEL licence request goes out
(Path A, the clean upgrade).

Engine installed under .engine/ (gitignored): uv venv py3.10, torch 2.13
MPS-green, ungated weights, smoke_test.py with Apple-Silicon bypasses
(detectron2 stubbed with full-frame bbox, headless pyrender stubbed -> QC in
Blender) -- see docs/B_ENGINE_SETUP.md.

Remaining before B ships: John's gated SKEL v1.1.1 download, then the
forward-pass smoke test (MPS end-to-end, timing, pose sanity). Known caveat:
HSMR is per-frame so root is camera-space cam_t, NOT world-grounded -> Lane C
derives root from foot-contact.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 02:25:55 +10:00
type-two
fefb1f71ff Lane B: free m3ultra disk — deleted GLM-4.6-GGUF (147G) HF cache
Confirmed prune. Space pinned by OS-update snapshots (macOS 26.5.2
staged); realizes on next reboot. Updated DECISIONS.md Decision 2.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 01:29:08 +10:00
type-two
7819e12a5a Lane B / B1: pose-model bake-off + SMPL decision gate
Fresh SOTA survey (per PLAN — distrust the snapshot). Findings:
- MoCapAnything V2 (MIT code+weights, no SMPL, BVH-direct onto arbitrary
  rig) is the pick — sidesteps the SMPL commercial-license hard-stop and
  collapses the SMPL->mixamorig mapper + most of Lane C retarget.
- GVHMR: best accuracy but non-commercial code license -> reference only.
- GEM-X: commercial-clean + perfect output but CUDA-only -> off-fleet.

Stopped at the John-decision gate (docs/DECISIONS.md): confirm engine,
clear m3ultra disk (97% full), supply test footage. Inference eyeball,
B2 service-ize, B3 jitter remain.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 01:10:26 +10:00
16 changed files with 1369 additions and 43 deletions

1
.gitignore vendored
View File

@ -1,5 +1,6 @@
# weights + runtime data stay out of git
weights/
.engine/
capture/
queue/
out/

18
PLAN.md
View File

@ -18,6 +18,10 @@ critical path. Lanes EF are follow-ons. Do NOT start E/F before D ships.
training. Python via conda/uv there. MODELBEAST lives here; mocap becomes a sibling
service, same patterns (job dirs, batch keys — see `MESHGOD/scripts/mb_*` for the
client convention).
- **v1 deviation (session 5):** HSMR is installed and MPS-verified on **ultra**, not
m3ultra, so v1 runs inference on ultra and `ingest.py` defaults to a **local** `queue/`.
`# ponytail: v1 inference on ultra; migrate to m3ultra when a real throughput need appears`
(`ingest.py --remote-queue` already ships to m3ultra when that day comes).
- **ultra** (this Mac, M1 Ultra 128GB) — GLUE + DCC. Blender headless retarget/cleanup/
export, orchestration, Unreal verification. Owns the canonical repo.
- **JING5** (M5 MBP) — capture companion + portable inference later. Not in v1 critical path.
@ -47,8 +51,11 @@ critical path. Lanes EF are follow-ons. Do NOT start E/F before D ships.
- Skeleton: **mixamorig 65-joint** as defined by `~/Documents/character_kit`
(`female_game.glb` / male base). Canonical bone list generated in Lane 0.
- Intermediate motion format: **BVH at 30fps** + sidecar JSON
(`{fps, joint_order, root_trajectory, source_video, model, version}`).
BVH because Blender imports it natively and it's human-diffable.
(`{fps, joint_order, root_space, root_trajectory, source_video, model, version}`).
**`root_space` ∈ {`"camera"`, `"world"`} is REQUIRED** (session-5 amendment): HSMR is a
per-frame recoverer so it emits `"camera"`; Lane C reads this and, for locomotion clips,
derives a world root from foot contacts. BVH because Blender imports it natively and it's
human-diffable.
- Units metres, Y-up, character faces Z (three.js convention; Unreal yaw-fix happens
at UE import, as established in the A_3D_STORE experiments).
- Final artifact: FBX (anim-only, Mixamo-style "Without Skin") dropped into
@ -153,8 +160,11 @@ Tasks:
`mixamorig_` variants — this bit us before), bake to keyframes.
- Foot-contact fix: detect frames where foot velocity ≈ 0 & height < threshold IK
pin. `# ponytail: threshold heuristic, upgrade to learned contact detection in Lane E if it annoys`.
- Trim the T-pose bookends (Lane A protocol guarantees them), root-motion option
(in-place vs traveling) as a flag.
- Trim the T-pose bookends (Lane A protocol guarantees them). Root motion is driven by the
sidecar **`root_space`** (session-5 treaty): `"camera"` (HSMR) → in-place by default, or
derive a world root from foot contacts for traveling clips; a `--in-place`/`--traveling`
flag overrides. v1 quality bar: **in-place clips are first-class; traveling is best-effort**
until a proper grounding pass exists.
- Export anim-only FBX (Mixamo "Without Skin" style — merge_anims.py already eats these).
2. QC render: 3-second turntable mp4 of the clip on the female_game body, written next
to the FBX. **Verify with own eyes** (the rule) — every clip gets its mp4 looked at

View File

@ -6,13 +6,124 @@ One lane per session. Tick on completion. See PLAN.md for acceptance checks.
spec/mixamorig_bones.json (65 bones, `mixamorig:` colon-prefixed). — 2026-07-17
- [x] **Lane A** — capture + ingest. docs/CAPTURE.md protocol, ingest.py
(probe → 30fps normalize → rename → rsync to m3ultra queue → LEDGER.tsv). — 2026-07-17
- [ ] **Lane B** — pose engine (m3ultra). Model bake-off, service-ize, jitter pass.
- [x] **Lane B** — pose engine. **DONE session 5 (2026-07-17).** `pose_engine.py` ships:
queue video → HSMR/SKEL → SKEL-24 BVH@fps + `motion.json` sidecar (`root_space:"camera"`),
`spec/skel_to_mixamorig.json` name-map for Lane C. Verified on the demo clip: Blender import
= 24 bones, **upright** (1.59 m), motion present, FK self-consistency **0.0 mm**; queue of 6
drained unattended (2 bad clips failed gracefully, 4 valid BVH, heartbeat per clip); runtime
**~1.6× realtime** steady-state (under the <2× bar). mobilenet detector + detect-every-20 +
single-subject box reuse retired the CPU-detector wall-clock finding. Axis fix: HSMR emits
Y-down (OpenCV) → `diag(1,-1,-1)` → Y-up/faces-Z. **History (sessions 24):**
**Decision made session 3: Path B** — engine =
**HSMR → SKEL**, run local, ship rig-motion only, John accepts NC-licence risk
(`docs/DECISIONS.md` Decision 1 RESOLVED). Env up in `.engine/HSMR` (uv venv, torch 2.13
**MPS green**, ungated weights pulled). **Detector RESOLVED (session 4):** torchvision
Faster R-CNN drop-in replaces the unbuildable detectron2; `test_detector.py` is green today
(7/7 demo imgs → 15 patches, **no SKEL needed**). `smoke_test.py` now runs real detection +
patch-crop and reaches the **exact SKEL gate**; chumpy cleared as a non-blocker. **Only open
before B ships:** (1) John downloads gated **SKEL v1.1.1** so
`.engine/HSMR/data_inputs/body_models/skel/skel_male.pkl` (+ `skel_female.pkl`) exist;
(2) run the forward-pass smoke test (MPS / timing <2× / NaN check); then B2 (service-ize
BVH) + B3 (jitter). Lanes C and D remain blocked on B.
- [ ] **Lane C** — retarget + cleanup (Blender). bvh → mixamorig FBX + QC turntable.
- [ ] **Lane D** — clip factory. `mocap` CLI end-to-end; games + Unreal integration.
- [ ] Lane E — ML upgrades (post-v1, gated).
- [ ] Lane F — UX (optional).
## SESSION 5 — DONE (2026-07-17). All six orders executed; Lane B ships.
1. ✅ **Durable-ized** the ARM fixes → `engine_patches/` + `setup_hsmr.sh --patch`. Verified: fresh
clone + `--patch``test_detector.py` green, no hand edits.
2. ✅ **Path-A email** drafted in `docs/DECISIONS.md` (John sends to ps-license@tue.mpg.de).
3. ✅ **Queue reconciled**: `ingest.py` defaults to local `queue/` on ultra; `--remote-queue` keeps
the m3ultra path. PLAN Ground-rule-1 deviation noted; CAPTURE.md updated.
4. ✅ **Treaty amended**: `root_space` (camera|world) is a REQUIRED sidecar field; Lane C reads it.
5. ✅ **B2 + B3**: `pose_engine.py` (SKEL-24 BVH + sidecar + jitter) + `spec/skel_to_mixamorig.json`.
Detector swapped to mobilenet; detect-every-20 + box reuse; build-once + per-video isolation.
Ran the smoke test's SKEL forward on real params — no re-verify needed (smoke already PASSED).
**HANDOFF TO LANE C (retarget + cleanup, Blender, on ultra):**
- Input is `out/<name>/motion.bvh` (SKEL-24 skeleton, **Y-up, metres, faces Z**) + `motion.json`.
Import in Blender with `import_anim.bvh(axis_forward='-Z', axis_up='Y')` (verified upright).
- Constraint-map SKEL→mixamorig via **`spec/skel_to_mixamorig.json`** (20 mapped, 4 null=twist/heel;
Spine1 interpolated, Head follows Neck, fingers are Lane E). mixamorig canon: `mixamorig:` colon.
- **`root_space` is `"camera"`** → root is NOT world-grounded (HSMR is per-frame). In-place clips are
first-class; for traveling clips derive a world root from foot contacts (PLAN Lane C task).
- Rest pose is a clean T-pose at zero rotation; T-pose bookends only exist if John filmed the protocol
(demo clips don't have them) — Lane C's trim must tolerate their absence.
- Re-run the SKEL→BVH stage cheaply with `pose_engine.py <clip> --from-params` (skips HMR) when
tuning; raw params are saved at `out/<name>/params.npz`.
## SESSION 5 ORDERS — from Fable's review (2026-07-17, supersede older notes where they conflict)
Review verdict: Lanes 0/A solid; session 2's reversal of the B1 animal-model blunder was the
right call, well executed; ARM engineering verified independently (test_detector.py re-run
green 7/7 by reviewer). Proceed as follows, in order:
1. **Durable-ize the gitignored engine fixes (do first, ~30 min).** The torchvision detector,
`_headless.py`, `test_detector.py`, `smoke_test.py` exist ONLY on ultra's disk inside the
gitignored `.engine/`. Move canonical copies into git at `engine_patches/` and make
`setup_hsmr.sh` apply them after clone. Acceptance: fresh clone in a temp dir + script →
`test_detector.py` green with no hand edits.
2. **Draft the Path-A email (parallel risk-retirement, ~15 min).** Write the SKEL indie
commercial-license request to ps-license@tue.mpg.de into `docs/DECISIONS.md` (project
description, solo Australian dev, indie revenue, SKEL-in-pipeline-not-redistributed usage).
John sends it himself. Path B continues meanwhile; a granted license converts B→A with zero
rework. Do not wait on the reply.
3. **Reconcile the split-brain queue (~30 min).** Lane A's ingest.py rsyncs to m3ultra's queue,
but the engine (and MPS verification) is on ultra — nothing consumes that queue. For v1:
default ingest to LOCAL `queue/` on ultra (`--remote-queue` flag preserves the m3ultra path),
note the deviation from PLAN Ground-rule 1 in PLAN.md with a `# ponytail: v1 inference on
ultra; migrate to m3ultra when a real throughput need appears` marker. Update CAPTURE.md.
4. **Treaty amendment (small, load-bearing).** `motion.json` sidecar gains REQUIRED field
`root_space: "camera" | "world"`. HSMR emits `"camera"`. Lane C must read it and, for
locomotion clips, derive world root from foot contacts (already in notes). v1 quality bar:
in-place clips are first-class; traveling clips best-effort until a grounding pass exists.
**SMOKE TEST RAN 2026-07-17 (Fable, after John's SKEL v1.1.1 download landed — installed at
the expected path): PASSED with findings.** device=mps, 263 frames → 1315 patches, recovery
31.2 fps, **nan=False**, `data_outputs/smoke/params.npz` written. Findings for B2: (1) the
CPU Faster R-CNN detector is the wall-clock hog (~1 fps, 4min of the 5min run) — B2 MUST add
detect-every-Nth-frame + box reuse/IoU tracking (single-subject footage makes this easy) or
the mobilenet swap; HMR itself is faster than realtime so the <2× acceptance is reachable.
(2) `prepare_mesh` MPS→numpy crash fixed in smoke_test.py (`.detach().cpu()`) — carry the
same pattern into pose_engine.py. Item 5 below is now UNBLOCKED (skip re-verifying: smoke
is done; go straight to B2/B3).
5. **If (and only if) SKEL v1.1.1 files are in place** (`data_inputs/body_models/skel/
skel_male.pkl`): run smoke_test.py — MPS forward pass, runtime <2× realtime on demo video,
NaN check — then proceed straight into B2 (params.npz → `spec/skel_to_mixamorig.json` map →
BVH@30fps + sidecar per treaty) and B3 (One-Euro/Sav-Gol jitter pass, root smoothed
separately). If SKEL absent: finish items 14, verify Decision-3 footage status, stop clean.
**John's homework (all three block different things):**
- SKEL v1.1.1 download (the ONLY hard blocker; instructions in `docs/B_ENGINE_SETUP.md`).
- Film walk / dance / pick-up-object per `docs/CAPTURE.md`, drop in `capture/inbox/` (blocks
the honest quality verdict + all of B's acceptance).
- Reboot m3ultra when convenient (applies staged 26.5.2 security update; housekeeping).
## Notes for the next lane
- **Lane D can't be started until B + C ship + test footage exists** — it's the CLI that wraps
ingest → pose_engine.py → retarget.py, and the latter two don't exist yet. Session 3 (the
"Lane D" session) correctly built nothing; it locked Decision 1 and de-risked MPS instead.
- **SKEL version trap:** HSMR needs **v1.1.1** exactly (`changelog_v1.1.1.txt`); older = silent
incompat. Source: `.engine/HSMR/docs/SETUP.md` (register skel.is.tue.mpg.de → download.php).
- **Engine wiring + ARM bypasses live in `docs/B_ENGINE_SETUP.md`** — all three ARM landmines
are now cleared (session 4): detectron2 → **torchvision drop-in** (real detection, not a
full-frame stub), the pyrender/EGL fake is centralized in `.engine/HSMR/_headless.py`, and
chumpy is confirmed a non-blocker. `test_detector.py` proves the detector path green, SKEL-free.
- **World-grounded gap (important):** HSMR is per-frame → root is camera-space `cam_t`, NOT the
treaty's world-grounded trajectory. Fine for in-place DJ moves; walking/locomotion needs Lane C
to derive root from foot-contact (or a grounding pass). Carry into Lane C design.
- **HSMR is installed on ultra, not m3ultra** (PLAN Ground-Rule 1 puts the pose engine on
m3ultra). Likely because m3ultra's disk reclaim waits on the reboot (Decision 2). Next B
session decides: move it to m3ultra, or consciously host inference on ultra.
- **VOID** (was: "MoCapAnything is BVH-direct, skip the retarget rig"). Reversed — the model
is animal-only and not our engine. Whatever human model we pick emits **SMPL or SKEL, not
mixamorig**, so **Lane C's constraint-retarget IS required** and
`spec/{smpl,skel}_to_mixamorig.json` is back on the menu. HSMR's SKEL output (a biomechanical
skeleton) is the likely source format if John goes SKEL.
- **Check MPS + global-trajectory FIRST** for whichever human model John's decision unlocks
(HSMR has an HF Space + demo to smoke-test before John's footage). These two risks are
unchanged by the engine swap.
- Ground-truth skeleton naming is **`mixamorig:`** (colon), armature object named `Armature`.
Lane C's name-map must handle colon vs underscore variants (PLAN warned about this).
- Blender headless entrypoint: `/Applications/Blender.app/Contents/MacOS/Blender --background --python <script>`.

107
docs/B1_BAKEOFF.md Normal file
View File

@ -0,0 +1,107 @@
# B1 — pose-model bake-off
**Status: SUPERSEDED 2026-07-17 (session 2).** The selection below (MoCapAnything V2) was a
**category error** — MoCapAnything is an **animal** mocap model (71 animal targets, no
human/biped/mixamorig, no fingers), so it cannot be MOCAPGOD's human v1 engine. Its "SMPL gate
dissolved" conclusion is therefore wrong; the gate is **LIVE**. See `docs/DECISIONS.md` for the
corrected, license-verified decision. The candidate research below is kept as history — its
*human*-model findings (GVHMR non-commercial, GEM-X CUDA-only) still stand; its recommendation
does not.
The PLAN told this lane to distrust the July-2024 snapshot and search fresh. It paid off:
two models that post-date the snapshot change the whole calculus — one of them **removes
the SMPL licensing hard-stop the plan was built around.**
## Selection criteria (PLAN order)
1. permissive code license (must ship in commercial games)
2. world-grounded global trajectory (not pose-in-place)
3. runs on Apple Silicon (m3ultra — MPS or acceptable CPU; no NVIDIA on the fleet)
4. SMPL-dependency status (the legal rail)
## Candidate matrix
| Model | World-grounded | Code lic. | Weights lic. | SMPL? | Apple Silicon | Output | Commercial-ship |
|---|---|---|---|---|---|---|---|
| **MoCapAnything V2** ⭐ | trajectories (global vs camera **unverified**) | MIT | **MIT** | **No** | standard PyTorch, no custom CUDA ops → MPS-portable *with effort* | **BVH rotations onto an arbitrary target rig** (feed it mixamorig) + hands | **YES (MIT/MIT)** |
| GVHMR | Yes — SOTA, beats WHAM 1828% WA-MPJPE | **research-only** | needs SMPL | Yes | plain PyTorch, MPS-portable w/ effort | SMPL params (rot) + global traj (.pt) | **NO** — code is non-commercial *and* SMPL-gated |
| GEM-X (NVIDIA) | Yes | Apache-2.0 | NVIDIA Open Model Lic. (NVIDIA states commercial-ready) | **No** (SOMA body model) | **CUDA 12.1+ only, NVIDIA arch** — will not run on the fleet | 77-joint axis-angle + world root (m), incl. hands | Yes, but **off-fleet** |
| WHAM | Yes (autoregressive; GVHMR's predecessor) | research/NC (not re-verified) | needs SMPL | Yes | PyTorch, MPS-portable | SMPL + traj | NO (gate) — dominated by GVHMR |
| TRAM | Yes (SLAM-based camera+human) | mixed; +DROID-SLAM | needs SMPL | Yes | heavier (SLAM stage) | SMPL + traj | NO (gate) — heaviest, not pursued |
Sources verified this session: GVHMR repo LICENSE (custom research-only, "prohibited for
commercial use", contact xwzhou@zju.edu.cn); GEM-X model card (Apache code / NVIDIA Open
Model License weights / SOMA / CUDA-only); MoCapAnything V2 weights card
(`kehong/MoCapAnythingV2-weights`, **License: mit**, author Kehong Gong = paper author) and
code repo `github.com/phongdaot/MocapAnything` (MIT). WHAM/TRAM licenses not re-verified —
both are SMPL-gated and dominated by GVHMR on the world-grounded metric, so not installed.
## The two findings that matter
**1. GVHMR (the accuracy leader) is double-blocked for us.** Its own code license is
non-commercial — that's a wall *independent of* SMPL, and the joints-only Path-B from the
PLAN can't get around a non-commercial *code* license. John's clips ship in commercial
games. GVHMR is therefore a research/accuracy *reference* only, unless John negotiates a
paid ZJU commercial license. It is **not** a v1 shipping engine.
**2. MoCapAnything V2 dissolves the SMPL gate.** MIT code + MIT weights, commercial use
permitted, **no SMPL files anywhere** in the pipeline or dependency list (torch/opencv/
trimesh only). It is category-agnostic: you give it a monocular video **plus the target
rig as a prompt**, and it emits a rotation-based BVH that drives *that rig* directly, via
constraint-aware IK. Cross-species too (it retargets across heterogeneous rigs — a free
preview of the Lane E4 centaur work).
### Why #2 is bigger than "gate cleared"
If we prompt MoCapAnything with the **mixamorig** skeleton (`spec/mixamorig_bones.json` /
`female_game.glb`), it emits **mixamorig BVH directly**. That means:
- `spec/smpl_to_mixamorig.json` (B2 task 2) is **unnecessary** — there is no SMPL
intermediate to map.
- Lane C's core constraint-retarget (bvh→mixamorig) **largely collapses** — the BVH is
already on-skeleton. Lane C shrinks to: import → foot-contact fix → jitter → trim →
export FBX.
The fresh search didn't just clear the legal hold-up; it deleted a spec file and most of a
lane. `# ponytail win: don't build the SMPL→mixamorig mapper or the Lane C retarget rig
until we've confirmed MoCapAnything can't hit mixamorig directly.`
**Caveat (honest):** getting it to emit *exactly* the treaty's mixamorig (joint order,
Y-up, faces Z, bone-roll) will still need a rig-prompt config and a convention
reconciliation pass. "Collapses" = "greatly simplifies," not "free."
## Recommendation
**Primary: MoCapAnything V2.** Only candidate that is commercial-clean AND on-fleet AND
BVH-direct. Accept two risks, both verifiable cheaply at inference (below).
**Reference (optional): GVHMR** under its research license, purely to benchmark how much
accuracy we're leaving on the table — informs whether a paid ZJU license is ever worth it.
**Shelved: GEM-X** — revisit only if the fleet ever gains an NVIDIA GPU (it's the cleanest
option the day CUDA is available; zero-cloud rules out renting it).
## Must-verify at the inference eyeball (the other half of B1)
These are why the lane isn't ticked. Cheap to settle — MoCapAnything ships a ~160 MB sample
dataset and an HF demo Space, so we don't even need John's footage to answer them:
1. **Global trajectory?** Treaty requires world-space root motion. MoCapAnything says
"temporally coherent trajectories" — confirm the character *travels through the world*,
not runs-in-place. If camera-space only, we need a grounding pass (or fall back to GVHMR
for locomotion clips). **This is the #1 risk.**
2. **MPS reality.** Standard PyTorch predicts yes, but the PLAN's rule is "check MPS early —
it decides everything." Expect some ops to need CPU fallback; time it (<2× realtime is
the bar).
3. **Quality.** The linked code is a re-implementation ("use as reference, not
reproduction"). Eyeball walk / dance / pick-up-object in Blender before trusting it.
## Preconditions before the install/inference session
- **m3ultra disk: 29 GB free of 926 GB (97% full).** Marginal for one torch env + weights;
too tight for two. Clear space first (see `DECISIONS.md`).
- **Test footage:** inbox is empty. Sample data covers the smoke test; John's walk / dance /
pick-up-object clips are needed for the real quality call and B2/B3/acceptance.
- **Engine decision:** confirm MoCapAnything-primary (vs. paying for GVHMR-grade). See
`DECISIONS.md`.
## Sources
- [GVHMR](https://github.com/zju3dv/GVHMR) · [GVHMR paper](https://arxiv.org/html/2409.06662v1)
- [NVIDIA GEM-X](https://huggingface.co/nvidia/GEM-X)
- [MoCapAnything project](https://animotionlab.github.io/MoCapAnything/) ·
[V2 weights (MIT)](https://huggingface.co/kehong/MoCapAnythingV2-weights) ·
[code](https://github.com/phongdaot/MocapAnything) ·
[paper](https://arxiv.org/abs/2512.10881)
- [TRAM](https://github.com/yufu-wang/tram)

87
docs/B_ENGINE_SETUP.md Normal file
View File

@ -0,0 +1,87 @@
# Lane B engine — HSMR install & smoke-test state (session 3, 2026-07-17)
Engine per Decision 1 = **Path B: HSMR → SKEL**, run local on **ultra**, MPS. Installed under
`.engine/HSMR` (gitignored). This doc = how it's wired + how to run once SKEL lands.
## State
- ✅ `uv venv` py3.10 at `.engine/HSMR/.venv`; torch **2.13.0**, `mps.is_available() == True`.
- ✅ Ungated weights in `.engine/HSMR/data_inputs/`: HSMR-ViTH ckpt
(`released_models/HSMR-ViTH-r1d1/checkpoints/hsmr.ckpt`), ViTPose backbone, 3 regressors.
- ✅ **Real person detector** (torchvision Faster R-CNN, no detectron2) — checked by
`.engine/HSMR/test_detector.py`, green **today with no SKEL**: 7/7 demo images → 15 patches
@256×256 through HSMR's own cropper.
- ✅ Smoke-test runner `.engine/HSMR/smoke_test.py`**PASSED 2026-07-17** (session 5): SKEL
v1.1.1 in place, device=mps, 263 frames → 1315 patches, recovery 31.2 fps, **nan=False**,
`data_outputs/smoke/params.npz` written (poses (N,46), betas (N,10), cam_t (N,3)).
- ✅ Install script (re-runnable): **`./setup_hsmr.sh`** at the repo root (tracked in git).
`./setup_hsmr.sh --patch` re-applies the ARM/headless fixes onto an existing/re-cloned HSMR.
- ✅ **The ARM fixes are now durable in git** (session 5, was: "on ultra's disk only"). Canonical
copies live in **`engine_patches/`** (`_headless.py`, `test_detector.py`, `smoke_test.py`,
`lib/modeling/pipelines/vitdet/__init__.py`); `setup_hsmr.sh` overlays them onto the gitignored
`.engine/HSMR` clone. Verified: fresh clone + `--patch``test_detector.py` green, no hand edits.
## Apple-Silicon blockers — all three resolved (session 4, "Lane B detector")
- **detectron2 won't build on ARM Mac****RESOLVED.** Replaced its whole ViTDet detector with
a torchvision Faster R-CNN drop-in in `lib/modeling/pipelines/vitdet/__init__.py` — same
`build_detector(...) -> (dets, downsample_ratios)` contract, torchvision's person class (1)
remapped to HSMR's human id (0), boxes as lurb, ratio 1.0. This also killed a **load-time
landmine**: the old `import detectron2` there exploded anything importing `lib.kits.hsmr_demo`
(why session 3 had to stub it). Detector pinned to **CPU** — cheap next to the ViT-H pass and
it dodges MPS RoIAlign/NMS gaps; an in-file `# ponytail` note marks the mobilenet swap if
per-frame detection ever dominates on long videos.
- **pyrender is dead headless on macOS** (`py_renderer` forces `PYOPENGL_PLATFORM=egl`, no EGL on
Mac; `pyrender` is even referenced in a type annotation, so a plain import-guard won't do).
Centralized the permissive `pyrender`/`OpenGL` fake into **`.engine/HSMR/_headless.py`** — every
entrypoint does `import _headless` before any `lib.*` import. QC still happens in Blender (fleet
renderer + Lane C's tool); the runner dumps `params.npz` (SKEL poses/betas/cam_t — the Path-B
payload) and `mesh_qc.npz` (verts/faces for Blender).
- **chumpy failed to build****NOT a blocker (verified).** The runtime `SKELWrapper` loads its
pkls with plain `pickle.load(encoding='latin1')` — no chumpy import. chumpy only appears in
SKEL's offline model-*build* script (`merge_smplh_mano.py`), which we never run. Ignore the WARN.
## John's one step — drop the SKEL files (login + licence, so it's yours to do)
Register **skel.is.tue.mpg.de** → Download → **"SKEL and BSM models"** → `skel_models_v1.1.zip`
(**must be v1.1.1** — check `changelog_v1.1.1.txt` inside; older = silent incompatibility).
Unzip, then:
```
mv /path/to/skel_models_v1.1 ~/Documents/MOCAPGOD/.engine/HSMR/data_inputs/body_models/skel
```
so that `…/data_inputs/body_models/skel/skel_male.pkl` exists.
## Run the smoke test (once SKEL is in place)
```
cd ~/Documents/MOCAPGOD/.engine/HSMR && source .venv/bin/activate
PYTHONPATH=. python smoke_test.py -i data_inputs/demo/example_videos/gymnasts.mp4 # or .../example_imgs
```
Outputs → `data_outputs/smoke/`: `params.npz` (rotations), `mesh_qc.npz` (Blender QC), and
console device/fps/NaN. Answers the Lane B smoke questions: **MPS forward OK? timing (<2×
realtime)? pose sane?**
⚠️ HSMR is a **per-frame** recoverer → root is **camera-space `cam_t`, not world-grounded**.
Fine for in-place DJ moves; locomotion (walk) needs Lane C's foot-contact root derivation or a
grounding pass. This is the treaty's global-trajectory caveat, carried into Path B.
## Run the pose engine (B2/B3, session 5) — the real service
`pose_engine.py` (repo root, tracked) wraps the engine into the pipeline. Run under the engine venv:
```
cd ~/Documents/MOCAPGOD
.engine/HSMR/.venv/bin/python pose_engine.py # drain queue/ → out/<name>/
.engine/HSMR/.venv/bin/python pose_engine.py queue/clip.mp4 # one clip
.engine/HSMR/.venv/bin/python pose_engine.py <clip> --from-params # re-run SKEL→BVH, skip HMR
.engine/HSMR/.venv/bin/python pose_engine.py --selfcheck # BVH-math self-test, no engine
```
Per clip → `out/<name>/`: `motion.bvh` (SKEL-24, **Y-up, metres, faces Z**), `motion.json`
(sidecar, `root_space:"camera"`), `params.npz` (raw poses/betas/cam_t for QC / `--from-params`).
Heartbeat: `~/.jobs/pose_engine`. Models build **once** per invocation (queue amortizes); a bad
clip is logged and skipped, not fatal.
Key engine facts learned this session (for whoever touches B again):
- **SKEL emits `poses` = (F, 46)** biomechanical q, NOT SMPL's 72 axis-angles. Don't decode q by
hand — run `SKEL.forward()` and consume `joints_ori` (F,24,3,3 global orientations) + `joints_tpose`.
- BVH is rigid-bone: rest-aware FK (`offset=rest_J[j]-rest_J[parent]`, `W=Rg@rest_R^T`,
`local=W[parent]^T@W[j]`) reconstructs SKEL joints to ~4 cm; the residual is SKEL's coupled
knee/ankle translations, which a rotation-only BVH cannot carry (accept it).
- **HSMR camera frame is Y-DOWN** (OpenCV; verified head.y < pelvis.y). `pose_engine` applies
`diag(1,-1,-1)` to reach the treaty's Y-up / faces-Z. Undo/adjust there if a future model differs.
- Detector is **mobilenet** now (was R50) — CPU detection was the wall-clock hog. It self-caps input
to ~640 px internally, so pre-downscaling does nothing; `--detect-every` is the real throughput knob.

View File

@ -25,8 +25,9 @@ uses to auto-trim the clip. No bookends → manual trimming.
- **AirDrop** the clip from the phone to **ultra**, save into
`~/Documents/MOCAPGOD/capture/inbox/`.
- Then run **`python3 ingest.py`** (or let the Lane D `mocap` CLI do it). Ingest validates,
normalizes to 30fps, and ships it to the m3ultra queue. Rejects (portrait, no feet, garbage)
land in `capture/rejected/` with a reason in `capture/LEDGER.tsv` — check there if a clip vanishes.
normalizes to 30fps, and puts it in the local pose queue (`queue/` on ultra, where the engine
runs). Rejects (portrait, no feet, garbage) land in `capture/rejected/` with a reason in
`capture/LEDGER.tsv` — check there if a clip vanishes. (`--remote-queue` ships to m3ultra instead.)
## Common rejections (and the fix)
| Symptom | Fix |

139
docs/DECISIONS.md Normal file
View File

@ -0,0 +1,139 @@
# Decisions needing John
**Status 2026-07-17 (session 2): Lane B1's engine pick is REVERSED** (Decision 1). The disk
work (Decision 2) from the concurrent session is preserved intact. Corrected, license-verified
analysis below + `LICENSES.md`. Prior "SMPL hard-stop avoided / go MoCapAnything" memo
superseded (git @ 7819e12).
## Decision 1 — pose engine: B1 picked an ANIMAL model for a HUMAN job; the gate is LIVE
> **RESOLVED 2026-07-17 — John chose Path B (run local, accept NC risk).** Engine = **HSMR
> → SKEL** on the fleet. Ship only the retargeted mixamorig motion (no Max-Planck file
> redistributed); John explicitly accepts the judgment call that authoring game clips from an
> NC-licensed model is his risk. Path A (email MPI for a commercial licence) stays on the shelf
> as the clean upgrade if that risk ever needs retiring. **MPS verified green on ultra this
> session** (torch 2.13.0, `mps_available=True`, `mps_built=True`). **Sole remaining blocker to
> a first forward pass: the gated SKEL download — John's manual step** (register at
> skel.is.tue.mpg.de → download.php → `skel_models_v1.1.zip`, **must be v1.1.1**, unzip into
> `.engine/HSMR/data_inputs/body_models/skel/` per HSMR `docs/SETUP.md`). Behind a login +
> NC-licence agreement, so it is John's to do — I can't create the account or accept the terms.
> Analysis below is retained for the record.
B1 chose **MoCapAnything V2** for being MIT/MIT with no SMPL, and concluded "the SMPL hard-stop
is avoided." Reading the actual repo this session shows why that's wrong: MoCapAnything is an
**animal** motion-capture system — **71 animal retarget targets** (Alligator…Tyranno), trained
on `zoo1030` (animals) + `obj1k` (objects), **no human/biped/mixamorig reference and no
fingers**. MOCAPGOD's v1 job is capturing **John (a human)** onto mixamorig (PLAN line 1; the
Lane B brief says "monocular **human** motion recovery"). MoCapAnything didn't dissolve the SMPL
gate — it swapped in a task the gate doesn't cover. **Wrong tool for the human path.** (It's a
clean, strong tool for *creature* NPCs — option C.)
**Verified from license text this session (links/dates in `LICENSES.md`):**
| Human-capable option | Code license | Body model | Body-model license | On the fleet? |
|---|---|---|---|---|
| GVHMR / WHAM / TRAM / OnlineHMR (accuracy leaders) | research/NC (GVHMR NC confirmed) | **SMPL** | NC — commercial via Meshcapade (quote) | PyTorch→MPS ✓ |
| **HSMR** (CVPR25 oral) / SKEL-CF (Nov 2025) | **MIT** (HSMR) | **SKEL** | NC — commercial via MPI `ps-license@tue.mpg.de` | PyTorch→MPS ✓ (unverified) |
| GEM-X (NVIDIA) | Apache | SOMA (not SMPL) | commercial-ready | **CUDA-only — OFF fleet** ✗ |
| MoCapAnything V2 | MIT | none | — (clean) | ✓ but **animal-only** ✗ for humans |
**No free-lunch, commercial-clean, on-fleet, world-grounded *human* model exists.** SMPL and
SKEL are both Max-Planck research licenses with an "email for commercial" path; the one
clean-licensed human model (GEM-X) needs NVIDIA hardware the fleet lacks. Pick a direction:
**A. License a body model, use a proven human model — RECOMMENDED.** Email for an indie
commercial license. Primary: **SKEL → HSMR** (code is MIT, outputs a biomechanical **skeleton**
ideal for the mixamorig retarget) via `ps-license@tue.mpg.de`. Fallback: **SMPL** via
`sales@meshcapade.com` (unlocks WHAM/TRAM, though their *code* licenses also need clearing).
One email + likely a modest fee + a few days. Clean and shippable. *I can draft the email; I
can't send it.*
**B. SMPL-free-outputs path (PLAN Path B) — run locally, ship only your rig's motion.** HSMR's
code is MIT: run on the fleet, extract joint **rotations**, retarget onto YOUR mixamorig, ship
only that — distribute no Max-Planck file. Zero cost, unblocks today. **Risk:** the NC licenses
forbid "production of artefacts for commercial purposes"; whether authoring game clips counts is
a legal judgment call. The clean mitigation (email MPI to ask) collapses into A. John-accepts-risk.
**C. Reconsider scope — creatures now, defer the human license.** MOCAPGOD's downstream is
stylized NPCs + a centaur RPG; some targets aren't human. If the near-term need is creature
NPCs, **MoCapAnything (MIT, clean, on-fleet) does that TODAY**; defer the human-capture license
until you need your own DJ clips.
**Recommendation A.** Honest, cheap, converts B's grey into a clear yes. In parallel I keep
MoCapAnything as the sanctioned *creature* engine (free upside, blocks nothing) and verify
HSMR's MPS-runnability so we're ready when a license lands. I will **not** build the human v1
pipeline on an NC model without your explicit go (that's B).
## Path A — SKEL commercial-licence email (READY TO SEND — John sends it himself)
Parallel risk-retirement (session-5 order 2). Path B ships meanwhile; if MPI grants a licence
the grey area in Decision 1 disappears with **zero pipeline rework** (same HSMR/SKEL engine).
**Do not wait on the reply.** John: fill the two `[brackets]`, send from your address to
`ps-license@tue.mpg.de`. I can't send email or create the account — this is yours.
> **To:** ps-license@tue.mpg.de
> **Subject:** Commercial licence enquiry — SKEL body model (indie game developer)
>
> Dear Perceiving Systems licensing team,
>
> I'm a solo independent game developer based in Australia. I'd like to enquire about a
> commercial licence for the **SKEL body model** (v1.1.1, from skel.is.tue.mpg.de).
>
> **Use case.** I'm building a small in-house tool that recovers 3D human motion from ordinary
> single-camera video of myself, to author animation clips for stylised characters in my own
> indie games. SKEL is used inside the pipeline at inference time (via the HSMR pose-recovery
> model) to estimate body pose.
>
> **Redistribution: none.** The SKEL/BSM model files stay on my own machine — I do not
> redistribute them or any Max-Planck-derived asset. What ships in the games is only the
> resulting joint **motion**, retargeted onto my own separately-authored character skeletons
> (a Mixamo-style rig). No SKEL mesh, shape space, or model file is included or exposed in any
> shipped product.
>
> **Scale.** Small and indie: modest commercial revenue, a single developer, no external funding.
>
> Could you let me know whether an **indie / small-developer commercial licence** is available
> for this kind of in-pipeline, non-redistributed use, and the terms and any fee? I'm happy to
> provide more detail on the pipeline or sign whatever agreement you need.
>
> Thank you,
> [your name]
> monsterrobot.party · [contact email]
If MPI replies with terms, record the outcome here and flip Decision 1 from Path B to Path A
(nothing in the pipeline changes — same engine, the licence just retires the NC risk).
## Decision 2 — clear m3ultra disk (RESOLVED pending reboot, 2026-07-17)
Was 98% full (20 GB free). Deleted `unsloth/GLM-4.6-GGUF` (147 GB, re-downloadable, 0 code
refs) from the HF cache with John's OK. **But `df` still shows 20 GB** — the freed blocks are
pinned by macOS **OS-update snapshots**: a **macOS Tahoe 26.5.2 update is staged** (recommended
/ security, "Action: restart") and the box has 18 days uptime. Space realizes on **next
reboot**, which also applies the pending update. No `~/.jobs` heartbeat job is running (load is
just the Claude desktop app), so a reboot kills nothing tracked.
**John: reboot m3ultra when convenient** → applies 26.5.2 + reclaims 147 GB (→ ~167 GB free,
ample for Lane B). Not doing this myself — it's your workstation. Kept the FLUX variants
(you deselected them); another ~200 GB is prunable there later if wanted.
## Decision 3 — test footage (blocker for the real quality call)
`capture/inbox/` is empty. Whichever human engine Decision 1 unlocks, the honest
walk/dance/pick-up-object quality verdict — and all of B2/B3 + the acceptance check — needs your
clips. (HSMR has an HF Space + demo images for the MPS/trajectory smoke-test before your
footage.) Film the PLAN's three per `docs/CAPTURE.md` (T-pose bookends, feet in frame), drop
them in the inbox, then `python3 ingest.py`.
→ **John: film + ingest walk / dance / pick-up-object when convenient.**
## Blast radius of the Decision-1 reversal (next session must know)
- **Lane B1 is NOT done** — wrong engine selected. Re-opened; PROGRESS un-ticked.
- **VOID the old note "MoCapAnything is BVH-direct, skip the retarget rig."** Any human model
emits **SMPL or SKEL, not mixamorig****Lane C's constraint-retarget IS required again**,
and `spec/{smpl,skel}_to_mixamorig.json` is back on the menu.
- Preconditions: disk resolves on the m3ultra reboot (Decision 2); the MPS/global-trajectory
smoke-test still stands for whichever engine John's pick unlocks.
---
Once Decision 1 is picked and 23 clear, the next session installs the chosen human engine,
proves MPS + global trajectory, and proceeds to B2 (service-ize) / B3 (jitter).

View File

@ -1,24 +1,43 @@
# Licenses & legal findings
Every third-party model/dataset/tool that touches the pipeline gets a row here with
its license, commercial status, link, and the date checked. Skeleton now; Lane B fills
the pose-model rows, Lane E fills any training-dataset rows. See PLAN.md §Ground rules.
Every third-party model/dataset/tool that touches the pipeline gets a row here with its
license, commercial status, link, and the date checked. See PLAN.md §Ground rules.
> **2026-07-17 correction (session 2):** B1's original row billed MoCapAnything as the human
> engine — it is an **animal** model (see `DECISIONS.md`). The SMPL/SKEL gate is **LIVE**, not
> avoided. Rows below corrected and license-verified.
## Components
| Component | Version | License | Commercial OK? | Notes | Checked |
|---|---|---|---|---|---|
| _pose model (Lane B1 winner)_ | — | — | — | fill after bake-off | — |
| SMPL / SMPL-X body model | — | research (Meshcapade) | **NO** by default | commercial needs Meshcapade license; or use joints-only path (PLAN §5) | — |
| AMASS dataset | — | academic | **verify** before any shipped training | Lane E only | — |
| Mixamo clips | — | Adobe ToS | ship-in-game OK; **not** ML training data | — | — |
| Component | License (code / body model) | Commercial OK? | Notes | Checked |
|---|---|---|---|---|
| **HSMR** (human — candidate engine) | code **MIT** / body **SKEL** | code yes / **model NO** | `IsshikiHugh/HSMR`, CVPR25 oral. Predicts **SKEL** params (biomechanical skeleton, great for retarget). Needs SKEL files (NC) at inference. MPS-runnability unverified. | 2026-07-17 |
| SKEL-CF (human — candidate) | code (verify) / body **SKEL** | **model NO** | Coarse-to-fine skeleton+mesh, Nov 2025. Same SKEL gate. | 2026-07-17 |
| **SKEL body model** | MPG research | **NO** (default) | "any use for commercial… purposes is prohibited"; commercial via **ps-license@tue.mpg.de**. Same family as SMPL. | 2026-07-17 |
| **SMPL / SMPL-X body model** | MPG research (Meshcapade sublicenses) | **NO** (default) | Commercial via **sales@meshcapade.com** (quote-based; incl. SMPL+H/-X/STAR). Gate for all SMPL-based human models. | 2026-07-17 |
| GVHMR / WHAM / TRAM / OnlineHMR (human, SMPL) | research/NC (GVHMR NC confirmed) / SMPL | **NO** | Accuracy leaders but **double gate**: NC *code* AND SMPL. GVHMR commercial via xwzhou@zju.edu.cn. Reference/yardstick unless licensed. | 2026-07-17 |
| GEM-X (human, shelved) | Apache / SOMA (not SMPL) | yes, but **off-fleet** | NVIDIA commercial-ready; **CUDA-only** → won't run on Apple Silicon. Revisit only if the fleet gains NVIDIA. | 2026-07-17 |
| MoCapAnything V2 (**animal** — creature NPCs only) | MIT / none | **YES** (creatures) | `kehong/MoCapAnythingV2-weights` MIT, no SMPL. **71 animal targets, no human/biped/fingers** → NOT the human engine. Keep for creature/centaur (Lane E4) work. | 2026-07-17 |
| AMASS dataset | academic | **verify** before shipped training | Lane E only | — |
| **BioAMASS v1.0** (downloaded, `~/Documents/skel/bioamass_v1.0`: CMU, DFAUST, MPI_Limits subsets, 376MB) | MPG research (same SKEL account/licence) | **NO** (default) — same NC family; Lane E training that ships needs the same Path-A licence email | SKEL-fitted mocap — ideal Lane E2 autoposer training data *format-wise*; legal gate identical to SKEL itself | 2026-07-17 |
| Mixamo clips | Adobe ToS | ship-in-game OK; **not** ML training data | — | — |
## The SMPL gate (hard stop — John decision)
## The SMPL/SKEL gate (hard stop — LIVE, John decision)
Most SOTA monocular mocap models emit SMPL parameters. Before any commercial use:
- **Path A:** buy a Meshcapade commercial license (cheap tiers exist).
- **Path B (SMPL-free):** use only the model's joint positions/rotations, never the SMPL
mesh/shape files, AND confirm the inference repo's own code license permits commercial
use of outputs.
Every accurate world-grounded **human** model depends on a Max-Planck body model — **SMPL**
(GVHMR/WHAM/TRAM/OnlineHMR) or **SKEL** (HSMR/SKEL-CF) — both research-licensed. Two paths:
- **Path A — license it.** Email MPI (SKEL, `ps-license@tue.mpg.de`) or Meshcapade (SMPL,
`sales@meshcapade.com`) for an indie commercial license. Clean; quote-based.
- **Path B — SMPL-free outputs.** Run an MIT-code model (HSMR) locally, keep only joint
rotations, retarget onto our own mixamorig, distribute no Max-Planck file. Grey area: the NC
license bars "production of artefacts for commercial purposes." John-accepts-risk.
Lane B writes the decision to `docs/DECISIONS.md` and stops there if unresolved.
Decision lives in `docs/DECISIONS.md`. The human path does not proceed until John picks.
## Sources (session 2, 2026-07-17)
- SKEL license (commercial prohibited): https://skel.is.tue.mpg.de/license.html
- HSMR (MIT code, SKEL-based): https://github.com/IsshikiHugh/HSMR ·
https://isshikihugh.github.io/HSMR/ · paper https://arxiv.org/html/2503.21751v1
- SKEL-CF: https://arxiv.org/html/2511.20157
- SMPL commercial licensing (Meshcapade): https://meshcapade.com/infopages/licensing.html
- GVHMR (NC): https://github.com/zju3dv/GVHMR · OnlineHMR: https://github.com/Tsukasane/Video-OnlineHMR

View File

@ -0,0 +1,34 @@
"""Headless-macOS import shims for the HSMR engine (our fleet is all Apple Silicon).
pyrender -> PyOpenGL -> EGL cannot load without a GL context, yet several vendored modules
`import pyrender` at load time some even in type annotations (`List[pyrender.Node]`) so
the import chain explodes before any of our code runs. We never render in-process; Blender
is the fleet renderer (Lane C). So we drop permissive fakes for the pyrender + OpenGL stack
into sys.modules, letting the chain import while keeping everything else (ColorPalette,
Wis3D, the models) real.
Import this BEFORE the first `lib.*` import: `import _headless # noqa: F401`
# ponytail: fakes, not a real headless GL backend (osmesa/egl). We don't render here, so a
# working GL context would be dead weight. Wire a real one only if in-process render is ever
# needed off-fleet.
"""
import sys
import types
class _Any:
def __call__(self, *a, **k): return self
def __getattr__(self, n): return self
class _AnyModule(types.ModuleType):
def __getattr__(self, n):
if n.startswith('__') and n.endswith('__'):
raise AttributeError(n)
return _Any()
# setdefault: never clobber a real module that a caller already imported on purpose.
for _n in ('pyrender', 'OpenGL', 'OpenGL.GL', 'OpenGL.error', 'OpenGL.platform'):
sys.modules.setdefault(_n, _AnyModule(_n))

View File

@ -0,0 +1,77 @@
"""
Person detector torchvision drop-in replacement for HSMR's detectron2/ViTDet.
Why: detectron2 + chumpy won't build from source on Apple Silicon (the Lane B blocker),
and this module's `import detectron2` was a load-time landmine for everything that touches
`lib.kits.hsmr_demo` (smoke_test.py, the future pose_engine.py). torchvision ships a
COCO-pretrained Faster R-CNN with ungated weights and no build step.
Same public contract as the original, so run_demo.py / smoke_test.py / pose_engine.py
import it unchanged:
build_detector(...) -> callable(raw_imgs) -> (dets, downsample_ratios)
dets[i] : dict of CPU tensors (empty tensors if no person)
pred_classes : all 0 (== hsmr_demo CLASS_HUMAN_ID)
scores : float
pred_boxes : (N, 4) left-upper-right-bottom pixels
downsample_ratios : one float per image; boxes are in native-resolution pixels -> 1.0
Contract is checked by test_detector.py (runs today, no SKEL needed).
# ponytail: ~35 lines replaces the whole detectron2 stack. Detection on CPU is the pose-engine
# wall-clock bottleneck, so this uses the fast mobilenet backbone and callers detect every-Nth frame.
"""
import numpy as np
import torch
from tqdm import tqdm
from torchvision.models.detection import (
fasterrcnn_mobilenet_v3_large_fpn,
FasterRCNN_MobileNet_V3_Large_FPN_Weights,
)
_COCO_PERSON = 1 # torchvision COCO label id for "person" (0 == background)
class _PersonDetector:
"""Callable over a list of RGB HxWx3 images (what HSMR's load_inputs yields).
uint8 or float, [0,255] or [0,1] are all handled."""
def __init__(self, device='cpu', score_thresh=0.25):
# score_thresh 0.25 mirrors the old detectron2 test_score_thresh; hsmr_demo's
# _img_det2patches re-filters at 0.5, so this only widens recall a little.
# mobilenet_v3 (not resnet50) — ~5-10x faster on CPU, which is the pose-engine wall-clock
# bottleneck (Lane B finding). Plenty accurate for one prominent, in-frame subject.
weights = FasterRCNN_MobileNet_V3_Large_FPN_Weights.DEFAULT
self.model = fasterrcnn_mobilenet_v3_large_fpn(
weights=weights, box_score_thresh=score_thresh,
).eval().to(device)
self.device = device
@torch.no_grad()
def __call__(self, raw_imgs):
dets, ratios = [], []
for img in tqdm(raw_imgs, desc='Detecting'):
t = torch.as_tensor(np.ascontiguousarray(img), device=self.device).float()
if float(t.max()) > 1.5: # [0,255] -> [0,1]
t = t / 255.0
t = t.permute(2, 0, 1) # HWC -> CHW (the model's own transform caps size)
out = self.model([t])[0]
keep = out['labels'] == _COCO_PERSON
dets.append({
'pred_classes': torch.zeros(int(keep.sum()), dtype=torch.long), # human -> 0
'scores' : out['scores'][keep].cpu(),
'pred_boxes' : out['boxes'][keep].cpu(), # xyxy == left-upper-right-bottom
})
ratios.append(1.0) # detected at native resolution
return dets, ratios
def build_detector(batch_size=1, max_img_size=512, device='cpu'):
"""Signature-compatible with the old detectron2 builder. batch_size / max_img_size are
accepted for parity but unused detection runs per-frame at native resolution.
# ponytail: detector pinned to CPU regardless of `device`. It's cheap next to the
# ViT-H HMR pass, and CPU sidesteps the MPS coverage gaps in RoIAlign/NMS. Flip to
# `device` only if per-frame detection ever dominates on long videos."""
return _PersonDetector(device='cpu')

View File

@ -0,0 +1,100 @@
#!/usr/bin/env python
"""
HSMR smoke test on Apple Silicon (MPS) Path B engine verification.
Answers the Lane B smoke-test questions on HSMR's bundled demo clips (no John footage,
no SKEL license needed beyond the model files John downloaded):
1. Does the HSMR forward run on MPS? (device + CPU-fallback)
2. Timing (target < 2x realtime).
3. Pose sanity (shape, NaNs) + an overlay render to eyeball quality.
Detection uses the torchvision person detector (lib.modeling.pipelines.vitdet, the
detectron2-free drop-in). The one remaining gate is SKEL: build_inference_pipeline
instantiates the SKEL body model, so this completes only once John's SKEL files are in place.
Run (once SKEL files are in data_inputs/body_models/skel/):
cd ~/Documents/MOCAPGOD/.engine/HSMR && source .venv/bin/activate
PYTHONPATH=. python smoke_test.py -i data_inputs/demo/example_videos/gymnasts.mp4
PYTHONPATH=. python smoke_test.py -i data_inputs/demo/example_imgs # image folder
"""
import os, time, argparse
os.environ.setdefault('PYTORCH_ENABLE_MPS_FALLBACK', '1') # some ops lack MPS kernels
import _headless # noqa: F401 fake pyrender/OpenGL for headless macOS — MUST precede lib.* imports
# One import line: every helper is an attribute of hsmr_demo (via its star-imports).
from lib.kits.hsmr_demo import (
load_inputs, imgs_det2patches, build_inference_pipeline, prepare_mesh,
visualize_full_img, IMG_MEAN_255, IMG_STD_255, asb, assemble_dict,
get_logger, save_video, save_img, np, torch, Path, DEFAULT_HSMR_ROOT,
build_detector,
)
def main():
ap = argparse.ArgumentParser()
ap.add_argument('-i', '--input_path', required=True)
ap.add_argument('-o', '--output_path', default='data_outputs/smoke')
ap.add_argument('-d', '--device', default='mps')
ap.add_argument('-m', '--model_root', default=None)
ap.add_argument('--rec_bs', type=int, default=64)
ap.add_argument('--ignore_skel', action='store_true')
args = ap.parse_args()
log = get_logger(brief=True)
model_root = Path(args.model_root) if args.model_root else DEFAULT_HSMR_ROOT
out = Path(args.output_path); out.mkdir(parents=True, exist_ok=True)
# Inputs (reuse the demo loader; it only reads .input_path / .input_type).
fake = argparse.Namespace(input_path=args.input_path, input_type='auto')
raw_imgs, meta = load_inputs(fake)
log.info(f'device={args.device} type={meta["type"]} frames={len(raw_imgs)}')
# Detection -> patches (real torchvision person detector, on CPU).
dets, ratios = build_detector(device=args.device)(raw_imgs)
patches, det_meta = imgs_det2patches(raw_imgs, dets, ratios, 5)
log.info(f'patches={len(patches)}')
# Build pipeline — instantiates SKEL, so this is where a missing SKEL file fails.
pipe = build_inference_pipeline(model_root=model_root, device=args.device)
# Recovery loop (mirrors run_demo), timed.
t0 = time.time()
pd_params, pd_cam_t = [], []
for bw in asb(total=len(patches), bs_scope=args.rec_bs, enable_tqdm=True):
p = patches[bw.sid:bw.eid]
pn = ((p - IMG_MEAN_255) / IMG_STD_255).transpose(0, 3, 1, 2)
with torch.no_grad():
o = pipe(pn)
pd_params.append({k: v.detach().cpu().clone() for k, v in o['pd_params'].items()})
pd_cam_t.append(o['pd_cam_t'].detach().cpu().clone())
dt = time.time() - t0
pd_params = assemble_dict(pd_params, expand_dim=False)
pd_cam_t = torch.cat(pd_cam_t, dim=0)
poses = pd_params['poses']
fps = len(patches) / dt if dt else 0
log.info(f'RECOVERY ok: {len(patches)} patches in {dt:.2f}s ({fps:.2f} fps) '
f'poses={tuple(poses.shape)} nan={bool(torch.isnan(poses).any())}')
# Dump raw params regardless (this is the Path-B payload: SKEL rotations).
np.savez(out / 'params.npz', poses=poses.numpy(),
betas=pd_params['betas'].numpy(), cam_t=pd_cam_t.numpy())
log.info(f'params -> {out/"params.npz"}')
# Mesh geometry for visual QC in Blender (pyrender/EGL is dead headless on macOS, and
# Blender is the fleet renderer + Lane C's tool). prepare_mesh uses skel_model, no GL.
try:
m_skin, m_skel = prepare_mesh(pipe, pd_params)
np.savez(out / 'mesh_qc.npz',
skin_v=m_skin['v'].detach().cpu().numpy(), skin_f=np.asarray(m_skin['f']),
skel_v=m_skel['v'].detach().cpu().numpy(), skel_f=np.asarray(m_skel['f']))
log.info(f'mesh geometry -> {out/"mesh_qc.npz"} (import in Blender to eyeball)')
except Exception as e:
log.warning(f'prepare_mesh failed (params still saved): {e!r}')
log.info('smoke test complete.')
if __name__ == '__main__':
main()

View File

@ -0,0 +1,71 @@
#!/usr/bin/env python
"""Detector smoke check — runs TODAY, no SKEL, no HMR forward pass.
Proves the torchvision person detector (lib/modeling/pipelines/vitdet) imports without
detectron2 and honors the contract lib.kits.hsmr_demo._img_det2patches depends on.
cd ~/Documents/MOCAPGOD/.engine/HSMR && source .venv/bin/activate
PYTHONPATH=. python test_detector.py
"""
from pathlib import Path
import cv2
import torch
import _headless # noqa: F401 fake pyrender/OpenGL for headless macOS — MUST precede lib.* imports
from lib.modeling.pipelines.vitdet import build_detector # must import without detectron2
from lib.kits.hsmr_demo import imgs_det2patches # the real consumer of detector output
DEMO = Path('data_inputs/demo/example_imgs')
DET_THRESHOLD_SCORE = 0.5 # matches hsmr_demo._img_det2patches
def main():
fns = sorted(p for p in DEMO.glob('*')
if p.suffix.lower() in {'.jpg', '.jpeg', '.png', '.webp'})
assert fns, f'no demo images under {DEMO}'
imgs = []
for fn in fns:
bgr = cv2.imread(str(fn))
assert bgr is not None, f'cv2 failed to read {fn}'
imgs.append(cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)) # HSMR feeds RGB
dets, ratios = build_detector(device='cpu')(imgs)
assert len(dets) == len(imgs) == len(ratios), 'per-image list length mismatch'
n_person_imgs = 0
for fn, img, d, r in zip(fns, imgs, dets, ratios):
H, W = img.shape[:2]
assert set(d) == {'pred_classes', 'scores', 'pred_boxes'}, f'bad keys {set(d)}'
assert d['pred_classes'].dtype == torch.long
assert (d['pred_classes'] == 0).all(), 'human class must be 0 (CLASS_HUMAN_ID)'
assert d['pred_boxes'].ndim == 2 and d['pred_boxes'].shape[1] == 4, 'boxes must be (N,4)'
assert r == 1.0, 'native-resolution detection -> ratio 1.0'
strong = d['scores'] > DET_THRESHOLD_SCORE
if strong.any():
n_person_imgs += 1
x1, y1, x2, y2 = d['pred_boxes'][strong].unbind(1)
assert (x2 > x1).all() and (y2 > y1).all(), 'boxes must be lurb (x2>x1, y2>y1)'
assert (x1 >= -1).all() and (y1 >= -1).all() \
and (x2 <= W + 1).all() and (y2 <= H + 1).all(), 'box out of frame'
print(f'{fn.name}: {int(strong.sum())} person(s), '
f'top score {float(d["scores"][strong].max()):.2f}')
else:
print(f' · {fn.name}: no confident person')
# Every demo image has an obvious human; allow one miss for robustness slack.
assert n_person_imgs >= len(imgs) - 1, \
f'expected a person in ~all demo imgs, got {n_person_imgs}/{len(imgs)}'
print(f'DETECTOR OK: person found in {n_person_imgs}/{len(imgs)} demo images.')
# Close the loop: feed detector output through HSMR's real cropper (SKEL-free) and
# confirm it yields the 256x256 patches the HMR forward pass expects.
patches, det_meta = imgs_det2patches(imgs, dets, ratios, max_instances_per_img=5)
assert patches.ndim == 4 and patches.shape[1:] == (256, 256, 3), f'bad patch shape {patches.shape}'
assert patches.shape[0] == sum(det_meta['n_patch_per_img']), 'patch count / meta mismatch'
print(f'PATCHES OK: {patches.shape[0]} human patches @ 256x256 — ready for the HMR pass.')
if __name__ == '__main__':
main()

View File

@ -1,15 +1,17 @@
#!/usr/bin/env python3
"""Lane A ingest — capture/inbox/*.{mp4,mov} → validate → 30fps h264 → rename → rsync to
the m3ultra pose queue capture/LEDGER.tsv.
"""Lane A ingest — capture/inbox/*.{mp4,mov} → validate → 30fps h264 → rename → pose queue
capture/LEDGER.tsv.
python3 ingest.py # scan capture/inbox/ and ship everything
python3 ingest.py # scan capture/inbox/ → local queue/ (the v1 default)
python3 ingest.py clip.mov --name dj_scratch_01 # one file, explicit clip name
python3 ingest.py --remote-queue # ship to the m3ultra queue instead (rsync)
python3 ingest.py --selfcheck # run built-in probe/validate self-test, no network
The pose engine (HSMR) currently runs on ultra, so ingest defaults to the LOCAL queue/ dir.
--remote-queue [user@host:/path] rsyncs to m3ultra instead (default target: MOCAP_REMOTE).
Rejects (portrait / malformed) move to capture/rejected/ with a reason in the ledger.
Successful originals move to capture/done/; on rsync failure the normalized file is kept
in capture/staged/ so a re-run retries the ship without re-encoding.
Remote defaults to m3ultra; override with MOCAP_REMOTE=user@host:/path.
Successful originals move to capture/done/; on a failed remote ship the normalized file is
kept in capture/staged/ so a re-run retries without re-encoding.
"""
import argparse, datetime, json, os, re, shutil, subprocess, sys, tempfile
from pathlib import Path
@ -20,7 +22,9 @@ STAGED = ROOT / "capture" / "staged"
DONE = ROOT / "capture" / "done"
REJECTED = ROOT / "capture" / "rejected"
LEDGER = ROOT / "capture" / "LEDGER.tsv"
REMOTE = os.environ.get("MOCAP_REMOTE", "m3ultra@100.89.131.57:~/Documents/MOCAPGOD/queue")
QUEUE = ROOT / "queue" # v1 default: pose engine (HSMR) runs on ultra, next to the repo
# ponytail: v1 inference on ultra; migrate to m3ultra (--remote-queue) when a real throughput need appears
REMOTE_DEFAULT = os.environ.get("MOCAP_REMOTE", "m3ultra@100.89.131.57:~/Documents/MOCAPGOD/queue")
VIDEO_EXT = {".mp4", ".mov", ".m4v"}
HEADERS = ["utc", "original", "output", "fps_in", "dur_s", "resolution", "audio", "status", "note"]
@ -90,13 +94,21 @@ def normalize(src, dst):
raise ValueError("ffmpeg normalize failed: " + tail)
def ship(staged):
host, _, rpath = REMOTE.partition(":")
run(["ssh", host, f"mkdir -p {rpath}"]) # queue dir may not exist yet
r = run(["rsync", "-az", str(staged), REMOTE + "/"])
def deliver(staged, remote):
"""Put the normalized clip in the pose queue. Local queue/ dir by default; rsync to a
remote (user@host:/path) when --remote-queue is used. Returns the destination string."""
if remote is None:
QUEUE.mkdir(parents=True, exist_ok=True)
dst = QUEUE / staged.name
shutil.move(str(staged), dst) # same-fs move; local delivery can't half-fail
return str(dst)
host, _, rpath = remote.partition(":")
run(["ssh", host, f"mkdir -p {rpath}"]) # remote queue dir may not exist yet
r = run(["rsync", "-az", str(staged), remote + "/"])
if r.returncode != 0:
tail = (r.stderr.strip().splitlines() or ["?"])[-1]
raise ConnectionError(f"rsync to {REMOTE} failed: {tail}")
raise ConnectionError(f"rsync to {remote} failed: {tail}")
return f"{remote}/{staged.name}"
def ledger(row):
@ -112,7 +124,7 @@ def slug(s):
return re.sub(r"[^A-Za-z0-9_-]+", "_", s).strip("_") or "clip"
def process_one(src, name=None):
def process_one(src, name=None, remote=None):
src = Path(src)
out_name = f"{datetime.date.today():%Y%m%d}_{slug(name or src.stem)}.mp4"
ts = datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
@ -139,9 +151,9 @@ def process_one(src, name=None):
res, audio = f"{info['w']}x{info['h']}", "yes" if info["audio"] else "no"
staged = STAGED / out_name
try:
if not staged.exists(): # skip re-encode when retrying a failed ship
if not staged.exists(): # skip re-encode when retrying a failed remote ship
normalize(src, staged)
ship(staged)
dest = deliver(staged, remote)
except (ValueError, ConnectionError) as e:
ledger([ts, src.name, out_name, info["fps"], f"{info['dur']:.1f}", res, audio, "FAIL", str(e)])
print(f" ! {src.name}: {e} (left in inbox; staged kept for retry)")
@ -152,7 +164,7 @@ def process_one(src, name=None):
staged.unlink(missing_ok=True)
ledger([ts, src.name, out_name, info["fps"], f"{info['dur']:.1f}", res, audio,
"OK", "" if info["audio"] else "audio stripped (none present)"])
print(f"{src.name}{REMOTE}/{out_name} ({res} {info['fps']}fps {info['dur']:.1f}s)")
print(f"{src.name}{dest} ({res} {info['fps']}fps {info['dur']:.1f}s)")
return True
@ -174,24 +186,29 @@ def selfcheck():
def main():
ap = argparse.ArgumentParser(description="Lane A: ingest phone videos into the m3ultra pose queue.")
ap = argparse.ArgumentParser(description="Lane A: ingest phone videos into the pose queue.")
ap.add_argument("video", nargs="?", help="single file to ingest (default: scan capture/inbox/)")
ap.add_argument("--name", help="clip name for the output (default: source filename)")
ap.add_argument("--remote-queue", nargs="?", const=REMOTE_DEFAULT, default=None,
metavar="USER@HOST:PATH",
help=f"rsync to a remote pose queue instead of the local queue/ dir "
f"(default target: {REMOTE_DEFAULT})")
ap.add_argument("--selfcheck", action="store_true", help="run the built-in self-test and exit")
args = ap.parse_args()
if args.selfcheck:
return selfcheck()
dest_label = args.remote_queue if args.remote_queue else str(QUEUE)
if args.video:
return 0 if process_one(args.video, args.name) else 1
return 0 if process_one(args.video, args.name, args.remote_queue) else 1
INBOX.mkdir(parents=True, exist_ok=True)
vids = sorted(p for p in INBOX.iterdir() if p.suffix.lower() in VIDEO_EXT)
if not vids:
print(f"inbox empty: {INBOX}")
return 0
print(f"ingesting {len(vids)} file(s) from {INBOX}{REMOTE}")
n_ok = sum(process_one(p) for p in vids)
print(f"ingesting {len(vids)} file(s) from {INBOX}{dest_label}")
n_ok = sum(process_one(p, remote=args.remote_queue) for p in vids)
print(f"done: {n_ok}/{len(vids)} ok")
return 0 if n_ok == len(vids) else 1

427
pose_engine.py Normal file
View File

@ -0,0 +1,427 @@
#!/usr/bin/env python3
"""Lane B (B2 + B3) pose engine — queue video → HSMR/SKEL → BVH@30fps + sidecar JSON.
# runs under the HSMR engine venv (torch/torchvision/skel live there):
.engine/HSMR/.venv/bin/python pose_engine.py # drain the queue/ dir
.engine/HSMR/.venv/bin/python pose_engine.py queue/clip.mp4 # one clip
python3 pose_engine.py --selfcheck # BVH math self-test, no engine
Pipeline per clip: person detection (torchvision, every-Nth-frame + box reuse single-subject
footage, this is the wall-clock win) single best box per frame HSMR ViT-H recovery SKEL
params (poses q46, betas, cam_t) SKEL forward for per-joint global orientations rigid-bone
BVH on SKEL's 24-joint skeleton + sidecar. Root is camera-space (root_space="camera"); Lane C
retargets onto mixamorig via spec/skel_to_mixamorig.json and grounds locomotion from foot contacts.
Output: out/<name>/motion.bvh + motion.json. Heartbeat: ~/.jobs/pose_engine.
"""
import argparse, datetime, json, os, subprocess, sys, time
from pathlib import Path
import numpy as np
from scipy.signal import savgol_filter
from scipy.spatial.transform import Rotation
REPO = Path(__file__).resolve().parent
ENGINE = REPO / ".engine" / "HSMR"
SKELDIR = ENGINE / "data_inputs" / "body_models" / "skel"
QUEUE = REPO / "queue"
OUT = REPO / "out"
JOB = Path.home() / ".jobs" / "pose_engine"
MODEL_TAG = "HSMR-ViTH-SKEL"
VERSION = "1.0"
EULER = "ZYX" # scipy intrinsic; BVH channels declared Z,Y,X to match
VIDEO_EXT = {".mp4", ".mov", ".m4v"}
def hb(msg):
JOB.parent.mkdir(parents=True, exist_ok=True)
JOB.write_text(f"{int(time.time())} {msg}\n")
print(f">>> [{datetime.datetime.now():%H:%M:%S}] {msg}", flush=True)
# ---------------------------------------------------------------- BVH math (engine-free) ----
def probe_fps(video, default=30.0):
r = subprocess.run(["ffprobe", "-v", "error", "-select_streams", "v:0",
"-show_entries", "stream=avg_frame_rate", "-of", "csv=p=0", str(video)],
capture_output=True, text=True)
try:
n, _, d = r.stdout.strip().partition("/")
f = float(n) / float(d or 1)
return round(f, 3) if f > 0 else default
except (ValueError, ZeroDivisionError):
return default
def smooth_rotations(W, win, poly):
"""SavitzkyGolay low-pass on a (F,3,3) global-rotation track, filtered as hemisphere-
aligned quaternions so it doesn't fight Euler wrap or quaternion double-cover."""
F = len(W)
q = Rotation.from_matrix(W).as_quat() # (F,4) xyzw
for i in range(1, F):
if float(q[i] @ q[i - 1]) < 0:
q[i] = -q[i]
w = _odd_window(win, F)
if w and w > poly:
q = savgol_filter(q, w, poly, axis=0)
q /= np.linalg.norm(q, axis=1, keepdims=True)
return Rotation.from_quat(q).as_matrix()
def smooth_xyz(P, win, poly):
w = _odd_window(win, len(P))
return savgol_filter(P, w, poly, axis=0) if (w and w > poly) else P
def _odd_window(win, n):
w = min(win, n if n % 2 else n - 1) # ≤ n, odd
return w if w >= 3 else 0 # too short to filter → skip
def locals_from_globals(W, parents):
"""BVH local rotations Q[j] = W[parent]^T · W[j] (root local == world)."""
Q = np.empty_like(W)
for j, p in enumerate(parents):
Q[j] = W[j] if p < 0 else W[p].T @ W[j]
return Q
def write_bvh(path, names, parents, offsets, root_pos, local_rot, fps):
"""SKEL-skeleton BVH. local_rot: (F,J,3,3) local rotations; root_pos: (F,3); offsets: (J,3)."""
F, J = local_rot.shape[:2]
children = {j: [c for c, p in enumerate(parents) if p == j] for j in range(J)}
ROT = "Zrotation Yrotation Xrotation" # order matches EULER='ZYX'
lines = []
def rec(j, depth):
pad = "\t" * depth
tag = "ROOT" if parents[j] < 0 else "JOINT"
lines.append(f"{pad}{tag} {names[j]}")
lines.append(f"{pad}{{")
ox, oy, oz = offsets[j]
lines.append(f"{pad}\tOFFSET {ox:.6f} {oy:.6f} {oz:.6f}")
if parents[j] < 0:
lines.append(f"{pad}\tCHANNELS 6 Xposition Yposition Zposition {ROT}")
else:
lines.append(f"{pad}\tCHANNELS 3 {ROT}")
for c in children[j]:
rec(c, depth + 1)
if not children[j]: # BVH needs an End Site for leaves
lines.append(f"{pad}\tEnd Site")
lines.append(f"{pad}\t{{")
lines.append(f"{pad}\t\tOFFSET 0.000000 0.100000 0.000000")
lines.append(f"{pad}\t}}")
lines.append(f"{pad}}}")
root = next(j for j, p in enumerate(parents) if p < 0)
lines.append("HIERARCHY")
rec(root, 0)
lines.append("MOTION")
lines.append(f"Frames: {F}")
lines.append(f"Frame Time: {1.0/fps:.6f}")
order_zyx = [0, 1, 2] # as_euler('ZYX') already returns Z,Y,X
dfs = [] # channel joint order = hierarchy DFS
def dfs_order(j):
dfs.append(j)
for c in children[j]:
dfs_order(c)
dfs_order(root)
for f in range(F):
row = []
for j in dfs:
zyx = Rotation.from_matrix(local_rot[f, j]).as_euler(EULER, degrees=True)[order_zyx]
if j == root:
px, py, pz = root_pos[f]
row += [f"{px:.6f}", f"{py:.6f}", f"{pz:.6f}"]
row += [f"{zyx[0]:.6f}", f"{zyx[1]:.6f}", f"{zyx[2]:.6f}"]
lines.append(" ".join(row))
Path(path).write_text("\n".join(lines) + "\n")
return dfs
def bvh_fk_positions(names, parents, offsets, root_pos, local_rot):
"""Reconstruct world joint positions from BVH data (verification / self-check)."""
F, J = local_rot.shape[:2]
pos = np.zeros((F, J, 3)); Wg = np.zeros((F, J, 3, 3))
for j, p in enumerate(parents):
if p < 0:
Wg[:, j] = local_rot[:, j]
pos[:, j] = root_pos + np.einsum("fab,b->fa", Wg[:, j], offsets[j])
else:
Wg[:, j] = Wg[:, p] @ local_rot[:, j]
pos[:, j] = pos[:, p] + np.einsum("fab,b->fa", Wg[:, p], offsets[j])
return pos
# ------------------------------------------------------------------ HSMR/SKEL (engine) ------
def build_engine(device, gender):
"""Load the engine once (imports, headless shims, SKEL body model). Detector + ViT-H pipeline
are built lazily on first recovery, so a --from-params re-run and the self-test skip them and
a queue amortizes the load across clips."""
os.environ.setdefault("PYTORCH_ENABLE_MPS_FALLBACK", "1")
sys.path.insert(0, str(ENGINE))
sys.path.insert(0, str(ENGINE / "thirdparty" / "SKEL"))
import _headless # noqa: F401 headless pyrender/OpenGL shims — before any lib.* import
from lib.kits.hsmr_demo import (
load_inputs, imgs_det2patches, build_inference_pipeline, build_detector,
asb, assemble_dict, IMG_MEAN_255, IMG_STD_255, DEFAULT_HSMR_ROOT,
)
import torch
from skel.skel_model import SKEL
from skel.kin_skel import skel_joints_name
os.chdir(ENGINE) # HSMR resolves model/cfg paths relative to its own root
skel = SKEL(gender=gender, model_path=str(SKELDIR)).eval()
return dict(load_inputs=load_inputs, imgs_det2patches=imgs_det2patches, asb=asb,
assemble_dict=assemble_dict, IMG_MEAN_255=IMG_MEAN_255, IMG_STD_255=IMG_STD_255,
torch=torch, device=device, skel=skel, skel_names=skel_joints_name,
_build_detector=build_detector, _build_pipe=build_inference_pipeline,
_hsmr_root=DEFAULT_HSMR_ROOT, det=None, pipe=None)
def _ensure_hmr(E):
if E["pipe"] is None: # build detector + ViT-H pipeline once, on first recovery
hb("build detector + HSMR ViT-H pipeline (once)")
E["det"] = E["_build_detector"](device=E["device"])
E["pipe"] = E["_build_pipe"](model_root=E["_hsmr_root"], device=E["device"])
def _single_subject_dets(det, frames, every, torch):
"""Detect on every-Nth frame, keep one best (largest, confident) box per frame, forward/
backward-fill gaps so patches align 1:1 with frames. Single-subject footage assumption."""
n = len(frames)
key = list(range(0, n, every))
kdets, _ = det([frames[i] for i in key])
best = []
for d in kdets:
s, b = d["scores"], d["pred_boxes"]
m = s > 0.5
if bool(m.any()):
bb = b[m]
area = (bb[:, 2] - bb[:, 0]) * (bb[:, 3] - bb[:, 1])
k = int(area.argmax())
best.append({"pred_classes": torch.zeros(1, dtype=torch.long),
"scores": bb.new_tensor([float(s[m][k])]), "pred_boxes": bb[k:k + 1]})
else:
best.append(None)
valid = [i for i, x in enumerate(best) if x is not None]
if not valid:
raise RuntimeError("no confident person detected in any sampled frame")
dets = []
for i in range(n):
ki = i // every
if best[ki] is None:
ki = min(valid, key=lambda v: abs(v - ki))
dets.append(best[ki])
return dets, [1.0] * n
def run_recovery(E, video, detect_every):
"""video → (poses (F,46), betas (F,10), cam_t (F,3)) with one pose per frame."""
torch = E["torch"]
_ensure_hmr(E)
fake = argparse.Namespace(input_path=str(video), input_type="auto")
frames, meta = E["load_inputs"](fake)
hb(f"detect (every {detect_every}f) over {len(frames)} frames")
dets, ratios = _single_subject_dets(E["det"], frames, detect_every, torch)
patches, det_meta = E["imgs_det2patches"](frames, dets, ratios, 1)
assert patches.shape[0] == len(frames), f"patch/frame misalign {patches.shape[0]}!={len(frames)}"
hb(f"recover {len(patches)} patches (HSMR ViT-H)")
params, cams = [], []
for bw in E["asb"](total=len(patches), bs_scope=64, enable_tqdm=True):
p = patches[bw.sid:bw.eid]
pn = ((p - E["IMG_MEAN_255"]) / E["IMG_STD_255"]).transpose(0, 3, 1, 2)
with torch.no_grad():
o = E["pipe"](pn)
params.append({k: v.detach().cpu().clone() for k, v in o["pd_params"].items()})
cams.append(o["pd_cam_t"].detach().cpu().clone())
params = E["assemble_dict"](params, expand_dim=False)
return params["poses"].numpy(), params["betas"].numpy(), torch.cat(cams, 0).numpy()
def skel_kinematics(E, poses, betas, cam_t):
"""SKEL forward on recovered params → posed joints/orientations + rest skeleton.
Returns names, parents(len J, root=-1), rest_offsets(J,3), root_pos(F,3), W(F,J,3,3)."""
torch = E["torch"]
skel = E["skel"]; skel_joints_name = E["skel_names"]
parents = np.concatenate([[-1], skel.parent.cpu().numpy()]).astype(int) # (J,), root=-1
P = torch.tensor(poses).float(); B = torch.tensor(betas).float(); T = torch.tensor(cam_t).float()
Rg, Jp = [], []
for s in range(0, len(P), 256): # batch to bound memory
with torch.no_grad():
o = skel(poses=P[s:s+256], betas=B[s:s+256], trans=T[s:s+256], skelmesh=False)
Rg.append(o.joints_ori.cpu().numpy()); Jp.append(o.joints.cpu().numpy())
Rg = np.concatenate(Rg); Jp = np.concatenate(Jp) # (F,J,3,3),(F,J,3)
with torch.no_grad(): # one rest skeleton from mean shape
r = skel(poses=torch.zeros(1, 46), betas=B.mean(0, keepdim=True),
trans=torch.zeros(1, 3), skelmesh=False)
rest_J = r.joints[0].cpu().numpy(); rest_R = r.joints_ori[0].cpu().numpy()
W = np.einsum("fjab,jcb->fjac", Rg, rest_R) # W[f,j] = Rg[f,j] @ rest_R[j]^T (camera frame)
root_pos = Jp[:, 0]
# HSMR emits camera space (X-right, Y-DOWN, Z-fwd — OpenCV image convention, verified: head.y <
# pelvis.y). Rotate 180° about X -> Y-up, character faces -Z, matching the treaty world axes.
R_CW = np.diag([1.0, -1.0, -1.0])
W = np.einsum("ab,fjbc->fjac", R_CW, W)
root_pos = root_pos @ R_CW.T
offsets = np.zeros((len(parents), 3))
for j, p in enumerate(parents):
offsets[j] = rest_J[j] if p < 0 else rest_J[j] - rest_J[p]
offsets[next(j for j, p in enumerate(parents) if p < 0)] = 0.0 # root at origin; abs pos in channels
return skel_joints_name, parents, offsets, root_pos, W
# --------------------------------------------------------------------- orchestration --------
def process(E, video, gender="male", detect_every=10,
filt_win=11, filt_poly=3, no_filter=False, from_params=False):
video = Path(video).resolve() # engine work chdir's into .engine/HSMR; keep paths absolute
name = video.stem
outdir = OUT / name; outdir.mkdir(parents=True, exist_ok=True)
pfile = outdir / "params.npz"
hb(f"START {name}")
if from_params and pfile.exists(): # re-run SKEL→BVH without the HMR pass (filter/axis tweaks)
z = np.load(pfile); poses, betas, cam_t = z["poses"], z["betas"], z["cam_t"]
hb(f"reuse params ({len(poses)} frames)")
else:
poses, betas, cam_t = run_recovery(E, video, detect_every)
np.savez(pfile, poses=poses, betas=betas, cam_t=cam_t) # raw payload for QC / re-processing
hb(f"skel forward ({len(poses)} frames)")
names, parents, offsets, root_pos, W = skel_kinematics(E, poses, betas, cam_t)
if not no_filter:
hb("jitter pass (savgol on rotations + root)")
W = np.stack([smooth_rotations(W[:, j], filt_win, filt_poly) for j in range(W.shape[1])], axis=1)
root_pos = smooth_xyz(root_pos, filt_win, filt_poly)
local_rot = np.stack([locals_from_globals(W[f], parents) for f in range(len(W))])
fps = probe_fps(video)
write_bvh(outdir / "motion.bvh", names, parents, offsets, root_pos, local_rot, fps)
# verify the BVH numerically encodes the motion we intended (rigid-bone tolerance)
recon = bvh_fk_positions(names, parents, offsets, root_pos, local_rot)
intended = _intended_positions(offsets, root_pos, W, parents)
err = float(np.abs(recon - intended).max())
assert err < 1e-3, f"BVH FK self-inconsistency {err:.4f}m (Euler/channel-order bug)"
sidecar = {
"fps": fps, "n_frames": int(len(local_rot)), "model": MODEL_TAG, "version": VERSION,
"gender": gender, "skeleton": "SKEL-24", "root_space": "camera", # HSMR is per-frame
"joint_order": list(names), "source_video": video.name,
"root_trajectory": np.round(root_pos, 5).tolist(),
"bvh_units": "metres, Y-up", "created": datetime.datetime.now().strftime("%Y-%m-%dT%H:%M:%S"),
"note": "camera-space root; Lane C grounds locomotion from foot contacts and retargets "
"to mixamorig via spec/skel_to_mixamorig.json",
}
(outdir / "motion.json").write_text(json.dumps(sidecar, indent=1))
hb(f"DONE {name}: {len(local_rot)} frames @ {fps}fps → {outdir} (FK err {err*1000:.1f}mm)")
return outdir
def _intended_positions(offsets, root_pos, W, parents):
"""World joint positions implied by the (unfiltered-or-filtered) global rotations W —
the ground truth the BVH must reproduce."""
F, J = W.shape[:2]
pos = np.zeros((F, J, 3))
root = next(j for j, p in enumerate(parents) if p < 0)
pos[:, root] = root_pos
order = _topo(parents)
for j in order:
p = parents[j]
if p >= 0:
pos[:, j] = pos[:, p] + np.einsum("fab,b->fa", W[:, p], offsets[j])
return pos
def _topo(parents):
order, seen = [], set()
def visit(j):
if j in seen:
return
if parents[j] >= 0:
visit(parents[j])
seen.add(j); order.append(j)
for j in range(len(parents)):
visit(j)
return order
def selfcheck():
"""Engine-free: a small tree + random global rotations → write BVH → parse-independent FK
reconstruction must match the intended positions. Proves Euler order channel declaration
FK are mutually consistent (the geometric crux)."""
import tempfile
rng = np.random.default_rng(0)
names = ["root", "spine", "head", "arm_l", "arm_r"]
parents = np.array([-1, 0, 1, 1, 1])
offsets = np.array([[0, 0, 0], [0, .5, 0], [0, .3, 0], [-.4, .1, 0], [.4, .1, 0]], float)
F = 20
W = np.stack([Rotation.random(len(names), random_state=rng).as_matrix() for _ in range(F)])
root_pos = np.cumsum(rng.normal(0, .01, (F, 3)), axis=0)
local = np.stack([locals_from_globals(W[f], parents) for f in range(F)])
d = Path(tempfile.mkdtemp())
dfs = write_bvh(d / "t.bvh", names, parents, offsets, root_pos, local, 30.0)
recon = bvh_fk_positions(names, parents, offsets, root_pos, local)
intended = _intended_positions(offsets, root_pos, W, parents)
err = float(np.abs(recon - intended).max())
assert err < 1e-9, f"BVH FK mismatch {err}"
assert (d / "t.bvh").read_text().count("JOINT") == 4 and dfs[0] == 0
# rotation round-trip in the declared order
R = Rotation.random(50, random_state=rng).as_matrix()
e = Rotation.from_matrix(R).as_euler(EULER, degrees=True)
back = Rotation.from_euler(EULER, e, degrees=True).as_matrix()
assert np.abs(back - R).max() < 1e-9
print(f"selfcheck OK — BVH FK exact ({err:.1e}m), Euler round-trip exact, hierarchy sane")
return 0
def main():
ap = argparse.ArgumentParser(description="Lane B pose engine: video → SKEL BVH + sidecar.")
ap.add_argument("video", nargs="?", help="single clip (default: drain queue/)")
ap.add_argument("--gender", default="male", choices=["male", "female"])
ap.add_argument("--device", default="mps")
ap.add_argument("--detect-every", type=int, default=20,
help="detect person every Nth frame; box reused between (single-subject footage)")
ap.add_argument("--filt-win", type=int, default=11, help="savgol window (odd); jitter pass")
ap.add_argument("--filt-poly", type=int, default=3, help="savgol polyorder")
ap.add_argument("--no-filter", action="store_true", help="skip the B3 jitter pass")
ap.add_argument("--from-params", action="store_true",
help="reuse out/<name>/params.npz, skip the HMR pass (filter/axis tweaks)")
ap.add_argument("--selfcheck", action="store_true", help="run the BVH-math self-test, no engine")
a = ap.parse_args()
if a.selfcheck:
return selfcheck()
kw = dict(gender=a.gender, detect_every=a.detect_every, filt_win=a.filt_win,
filt_poly=a.filt_poly, no_filter=a.no_filter, from_params=a.from_params)
video = Path(a.video).resolve() if a.video else None # resolve before build_engine chdir's away
E = build_engine(a.device, a.gender) # load models once; queue reuses them
if video:
process(E, video, **kw)
return 0
QUEUE.mkdir(parents=True, exist_ok=True)
vids = sorted(p for p in QUEUE.iterdir() if p.suffix.lower() in VIDEO_EXT)
if not vids:
print(f"queue empty: {QUEUE}")
return 0
hb(f"draining {len(vids)} clip(s) from {QUEUE}")
fails = 0
for v in vids:
try: # one bad clip must not sink the queue
process(E, v, **kw)
except Exception as e:
fails += 1
hb(f"FAIL {v.name}: {e}")
import traceback; traceback.print_exc()
hb(f"queue drained: {len(vids) - fails}/{len(vids)} clip(s) ok")
return 1 if fails else 0
if __name__ == "__main__":
sys.exit(main())

87
setup_hsmr.sh Executable file
View File

@ -0,0 +1,87 @@
#!/bin/bash
# HSMR pose-engine install on Apple Silicon (MPS) — Path B (local HSMR/SKEL).
# Clone -> uv venv(3.10) -> deps -> ungated weights -> apply engine_patches/ -> STOP at SKEL gate.
#
# ./setup_hsmr.sh full install (re-runnable; STOPS at the SKEL download gate)
# ./setup_hsmr.sh --patch re-apply engine_patches/ onto an existing clone and exit
#
# engine_patches/ holds our ARM/headless fixes (torchvision detector drop-in, headless
# pyrender/OpenGL shims, smoke + detector tests). HSMR itself is gitignored under .engine/,
# so a re-clone would drop those fixes — this script restores them. Override the clone
# location with HSMR_DIR=... (used by the durability test against a temp clone).
#
# ponytail: no `set -e` — optional deps (detectron2/chumpy/render) WARN and we still want the
# env + weights down so the smoke test can run and any failure is diagnosable.
set -uo pipefail
REPO="$(cd "$(dirname "$0")" && pwd)"
ENGINE="$REPO/.engine"
HSMR="${HSMR_DIR:-$ENGINE/HSMR}"
PATCHES="$REPO/engine_patches"
LOG="$ENGINE/setup.log"
JOB="$HOME/.jobs/hsmr_setup"
apply_patches(){
[ -d "$HSMR" ] || { echo "no HSMR clone at $HSMR — clone first"; exit 1; }
[ -d "$PATCHES" ] || { echo "no engine_patches/ at $PATCHES"; exit 1; }
( cd "$PATCHES" && find . -type f ! -name '.DS_Store' ) | sed 's|^\./||' | while read -r f; do
mkdir -p "$HSMR/$(dirname "$f")"
cp "$PATCHES/$f" "$HSMR/$f"
echo " patched $f"
done
}
# Standalone re-apply (after a manual HSMR re-clone).
if [ "${1:-}" = "--patch" ]; then
echo ">>> applying engine_patches/ -> $HSMR"
apply_patches
echo "patches applied."
exit 0
fi
mkdir -p "$ENGINE" "$HOME/.jobs"
exec > >(tee -a "$LOG") 2>&1
hb(){ echo "$(date +%s) $*" > "$JOB"; echo ">>> [$(date +%H:%M:%S)] $*"; }
hb "clone HSMR + submodules (SKEL)"
if [ ! -d "$HSMR/.git" ]; then
git clone --recurse-submodules https://github.com/IsshikiHugh/HSMR "$HSMR" || { hb "FAIL clone"; exit 1; }
else
git -C "$HSMR" submodule update --init || true
fi
cd "$HSMR" || { hb "FAIL cd"; exit 1; }
hb "uv venv python 3.10"
uv venv --python 3.10 .venv || { hb "FAIL venv"; exit 1; }
source .venv/bin/activate
hb "install torch + torchvision (MPS wheel)"
uv pip install torch torchvision || { hb "FAIL torch"; exit 1; }
hb "install requirements.txt"
uv pip install -r requirements.txt || hb "WARN requirements had issues"
hb "pin numpy<2 (chumpy/detectron2/smplx need 1.x)"
uv pip install "numpy==1.26.4"
# detectron2/chumpy are NOT needed — engine_patches replaces the detector and the SKEL runtime
# loads pkls without chumpy (see docs/B_ENGINE_SETUP.md). Left out on purpose; they don't build on ARM.
hb "pip install -e HSMR + thirdparty/SKEL"
uv pip install -e . || hb "WARN pip -e . failed"
uv pip install -e thirdparty/SKEL || hb "WARN SKEL editable install failed"
hb "download UNGATED weights from HuggingFace (regressors + HSMR ckpt + ViTPose backbone)"
hf download IsshikiHugh/HSMR-data_inputs \
--include "body_models/SMPL_to_J19.pkl" "body_models/J_regressor_SKEL_mix_MALE.pkl" "body_models/J_regressor_SMPL_MALE.pkl" "released_models/HSMR-ViTH-r1d1.tar.gz" "backbone/vitpose_backbone.pth" \
--local-dir data_inputs || hb "WARN hf download issue"
if [ -f data_inputs/released_models/HSMR-ViTH-r1d1.tar.gz ]; then
hb "extract HSMR checkpoint"
tar -xzf data_inputs/released_models/HSMR-ViTH-r1d1.tar.gz -C data_inputs/released_models/ && rm -f data_inputs/released_models/HSMR-ViTH-r1d1.tar.gz
fi
mkdir -p data_inputs/body_models/skel
hb "apply engine_patches/ (ARM detector, headless shims, tests)"
apply_patches
hb "DONE-ENV — SKEL gate: John drops skel_models_v1.1/ (v1.1.1) into data_inputs/body_models/skel/"
echo "=== pip freeze (key) ===" ; uv pip freeze | grep -iE 'torch|numpy|smplx|pytorch-lightning|timm' || true

View File

@ -0,0 +1,38 @@
{
"comment": "SKEL 24-joint -> mixamorig name map for Lane C's constraint retarget (pose_engine.py emits a SKEL-native BVH; Lane C maps it onto the character_kit mixamorig rig). null = a SKEL joint with no direct mixamorig target (twist/heel DoF folded elsewhere). mixamorig bones absent here (Spine1, Head, fingers, *_End) are not driven by SKEL — see notes.",
"source_skeleton": "SKEL-24 (skel/kin_skel.py order)",
"target_skeleton": "mixamorig (spec/mixamorig_bones.json, colon-prefixed, 65 joints)",
"skel_to_mixamorig": {
"pelvis": "mixamorig:Hips",
"lumbar_body": "mixamorig:Spine",
"thorax": "mixamorig:Spine2",
"head": "mixamorig:Neck",
"scapula_r": "mixamorig:RightShoulder",
"humerus_r": "mixamorig:RightArm",
"ulna_r": "mixamorig:RightForeArm",
"radius_r": null,
"hand_r": "mixamorig:RightHand",
"scapula_l": "mixamorig:LeftShoulder",
"humerus_l": "mixamorig:LeftArm",
"ulna_l": "mixamorig:LeftForeArm",
"radius_l": null,
"hand_l": "mixamorig:LeftHand",
"femur_r": "mixamorig:RightUpLeg",
"tibia_r": "mixamorig:RightLeg",
"talus_r": "mixamorig:RightFoot",
"calcn_r": null,
"toes_r": "mixamorig:RightToeBase",
"femur_l": "mixamorig:LeftUpLeg",
"tibia_l": "mixamorig:LeftLeg",
"talus_l": "mixamorig:LeftFoot",
"calcn_l": null,
"toes_l": "mixamorig:LeftToeBase"
},
"notes": {
"spine": "SKEL has 2 trunk joints (lumbar_body, thorax); mixamorig has 3 (Spine, Spine1, Spine2). lumbar_body->Spine, thorax->Spine2; Lane C interpolates Spine1 (or leaves identity).",
"neck_head": "SKEL 'head' is a single thorax-relative joint -> mapped to mixamorig:Neck. mixamorig:Head is left to Lane C (identity, or a small follow of Neck).",
"radius_calcn": "radius_* (forearm pronation/supination twist) and calcn_* (heel) have no direct mixamorig bone; their DoF fold into ForeArm/Foot in mixamorig. null = skip in the constraint map.",
"fingers": "SKEL has no fingers -> mixamorig hand/finger bones stay undriven; fingers arrive in Lane E1.",
"toe_end_headtop": "mixamorig *_End tips (HeadTop_End, LeftToe_End, RightToe_End) are leaf markers, not driven."
}
}