Compare commits
11 Commits
468f35be7d
...
35403f8f05
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
35403f8f05 | ||
|
|
7d0a3b7f7c | ||
|
|
0e2f7d0c12 | ||
|
|
0cf673aa82 | ||
|
|
9db4fec988 | ||
|
|
f8037f7b85 | ||
|
|
60b49453b1 | ||
|
|
2dbd99579b | ||
|
|
4cc67b90fa | ||
|
|
fefb1f71ff | ||
|
|
7819e12a5a |
1
.gitignore
vendored
1
.gitignore
vendored
@ -1,5 +1,6 @@
|
||||
# weights + runtime data stay out of git
|
||||
weights/
|
||||
.engine/
|
||||
capture/
|
||||
queue/
|
||||
out/
|
||||
|
||||
18
PLAN.md
18
PLAN.md
@ -18,6 +18,10 @@ critical path. Lanes E–F are follow-ons. Do NOT start E/F before D ships.
|
||||
training. Python via conda/uv there. MODELBEAST lives here; mocap becomes a sibling
|
||||
service, same patterns (job dirs, batch keys — see `MESHGOD/scripts/mb_*` for the
|
||||
client convention).
|
||||
- **v1 deviation (session 5):** HSMR is installed and MPS-verified on **ultra**, not
|
||||
m3ultra, so v1 runs inference on ultra and `ingest.py` defaults to a **local** `queue/`.
|
||||
`# ponytail: v1 inference on ultra; migrate to m3ultra when a real throughput need appears`
|
||||
(`ingest.py --remote-queue` already ships to m3ultra when that day comes).
|
||||
- **ultra** (this Mac, M1 Ultra 128GB) — GLUE + DCC. Blender headless retarget/cleanup/
|
||||
export, orchestration, Unreal verification. Owns the canonical repo.
|
||||
- **JING5** (M5 MBP) — capture companion + portable inference later. Not in v1 critical path.
|
||||
@ -47,8 +51,11 @@ critical path. Lanes E–F are follow-ons. Do NOT start E/F before D ships.
|
||||
- Skeleton: **mixamorig 65-joint** as defined by `~/Documents/character_kit`
|
||||
(`female_game.glb` / male base). Canonical bone list generated in Lane 0.
|
||||
- Intermediate motion format: **BVH at 30fps** + sidecar JSON
|
||||
(`{fps, joint_order, root_trajectory, source_video, model, version}`).
|
||||
BVH because Blender imports it natively and it's human-diffable.
|
||||
(`{fps, joint_order, root_space, root_trajectory, source_video, model, version}`).
|
||||
**`root_space` ∈ {`"camera"`, `"world"`} is REQUIRED** (session-5 amendment): HSMR is a
|
||||
per-frame recoverer so it emits `"camera"`; Lane C reads this and, for locomotion clips,
|
||||
derives a world root from foot contacts. BVH because Blender imports it natively and it's
|
||||
human-diffable.
|
||||
- Units metres, Y-up, character faces −Z (three.js convention; Unreal yaw-fix happens
|
||||
at UE import, as established in the A_3D_STORE experiments).
|
||||
- Final artifact: FBX (anim-only, Mixamo-style "Without Skin") dropped into
|
||||
@ -153,8 +160,11 @@ Tasks:
|
||||
`mixamorig_` variants — this bit us before), bake to keyframes.
|
||||
- Foot-contact fix: detect frames where foot velocity ≈ 0 & height < threshold → IK
|
||||
pin. `# ponytail: threshold heuristic, upgrade to learned contact detection in Lane E if it annoys`.
|
||||
- Trim the T-pose bookends (Lane A protocol guarantees them), root-motion option
|
||||
(in-place vs traveling) as a flag.
|
||||
- Trim the T-pose bookends (Lane A protocol guarantees them). Root motion is driven by the
|
||||
sidecar **`root_space`** (session-5 treaty): `"camera"` (HSMR) → in-place by default, or
|
||||
derive a world root from foot contacts for traveling clips; a `--in-place`/`--traveling`
|
||||
flag overrides. v1 quality bar: **in-place clips are first-class; traveling is best-effort**
|
||||
until a proper grounding pass exists.
|
||||
- Export anim-only FBX (Mixamo "Without Skin" style — merge_anims.py already eats these).
|
||||
2. QC render: 3-second turntable mp4 of the clip on the female_game body, written next
|
||||
to the FBX. **Verify with own eyes** (the rule) — every clip gets its mp4 looked at
|
||||
|
||||
113
PROGRESS.md
113
PROGRESS.md
@ -6,13 +6,124 @@ One lane per session. Tick on completion. See PLAN.md for acceptance checks.
|
||||
spec/mixamorig_bones.json (65 bones, `mixamorig:` colon-prefixed). — 2026-07-17
|
||||
- [x] **Lane A** — capture + ingest. docs/CAPTURE.md protocol, ingest.py
|
||||
(probe → 30fps normalize → rename → rsync to m3ultra queue → LEDGER.tsv). — 2026-07-17
|
||||
- [ ] **Lane B** — pose engine (m3ultra). Model bake-off, service-ize, jitter pass.
|
||||
- [x] **Lane B** — pose engine. **DONE session 5 (2026-07-17).** `pose_engine.py` ships:
|
||||
queue video → HSMR/SKEL → SKEL-24 BVH@fps + `motion.json` sidecar (`root_space:"camera"`),
|
||||
`spec/skel_to_mixamorig.json` name-map for Lane C. Verified on the demo clip: Blender import
|
||||
= 24 bones, **upright** (1.59 m), motion present, FK self-consistency **0.0 mm**; queue of 6
|
||||
drained unattended (2 bad clips failed gracefully, 4 valid BVH, heartbeat per clip); runtime
|
||||
**~1.6× realtime** steady-state (under the <2× bar). mobilenet detector + detect-every-20 +
|
||||
single-subject box reuse retired the CPU-detector wall-clock finding. Axis fix: HSMR emits
|
||||
Y-down (OpenCV) → `diag(1,-1,-1)` → Y-up/faces-−Z. **History (sessions 2–4):**
|
||||
**Decision made session 3: Path B** — engine =
|
||||
**HSMR → SKEL**, run local, ship rig-motion only, John accepts NC-licence risk
|
||||
(`docs/DECISIONS.md` Decision 1 RESOLVED). Env up in `.engine/HSMR` (uv venv, torch 2.13
|
||||
**MPS green**, ungated weights pulled). **Detector RESOLVED (session 4):** torchvision
|
||||
Faster R-CNN drop-in replaces the unbuildable detectron2; `test_detector.py` is green today
|
||||
(7/7 demo imgs → 15 patches, **no SKEL needed**). `smoke_test.py` now runs real detection +
|
||||
patch-crop and reaches the **exact SKEL gate**; chumpy cleared as a non-blocker. **Only open
|
||||
before B ships:** (1) John downloads gated **SKEL v1.1.1** so
|
||||
`.engine/HSMR/data_inputs/body_models/skel/skel_male.pkl` (+ `skel_female.pkl`) exist;
|
||||
(2) run the forward-pass smoke test (MPS / timing <2× / NaN check); then B2 (service-ize →
|
||||
BVH) + B3 (jitter). Lanes C and D remain blocked on B.
|
||||
- [ ] **Lane C** — retarget + cleanup (Blender). bvh → mixamorig FBX + QC turntable.
|
||||
- [ ] **Lane D** — clip factory. `mocap` CLI end-to-end; games + Unreal integration.
|
||||
- [ ] Lane E — ML upgrades (post-v1, gated).
|
||||
- [ ] Lane F — UX (optional).
|
||||
|
||||
## SESSION 5 — DONE (2026-07-17). All six orders executed; Lane B ships.
|
||||
1. ✅ **Durable-ized** the ARM fixes → `engine_patches/` + `setup_hsmr.sh --patch`. Verified: fresh
|
||||
clone + `--patch` → `test_detector.py` green, no hand edits.
|
||||
2. ✅ **Path-A email** drafted in `docs/DECISIONS.md` (John sends to ps-license@tue.mpg.de).
|
||||
3. ✅ **Queue reconciled**: `ingest.py` defaults to local `queue/` on ultra; `--remote-queue` keeps
|
||||
the m3ultra path. PLAN Ground-rule-1 deviation noted; CAPTURE.md updated.
|
||||
4. ✅ **Treaty amended**: `root_space` (camera|world) is a REQUIRED sidecar field; Lane C reads it.
|
||||
5. ✅ **B2 + B3**: `pose_engine.py` (SKEL-24 BVH + sidecar + jitter) + `spec/skel_to_mixamorig.json`.
|
||||
Detector swapped to mobilenet; detect-every-20 + box reuse; build-once + per-video isolation.
|
||||
Ran the smoke test's SKEL forward on real params — no re-verify needed (smoke already PASSED).
|
||||
|
||||
**HANDOFF TO LANE C (retarget + cleanup, Blender, on ultra):**
|
||||
- Input is `out/<name>/motion.bvh` (SKEL-24 skeleton, **Y-up, metres, faces −Z**) + `motion.json`.
|
||||
Import in Blender with `import_anim.bvh(axis_forward='-Z', axis_up='Y')` (verified upright).
|
||||
- Constraint-map SKEL→mixamorig via **`spec/skel_to_mixamorig.json`** (20 mapped, 4 null=twist/heel;
|
||||
Spine1 interpolated, Head follows Neck, fingers are Lane E). mixamorig canon: `mixamorig:` colon.
|
||||
- **`root_space` is `"camera"`** → root is NOT world-grounded (HSMR is per-frame). In-place clips are
|
||||
first-class; for traveling clips derive a world root from foot contacts (PLAN Lane C task).
|
||||
- Rest pose is a clean T-pose at zero rotation; T-pose bookends only exist if John filmed the protocol
|
||||
(demo clips don't have them) — Lane C's trim must tolerate their absence.
|
||||
- Re-run the SKEL→BVH stage cheaply with `pose_engine.py <clip> --from-params` (skips HMR) when
|
||||
tuning; raw params are saved at `out/<name>/params.npz`.
|
||||
|
||||
## SESSION 5 ORDERS — from Fable's review (2026-07-17, supersede older notes where they conflict)
|
||||
|
||||
Review verdict: Lanes 0/A solid; session 2's reversal of the B1 animal-model blunder was the
|
||||
right call, well executed; ARM engineering verified independently (test_detector.py re-run
|
||||
green 7/7 by reviewer). Proceed as follows, in order:
|
||||
|
||||
1. **Durable-ize the gitignored engine fixes (do first, ~30 min).** The torchvision detector,
|
||||
`_headless.py`, `test_detector.py`, `smoke_test.py` exist ONLY on ultra's disk inside the
|
||||
gitignored `.engine/`. Move canonical copies into git at `engine_patches/` and make
|
||||
`setup_hsmr.sh` apply them after clone. Acceptance: fresh clone in a temp dir + script →
|
||||
`test_detector.py` green with no hand edits.
|
||||
2. **Draft the Path-A email (parallel risk-retirement, ~15 min).** Write the SKEL indie
|
||||
commercial-license request to ps-license@tue.mpg.de into `docs/DECISIONS.md` (project
|
||||
description, solo Australian dev, indie revenue, SKEL-in-pipeline-not-redistributed usage).
|
||||
John sends it himself. Path B continues meanwhile; a granted license converts B→A with zero
|
||||
rework. Do not wait on the reply.
|
||||
3. **Reconcile the split-brain queue (~30 min).** Lane A's ingest.py rsyncs to m3ultra's queue,
|
||||
but the engine (and MPS verification) is on ultra — nothing consumes that queue. For v1:
|
||||
default ingest to LOCAL `queue/` on ultra (`--remote-queue` flag preserves the m3ultra path),
|
||||
note the deviation from PLAN Ground-rule 1 in PLAN.md with a `# ponytail: v1 inference on
|
||||
ultra; migrate to m3ultra when a real throughput need appears` marker. Update CAPTURE.md.
|
||||
4. **Treaty amendment (small, load-bearing).** `motion.json` sidecar gains REQUIRED field
|
||||
`root_space: "camera" | "world"`. HSMR emits `"camera"`. Lane C must read it and, for
|
||||
locomotion clips, derive world root from foot contacts (already in notes). v1 quality bar:
|
||||
in-place clips are first-class; traveling clips best-effort until a grounding pass exists.
|
||||
**SMOKE TEST RAN 2026-07-17 (Fable, after John's SKEL v1.1.1 download landed — installed at
|
||||
the expected path): PASSED with findings.** device=mps, 263 frames → 1315 patches, recovery
|
||||
31.2 fps, **nan=False**, `data_outputs/smoke/params.npz` written. Findings for B2: (1) the
|
||||
CPU Faster R-CNN detector is the wall-clock hog (~1 fps, 4min of the 5min run) — B2 MUST add
|
||||
detect-every-Nth-frame + box reuse/IoU tracking (single-subject footage makes this easy) or
|
||||
the mobilenet swap; HMR itself is faster than realtime so the <2× acceptance is reachable.
|
||||
(2) `prepare_mesh` MPS→numpy crash fixed in smoke_test.py (`.detach().cpu()`) — carry the
|
||||
same pattern into pose_engine.py. Item 5 below is now UNBLOCKED (skip re-verifying: smoke
|
||||
is done; go straight to B2/B3).
|
||||
|
||||
5. **If (and only if) SKEL v1.1.1 files are in place** (`data_inputs/body_models/skel/
|
||||
skel_male.pkl`): run smoke_test.py — MPS forward pass, runtime <2× realtime on demo video,
|
||||
NaN check — then proceed straight into B2 (params.npz → `spec/skel_to_mixamorig.json` map →
|
||||
BVH@30fps + sidecar per treaty) and B3 (One-Euro/Sav-Gol jitter pass, root smoothed
|
||||
separately). If SKEL absent: finish items 1–4, verify Decision-3 footage status, stop clean.
|
||||
|
||||
**John's homework (all three block different things):**
|
||||
- SKEL v1.1.1 download (the ONLY hard blocker; instructions in `docs/B_ENGINE_SETUP.md`).
|
||||
- Film walk / dance / pick-up-object per `docs/CAPTURE.md`, drop in `capture/inbox/` (blocks
|
||||
the honest quality verdict + all of B's acceptance).
|
||||
- Reboot m3ultra when convenient (applies staged 26.5.2 security update; housekeeping).
|
||||
|
||||
## Notes for the next lane
|
||||
- **Lane D can't be started until B + C ship + test footage exists** — it's the CLI that wraps
|
||||
ingest → pose_engine.py → retarget.py, and the latter two don't exist yet. Session 3 (the
|
||||
"Lane D" session) correctly built nothing; it locked Decision 1 and de-risked MPS instead.
|
||||
- **SKEL version trap:** HSMR needs **v1.1.1** exactly (`changelog_v1.1.1.txt`); older = silent
|
||||
incompat. Source: `.engine/HSMR/docs/SETUP.md` (register skel.is.tue.mpg.de → download.php).
|
||||
- **Engine wiring + ARM bypasses live in `docs/B_ENGINE_SETUP.md`** — all three ARM landmines
|
||||
are now cleared (session 4): detectron2 → **torchvision drop-in** (real detection, not a
|
||||
full-frame stub), the pyrender/EGL fake is centralized in `.engine/HSMR/_headless.py`, and
|
||||
chumpy is confirmed a non-blocker. `test_detector.py` proves the detector path green, SKEL-free.
|
||||
- **World-grounded gap (important):** HSMR is per-frame → root is camera-space `cam_t`, NOT the
|
||||
treaty's world-grounded trajectory. Fine for in-place DJ moves; walking/locomotion needs Lane C
|
||||
to derive root from foot-contact (or a grounding pass). Carry into Lane C design.
|
||||
- **HSMR is installed on ultra, not m3ultra** (PLAN Ground-Rule 1 puts the pose engine on
|
||||
m3ultra). Likely because m3ultra's disk reclaim waits on the reboot (Decision 2). Next B
|
||||
session decides: move it to m3ultra, or consciously host inference on ultra.
|
||||
- **VOID** (was: "MoCapAnything is BVH-direct, skip the retarget rig"). Reversed — the model
|
||||
is animal-only and not our engine. Whatever human model we pick emits **SMPL or SKEL, not
|
||||
mixamorig**, so **Lane C's constraint-retarget IS required** and
|
||||
`spec/{smpl,skel}_to_mixamorig.json` is back on the menu. HSMR's SKEL output (a biomechanical
|
||||
skeleton) is the likely source format if John goes SKEL.
|
||||
- **Check MPS + global-trajectory FIRST** for whichever human model John's decision unlocks
|
||||
(HSMR has an HF Space + demo to smoke-test before John's footage). These two risks are
|
||||
unchanged by the engine swap.
|
||||
- Ground-truth skeleton naming is **`mixamorig:`** (colon), armature object named `Armature`.
|
||||
Lane C's name-map must handle colon vs underscore variants (PLAN warned about this).
|
||||
- Blender headless entrypoint: `/Applications/Blender.app/Contents/MacOS/Blender --background --python <script>`.
|
||||
|
||||
107
docs/B1_BAKEOFF.md
Normal file
107
docs/B1_BAKEOFF.md
Normal file
@ -0,0 +1,107 @@
|
||||
# B1 — pose-model bake-off
|
||||
|
||||
**Status: SUPERSEDED 2026-07-17 (session 2).** The selection below (MoCapAnything V2) was a
|
||||
**category error** — MoCapAnything is an **animal** mocap model (71 animal targets, no
|
||||
human/biped/mixamorig, no fingers), so it cannot be MOCAPGOD's human v1 engine. Its "SMPL gate
|
||||
dissolved" conclusion is therefore wrong; the gate is **LIVE**. See `docs/DECISIONS.md` for the
|
||||
corrected, license-verified decision. The candidate research below is kept as history — its
|
||||
*human*-model findings (GVHMR non-commercial, GEM-X CUDA-only) still stand; its recommendation
|
||||
does not.
|
||||
|
||||
The PLAN told this lane to distrust the July-2024 snapshot and search fresh. It paid off:
|
||||
two models that post-date the snapshot change the whole calculus — one of them **removes
|
||||
the SMPL licensing hard-stop the plan was built around.**
|
||||
|
||||
## Selection criteria (PLAN order)
|
||||
1. permissive code license (must ship in commercial games)
|
||||
2. world-grounded global trajectory (not pose-in-place)
|
||||
3. runs on Apple Silicon (m3ultra — MPS or acceptable CPU; no NVIDIA on the fleet)
|
||||
4. SMPL-dependency status (the legal rail)
|
||||
|
||||
## Candidate matrix
|
||||
|
||||
| Model | World-grounded | Code lic. | Weights lic. | SMPL? | Apple Silicon | Output | Commercial-ship |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **MoCapAnything V2** ⭐ | trajectories (global vs camera **unverified**) | MIT | **MIT** | **No** | standard PyTorch, no custom CUDA ops → MPS-portable *with effort* | **BVH rotations onto an arbitrary target rig** (feed it mixamorig) + hands | **YES (MIT/MIT)** |
|
||||
| GVHMR | Yes — SOTA, beats WHAM 18–28% WA-MPJPE | **research-only** | needs SMPL | Yes | plain PyTorch, MPS-portable w/ effort | SMPL params (rot) + global traj (.pt) | **NO** — code is non-commercial *and* SMPL-gated |
|
||||
| GEM-X (NVIDIA) | Yes | Apache-2.0 | NVIDIA Open Model Lic. (NVIDIA states commercial-ready) | **No** (SOMA body model) | **CUDA 12.1+ only, NVIDIA arch** — will not run on the fleet | 77-joint axis-angle + world root (m), incl. hands | Yes, but **off-fleet** |
|
||||
| WHAM | Yes (autoregressive; GVHMR's predecessor) | research/NC (not re-verified) | needs SMPL | Yes | PyTorch, MPS-portable | SMPL + traj | NO (gate) — dominated by GVHMR |
|
||||
| TRAM | Yes (SLAM-based camera+human) | mixed; +DROID-SLAM | needs SMPL | Yes | heavier (SLAM stage) | SMPL + traj | NO (gate) — heaviest, not pursued |
|
||||
|
||||
Sources verified this session: GVHMR repo LICENSE (custom research-only, "prohibited for
|
||||
commercial use", contact xwzhou@zju.edu.cn); GEM-X model card (Apache code / NVIDIA Open
|
||||
Model License weights / SOMA / CUDA-only); MoCapAnything V2 weights card
|
||||
(`kehong/MoCapAnythingV2-weights`, **License: mit**, author Kehong Gong = paper author) and
|
||||
code repo `github.com/phongdaot/MocapAnything` (MIT). WHAM/TRAM licenses not re-verified —
|
||||
both are SMPL-gated and dominated by GVHMR on the world-grounded metric, so not installed.
|
||||
|
||||
## The two findings that matter
|
||||
|
||||
**1. GVHMR (the accuracy leader) is double-blocked for us.** Its own code license is
|
||||
non-commercial — that's a wall *independent of* SMPL, and the joints-only Path-B from the
|
||||
PLAN can't get around a non-commercial *code* license. John's clips ship in commercial
|
||||
games. GVHMR is therefore a research/accuracy *reference* only, unless John negotiates a
|
||||
paid ZJU commercial license. It is **not** a v1 shipping engine.
|
||||
|
||||
**2. MoCapAnything V2 dissolves the SMPL gate.** MIT code + MIT weights, commercial use
|
||||
permitted, **no SMPL files anywhere** in the pipeline or dependency list (torch/opencv/
|
||||
trimesh only). It is category-agnostic: you give it a monocular video **plus the target
|
||||
rig as a prompt**, and it emits a rotation-based BVH that drives *that rig* directly, via
|
||||
constraint-aware IK. Cross-species too (it retargets across heterogeneous rigs — a free
|
||||
preview of the Lane E4 centaur work).
|
||||
|
||||
### Why #2 is bigger than "gate cleared"
|
||||
If we prompt MoCapAnything with the **mixamorig** skeleton (`spec/mixamorig_bones.json` /
|
||||
`female_game.glb`), it emits **mixamorig BVH directly**. That means:
|
||||
- `spec/smpl_to_mixamorig.json` (B2 task 2) is **unnecessary** — there is no SMPL
|
||||
intermediate to map.
|
||||
- Lane C's core constraint-retarget (bvh→mixamorig) **largely collapses** — the BVH is
|
||||
already on-skeleton. Lane C shrinks to: import → foot-contact fix → jitter → trim →
|
||||
export FBX.
|
||||
|
||||
The fresh search didn't just clear the legal hold-up; it deleted a spec file and most of a
|
||||
lane. `# ponytail win: don't build the SMPL→mixamorig mapper or the Lane C retarget rig
|
||||
until we've confirmed MoCapAnything can't hit mixamorig directly.`
|
||||
|
||||
**Caveat (honest):** getting it to emit *exactly* the treaty's mixamorig (joint order,
|
||||
Y-up, faces −Z, bone-roll) will still need a rig-prompt config and a convention
|
||||
reconciliation pass. "Collapses" = "greatly simplifies," not "free."
|
||||
|
||||
## Recommendation
|
||||
|
||||
**Primary: MoCapAnything V2.** Only candidate that is commercial-clean AND on-fleet AND
|
||||
BVH-direct. Accept two risks, both verifiable cheaply at inference (below).
|
||||
**Reference (optional): GVHMR** under its research license, purely to benchmark how much
|
||||
accuracy we're leaving on the table — informs whether a paid ZJU license is ever worth it.
|
||||
**Shelved: GEM-X** — revisit only if the fleet ever gains an NVIDIA GPU (it's the cleanest
|
||||
option the day CUDA is available; zero-cloud rules out renting it).
|
||||
|
||||
## Must-verify at the inference eyeball (the other half of B1)
|
||||
These are why the lane isn't ticked. Cheap to settle — MoCapAnything ships a ~160 MB sample
|
||||
dataset and an HF demo Space, so we don't even need John's footage to answer them:
|
||||
1. **Global trajectory?** Treaty requires world-space root motion. MoCapAnything says
|
||||
"temporally coherent trajectories" — confirm the character *travels through the world*,
|
||||
not runs-in-place. If camera-space only, we need a grounding pass (or fall back to GVHMR
|
||||
for locomotion clips). **This is the #1 risk.**
|
||||
2. **MPS reality.** Standard PyTorch predicts yes, but the PLAN's rule is "check MPS early —
|
||||
it decides everything." Expect some ops to need CPU fallback; time it (<2× realtime is
|
||||
the bar).
|
||||
3. **Quality.** The linked code is a re-implementation ("use as reference, not
|
||||
reproduction"). Eyeball walk / dance / pick-up-object in Blender before trusting it.
|
||||
|
||||
## Preconditions before the install/inference session
|
||||
- **m3ultra disk: 29 GB free of 926 GB (97% full).** Marginal for one torch env + weights;
|
||||
too tight for two. Clear space first (see `DECISIONS.md`).
|
||||
- **Test footage:** inbox is empty. Sample data covers the smoke test; John's walk / dance /
|
||||
pick-up-object clips are needed for the real quality call and B2/B3/acceptance.
|
||||
- **Engine decision:** confirm MoCapAnything-primary (vs. paying for GVHMR-grade). See
|
||||
`DECISIONS.md`.
|
||||
|
||||
## Sources
|
||||
- [GVHMR](https://github.com/zju3dv/GVHMR) · [GVHMR paper](https://arxiv.org/html/2409.06662v1)
|
||||
- [NVIDIA GEM-X](https://huggingface.co/nvidia/GEM-X)
|
||||
- [MoCapAnything project](https://animotionlab.github.io/MoCapAnything/) ·
|
||||
[V2 weights (MIT)](https://huggingface.co/kehong/MoCapAnythingV2-weights) ·
|
||||
[code](https://github.com/phongdaot/MocapAnything) ·
|
||||
[paper](https://arxiv.org/abs/2512.10881)
|
||||
- [TRAM](https://github.com/yufu-wang/tram)
|
||||
87
docs/B_ENGINE_SETUP.md
Normal file
87
docs/B_ENGINE_SETUP.md
Normal file
@ -0,0 +1,87 @@
|
||||
# Lane B engine — HSMR install & smoke-test state (session 3, 2026-07-17)
|
||||
|
||||
Engine per Decision 1 = **Path B: HSMR → SKEL**, run local on **ultra**, MPS. Installed under
|
||||
`.engine/HSMR` (gitignored). This doc = how it's wired + how to run once SKEL lands.
|
||||
|
||||
## State
|
||||
- ✅ `uv venv` py3.10 at `.engine/HSMR/.venv`; torch **2.13.0**, `mps.is_available() == True`.
|
||||
- ✅ Ungated weights in `.engine/HSMR/data_inputs/`: HSMR-ViTH ckpt
|
||||
(`released_models/HSMR-ViTH-r1d1/checkpoints/hsmr.ckpt`), ViTPose backbone, 3 regressors.
|
||||
- ✅ **Real person detector** (torchvision Faster R-CNN, no detectron2) — checked by
|
||||
`.engine/HSMR/test_detector.py`, green **today with no SKEL**: 7/7 demo images → 15 patches
|
||||
@256×256 through HSMR's own cropper.
|
||||
- ✅ Smoke-test runner `.engine/HSMR/smoke_test.py` — **PASSED 2026-07-17** (session 5): SKEL
|
||||
v1.1.1 in place, device=mps, 263 frames → 1315 patches, recovery 31.2 fps, **nan=False**,
|
||||
`data_outputs/smoke/params.npz` written (poses (N,46), betas (N,10), cam_t (N,3)).
|
||||
- ✅ Install script (re-runnable): **`./setup_hsmr.sh`** at the repo root (tracked in git).
|
||||
`./setup_hsmr.sh --patch` re-applies the ARM/headless fixes onto an existing/re-cloned HSMR.
|
||||
- ✅ **The ARM fixes are now durable in git** (session 5, was: "on ultra's disk only"). Canonical
|
||||
copies live in **`engine_patches/`** (`_headless.py`, `test_detector.py`, `smoke_test.py`,
|
||||
`lib/modeling/pipelines/vitdet/__init__.py`); `setup_hsmr.sh` overlays them onto the gitignored
|
||||
`.engine/HSMR` clone. Verified: fresh clone + `--patch` → `test_detector.py` green, no hand edits.
|
||||
|
||||
## Apple-Silicon blockers — all three resolved (session 4, "Lane B detector")
|
||||
- **detectron2 won't build on ARM Mac** → **RESOLVED.** Replaced its whole ViTDet detector with
|
||||
a torchvision Faster R-CNN drop-in in `lib/modeling/pipelines/vitdet/__init__.py` — same
|
||||
`build_detector(...) -> (dets, downsample_ratios)` contract, torchvision's person class (1)
|
||||
remapped to HSMR's human id (0), boxes as lurb, ratio 1.0. This also killed a **load-time
|
||||
landmine**: the old `import detectron2` there exploded anything importing `lib.kits.hsmr_demo`
|
||||
(why session 3 had to stub it). Detector pinned to **CPU** — cheap next to the ViT-H pass and
|
||||
it dodges MPS RoIAlign/NMS gaps; an in-file `# ponytail` note marks the mobilenet swap if
|
||||
per-frame detection ever dominates on long videos.
|
||||
- **pyrender is dead headless on macOS** (`py_renderer` forces `PYOPENGL_PLATFORM=egl`, no EGL on
|
||||
Mac; `pyrender` is even referenced in a type annotation, so a plain import-guard won't do).
|
||||
Centralized the permissive `pyrender`/`OpenGL` fake into **`.engine/HSMR/_headless.py`** — every
|
||||
entrypoint does `import _headless` before any `lib.*` import. QC still happens in Blender (fleet
|
||||
renderer + Lane C's tool); the runner dumps `params.npz` (SKEL poses/betas/cam_t — the Path-B
|
||||
payload) and `mesh_qc.npz` (verts/faces for Blender).
|
||||
- **chumpy failed to build** → **NOT a blocker (verified).** The runtime `SKELWrapper` loads its
|
||||
pkls with plain `pickle.load(encoding='latin1')` — no chumpy import. chumpy only appears in
|
||||
SKEL's offline model-*build* script (`merge_smplh_mano.py`), which we never run. Ignore the WARN.
|
||||
|
||||
## John's one step — drop the SKEL files (login + licence, so it's yours to do)
|
||||
Register **skel.is.tue.mpg.de** → Download → **"SKEL and BSM models"** → `skel_models_v1.1.zip`
|
||||
(**must be v1.1.1** — check `changelog_v1.1.1.txt` inside; older = silent incompatibility).
|
||||
Unzip, then:
|
||||
```
|
||||
mv /path/to/skel_models_v1.1 ~/Documents/MOCAPGOD/.engine/HSMR/data_inputs/body_models/skel
|
||||
```
|
||||
so that `…/data_inputs/body_models/skel/skel_male.pkl` exists.
|
||||
|
||||
## Run the smoke test (once SKEL is in place)
|
||||
```
|
||||
cd ~/Documents/MOCAPGOD/.engine/HSMR && source .venv/bin/activate
|
||||
PYTHONPATH=. python smoke_test.py -i data_inputs/demo/example_videos/gymnasts.mp4 # or .../example_imgs
|
||||
```
|
||||
Outputs → `data_outputs/smoke/`: `params.npz` (rotations), `mesh_qc.npz` (Blender QC), and
|
||||
console device/fps/NaN. Answers the Lane B smoke questions: **MPS forward OK? timing (<2×
|
||||
realtime)? pose sane?**
|
||||
|
||||
⚠️ HSMR is a **per-frame** recoverer → root is **camera-space `cam_t`, not world-grounded**.
|
||||
Fine for in-place DJ moves; locomotion (walk) needs Lane C's foot-contact root derivation or a
|
||||
grounding pass. This is the treaty's global-trajectory caveat, carried into Path B.
|
||||
|
||||
## Run the pose engine (B2/B3, session 5) — the real service
|
||||
`pose_engine.py` (repo root, tracked) wraps the engine into the pipeline. Run under the engine venv:
|
||||
```
|
||||
cd ~/Documents/MOCAPGOD
|
||||
.engine/HSMR/.venv/bin/python pose_engine.py # drain queue/ → out/<name>/
|
||||
.engine/HSMR/.venv/bin/python pose_engine.py queue/clip.mp4 # one clip
|
||||
.engine/HSMR/.venv/bin/python pose_engine.py <clip> --from-params # re-run SKEL→BVH, skip HMR
|
||||
.engine/HSMR/.venv/bin/python pose_engine.py --selfcheck # BVH-math self-test, no engine
|
||||
```
|
||||
Per clip → `out/<name>/`: `motion.bvh` (SKEL-24, **Y-up, metres, faces −Z**), `motion.json`
|
||||
(sidecar, `root_space:"camera"`), `params.npz` (raw poses/betas/cam_t for QC / `--from-params`).
|
||||
Heartbeat: `~/.jobs/pose_engine`. Models build **once** per invocation (queue amortizes); a bad
|
||||
clip is logged and skipped, not fatal.
|
||||
|
||||
Key engine facts learned this session (for whoever touches B again):
|
||||
- **SKEL emits `poses` = (F, 46)** biomechanical q, NOT SMPL's 72 axis-angles. Don't decode q by
|
||||
hand — run `SKEL.forward()` and consume `joints_ori` (F,24,3,3 global orientations) + `joints_tpose`.
|
||||
- BVH is rigid-bone: rest-aware FK (`offset=rest_J[j]-rest_J[parent]`, `W=Rg@rest_R^T`,
|
||||
`local=W[parent]^T@W[j]`) reconstructs SKEL joints to ~4 cm; the residual is SKEL's coupled
|
||||
knee/ankle translations, which a rotation-only BVH cannot carry (accept it).
|
||||
- **HSMR camera frame is Y-DOWN** (OpenCV; verified head.y < pelvis.y). `pose_engine` applies
|
||||
`diag(1,-1,-1)` to reach the treaty's Y-up / faces-−Z. Undo/adjust there if a future model differs.
|
||||
- Detector is **mobilenet** now (was R50) — CPU detection was the wall-clock hog. It self-caps input
|
||||
to ~640 px internally, so pre-downscaling does nothing; `--detect-every` is the real throughput knob.
|
||||
@ -25,8 +25,9 @@ uses to auto-trim the clip. No bookends → manual trimming.
|
||||
- **AirDrop** the clip from the phone to **ultra**, save into
|
||||
`~/Documents/MOCAPGOD/capture/inbox/`.
|
||||
- Then run **`python3 ingest.py`** (or let the Lane D `mocap` CLI do it). Ingest validates,
|
||||
normalizes to 30fps, and ships it to the m3ultra queue. Rejects (portrait, no feet, garbage)
|
||||
land in `capture/rejected/` with a reason in `capture/LEDGER.tsv` — check there if a clip vanishes.
|
||||
normalizes to 30fps, and puts it in the local pose queue (`queue/` on ultra, where the engine
|
||||
runs). Rejects (portrait, no feet, garbage) land in `capture/rejected/` with a reason in
|
||||
`capture/LEDGER.tsv` — check there if a clip vanishes. (`--remote-queue` ships to m3ultra instead.)
|
||||
|
||||
## Common rejections (and the fix)
|
||||
| Symptom | Fix |
|
||||
|
||||
139
docs/DECISIONS.md
Normal file
139
docs/DECISIONS.md
Normal file
@ -0,0 +1,139 @@
|
||||
# Decisions needing John
|
||||
|
||||
**Status 2026-07-17 (session 2): Lane B1's engine pick is REVERSED** (Decision 1). The disk
|
||||
work (Decision 2) from the concurrent session is preserved intact. Corrected, license-verified
|
||||
analysis below + `LICENSES.md`. Prior "SMPL hard-stop avoided / go MoCapAnything" memo
|
||||
superseded (git @ 7819e12).
|
||||
|
||||
## Decision 1 — pose engine: B1 picked an ANIMAL model for a HUMAN job; the gate is LIVE
|
||||
|
||||
> **RESOLVED 2026-07-17 — John chose Path B (run local, accept NC risk).** Engine = **HSMR
|
||||
> → SKEL** on the fleet. Ship only the retargeted mixamorig motion (no Max-Planck file
|
||||
> redistributed); John explicitly accepts the judgment call that authoring game clips from an
|
||||
> NC-licensed model is his risk. Path A (email MPI for a commercial licence) stays on the shelf
|
||||
> as the clean upgrade if that risk ever needs retiring. **MPS verified green on ultra this
|
||||
> session** (torch 2.13.0, `mps_available=True`, `mps_built=True`). **Sole remaining blocker to
|
||||
> a first forward pass: the gated SKEL download — John's manual step** (register at
|
||||
> skel.is.tue.mpg.de → download.php → `skel_models_v1.1.zip`, **must be v1.1.1**, unzip into
|
||||
> `.engine/HSMR/data_inputs/body_models/skel/` per HSMR `docs/SETUP.md`). Behind a login +
|
||||
> NC-licence agreement, so it is John's to do — I can't create the account or accept the terms.
|
||||
> Analysis below is retained for the record.
|
||||
|
||||
B1 chose **MoCapAnything V2** for being MIT/MIT with no SMPL, and concluded "the SMPL hard-stop
|
||||
is avoided." Reading the actual repo this session shows why that's wrong: MoCapAnything is an
|
||||
**animal** motion-capture system — **71 animal retarget targets** (Alligator…Tyranno), trained
|
||||
on `zoo1030` (animals) + `obj1k` (objects), **no human/biped/mixamorig reference and no
|
||||
fingers**. MOCAPGOD's v1 job is capturing **John (a human)** onto mixamorig (PLAN line 1; the
|
||||
Lane B brief says "monocular **human** motion recovery"). MoCapAnything didn't dissolve the SMPL
|
||||
gate — it swapped in a task the gate doesn't cover. **Wrong tool for the human path.** (It's a
|
||||
clean, strong tool for *creature* NPCs — option C.)
|
||||
|
||||
**Verified from license text this session (links/dates in `LICENSES.md`):**
|
||||
|
||||
| Human-capable option | Code license | Body model | Body-model license | On the fleet? |
|
||||
|---|---|---|---|---|
|
||||
| GVHMR / WHAM / TRAM / OnlineHMR (accuracy leaders) | research/NC (GVHMR NC confirmed) | **SMPL** | NC — commercial via Meshcapade (quote) | PyTorch→MPS ✓ |
|
||||
| **HSMR** (CVPR25 oral) / SKEL-CF (Nov 2025) | **MIT** (HSMR) | **SKEL** | NC — commercial via MPI `ps-license@tue.mpg.de` | PyTorch→MPS ✓ (unverified) |
|
||||
| GEM-X (NVIDIA) | Apache | SOMA (not SMPL) | commercial-ready | **CUDA-only — OFF fleet** ✗ |
|
||||
| MoCapAnything V2 | MIT | none | — (clean) | ✓ but **animal-only** ✗ for humans |
|
||||
|
||||
**No free-lunch, commercial-clean, on-fleet, world-grounded *human* model exists.** SMPL and
|
||||
SKEL are both Max-Planck research licenses with an "email for commercial" path; the one
|
||||
clean-licensed human model (GEM-X) needs NVIDIA hardware the fleet lacks. Pick a direction:
|
||||
|
||||
**A. License a body model, use a proven human model — RECOMMENDED.** Email for an indie
|
||||
commercial license. Primary: **SKEL → HSMR** (code is MIT, outputs a biomechanical **skeleton**
|
||||
ideal for the mixamorig retarget) via `ps-license@tue.mpg.de`. Fallback: **SMPL** via
|
||||
`sales@meshcapade.com` (unlocks WHAM/TRAM, though their *code* licenses also need clearing).
|
||||
One email + likely a modest fee + a few days. Clean and shippable. *I can draft the email; I
|
||||
can't send it.*
|
||||
|
||||
**B. SMPL-free-outputs path (PLAN Path B) — run locally, ship only your rig's motion.** HSMR's
|
||||
code is MIT: run on the fleet, extract joint **rotations**, retarget onto YOUR mixamorig, ship
|
||||
only that — distribute no Max-Planck file. Zero cost, unblocks today. **Risk:** the NC licenses
|
||||
forbid "production of artefacts for commercial purposes"; whether authoring game clips counts is
|
||||
a legal judgment call. The clean mitigation (email MPI to ask) collapses into A. John-accepts-risk.
|
||||
|
||||
**C. Reconsider scope — creatures now, defer the human license.** MOCAPGOD's downstream is
|
||||
stylized NPCs + a centaur RPG; some targets aren't human. If the near-term need is creature
|
||||
NPCs, **MoCapAnything (MIT, clean, on-fleet) does that TODAY**; defer the human-capture license
|
||||
until you need your own DJ clips.
|
||||
|
||||
→ **Recommendation A.** Honest, cheap, converts B's grey into a clear yes. In parallel I keep
|
||||
MoCapAnything as the sanctioned *creature* engine (free upside, blocks nothing) and verify
|
||||
HSMR's MPS-runnability so we're ready when a license lands. I will **not** build the human v1
|
||||
pipeline on an NC model without your explicit go (that's B).
|
||||
|
||||
## Path A — SKEL commercial-licence email (READY TO SEND — John sends it himself)
|
||||
|
||||
Parallel risk-retirement (session-5 order 2). Path B ships meanwhile; if MPI grants a licence
|
||||
the grey area in Decision 1 disappears with **zero pipeline rework** (same HSMR/SKEL engine).
|
||||
**Do not wait on the reply.** John: fill the two `[brackets]`, send from your address to
|
||||
`ps-license@tue.mpg.de`. I can't send email or create the account — this is yours.
|
||||
|
||||
> **To:** ps-license@tue.mpg.de
|
||||
> **Subject:** Commercial licence enquiry — SKEL body model (indie game developer)
|
||||
>
|
||||
> Dear Perceiving Systems licensing team,
|
||||
>
|
||||
> I'm a solo independent game developer based in Australia. I'd like to enquire about a
|
||||
> commercial licence for the **SKEL body model** (v1.1.1, from skel.is.tue.mpg.de).
|
||||
>
|
||||
> **Use case.** I'm building a small in-house tool that recovers 3D human motion from ordinary
|
||||
> single-camera video of myself, to author animation clips for stylised characters in my own
|
||||
> indie games. SKEL is used inside the pipeline at inference time (via the HSMR pose-recovery
|
||||
> model) to estimate body pose.
|
||||
>
|
||||
> **Redistribution: none.** The SKEL/BSM model files stay on my own machine — I do not
|
||||
> redistribute them or any Max-Planck-derived asset. What ships in the games is only the
|
||||
> resulting joint **motion**, retargeted onto my own separately-authored character skeletons
|
||||
> (a Mixamo-style rig). No SKEL mesh, shape space, or model file is included or exposed in any
|
||||
> shipped product.
|
||||
>
|
||||
> **Scale.** Small and indie: modest commercial revenue, a single developer, no external funding.
|
||||
>
|
||||
> Could you let me know whether an **indie / small-developer commercial licence** is available
|
||||
> for this kind of in-pipeline, non-redistributed use, and the terms and any fee? I'm happy to
|
||||
> provide more detail on the pipeline or sign whatever agreement you need.
|
||||
>
|
||||
> Thank you,
|
||||
> [your name]
|
||||
> monsterrobot.party · [contact email]
|
||||
|
||||
If MPI replies with terms, record the outcome here and flip Decision 1 from Path B to Path A
|
||||
(nothing in the pipeline changes — same engine, the licence just retires the NC risk).
|
||||
|
||||
## Decision 2 — clear m3ultra disk (RESOLVED pending reboot, 2026-07-17)
|
||||
|
||||
Was 98% full (20 GB free). Deleted `unsloth/GLM-4.6-GGUF` (147 GB, re-downloadable, 0 code
|
||||
refs) from the HF cache with John's OK. **But `df` still shows 20 GB** — the freed blocks are
|
||||
pinned by macOS **OS-update snapshots**: a **macOS Tahoe 26.5.2 update is staged** (recommended
|
||||
/ security, "Action: restart") and the box has 18 days uptime. Space realizes on **next
|
||||
reboot**, which also applies the pending update. No `~/.jobs` heartbeat job is running (load is
|
||||
just the Claude desktop app), so a reboot kills nothing tracked.
|
||||
|
||||
→ **John: reboot m3ultra when convenient** → applies 26.5.2 + reclaims 147 GB (→ ~167 GB free,
|
||||
ample for Lane B). Not doing this myself — it's your workstation. Kept the FLUX variants
|
||||
(you deselected them); another ~200 GB is prunable there later if wanted.
|
||||
|
||||
## Decision 3 — test footage (blocker for the real quality call)
|
||||
|
||||
`capture/inbox/` is empty. Whichever human engine Decision 1 unlocks, the honest
|
||||
walk/dance/pick-up-object quality verdict — and all of B2/B3 + the acceptance check — needs your
|
||||
clips. (HSMR has an HF Space + demo images for the MPS/trajectory smoke-test before your
|
||||
footage.) Film the PLAN's three per `docs/CAPTURE.md` (T-pose bookends, feet in frame), drop
|
||||
them in the inbox, then `python3 ingest.py`.
|
||||
|
||||
→ **John: film + ingest walk / dance / pick-up-object when convenient.**
|
||||
|
||||
## Blast radius of the Decision-1 reversal (next session must know)
|
||||
- **Lane B1 is NOT done** — wrong engine selected. Re-opened; PROGRESS un-ticked.
|
||||
- **VOID the old note "MoCapAnything is BVH-direct, skip the retarget rig."** Any human model
|
||||
emits **SMPL or SKEL, not mixamorig** → **Lane C's constraint-retarget IS required again**,
|
||||
and `spec/{smpl,skel}_to_mixamorig.json` is back on the menu.
|
||||
- Preconditions: disk resolves on the m3ultra reboot (Decision 2); the MPS/global-trajectory
|
||||
smoke-test still stands for whichever engine John's pick unlocks.
|
||||
|
||||
---
|
||||
Once Decision 1 is picked and 2–3 clear, the next session installs the chosen human engine,
|
||||
proves MPS + global trajectory, and proceeds to B2 (service-ize) / B3 (jitter).
|
||||
@ -1,24 +1,43 @@
|
||||
# Licenses & legal findings
|
||||
|
||||
Every third-party model/dataset/tool that touches the pipeline gets a row here with
|
||||
its license, commercial status, link, and the date checked. Skeleton now; Lane B fills
|
||||
the pose-model rows, Lane E fills any training-dataset rows. See PLAN.md §Ground rules.
|
||||
Every third-party model/dataset/tool that touches the pipeline gets a row here with its
|
||||
license, commercial status, link, and the date checked. See PLAN.md §Ground rules.
|
||||
|
||||
> **2026-07-17 correction (session 2):** B1's original row billed MoCapAnything as the human
|
||||
> engine — it is an **animal** model (see `DECISIONS.md`). The SMPL/SKEL gate is **LIVE**, not
|
||||
> avoided. Rows below corrected and license-verified.
|
||||
|
||||
## Components
|
||||
|
||||
| Component | Version | License | Commercial OK? | Notes | Checked |
|
||||
|---|---|---|---|---|---|
|
||||
| _pose model (Lane B1 winner)_ | — | — | — | fill after bake-off | — |
|
||||
| SMPL / SMPL-X body model | — | research (Meshcapade) | **NO** by default | commercial needs Meshcapade license; or use joints-only path (PLAN §5) | — |
|
||||
| AMASS dataset | — | academic | **verify** before any shipped training | Lane E only | — |
|
||||
| Mixamo clips | — | Adobe ToS | ship-in-game OK; **not** ML training data | — | — |
|
||||
| Component | License (code / body model) | Commercial OK? | Notes | Checked |
|
||||
|---|---|---|---|---|
|
||||
| **HSMR** (human — candidate engine) | code **MIT** / body **SKEL** | code yes / **model NO** | `IsshikiHugh/HSMR`, CVPR25 oral. Predicts **SKEL** params (biomechanical skeleton, great for retarget). Needs SKEL files (NC) at inference. MPS-runnability unverified. | 2026-07-17 |
|
||||
| SKEL-CF (human — candidate) | code (verify) / body **SKEL** | **model NO** | Coarse-to-fine skeleton+mesh, Nov 2025. Same SKEL gate. | 2026-07-17 |
|
||||
| **SKEL body model** | MPG research | **NO** (default) | "any use for commercial… purposes is prohibited"; commercial via **ps-license@tue.mpg.de**. Same family as SMPL. | 2026-07-17 |
|
||||
| **SMPL / SMPL-X body model** | MPG research (Meshcapade sublicenses) | **NO** (default) | Commercial via **sales@meshcapade.com** (quote-based; incl. SMPL+H/-X/STAR). Gate for all SMPL-based human models. | 2026-07-17 |
|
||||
| GVHMR / WHAM / TRAM / OnlineHMR (human, SMPL) | research/NC (GVHMR NC confirmed) / SMPL | **NO** | Accuracy leaders but **double gate**: NC *code* AND SMPL. GVHMR commercial via xwzhou@zju.edu.cn. Reference/yardstick unless licensed. | 2026-07-17 |
|
||||
| GEM-X (human, shelved) | Apache / SOMA (not SMPL) | yes, but **off-fleet** | NVIDIA commercial-ready; **CUDA-only** → won't run on Apple Silicon. Revisit only if the fleet gains NVIDIA. | 2026-07-17 |
|
||||
| MoCapAnything V2 (**animal** — creature NPCs only) | MIT / none | **YES** (creatures) | `kehong/MoCapAnythingV2-weights` MIT, no SMPL. **71 animal targets, no human/biped/fingers** → NOT the human engine. Keep for creature/centaur (Lane E4) work. | 2026-07-17 |
|
||||
| AMASS dataset | academic | **verify** before shipped training | Lane E only | — |
|
||||
| **BioAMASS v1.0** (downloaded, `~/Documents/skel/bioamass_v1.0`: CMU, DFAUST, MPI_Limits subsets, 376MB) | MPG research (same SKEL account/licence) | **NO** (default) — same NC family; Lane E training that ships needs the same Path-A licence email | SKEL-fitted mocap — ideal Lane E2 autoposer training data *format-wise*; legal gate identical to SKEL itself | 2026-07-17 |
|
||||
| Mixamo clips | Adobe ToS | ship-in-game OK; **not** ML training data | — | — |
|
||||
|
||||
## The SMPL gate (hard stop — John decision)
|
||||
## The SMPL/SKEL gate (hard stop — LIVE, John decision)
|
||||
|
||||
Most SOTA monocular mocap models emit SMPL parameters. Before any commercial use:
|
||||
- **Path A:** buy a Meshcapade commercial license (cheap tiers exist).
|
||||
- **Path B (SMPL-free):** use only the model's joint positions/rotations, never the SMPL
|
||||
mesh/shape files, AND confirm the inference repo's own code license permits commercial
|
||||
use of outputs.
|
||||
Every accurate world-grounded **human** model depends on a Max-Planck body model — **SMPL**
|
||||
(GVHMR/WHAM/TRAM/OnlineHMR) or **SKEL** (HSMR/SKEL-CF) — both research-licensed. Two paths:
|
||||
- **Path A — license it.** Email MPI (SKEL, `ps-license@tue.mpg.de`) or Meshcapade (SMPL,
|
||||
`sales@meshcapade.com`) for an indie commercial license. Clean; quote-based.
|
||||
- **Path B — SMPL-free outputs.** Run an MIT-code model (HSMR) locally, keep only joint
|
||||
rotations, retarget onto our own mixamorig, distribute no Max-Planck file. Grey area: the NC
|
||||
license bars "production of artefacts for commercial purposes." John-accepts-risk.
|
||||
|
||||
Lane B writes the decision to `docs/DECISIONS.md` and stops there if unresolved.
|
||||
Decision lives in `docs/DECISIONS.md`. The human path does not proceed until John picks.
|
||||
|
||||
## Sources (session 2, 2026-07-17)
|
||||
- SKEL license (commercial prohibited): https://skel.is.tue.mpg.de/license.html
|
||||
- HSMR (MIT code, SKEL-based): https://github.com/IsshikiHugh/HSMR ·
|
||||
https://isshikihugh.github.io/HSMR/ · paper https://arxiv.org/html/2503.21751v1
|
||||
- SKEL-CF: https://arxiv.org/html/2511.20157
|
||||
- SMPL commercial licensing (Meshcapade): https://meshcapade.com/infopages/licensing.html
|
||||
- GVHMR (NC): https://github.com/zju3dv/GVHMR · OnlineHMR: https://github.com/Tsukasane/Video-OnlineHMR
|
||||
|
||||
34
engine_patches/_headless.py
Normal file
34
engine_patches/_headless.py
Normal file
@ -0,0 +1,34 @@
|
||||
"""Headless-macOS import shims for the HSMR engine (our fleet is all Apple Silicon).
|
||||
|
||||
pyrender -> PyOpenGL -> EGL cannot load without a GL context, yet several vendored modules
|
||||
`import pyrender` at load time — some even in type annotations (`List[pyrender.Node]`) — so
|
||||
the import chain explodes before any of our code runs. We never render in-process; Blender
|
||||
is the fleet renderer (Lane C). So we drop permissive fakes for the pyrender + OpenGL stack
|
||||
into sys.modules, letting the chain import while keeping everything else (ColorPalette,
|
||||
Wis3D, the models) real.
|
||||
|
||||
Import this BEFORE the first `lib.*` import: `import _headless # noqa: F401`
|
||||
|
||||
# ponytail: fakes, not a real headless GL backend (osmesa/egl). We don't render here, so a
|
||||
# working GL context would be dead weight. Wire a real one only if in-process render is ever
|
||||
# needed off-fleet.
|
||||
"""
|
||||
import sys
|
||||
import types
|
||||
|
||||
|
||||
class _Any:
|
||||
def __call__(self, *a, **k): return self
|
||||
def __getattr__(self, n): return self
|
||||
|
||||
|
||||
class _AnyModule(types.ModuleType):
|
||||
def __getattr__(self, n):
|
||||
if n.startswith('__') and n.endswith('__'):
|
||||
raise AttributeError(n)
|
||||
return _Any()
|
||||
|
||||
|
||||
# setdefault: never clobber a real module that a caller already imported on purpose.
|
||||
for _n in ('pyrender', 'OpenGL', 'OpenGL.GL', 'OpenGL.error', 'OpenGL.platform'):
|
||||
sys.modules.setdefault(_n, _AnyModule(_n))
|
||||
77
engine_patches/lib/modeling/pipelines/vitdet/__init__.py
Normal file
77
engine_patches/lib/modeling/pipelines/vitdet/__init__.py
Normal file
@ -0,0 +1,77 @@
|
||||
"""
|
||||
Person detector — torchvision drop-in replacement for HSMR's detectron2/ViTDet.
|
||||
|
||||
Why: detectron2 + chumpy won't build from source on Apple Silicon (the Lane B blocker),
|
||||
and this module's `import detectron2` was a load-time landmine for everything that touches
|
||||
`lib.kits.hsmr_demo` (smoke_test.py, the future pose_engine.py). torchvision ships a
|
||||
COCO-pretrained Faster R-CNN with ungated weights and no build step.
|
||||
|
||||
Same public contract as the original, so run_demo.py / smoke_test.py / pose_engine.py
|
||||
import it unchanged:
|
||||
|
||||
build_detector(...) -> callable(raw_imgs) -> (dets, downsample_ratios)
|
||||
|
||||
dets[i] : dict of CPU tensors (empty tensors if no person) —
|
||||
pred_classes : all 0 (== hsmr_demo CLASS_HUMAN_ID)
|
||||
scores : float
|
||||
pred_boxes : (N, 4) left-upper-right-bottom pixels
|
||||
downsample_ratios : one float per image; boxes are in native-resolution pixels -> 1.0
|
||||
|
||||
Contract is checked by test_detector.py (runs today, no SKEL needed).
|
||||
|
||||
# ponytail: ~35 lines replaces the whole detectron2 stack. Detection on CPU is the pose-engine
|
||||
# wall-clock bottleneck, so this uses the fast mobilenet backbone and callers detect every-Nth frame.
|
||||
"""
|
||||
import numpy as np
|
||||
import torch
|
||||
from tqdm import tqdm
|
||||
from torchvision.models.detection import (
|
||||
fasterrcnn_mobilenet_v3_large_fpn,
|
||||
FasterRCNN_MobileNet_V3_Large_FPN_Weights,
|
||||
)
|
||||
|
||||
_COCO_PERSON = 1 # torchvision COCO label id for "person" (0 == background)
|
||||
|
||||
|
||||
class _PersonDetector:
|
||||
"""Callable over a list of RGB HxWx3 images (what HSMR's load_inputs yields).
|
||||
uint8 or float, [0,255] or [0,1] are all handled."""
|
||||
|
||||
def __init__(self, device='cpu', score_thresh=0.25):
|
||||
# score_thresh 0.25 mirrors the old detectron2 test_score_thresh; hsmr_demo's
|
||||
# _img_det2patches re-filters at 0.5, so this only widens recall a little.
|
||||
# mobilenet_v3 (not resnet50) — ~5-10x faster on CPU, which is the pose-engine wall-clock
|
||||
# bottleneck (Lane B finding). Plenty accurate for one prominent, in-frame subject.
|
||||
weights = FasterRCNN_MobileNet_V3_Large_FPN_Weights.DEFAULT
|
||||
self.model = fasterrcnn_mobilenet_v3_large_fpn(
|
||||
weights=weights, box_score_thresh=score_thresh,
|
||||
).eval().to(device)
|
||||
self.device = device
|
||||
|
||||
@torch.no_grad()
|
||||
def __call__(self, raw_imgs):
|
||||
dets, ratios = [], []
|
||||
for img in tqdm(raw_imgs, desc='Detecting'):
|
||||
t = torch.as_tensor(np.ascontiguousarray(img), device=self.device).float()
|
||||
if float(t.max()) > 1.5: # [0,255] -> [0,1]
|
||||
t = t / 255.0
|
||||
t = t.permute(2, 0, 1) # HWC -> CHW (the model's own transform caps size)
|
||||
out = self.model([t])[0]
|
||||
keep = out['labels'] == _COCO_PERSON
|
||||
dets.append({
|
||||
'pred_classes': torch.zeros(int(keep.sum()), dtype=torch.long), # human -> 0
|
||||
'scores' : out['scores'][keep].cpu(),
|
||||
'pred_boxes' : out['boxes'][keep].cpu(), # xyxy == left-upper-right-bottom
|
||||
})
|
||||
ratios.append(1.0) # detected at native resolution
|
||||
return dets, ratios
|
||||
|
||||
|
||||
def build_detector(batch_size=1, max_img_size=512, device='cpu'):
|
||||
"""Signature-compatible with the old detectron2 builder. batch_size / max_img_size are
|
||||
accepted for parity but unused — detection runs per-frame at native resolution.
|
||||
|
||||
# ponytail: detector pinned to CPU regardless of `device`. It's cheap next to the
|
||||
# ViT-H HMR pass, and CPU sidesteps the MPS coverage gaps in RoIAlign/NMS. Flip to
|
||||
# `device` only if per-frame detection ever dominates on long videos."""
|
||||
return _PersonDetector(device='cpu')
|
||||
100
engine_patches/smoke_test.py
Normal file
100
engine_patches/smoke_test.py
Normal file
@ -0,0 +1,100 @@
|
||||
#!/usr/bin/env python
|
||||
"""
|
||||
HSMR smoke test on Apple Silicon (MPS) — Path B engine verification.
|
||||
|
||||
Answers the Lane B smoke-test questions on HSMR's bundled demo clips (no John footage,
|
||||
no SKEL license needed beyond the model files John downloaded):
|
||||
1. Does the HSMR forward run on MPS? (device + CPU-fallback)
|
||||
2. Timing (target < 2x realtime).
|
||||
3. Pose sanity (shape, NaNs) + an overlay render to eyeball quality.
|
||||
|
||||
Detection uses the torchvision person detector (lib.modeling.pipelines.vitdet, the
|
||||
detectron2-free drop-in). The one remaining gate is SKEL: build_inference_pipeline
|
||||
instantiates the SKEL body model, so this completes only once John's SKEL files are in place.
|
||||
|
||||
Run (once SKEL files are in data_inputs/body_models/skel/):
|
||||
cd ~/Documents/MOCAPGOD/.engine/HSMR && source .venv/bin/activate
|
||||
PYTHONPATH=. python smoke_test.py -i data_inputs/demo/example_videos/gymnasts.mp4
|
||||
PYTHONPATH=. python smoke_test.py -i data_inputs/demo/example_imgs # image folder
|
||||
"""
|
||||
import os, time, argparse
|
||||
os.environ.setdefault('PYTORCH_ENABLE_MPS_FALLBACK', '1') # some ops lack MPS kernels
|
||||
|
||||
import _headless # noqa: F401 fake pyrender/OpenGL for headless macOS — MUST precede lib.* imports
|
||||
|
||||
# One import line: every helper is an attribute of hsmr_demo (via its star-imports).
|
||||
from lib.kits.hsmr_demo import (
|
||||
load_inputs, imgs_det2patches, build_inference_pipeline, prepare_mesh,
|
||||
visualize_full_img, IMG_MEAN_255, IMG_STD_255, asb, assemble_dict,
|
||||
get_logger, save_video, save_img, np, torch, Path, DEFAULT_HSMR_ROOT,
|
||||
build_detector,
|
||||
)
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('-i', '--input_path', required=True)
|
||||
ap.add_argument('-o', '--output_path', default='data_outputs/smoke')
|
||||
ap.add_argument('-d', '--device', default='mps')
|
||||
ap.add_argument('-m', '--model_root', default=None)
|
||||
ap.add_argument('--rec_bs', type=int, default=64)
|
||||
ap.add_argument('--ignore_skel', action='store_true')
|
||||
args = ap.parse_args()
|
||||
|
||||
log = get_logger(brief=True)
|
||||
model_root = Path(args.model_root) if args.model_root else DEFAULT_HSMR_ROOT
|
||||
out = Path(args.output_path); out.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Inputs (reuse the demo loader; it only reads .input_path / .input_type).
|
||||
fake = argparse.Namespace(input_path=args.input_path, input_type='auto')
|
||||
raw_imgs, meta = load_inputs(fake)
|
||||
log.info(f'device={args.device} type={meta["type"]} frames={len(raw_imgs)}')
|
||||
|
||||
# Detection -> patches (real torchvision person detector, on CPU).
|
||||
dets, ratios = build_detector(device=args.device)(raw_imgs)
|
||||
patches, det_meta = imgs_det2patches(raw_imgs, dets, ratios, 5)
|
||||
log.info(f'patches={len(patches)}')
|
||||
|
||||
# Build pipeline — instantiates SKEL, so this is where a missing SKEL file fails.
|
||||
pipe = build_inference_pipeline(model_root=model_root, device=args.device)
|
||||
|
||||
# Recovery loop (mirrors run_demo), timed.
|
||||
t0 = time.time()
|
||||
pd_params, pd_cam_t = [], []
|
||||
for bw in asb(total=len(patches), bs_scope=args.rec_bs, enable_tqdm=True):
|
||||
p = patches[bw.sid:bw.eid]
|
||||
pn = ((p - IMG_MEAN_255) / IMG_STD_255).transpose(0, 3, 1, 2)
|
||||
with torch.no_grad():
|
||||
o = pipe(pn)
|
||||
pd_params.append({k: v.detach().cpu().clone() for k, v in o['pd_params'].items()})
|
||||
pd_cam_t.append(o['pd_cam_t'].detach().cpu().clone())
|
||||
dt = time.time() - t0
|
||||
pd_params = assemble_dict(pd_params, expand_dim=False)
|
||||
pd_cam_t = torch.cat(pd_cam_t, dim=0)
|
||||
|
||||
poses = pd_params['poses']
|
||||
fps = len(patches) / dt if dt else 0
|
||||
log.info(f'RECOVERY ok: {len(patches)} patches in {dt:.2f}s ({fps:.2f} fps) '
|
||||
f'poses={tuple(poses.shape)} nan={bool(torch.isnan(poses).any())}')
|
||||
|
||||
# Dump raw params regardless (this is the Path-B payload: SKEL rotations).
|
||||
np.savez(out / 'params.npz', poses=poses.numpy(),
|
||||
betas=pd_params['betas'].numpy(), cam_t=pd_cam_t.numpy())
|
||||
log.info(f'params -> {out/"params.npz"}')
|
||||
|
||||
# Mesh geometry for visual QC in Blender (pyrender/EGL is dead headless on macOS, and
|
||||
# Blender is the fleet renderer + Lane C's tool). prepare_mesh uses skel_model, no GL.
|
||||
try:
|
||||
m_skin, m_skel = prepare_mesh(pipe, pd_params)
|
||||
np.savez(out / 'mesh_qc.npz',
|
||||
skin_v=m_skin['v'].detach().cpu().numpy(), skin_f=np.asarray(m_skin['f']),
|
||||
skel_v=m_skel['v'].detach().cpu().numpy(), skel_f=np.asarray(m_skel['f']))
|
||||
log.info(f'mesh geometry -> {out/"mesh_qc.npz"} (import in Blender to eyeball)')
|
||||
except Exception as e:
|
||||
log.warning(f'prepare_mesh failed (params still saved): {e!r}')
|
||||
|
||||
log.info('smoke test complete.')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
71
engine_patches/test_detector.py
Normal file
71
engine_patches/test_detector.py
Normal file
@ -0,0 +1,71 @@
|
||||
#!/usr/bin/env python
|
||||
"""Detector smoke check — runs TODAY, no SKEL, no HMR forward pass.
|
||||
|
||||
Proves the torchvision person detector (lib/modeling/pipelines/vitdet) imports without
|
||||
detectron2 and honors the contract lib.kits.hsmr_demo._img_det2patches depends on.
|
||||
|
||||
cd ~/Documents/MOCAPGOD/.engine/HSMR && source .venv/bin/activate
|
||||
PYTHONPATH=. python test_detector.py
|
||||
"""
|
||||
from pathlib import Path
|
||||
import cv2
|
||||
import torch
|
||||
|
||||
import _headless # noqa: F401 fake pyrender/OpenGL for headless macOS — MUST precede lib.* imports
|
||||
from lib.modeling.pipelines.vitdet import build_detector # must import without detectron2
|
||||
from lib.kits.hsmr_demo import imgs_det2patches # the real consumer of detector output
|
||||
|
||||
DEMO = Path('data_inputs/demo/example_imgs')
|
||||
DET_THRESHOLD_SCORE = 0.5 # matches hsmr_demo._img_det2patches
|
||||
|
||||
|
||||
def main():
|
||||
fns = sorted(p for p in DEMO.glob('*')
|
||||
if p.suffix.lower() in {'.jpg', '.jpeg', '.png', '.webp'})
|
||||
assert fns, f'no demo images under {DEMO}'
|
||||
|
||||
imgs = []
|
||||
for fn in fns:
|
||||
bgr = cv2.imread(str(fn))
|
||||
assert bgr is not None, f'cv2 failed to read {fn}'
|
||||
imgs.append(cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)) # HSMR feeds RGB
|
||||
|
||||
dets, ratios = build_detector(device='cpu')(imgs)
|
||||
assert len(dets) == len(imgs) == len(ratios), 'per-image list length mismatch'
|
||||
|
||||
n_person_imgs = 0
|
||||
for fn, img, d, r in zip(fns, imgs, dets, ratios):
|
||||
H, W = img.shape[:2]
|
||||
assert set(d) == {'pred_classes', 'scores', 'pred_boxes'}, f'bad keys {set(d)}'
|
||||
assert d['pred_classes'].dtype == torch.long
|
||||
assert (d['pred_classes'] == 0).all(), 'human class must be 0 (CLASS_HUMAN_ID)'
|
||||
assert d['pred_boxes'].ndim == 2 and d['pred_boxes'].shape[1] == 4, 'boxes must be (N,4)'
|
||||
assert r == 1.0, 'native-resolution detection -> ratio 1.0'
|
||||
|
||||
strong = d['scores'] > DET_THRESHOLD_SCORE
|
||||
if strong.any():
|
||||
n_person_imgs += 1
|
||||
x1, y1, x2, y2 = d['pred_boxes'][strong].unbind(1)
|
||||
assert (x2 > x1).all() and (y2 > y1).all(), 'boxes must be lurb (x2>x1, y2>y1)'
|
||||
assert (x1 >= -1).all() and (y1 >= -1).all() \
|
||||
and (x2 <= W + 1).all() and (y2 <= H + 1).all(), 'box out of frame'
|
||||
print(f' ✓ {fn.name}: {int(strong.sum())} person(s), '
|
||||
f'top score {float(d["scores"][strong].max()):.2f}')
|
||||
else:
|
||||
print(f' · {fn.name}: no confident person')
|
||||
|
||||
# Every demo image has an obvious human; allow one miss for robustness slack.
|
||||
assert n_person_imgs >= len(imgs) - 1, \
|
||||
f'expected a person in ~all demo imgs, got {n_person_imgs}/{len(imgs)}'
|
||||
print(f'DETECTOR OK: person found in {n_person_imgs}/{len(imgs)} demo images.')
|
||||
|
||||
# Close the loop: feed detector output through HSMR's real cropper (SKEL-free) and
|
||||
# confirm it yields the 256x256 patches the HMR forward pass expects.
|
||||
patches, det_meta = imgs_det2patches(imgs, dets, ratios, max_instances_per_img=5)
|
||||
assert patches.ndim == 4 and patches.shape[1:] == (256, 256, 3), f'bad patch shape {patches.shape}'
|
||||
assert patches.shape[0] == sum(det_meta['n_patch_per_img']), 'patch count / meta mismatch'
|
||||
print(f'PATCHES OK: {patches.shape[0]} human patches @ 256x256 — ready for the HMR pass.')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
57
ingest.py
57
ingest.py
@ -1,15 +1,17 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Lane A ingest — capture/inbox/*.{mp4,mov} → validate → 30fps h264 → rename → rsync to
|
||||
the m3ultra pose queue → capture/LEDGER.tsv.
|
||||
"""Lane A ingest — capture/inbox/*.{mp4,mov} → validate → 30fps h264 → rename → pose queue
|
||||
→ capture/LEDGER.tsv.
|
||||
|
||||
python3 ingest.py # scan capture/inbox/ and ship everything
|
||||
python3 ingest.py # scan capture/inbox/ → local queue/ (the v1 default)
|
||||
python3 ingest.py clip.mov --name dj_scratch_01 # one file, explicit clip name
|
||||
python3 ingest.py --remote-queue # ship to the m3ultra queue instead (rsync)
|
||||
python3 ingest.py --selfcheck # run built-in probe/validate self-test, no network
|
||||
|
||||
The pose engine (HSMR) currently runs on ultra, so ingest defaults to the LOCAL queue/ dir.
|
||||
--remote-queue [user@host:/path] rsyncs to m3ultra instead (default target: MOCAP_REMOTE).
|
||||
Rejects (portrait / malformed) move to capture/rejected/ with a reason in the ledger.
|
||||
Successful originals move to capture/done/; on rsync failure the normalized file is kept
|
||||
in capture/staged/ so a re-run retries the ship without re-encoding.
|
||||
Remote defaults to m3ultra; override with MOCAP_REMOTE=user@host:/path.
|
||||
Successful originals move to capture/done/; on a failed remote ship the normalized file is
|
||||
kept in capture/staged/ so a re-run retries without re-encoding.
|
||||
"""
|
||||
import argparse, datetime, json, os, re, shutil, subprocess, sys, tempfile
|
||||
from pathlib import Path
|
||||
@ -20,7 +22,9 @@ STAGED = ROOT / "capture" / "staged"
|
||||
DONE = ROOT / "capture" / "done"
|
||||
REJECTED = ROOT / "capture" / "rejected"
|
||||
LEDGER = ROOT / "capture" / "LEDGER.tsv"
|
||||
REMOTE = os.environ.get("MOCAP_REMOTE", "m3ultra@100.89.131.57:~/Documents/MOCAPGOD/queue")
|
||||
QUEUE = ROOT / "queue" # v1 default: pose engine (HSMR) runs on ultra, next to the repo
|
||||
# ponytail: v1 inference on ultra; migrate to m3ultra (--remote-queue) when a real throughput need appears
|
||||
REMOTE_DEFAULT = os.environ.get("MOCAP_REMOTE", "m3ultra@100.89.131.57:~/Documents/MOCAPGOD/queue")
|
||||
VIDEO_EXT = {".mp4", ".mov", ".m4v"}
|
||||
HEADERS = ["utc", "original", "output", "fps_in", "dur_s", "resolution", "audio", "status", "note"]
|
||||
|
||||
@ -90,13 +94,21 @@ def normalize(src, dst):
|
||||
raise ValueError("ffmpeg normalize failed: " + tail)
|
||||
|
||||
|
||||
def ship(staged):
|
||||
host, _, rpath = REMOTE.partition(":")
|
||||
run(["ssh", host, f"mkdir -p {rpath}"]) # queue dir may not exist yet
|
||||
r = run(["rsync", "-az", str(staged), REMOTE + "/"])
|
||||
def deliver(staged, remote):
|
||||
"""Put the normalized clip in the pose queue. Local queue/ dir by default; rsync to a
|
||||
remote (user@host:/path) when --remote-queue is used. Returns the destination string."""
|
||||
if remote is None:
|
||||
QUEUE.mkdir(parents=True, exist_ok=True)
|
||||
dst = QUEUE / staged.name
|
||||
shutil.move(str(staged), dst) # same-fs move; local delivery can't half-fail
|
||||
return str(dst)
|
||||
host, _, rpath = remote.partition(":")
|
||||
run(["ssh", host, f"mkdir -p {rpath}"]) # remote queue dir may not exist yet
|
||||
r = run(["rsync", "-az", str(staged), remote + "/"])
|
||||
if r.returncode != 0:
|
||||
tail = (r.stderr.strip().splitlines() or ["?"])[-1]
|
||||
raise ConnectionError(f"rsync to {REMOTE} failed: {tail}")
|
||||
raise ConnectionError(f"rsync to {remote} failed: {tail}")
|
||||
return f"{remote}/{staged.name}"
|
||||
|
||||
|
||||
def ledger(row):
|
||||
@ -112,7 +124,7 @@ def slug(s):
|
||||
return re.sub(r"[^A-Za-z0-9_-]+", "_", s).strip("_") or "clip"
|
||||
|
||||
|
||||
def process_one(src, name=None):
|
||||
def process_one(src, name=None, remote=None):
|
||||
src = Path(src)
|
||||
out_name = f"{datetime.date.today():%Y%m%d}_{slug(name or src.stem)}.mp4"
|
||||
ts = datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
@ -139,9 +151,9 @@ def process_one(src, name=None):
|
||||
res, audio = f"{info['w']}x{info['h']}", "yes" if info["audio"] else "no"
|
||||
staged = STAGED / out_name
|
||||
try:
|
||||
if not staged.exists(): # skip re-encode when retrying a failed ship
|
||||
if not staged.exists(): # skip re-encode when retrying a failed remote ship
|
||||
normalize(src, staged)
|
||||
ship(staged)
|
||||
dest = deliver(staged, remote)
|
||||
except (ValueError, ConnectionError) as e:
|
||||
ledger([ts, src.name, out_name, info["fps"], f"{info['dur']:.1f}", res, audio, "FAIL", str(e)])
|
||||
print(f" ! {src.name}: {e} (left in inbox; staged kept for retry)")
|
||||
@ -152,7 +164,7 @@ def process_one(src, name=None):
|
||||
staged.unlink(missing_ok=True)
|
||||
ledger([ts, src.name, out_name, info["fps"], f"{info['dur']:.1f}", res, audio,
|
||||
"OK", "" if info["audio"] else "audio stripped (none present)"])
|
||||
print(f" ✓ {src.name} → {REMOTE}/{out_name} ({res} {info['fps']}fps {info['dur']:.1f}s)")
|
||||
print(f" ✓ {src.name} → {dest} ({res} {info['fps']}fps {info['dur']:.1f}s)")
|
||||
return True
|
||||
|
||||
|
||||
@ -174,24 +186,29 @@ def selfcheck():
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description="Lane A: ingest phone videos into the m3ultra pose queue.")
|
||||
ap = argparse.ArgumentParser(description="Lane A: ingest phone videos into the pose queue.")
|
||||
ap.add_argument("video", nargs="?", help="single file to ingest (default: scan capture/inbox/)")
|
||||
ap.add_argument("--name", help="clip name for the output (default: source filename)")
|
||||
ap.add_argument("--remote-queue", nargs="?", const=REMOTE_DEFAULT, default=None,
|
||||
metavar="USER@HOST:PATH",
|
||||
help=f"rsync to a remote pose queue instead of the local queue/ dir "
|
||||
f"(default target: {REMOTE_DEFAULT})")
|
||||
ap.add_argument("--selfcheck", action="store_true", help="run the built-in self-test and exit")
|
||||
args = ap.parse_args()
|
||||
|
||||
if args.selfcheck:
|
||||
return selfcheck()
|
||||
dest_label = args.remote_queue if args.remote_queue else str(QUEUE)
|
||||
if args.video:
|
||||
return 0 if process_one(args.video, args.name) else 1
|
||||
return 0 if process_one(args.video, args.name, args.remote_queue) else 1
|
||||
|
||||
INBOX.mkdir(parents=True, exist_ok=True)
|
||||
vids = sorted(p for p in INBOX.iterdir() if p.suffix.lower() in VIDEO_EXT)
|
||||
if not vids:
|
||||
print(f"inbox empty: {INBOX}")
|
||||
return 0
|
||||
print(f"ingesting {len(vids)} file(s) from {INBOX} → {REMOTE}")
|
||||
n_ok = sum(process_one(p) for p in vids)
|
||||
print(f"ingesting {len(vids)} file(s) from {INBOX} → {dest_label}")
|
||||
n_ok = sum(process_one(p, remote=args.remote_queue) for p in vids)
|
||||
print(f"done: {n_ok}/{len(vids)} ok")
|
||||
return 0 if n_ok == len(vids) else 1
|
||||
|
||||
|
||||
427
pose_engine.py
Normal file
427
pose_engine.py
Normal file
@ -0,0 +1,427 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Lane B (B2 + B3) pose engine — queue video → HSMR/SKEL → BVH@30fps + sidecar JSON.
|
||||
|
||||
# runs under the HSMR engine venv (torch/torchvision/skel live there):
|
||||
.engine/HSMR/.venv/bin/python pose_engine.py # drain the queue/ dir
|
||||
.engine/HSMR/.venv/bin/python pose_engine.py queue/clip.mp4 # one clip
|
||||
python3 pose_engine.py --selfcheck # BVH math self-test, no engine
|
||||
|
||||
Pipeline per clip: person detection (torchvision, every-Nth-frame + box reuse — single-subject
|
||||
footage, this is the wall-clock win) → single best box per frame → HSMR ViT-H recovery → SKEL
|
||||
params (poses q46, betas, cam_t) → SKEL forward for per-joint global orientations → rigid-bone
|
||||
BVH on SKEL's 24-joint skeleton + sidecar. Root is camera-space (root_space="camera"); Lane C
|
||||
retargets onto mixamorig via spec/skel_to_mixamorig.json and grounds locomotion from foot contacts.
|
||||
|
||||
Output: out/<name>/motion.bvh + motion.json. Heartbeat: ~/.jobs/pose_engine.
|
||||
"""
|
||||
import argparse, datetime, json, os, subprocess, sys, time
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
from scipy.signal import savgol_filter
|
||||
from scipy.spatial.transform import Rotation
|
||||
|
||||
REPO = Path(__file__).resolve().parent
|
||||
ENGINE = REPO / ".engine" / "HSMR"
|
||||
SKELDIR = ENGINE / "data_inputs" / "body_models" / "skel"
|
||||
QUEUE = REPO / "queue"
|
||||
OUT = REPO / "out"
|
||||
JOB = Path.home() / ".jobs" / "pose_engine"
|
||||
MODEL_TAG = "HSMR-ViTH-SKEL"
|
||||
VERSION = "1.0"
|
||||
EULER = "ZYX" # scipy intrinsic; BVH channels declared Z,Y,X to match
|
||||
VIDEO_EXT = {".mp4", ".mov", ".m4v"}
|
||||
|
||||
|
||||
def hb(msg):
|
||||
JOB.parent.mkdir(parents=True, exist_ok=True)
|
||||
JOB.write_text(f"{int(time.time())} {msg}\n")
|
||||
print(f">>> [{datetime.datetime.now():%H:%M:%S}] {msg}", flush=True)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- BVH math (engine-free) ----
|
||||
|
||||
def probe_fps(video, default=30.0):
|
||||
r = subprocess.run(["ffprobe", "-v", "error", "-select_streams", "v:0",
|
||||
"-show_entries", "stream=avg_frame_rate", "-of", "csv=p=0", str(video)],
|
||||
capture_output=True, text=True)
|
||||
try:
|
||||
n, _, d = r.stdout.strip().partition("/")
|
||||
f = float(n) / float(d or 1)
|
||||
return round(f, 3) if f > 0 else default
|
||||
except (ValueError, ZeroDivisionError):
|
||||
return default
|
||||
|
||||
|
||||
def smooth_rotations(W, win, poly):
|
||||
"""Savitzky–Golay low-pass on a (F,3,3) global-rotation track, filtered as hemisphere-
|
||||
aligned quaternions so it doesn't fight Euler wrap or quaternion double-cover."""
|
||||
F = len(W)
|
||||
q = Rotation.from_matrix(W).as_quat() # (F,4) xyzw
|
||||
for i in range(1, F):
|
||||
if float(q[i] @ q[i - 1]) < 0:
|
||||
q[i] = -q[i]
|
||||
w = _odd_window(win, F)
|
||||
if w and w > poly:
|
||||
q = savgol_filter(q, w, poly, axis=0)
|
||||
q /= np.linalg.norm(q, axis=1, keepdims=True)
|
||||
return Rotation.from_quat(q).as_matrix()
|
||||
|
||||
|
||||
def smooth_xyz(P, win, poly):
|
||||
w = _odd_window(win, len(P))
|
||||
return savgol_filter(P, w, poly, axis=0) if (w and w > poly) else P
|
||||
|
||||
|
||||
def _odd_window(win, n):
|
||||
w = min(win, n if n % 2 else n - 1) # ≤ n, odd
|
||||
return w if w >= 3 else 0 # too short to filter → skip
|
||||
|
||||
|
||||
def locals_from_globals(W, parents):
|
||||
"""BVH local rotations Q[j] = W[parent]^T · W[j] (root local == world)."""
|
||||
Q = np.empty_like(W)
|
||||
for j, p in enumerate(parents):
|
||||
Q[j] = W[j] if p < 0 else W[p].T @ W[j]
|
||||
return Q
|
||||
|
||||
|
||||
def write_bvh(path, names, parents, offsets, root_pos, local_rot, fps):
|
||||
"""SKEL-skeleton BVH. local_rot: (F,J,3,3) local rotations; root_pos: (F,3); offsets: (J,3)."""
|
||||
F, J = local_rot.shape[:2]
|
||||
children = {j: [c for c, p in enumerate(parents) if p == j] for j in range(J)}
|
||||
ROT = "Zrotation Yrotation Xrotation" # order matches EULER='ZYX'
|
||||
lines = []
|
||||
|
||||
def rec(j, depth):
|
||||
pad = "\t" * depth
|
||||
tag = "ROOT" if parents[j] < 0 else "JOINT"
|
||||
lines.append(f"{pad}{tag} {names[j]}")
|
||||
lines.append(f"{pad}{{")
|
||||
ox, oy, oz = offsets[j]
|
||||
lines.append(f"{pad}\tOFFSET {ox:.6f} {oy:.6f} {oz:.6f}")
|
||||
if parents[j] < 0:
|
||||
lines.append(f"{pad}\tCHANNELS 6 Xposition Yposition Zposition {ROT}")
|
||||
else:
|
||||
lines.append(f"{pad}\tCHANNELS 3 {ROT}")
|
||||
for c in children[j]:
|
||||
rec(c, depth + 1)
|
||||
if not children[j]: # BVH needs an End Site for leaves
|
||||
lines.append(f"{pad}\tEnd Site")
|
||||
lines.append(f"{pad}\t{{")
|
||||
lines.append(f"{pad}\t\tOFFSET 0.000000 0.100000 0.000000")
|
||||
lines.append(f"{pad}\t}}")
|
||||
lines.append(f"{pad}}}")
|
||||
|
||||
root = next(j for j, p in enumerate(parents) if p < 0)
|
||||
lines.append("HIERARCHY")
|
||||
rec(root, 0)
|
||||
lines.append("MOTION")
|
||||
lines.append(f"Frames: {F}")
|
||||
lines.append(f"Frame Time: {1.0/fps:.6f}")
|
||||
|
||||
order_zyx = [0, 1, 2] # as_euler('ZYX') already returns Z,Y,X
|
||||
dfs = [] # channel joint order = hierarchy DFS
|
||||
def dfs_order(j):
|
||||
dfs.append(j)
|
||||
for c in children[j]:
|
||||
dfs_order(c)
|
||||
dfs_order(root)
|
||||
|
||||
for f in range(F):
|
||||
row = []
|
||||
for j in dfs:
|
||||
zyx = Rotation.from_matrix(local_rot[f, j]).as_euler(EULER, degrees=True)[order_zyx]
|
||||
if j == root:
|
||||
px, py, pz = root_pos[f]
|
||||
row += [f"{px:.6f}", f"{py:.6f}", f"{pz:.6f}"]
|
||||
row += [f"{zyx[0]:.6f}", f"{zyx[1]:.6f}", f"{zyx[2]:.6f}"]
|
||||
lines.append(" ".join(row))
|
||||
Path(path).write_text("\n".join(lines) + "\n")
|
||||
return dfs
|
||||
|
||||
|
||||
def bvh_fk_positions(names, parents, offsets, root_pos, local_rot):
|
||||
"""Reconstruct world joint positions from BVH data (verification / self-check)."""
|
||||
F, J = local_rot.shape[:2]
|
||||
pos = np.zeros((F, J, 3)); Wg = np.zeros((F, J, 3, 3))
|
||||
for j, p in enumerate(parents):
|
||||
if p < 0:
|
||||
Wg[:, j] = local_rot[:, j]
|
||||
pos[:, j] = root_pos + np.einsum("fab,b->fa", Wg[:, j], offsets[j])
|
||||
else:
|
||||
Wg[:, j] = Wg[:, p] @ local_rot[:, j]
|
||||
pos[:, j] = pos[:, p] + np.einsum("fab,b->fa", Wg[:, p], offsets[j])
|
||||
return pos
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ HSMR/SKEL (engine) ------
|
||||
|
||||
def build_engine(device, gender):
|
||||
"""Load the engine once (imports, headless shims, SKEL body model). Detector + ViT-H pipeline
|
||||
are built lazily on first recovery, so a --from-params re-run and the self-test skip them and
|
||||
a queue amortizes the load across clips."""
|
||||
os.environ.setdefault("PYTORCH_ENABLE_MPS_FALLBACK", "1")
|
||||
sys.path.insert(0, str(ENGINE))
|
||||
sys.path.insert(0, str(ENGINE / "thirdparty" / "SKEL"))
|
||||
import _headless # noqa: F401 headless pyrender/OpenGL shims — before any lib.* import
|
||||
from lib.kits.hsmr_demo import (
|
||||
load_inputs, imgs_det2patches, build_inference_pipeline, build_detector,
|
||||
asb, assemble_dict, IMG_MEAN_255, IMG_STD_255, DEFAULT_HSMR_ROOT,
|
||||
)
|
||||
import torch
|
||||
from skel.skel_model import SKEL
|
||||
from skel.kin_skel import skel_joints_name
|
||||
os.chdir(ENGINE) # HSMR resolves model/cfg paths relative to its own root
|
||||
skel = SKEL(gender=gender, model_path=str(SKELDIR)).eval()
|
||||
return dict(load_inputs=load_inputs, imgs_det2patches=imgs_det2patches, asb=asb,
|
||||
assemble_dict=assemble_dict, IMG_MEAN_255=IMG_MEAN_255, IMG_STD_255=IMG_STD_255,
|
||||
torch=torch, device=device, skel=skel, skel_names=skel_joints_name,
|
||||
_build_detector=build_detector, _build_pipe=build_inference_pipeline,
|
||||
_hsmr_root=DEFAULT_HSMR_ROOT, det=None, pipe=None)
|
||||
|
||||
|
||||
def _ensure_hmr(E):
|
||||
if E["pipe"] is None: # build detector + ViT-H pipeline once, on first recovery
|
||||
hb("build detector + HSMR ViT-H pipeline (once)")
|
||||
E["det"] = E["_build_detector"](device=E["device"])
|
||||
E["pipe"] = E["_build_pipe"](model_root=E["_hsmr_root"], device=E["device"])
|
||||
|
||||
|
||||
def _single_subject_dets(det, frames, every, torch):
|
||||
"""Detect on every-Nth frame, keep one best (largest, confident) box per frame, forward/
|
||||
backward-fill gaps so patches align 1:1 with frames. Single-subject footage assumption."""
|
||||
n = len(frames)
|
||||
key = list(range(0, n, every))
|
||||
kdets, _ = det([frames[i] for i in key])
|
||||
best = []
|
||||
for d in kdets:
|
||||
s, b = d["scores"], d["pred_boxes"]
|
||||
m = s > 0.5
|
||||
if bool(m.any()):
|
||||
bb = b[m]
|
||||
area = (bb[:, 2] - bb[:, 0]) * (bb[:, 3] - bb[:, 1])
|
||||
k = int(area.argmax())
|
||||
best.append({"pred_classes": torch.zeros(1, dtype=torch.long),
|
||||
"scores": bb.new_tensor([float(s[m][k])]), "pred_boxes": bb[k:k + 1]})
|
||||
else:
|
||||
best.append(None)
|
||||
valid = [i for i, x in enumerate(best) if x is not None]
|
||||
if not valid:
|
||||
raise RuntimeError("no confident person detected in any sampled frame")
|
||||
dets = []
|
||||
for i in range(n):
|
||||
ki = i // every
|
||||
if best[ki] is None:
|
||||
ki = min(valid, key=lambda v: abs(v - ki))
|
||||
dets.append(best[ki])
|
||||
return dets, [1.0] * n
|
||||
|
||||
|
||||
def run_recovery(E, video, detect_every):
|
||||
"""video → (poses (F,46), betas (F,10), cam_t (F,3)) with one pose per frame."""
|
||||
torch = E["torch"]
|
||||
_ensure_hmr(E)
|
||||
fake = argparse.Namespace(input_path=str(video), input_type="auto")
|
||||
frames, meta = E["load_inputs"](fake)
|
||||
hb(f"detect (every {detect_every}f) over {len(frames)} frames")
|
||||
dets, ratios = _single_subject_dets(E["det"], frames, detect_every, torch)
|
||||
patches, det_meta = E["imgs_det2patches"](frames, dets, ratios, 1)
|
||||
assert patches.shape[0] == len(frames), f"patch/frame misalign {patches.shape[0]}!={len(frames)}"
|
||||
|
||||
hb(f"recover {len(patches)} patches (HSMR ViT-H)")
|
||||
params, cams = [], []
|
||||
for bw in E["asb"](total=len(patches), bs_scope=64, enable_tqdm=True):
|
||||
p = patches[bw.sid:bw.eid]
|
||||
pn = ((p - E["IMG_MEAN_255"]) / E["IMG_STD_255"]).transpose(0, 3, 1, 2)
|
||||
with torch.no_grad():
|
||||
o = E["pipe"](pn)
|
||||
params.append({k: v.detach().cpu().clone() for k, v in o["pd_params"].items()})
|
||||
cams.append(o["pd_cam_t"].detach().cpu().clone())
|
||||
params = E["assemble_dict"](params, expand_dim=False)
|
||||
return params["poses"].numpy(), params["betas"].numpy(), torch.cat(cams, 0).numpy()
|
||||
|
||||
|
||||
def skel_kinematics(E, poses, betas, cam_t):
|
||||
"""SKEL forward on recovered params → posed joints/orientations + rest skeleton.
|
||||
Returns names, parents(len J, root=-1), rest_offsets(J,3), root_pos(F,3), W(F,J,3,3)."""
|
||||
torch = E["torch"]
|
||||
skel = E["skel"]; skel_joints_name = E["skel_names"]
|
||||
parents = np.concatenate([[-1], skel.parent.cpu().numpy()]).astype(int) # (J,), root=-1
|
||||
P = torch.tensor(poses).float(); B = torch.tensor(betas).float(); T = torch.tensor(cam_t).float()
|
||||
|
||||
Rg, Jp = [], []
|
||||
for s in range(0, len(P), 256): # batch to bound memory
|
||||
with torch.no_grad():
|
||||
o = skel(poses=P[s:s+256], betas=B[s:s+256], trans=T[s:s+256], skelmesh=False)
|
||||
Rg.append(o.joints_ori.cpu().numpy()); Jp.append(o.joints.cpu().numpy())
|
||||
Rg = np.concatenate(Rg); Jp = np.concatenate(Jp) # (F,J,3,3),(F,J,3)
|
||||
|
||||
with torch.no_grad(): # one rest skeleton from mean shape
|
||||
r = skel(poses=torch.zeros(1, 46), betas=B.mean(0, keepdim=True),
|
||||
trans=torch.zeros(1, 3), skelmesh=False)
|
||||
rest_J = r.joints[0].cpu().numpy(); rest_R = r.joints_ori[0].cpu().numpy()
|
||||
|
||||
W = np.einsum("fjab,jcb->fjac", Rg, rest_R) # W[f,j] = Rg[f,j] @ rest_R[j]^T (camera frame)
|
||||
root_pos = Jp[:, 0]
|
||||
# HSMR emits camera space (X-right, Y-DOWN, Z-fwd — OpenCV image convention, verified: head.y <
|
||||
# pelvis.y). Rotate 180° about X -> Y-up, character faces -Z, matching the treaty world axes.
|
||||
R_CW = np.diag([1.0, -1.0, -1.0])
|
||||
W = np.einsum("ab,fjbc->fjac", R_CW, W)
|
||||
root_pos = root_pos @ R_CW.T
|
||||
|
||||
offsets = np.zeros((len(parents), 3))
|
||||
for j, p in enumerate(parents):
|
||||
offsets[j] = rest_J[j] if p < 0 else rest_J[j] - rest_J[p]
|
||||
offsets[next(j for j, p in enumerate(parents) if p < 0)] = 0.0 # root at origin; abs pos in channels
|
||||
return skel_joints_name, parents, offsets, root_pos, W
|
||||
|
||||
|
||||
# --------------------------------------------------------------------- orchestration --------
|
||||
|
||||
def process(E, video, gender="male", detect_every=10,
|
||||
filt_win=11, filt_poly=3, no_filter=False, from_params=False):
|
||||
video = Path(video).resolve() # engine work chdir's into .engine/HSMR; keep paths absolute
|
||||
name = video.stem
|
||||
outdir = OUT / name; outdir.mkdir(parents=True, exist_ok=True)
|
||||
pfile = outdir / "params.npz"
|
||||
hb(f"START {name}")
|
||||
if from_params and pfile.exists(): # re-run SKEL→BVH without the HMR pass (filter/axis tweaks)
|
||||
z = np.load(pfile); poses, betas, cam_t = z["poses"], z["betas"], z["cam_t"]
|
||||
hb(f"reuse params ({len(poses)} frames)")
|
||||
else:
|
||||
poses, betas, cam_t = run_recovery(E, video, detect_every)
|
||||
np.savez(pfile, poses=poses, betas=betas, cam_t=cam_t) # raw payload for QC / re-processing
|
||||
hb(f"skel forward ({len(poses)} frames)")
|
||||
names, parents, offsets, root_pos, W = skel_kinematics(E, poses, betas, cam_t)
|
||||
|
||||
if not no_filter:
|
||||
hb("jitter pass (savgol on rotations + root)")
|
||||
W = np.stack([smooth_rotations(W[:, j], filt_win, filt_poly) for j in range(W.shape[1])], axis=1)
|
||||
root_pos = smooth_xyz(root_pos, filt_win, filt_poly)
|
||||
|
||||
local_rot = np.stack([locals_from_globals(W[f], parents) for f in range(len(W))])
|
||||
fps = probe_fps(video)
|
||||
write_bvh(outdir / "motion.bvh", names, parents, offsets, root_pos, local_rot, fps)
|
||||
|
||||
# verify the BVH numerically encodes the motion we intended (rigid-bone tolerance)
|
||||
recon = bvh_fk_positions(names, parents, offsets, root_pos, local_rot)
|
||||
intended = _intended_positions(offsets, root_pos, W, parents)
|
||||
err = float(np.abs(recon - intended).max())
|
||||
assert err < 1e-3, f"BVH FK self-inconsistency {err:.4f}m (Euler/channel-order bug)"
|
||||
|
||||
sidecar = {
|
||||
"fps": fps, "n_frames": int(len(local_rot)), "model": MODEL_TAG, "version": VERSION,
|
||||
"gender": gender, "skeleton": "SKEL-24", "root_space": "camera", # HSMR is per-frame
|
||||
"joint_order": list(names), "source_video": video.name,
|
||||
"root_trajectory": np.round(root_pos, 5).tolist(),
|
||||
"bvh_units": "metres, Y-up", "created": datetime.datetime.now().strftime("%Y-%m-%dT%H:%M:%S"),
|
||||
"note": "camera-space root; Lane C grounds locomotion from foot contacts and retargets "
|
||||
"to mixamorig via spec/skel_to_mixamorig.json",
|
||||
}
|
||||
(outdir / "motion.json").write_text(json.dumps(sidecar, indent=1))
|
||||
hb(f"DONE {name}: {len(local_rot)} frames @ {fps}fps → {outdir} (FK err {err*1000:.1f}mm)")
|
||||
return outdir
|
||||
|
||||
|
||||
def _intended_positions(offsets, root_pos, W, parents):
|
||||
"""World joint positions implied by the (unfiltered-or-filtered) global rotations W —
|
||||
the ground truth the BVH must reproduce."""
|
||||
F, J = W.shape[:2]
|
||||
pos = np.zeros((F, J, 3))
|
||||
root = next(j for j, p in enumerate(parents) if p < 0)
|
||||
pos[:, root] = root_pos
|
||||
order = _topo(parents)
|
||||
for j in order:
|
||||
p = parents[j]
|
||||
if p >= 0:
|
||||
pos[:, j] = pos[:, p] + np.einsum("fab,b->fa", W[:, p], offsets[j])
|
||||
return pos
|
||||
|
||||
|
||||
def _topo(parents):
|
||||
order, seen = [], set()
|
||||
def visit(j):
|
||||
if j in seen:
|
||||
return
|
||||
if parents[j] >= 0:
|
||||
visit(parents[j])
|
||||
seen.add(j); order.append(j)
|
||||
for j in range(len(parents)):
|
||||
visit(j)
|
||||
return order
|
||||
|
||||
|
||||
def selfcheck():
|
||||
"""Engine-free: a small tree + random global rotations → write BVH → parse-independent FK
|
||||
reconstruction must match the intended positions. Proves Euler order ↔ channel declaration
|
||||
↔ FK are mutually consistent (the geometric crux)."""
|
||||
import tempfile
|
||||
rng = np.random.default_rng(0)
|
||||
names = ["root", "spine", "head", "arm_l", "arm_r"]
|
||||
parents = np.array([-1, 0, 1, 1, 1])
|
||||
offsets = np.array([[0, 0, 0], [0, .5, 0], [0, .3, 0], [-.4, .1, 0], [.4, .1, 0]], float)
|
||||
F = 20
|
||||
W = np.stack([Rotation.random(len(names), random_state=rng).as_matrix() for _ in range(F)])
|
||||
root_pos = np.cumsum(rng.normal(0, .01, (F, 3)), axis=0)
|
||||
local = np.stack([locals_from_globals(W[f], parents) for f in range(F)])
|
||||
|
||||
d = Path(tempfile.mkdtemp())
|
||||
dfs = write_bvh(d / "t.bvh", names, parents, offsets, root_pos, local, 30.0)
|
||||
recon = bvh_fk_positions(names, parents, offsets, root_pos, local)
|
||||
intended = _intended_positions(offsets, root_pos, W, parents)
|
||||
err = float(np.abs(recon - intended).max())
|
||||
assert err < 1e-9, f"BVH FK mismatch {err}"
|
||||
assert (d / "t.bvh").read_text().count("JOINT") == 4 and dfs[0] == 0
|
||||
# rotation round-trip in the declared order
|
||||
R = Rotation.random(50, random_state=rng).as_matrix()
|
||||
e = Rotation.from_matrix(R).as_euler(EULER, degrees=True)
|
||||
back = Rotation.from_euler(EULER, e, degrees=True).as_matrix()
|
||||
assert np.abs(back - R).max() < 1e-9
|
||||
print(f"selfcheck OK — BVH FK exact ({err:.1e}m), Euler round-trip exact, hierarchy sane")
|
||||
return 0
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description="Lane B pose engine: video → SKEL BVH + sidecar.")
|
||||
ap.add_argument("video", nargs="?", help="single clip (default: drain queue/)")
|
||||
ap.add_argument("--gender", default="male", choices=["male", "female"])
|
||||
ap.add_argument("--device", default="mps")
|
||||
ap.add_argument("--detect-every", type=int, default=20,
|
||||
help="detect person every Nth frame; box reused between (single-subject footage)")
|
||||
ap.add_argument("--filt-win", type=int, default=11, help="savgol window (odd); jitter pass")
|
||||
ap.add_argument("--filt-poly", type=int, default=3, help="savgol polyorder")
|
||||
ap.add_argument("--no-filter", action="store_true", help="skip the B3 jitter pass")
|
||||
ap.add_argument("--from-params", action="store_true",
|
||||
help="reuse out/<name>/params.npz, skip the HMR pass (filter/axis tweaks)")
|
||||
ap.add_argument("--selfcheck", action="store_true", help="run the BVH-math self-test, no engine")
|
||||
a = ap.parse_args()
|
||||
if a.selfcheck:
|
||||
return selfcheck()
|
||||
|
||||
kw = dict(gender=a.gender, detect_every=a.detect_every, filt_win=a.filt_win,
|
||||
filt_poly=a.filt_poly, no_filter=a.no_filter, from_params=a.from_params)
|
||||
video = Path(a.video).resolve() if a.video else None # resolve before build_engine chdir's away
|
||||
E = build_engine(a.device, a.gender) # load models once; queue reuses them
|
||||
if video:
|
||||
process(E, video, **kw)
|
||||
return 0
|
||||
QUEUE.mkdir(parents=True, exist_ok=True)
|
||||
vids = sorted(p for p in QUEUE.iterdir() if p.suffix.lower() in VIDEO_EXT)
|
||||
if not vids:
|
||||
print(f"queue empty: {QUEUE}")
|
||||
return 0
|
||||
hb(f"draining {len(vids)} clip(s) from {QUEUE}")
|
||||
fails = 0
|
||||
for v in vids:
|
||||
try: # one bad clip must not sink the queue
|
||||
process(E, v, **kw)
|
||||
except Exception as e:
|
||||
fails += 1
|
||||
hb(f"FAIL {v.name}: {e}")
|
||||
import traceback; traceback.print_exc()
|
||||
hb(f"queue drained: {len(vids) - fails}/{len(vids)} clip(s) ok")
|
||||
return 1 if fails else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
87
setup_hsmr.sh
Executable file
87
setup_hsmr.sh
Executable file
@ -0,0 +1,87 @@
|
||||
#!/bin/bash
|
||||
# HSMR pose-engine install on Apple Silicon (MPS) — Path B (local HSMR/SKEL).
|
||||
# Clone -> uv venv(3.10) -> deps -> ungated weights -> apply engine_patches/ -> STOP at SKEL gate.
|
||||
#
|
||||
# ./setup_hsmr.sh full install (re-runnable; STOPS at the SKEL download gate)
|
||||
# ./setup_hsmr.sh --patch re-apply engine_patches/ onto an existing clone and exit
|
||||
#
|
||||
# engine_patches/ holds our ARM/headless fixes (torchvision detector drop-in, headless
|
||||
# pyrender/OpenGL shims, smoke + detector tests). HSMR itself is gitignored under .engine/,
|
||||
# so a re-clone would drop those fixes — this script restores them. Override the clone
|
||||
# location with HSMR_DIR=... (used by the durability test against a temp clone).
|
||||
#
|
||||
# ponytail: no `set -e` — optional deps (detectron2/chumpy/render) WARN and we still want the
|
||||
# env + weights down so the smoke test can run and any failure is diagnosable.
|
||||
set -uo pipefail
|
||||
REPO="$(cd "$(dirname "$0")" && pwd)"
|
||||
ENGINE="$REPO/.engine"
|
||||
HSMR="${HSMR_DIR:-$ENGINE/HSMR}"
|
||||
PATCHES="$REPO/engine_patches"
|
||||
LOG="$ENGINE/setup.log"
|
||||
JOB="$HOME/.jobs/hsmr_setup"
|
||||
|
||||
apply_patches(){
|
||||
[ -d "$HSMR" ] || { echo "no HSMR clone at $HSMR — clone first"; exit 1; }
|
||||
[ -d "$PATCHES" ] || { echo "no engine_patches/ at $PATCHES"; exit 1; }
|
||||
( cd "$PATCHES" && find . -type f ! -name '.DS_Store' ) | sed 's|^\./||' | while read -r f; do
|
||||
mkdir -p "$HSMR/$(dirname "$f")"
|
||||
cp "$PATCHES/$f" "$HSMR/$f"
|
||||
echo " patched $f"
|
||||
done
|
||||
}
|
||||
|
||||
# Standalone re-apply (after a manual HSMR re-clone).
|
||||
if [ "${1:-}" = "--patch" ]; then
|
||||
echo ">>> applying engine_patches/ -> $HSMR"
|
||||
apply_patches
|
||||
echo "patches applied."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
mkdir -p "$ENGINE" "$HOME/.jobs"
|
||||
exec > >(tee -a "$LOG") 2>&1
|
||||
hb(){ echo "$(date +%s) $*" > "$JOB"; echo ">>> [$(date +%H:%M:%S)] $*"; }
|
||||
|
||||
hb "clone HSMR + submodules (SKEL)"
|
||||
if [ ! -d "$HSMR/.git" ]; then
|
||||
git clone --recurse-submodules https://github.com/IsshikiHugh/HSMR "$HSMR" || { hb "FAIL clone"; exit 1; }
|
||||
else
|
||||
git -C "$HSMR" submodule update --init || true
|
||||
fi
|
||||
cd "$HSMR" || { hb "FAIL cd"; exit 1; }
|
||||
|
||||
hb "uv venv python 3.10"
|
||||
uv venv --python 3.10 .venv || { hb "FAIL venv"; exit 1; }
|
||||
source .venv/bin/activate
|
||||
|
||||
hb "install torch + torchvision (MPS wheel)"
|
||||
uv pip install torch torchvision || { hb "FAIL torch"; exit 1; }
|
||||
|
||||
hb "install requirements.txt"
|
||||
uv pip install -r requirements.txt || hb "WARN requirements had issues"
|
||||
|
||||
hb "pin numpy<2 (chumpy/detectron2/smplx need 1.x)"
|
||||
uv pip install "numpy==1.26.4"
|
||||
|
||||
# detectron2/chumpy are NOT needed — engine_patches replaces the detector and the SKEL runtime
|
||||
# loads pkls without chumpy (see docs/B_ENGINE_SETUP.md). Left out on purpose; they don't build on ARM.
|
||||
|
||||
hb "pip install -e HSMR + thirdparty/SKEL"
|
||||
uv pip install -e . || hb "WARN pip -e . failed"
|
||||
uv pip install -e thirdparty/SKEL || hb "WARN SKEL editable install failed"
|
||||
|
||||
hb "download UNGATED weights from HuggingFace (regressors + HSMR ckpt + ViTPose backbone)"
|
||||
hf download IsshikiHugh/HSMR-data_inputs \
|
||||
--include "body_models/SMPL_to_J19.pkl" "body_models/J_regressor_SKEL_mix_MALE.pkl" "body_models/J_regressor_SMPL_MALE.pkl" "released_models/HSMR-ViTH-r1d1.tar.gz" "backbone/vitpose_backbone.pth" \
|
||||
--local-dir data_inputs || hb "WARN hf download issue"
|
||||
if [ -f data_inputs/released_models/HSMR-ViTH-r1d1.tar.gz ]; then
|
||||
hb "extract HSMR checkpoint"
|
||||
tar -xzf data_inputs/released_models/HSMR-ViTH-r1d1.tar.gz -C data_inputs/released_models/ && rm -f data_inputs/released_models/HSMR-ViTH-r1d1.tar.gz
|
||||
fi
|
||||
mkdir -p data_inputs/body_models/skel
|
||||
|
||||
hb "apply engine_patches/ (ARM detector, headless shims, tests)"
|
||||
apply_patches
|
||||
|
||||
hb "DONE-ENV — SKEL gate: John drops skel_models_v1.1/ (v1.1.1) into data_inputs/body_models/skel/"
|
||||
echo "=== pip freeze (key) ===" ; uv pip freeze | grep -iE 'torch|numpy|smplx|pytorch-lightning|timm' || true
|
||||
38
spec/skel_to_mixamorig.json
Normal file
38
spec/skel_to_mixamorig.json
Normal file
@ -0,0 +1,38 @@
|
||||
{
|
||||
"comment": "SKEL 24-joint -> mixamorig name map for Lane C's constraint retarget (pose_engine.py emits a SKEL-native BVH; Lane C maps it onto the character_kit mixamorig rig). null = a SKEL joint with no direct mixamorig target (twist/heel DoF folded elsewhere). mixamorig bones absent here (Spine1, Head, fingers, *_End) are not driven by SKEL — see notes.",
|
||||
"source_skeleton": "SKEL-24 (skel/kin_skel.py order)",
|
||||
"target_skeleton": "mixamorig (spec/mixamorig_bones.json, colon-prefixed, 65 joints)",
|
||||
"skel_to_mixamorig": {
|
||||
"pelvis": "mixamorig:Hips",
|
||||
"lumbar_body": "mixamorig:Spine",
|
||||
"thorax": "mixamorig:Spine2",
|
||||
"head": "mixamorig:Neck",
|
||||
"scapula_r": "mixamorig:RightShoulder",
|
||||
"humerus_r": "mixamorig:RightArm",
|
||||
"ulna_r": "mixamorig:RightForeArm",
|
||||
"radius_r": null,
|
||||
"hand_r": "mixamorig:RightHand",
|
||||
"scapula_l": "mixamorig:LeftShoulder",
|
||||
"humerus_l": "mixamorig:LeftArm",
|
||||
"ulna_l": "mixamorig:LeftForeArm",
|
||||
"radius_l": null,
|
||||
"hand_l": "mixamorig:LeftHand",
|
||||
"femur_r": "mixamorig:RightUpLeg",
|
||||
"tibia_r": "mixamorig:RightLeg",
|
||||
"talus_r": "mixamorig:RightFoot",
|
||||
"calcn_r": null,
|
||||
"toes_r": "mixamorig:RightToeBase",
|
||||
"femur_l": "mixamorig:LeftUpLeg",
|
||||
"tibia_l": "mixamorig:LeftLeg",
|
||||
"talus_l": "mixamorig:LeftFoot",
|
||||
"calcn_l": null,
|
||||
"toes_l": "mixamorig:LeftToeBase"
|
||||
},
|
||||
"notes": {
|
||||
"spine": "SKEL has 2 trunk joints (lumbar_body, thorax); mixamorig has 3 (Spine, Spine1, Spine2). lumbar_body->Spine, thorax->Spine2; Lane C interpolates Spine1 (or leaves identity).",
|
||||
"neck_head": "SKEL 'head' is a single thorax-relative joint -> mapped to mixamorig:Neck. mixamorig:Head is left to Lane C (identity, or a small follow of Neck).",
|
||||
"radius_calcn": "radius_* (forearm pronation/supination twist) and calcn_* (heel) have no direct mixamorig bone; their DoF fold into ForeArm/Foot in mixamorig. null = skip in the constraint map.",
|
||||
"fingers": "SKEL has no fingers -> mixamorig hand/finger bones stay undriven; fingers arrive in Lane E1.",
|
||||
"toe_end_headtop": "mixamorig *_End tips (HeadTop_End, LeftToe_End, RightToe_End) are leaf markers, not driven."
|
||||
}
|
||||
}
|
||||
Loading…
Reference in New Issue
Block a user