Lane B / B1: pose-model bake-off + SMPL decision gate

Fresh SOTA survey (per PLAN — distrust the snapshot). Findings:
- MoCapAnything V2 (MIT code+weights, no SMPL, BVH-direct onto arbitrary
  rig) is the pick — sidesteps the SMPL commercial-license hard-stop and
  collapses the SMPL->mixamorig mapper + most of Lane C retarget.
- GVHMR: best accuracy but non-commercial code license -> reference only.
- GEM-X: commercial-clean + perfect output but CUDA-only -> off-fleet.

Stopped at the John-decision gate (docs/DECISIONS.md): confirm engine,
clear m3ultra disk (97% full), supply test footage. Inference eyeball,
B2 service-ize, B3 jitter remain.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
type-two 2026-07-17 01:10:26 +10:00
parent 468f35be7d
commit 7819e12a5a
4 changed files with 165 additions and 3 deletions

View File

@ -6,13 +6,23 @@ One lane per session. Tick on completion. See PLAN.md for acceptance checks.
spec/mixamorig_bones.json (65 bones, `mixamorig:` colon-prefixed). — 2026-07-17
- [x] **Lane A** — capture + ingest. docs/CAPTURE.md protocol, ingest.py
(probe → 30fps normalize → rename → rsync to m3ultra queue → LEDGER.tsv). — 2026-07-17
- [ ] **Lane B** — pose engine (m3ultra). Model bake-off, service-ize, jitter pass.
- [~] **Lane B** — pose engine (m3ultra). B1 research + selection DONE (2026-07-17):
picked **MoCapAnything V2** (MIT/MIT, no SMPL, BVH-direct onto mixamorig) — see
`docs/B1_BAKEOFF.md`. **Stopped at the John-decision gate** (`docs/DECISIONS.md`):
confirm engine + clear m3ultra disk (97% full) + supply test footage. Inference
eyeball, B2 (service-ize), B3 (jitter) remain.
- [ ] **Lane C** — retarget + cleanup (Blender). bvh → mixamorig FBX + QC turntable.
- [ ] **Lane D** — clip factory. `mocap` CLI end-to-end; games + Unreal integration.
- [ ] Lane E — ML upgrades (post-v1, gated).
- [ ] Lane F — UX (optional).
## Notes for the next lane
- **MoCapAnything is BVH-direct onto an arbitrary rig.** If it hits mixamorig directly, do
NOT build `spec/smpl_to_mixamorig.json` (B2 task 2) or Lane C's constraint-retarget rig —
both are obsoleted. Verify the direct-to-mixamorig path first; only build a mapper if it
can't. (`# ponytail: don't write the retarget machinery a BVH-direct model deletes.`)
- **Check MPS + global-trajectory FIRST next session** (the two unproven risks in B1_BAKEOFF).
MoCapAnything ships a ~160MB sample + HF demo — smoke-test without waiting on John's footage.
- Ground-truth skeleton naming is **`mixamorig:`** (colon), armature object named `Armature`.
Lane C's name-map must handle colon vs underscore variants (PLAN warned about this).
- Blender headless entrypoint: `/Applications/Blender.app/Contents/MacOS/Blender --background --python <script>`.

103
docs/B1_BAKEOFF.md Normal file
View File

@ -0,0 +1,103 @@
# B1 — pose-model bake-off
**Status:** research + selection done (2026-07-17). Inference eyeball NOT yet run —
blocked on 3 preconditions (see bottom + `DECISIONS.md`). This lane does **not** tick until
the chosen model produces a right-way-up BVH on a real clip.
The PLAN told this lane to distrust the July-2024 snapshot and search fresh. It paid off:
two models that post-date the snapshot change the whole calculus — one of them **removes
the SMPL licensing hard-stop the plan was built around.**
## Selection criteria (PLAN order)
1. permissive code license (must ship in commercial games)
2. world-grounded global trajectory (not pose-in-place)
3. runs on Apple Silicon (m3ultra — MPS or acceptable CPU; no NVIDIA on the fleet)
4. SMPL-dependency status (the legal rail)
## Candidate matrix
| Model | World-grounded | Code lic. | Weights lic. | SMPL? | Apple Silicon | Output | Commercial-ship |
|---|---|---|---|---|---|---|---|
| **MoCapAnything V2** ⭐ | trajectories (global vs camera **unverified**) | MIT | **MIT** | **No** | standard PyTorch, no custom CUDA ops → MPS-portable *with effort* | **BVH rotations onto an arbitrary target rig** (feed it mixamorig) + hands | **YES (MIT/MIT)** |
| GVHMR | Yes — SOTA, beats WHAM 1828% WA-MPJPE | **research-only** | needs SMPL | Yes | plain PyTorch, MPS-portable w/ effort | SMPL params (rot) + global traj (.pt) | **NO** — code is non-commercial *and* SMPL-gated |
| GEM-X (NVIDIA) | Yes | Apache-2.0 | NVIDIA Open Model Lic. (NVIDIA states commercial-ready) | **No** (SOMA body model) | **CUDA 12.1+ only, NVIDIA arch** — will not run on the fleet | 77-joint axis-angle + world root (m), incl. hands | Yes, but **off-fleet** |
| WHAM | Yes (autoregressive; GVHMR's predecessor) | research/NC (not re-verified) | needs SMPL | Yes | PyTorch, MPS-portable | SMPL + traj | NO (gate) — dominated by GVHMR |
| TRAM | Yes (SLAM-based camera+human) | mixed; +DROID-SLAM | needs SMPL | Yes | heavier (SLAM stage) | SMPL + traj | NO (gate) — heaviest, not pursued |
Sources verified this session: GVHMR repo LICENSE (custom research-only, "prohibited for
commercial use", contact xwzhou@zju.edu.cn); GEM-X model card (Apache code / NVIDIA Open
Model License weights / SOMA / CUDA-only); MoCapAnything V2 weights card
(`kehong/MoCapAnythingV2-weights`, **License: mit**, author Kehong Gong = paper author) and
code repo `github.com/phongdaot/MocapAnything` (MIT). WHAM/TRAM licenses not re-verified —
both are SMPL-gated and dominated by GVHMR on the world-grounded metric, so not installed.
## The two findings that matter
**1. GVHMR (the accuracy leader) is double-blocked for us.** Its own code license is
non-commercial — that's a wall *independent of* SMPL, and the joints-only Path-B from the
PLAN can't get around a non-commercial *code* license. John's clips ship in commercial
games. GVHMR is therefore a research/accuracy *reference* only, unless John negotiates a
paid ZJU commercial license. It is **not** a v1 shipping engine.
**2. MoCapAnything V2 dissolves the SMPL gate.** MIT code + MIT weights, commercial use
permitted, **no SMPL files anywhere** in the pipeline or dependency list (torch/opencv/
trimesh only). It is category-agnostic: you give it a monocular video **plus the target
rig as a prompt**, and it emits a rotation-based BVH that drives *that rig* directly, via
constraint-aware IK. Cross-species too (it retargets across heterogeneous rigs — a free
preview of the Lane E4 centaur work).
### Why #2 is bigger than "gate cleared"
If we prompt MoCapAnything with the **mixamorig** skeleton (`spec/mixamorig_bones.json` /
`female_game.glb`), it emits **mixamorig BVH directly**. That means:
- `spec/smpl_to_mixamorig.json` (B2 task 2) is **unnecessary** — there is no SMPL
intermediate to map.
- Lane C's core constraint-retarget (bvh→mixamorig) **largely collapses** — the BVH is
already on-skeleton. Lane C shrinks to: import → foot-contact fix → jitter → trim →
export FBX.
The fresh search didn't just clear the legal hold-up; it deleted a spec file and most of a
lane. `# ponytail win: don't build the SMPL→mixamorig mapper or the Lane C retarget rig
until we've confirmed MoCapAnything can't hit mixamorig directly.`
**Caveat (honest):** getting it to emit *exactly* the treaty's mixamorig (joint order,
Y-up, faces Z, bone-roll) will still need a rig-prompt config and a convention
reconciliation pass. "Collapses" = "greatly simplifies," not "free."
## Recommendation
**Primary: MoCapAnything V2.** Only candidate that is commercial-clean AND on-fleet AND
BVH-direct. Accept two risks, both verifiable cheaply at inference (below).
**Reference (optional): GVHMR** under its research license, purely to benchmark how much
accuracy we're leaving on the table — informs whether a paid ZJU license is ever worth it.
**Shelved: GEM-X** — revisit only if the fleet ever gains an NVIDIA GPU (it's the cleanest
option the day CUDA is available; zero-cloud rules out renting it).
## Must-verify at the inference eyeball (the other half of B1)
These are why the lane isn't ticked. Cheap to settle — MoCapAnything ships a ~160 MB sample
dataset and an HF demo Space, so we don't even need John's footage to answer them:
1. **Global trajectory?** Treaty requires world-space root motion. MoCapAnything says
"temporally coherent trajectories" — confirm the character *travels through the world*,
not runs-in-place. If camera-space only, we need a grounding pass (or fall back to GVHMR
for locomotion clips). **This is the #1 risk.**
2. **MPS reality.** Standard PyTorch predicts yes, but the PLAN's rule is "check MPS early —
it decides everything." Expect some ops to need CPU fallback; time it (<2× realtime is
the bar).
3. **Quality.** The linked code is a re-implementation ("use as reference, not
reproduction"). Eyeball walk / dance / pick-up-object in Blender before trusting it.
## Preconditions before the install/inference session
- **m3ultra disk: 29 GB free of 926 GB (97% full).** Marginal for one torch env + weights;
too tight for two. Clear space first (see `DECISIONS.md`).
- **Test footage:** inbox is empty. Sample data covers the smoke test; John's walk / dance /
pick-up-object clips are needed for the real quality call and B2/B3/acceptance.
- **Engine decision:** confirm MoCapAnything-primary (vs. paying for GVHMR-grade). See
`DECISIONS.md`.
## Sources
- [GVHMR](https://github.com/zju3dv/GVHMR) · [GVHMR paper](https://arxiv.org/html/2409.06662v1)
- [NVIDIA GEM-X](https://huggingface.co/nvidia/GEM-X)
- [MoCapAnything project](https://animotionlab.github.io/MoCapAnything/) ·
[V2 weights (MIT)](https://huggingface.co/kehong/MoCapAnythingV2-weights) ·
[code](https://github.com/phongdaot/MocapAnything) ·
[paper](https://arxiv.org/abs/2512.10881)
- [TRAM](https://github.com/yufu-wang/tram)

47
docs/DECISIONS.md Normal file
View File

@ -0,0 +1,47 @@
# Decisions needing John
Lane B stops here per the PLAN's John-decision protocol. Full analysis in `B1_BAKEOFF.md`.
## Headline: the SMPL hard-stop is (probably) avoided
The gate the plan feared — "SOTA model needs SMPL, which is research-licensed, so a
commercial license decision blocks everything" — **does not fire for the recommended
engine.** The fresh search turned up **MoCapAnything V2**: MIT code + MIT weights,
commercial use permitted, **no SMPL anywhere**, and it emits BVH straight onto our
mixamorig rig. So the remaining decision is lighter and strategic, not a legal wall.
## Decision 1 — pose engine (the real choice)
| Option | Pros | Cons |
|---|---|---|
| **A. MoCapAnything V2** (recommended) | commercial-clean (MIT/MIT), no SMPL, BVH-direct onto mixamorig (kills a spec file + most of Lane C), runs on the fleet, cross-species bonus for centaurs | newer/less battle-tested; linked code is a re-implementation; global-trajectory + MPS not yet proven (cheap to verify) |
| B. Pay for GVHMR-grade | best measured accuracy (beats WHAM 1828%) | GVHMR code is **non-commercial** → must negotiate a paid ZJU license *and* separately clear SMPL. Two negotiations for one engine. |
| C. GEM-X | cleanest license + best output format | **CUDA-only** — needs an NVIDIA box the fleet doesn't have; renting cloud CUDA breaks the zero-cloud rule |
**Recommendation: A.** Go MoCapAnything V2. It's the only option that is commercial-clean,
on-fleet, and BVH-direct at once. Verify its two unknowns (global trajectory, MPS) on the
sample data next session before committing B2 code to it. Keep GVHMR as a research-only
accuracy yardstick if you ever want to know what B would buy.
→ **John: confirm "go MoCapAnything," or say if you want the paid-GVHMR route explored.**
## Decision 2 — clear m3ultra disk (blocker)
`~` on m3ultra is **97% full — 29 GB free of 926 GB.** Enough for one torch env + weights,
too tight to be comfortable, and MODELBEAST shares the box. Please free some space (or point
me at what's safe to prune) before the install session.
→ **John: free space on m3ultra, or greenlight me to find prunable caches.**
## Decision 3 — test footage (blocker for the real quality call)
`capture/inbox/` is empty. I can smoke-test MoCapAnything on its bundled sample, but the
honest walk/dance/pick-up-object quality verdict — and all of B2/B3 + the acceptance check —
needs your clips. Film the PLAN's three per `docs/CAPTURE.md` (T-pose bookends, feet in
frame) and drop them in the inbox, then `python3 ingest.py`.
→ **John: film + ingest walk / dance / pick-up-object when convenient.**
---
Once Decision 1 is confirmed and 23 are unblocked, the next session installs MoCapAnything
on m3ultra, proves MPS + global trajectory, and proceeds to B2 (service-ize) / B3 (jitter).

View File

@ -8,8 +8,10 @@ the pose-model rows, Lane E fills any training-dataset rows. See PLAN.md §Groun
| Component | Version | License | Commercial OK? | Notes | Checked |
|---|---|---|---|---|---|
| _pose model (Lane B1 winner)_ | — | — | — | fill after bake-off | — |
| SMPL / SMPL-X body model | — | research (Meshcapade) | **NO** by default | commercial needs Meshcapade license; or use joints-only path (PLAN §5) | — |
| **MoCapAnything** (B1 pick) | V2 | code MIT + weights MIT | **YES** | `kehong/MoCapAnythingV2-weights` states `License: mit`; code `github.com/phongdaot/MocapAnything` MIT. No SMPL. BVH-direct onto arbitrary rig. Re-verify no SMPL/AMASS-derived weights taint at install. | 2026-07-17 |
| GVHMR (rejected — NC) | siga24 | custom research-only | **NO** | "prohibited for commercial use"; commercial via xwzhou@zju.edu.cn. Also SMPL-gated. Reference/accuracy yardstick only. | 2026-07-17 |
| GEM-X (shelved — CUDA) | X | code Apache-2.0 + weights NVIDIA Open Model Lic. | yes, but off-fleet | Uses NVIDIA **SOMA** body model, not SMPL; NVIDIA states commercial-ready. CUDA-only → won't run on Apple Silicon. | 2026-07-17 |
| SMPL / SMPL-X body model | — | research (Meshcapade) | **NO** by default | **Avoided** by the MoCapAnything path — no SMPL files touched. Gate only returns if we ever switch to a SMPL-based engine (GVHMR/WHAM/TRAM). | 2026-07-17 |
| AMASS dataset | — | academic | **verify** before any shipped training | Lane E only | — |
| Mixamo clips | — | Adobe ToS | ship-in-game OK; **not** ML training data | — | — |