7819e12a5a
Fresh SOTA survey (per PLAN — distrust the snapshot). Findings: - MoCapAnything V2 (MIT code+weights, no SMPL, BVH-direct onto arbitrary rig) is the pick — sidesteps the SMPL commercial-license hard-stop and collapses the SMPL->mixamorig mapper + most of Lane C retarget. - GVHMR: best accuracy but non-commercial code license -> reference only. - GEM-X: commercial-clean + perfect output but CUDA-only -> off-fleet. Stopped at the John-decision gate (docs/DECISIONS.md): confirm engine, clear m3ultra disk (97% full), supply test footage. Inference eyeball, B2 service-ize, B3 jitter remain. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
104 lines
6.7 KiB
Markdown
104 lines
6.7 KiB
Markdown
# B1 — pose-model bake-off
|
||
|
||
**Status:** research + selection done (2026-07-17). Inference eyeball NOT yet run —
|
||
blocked on 3 preconditions (see bottom + `DECISIONS.md`). This lane does **not** tick until
|
||
the chosen model produces a right-way-up BVH on a real clip.
|
||
|
||
The PLAN told this lane to distrust the July-2024 snapshot and search fresh. It paid off:
|
||
two models that post-date the snapshot change the whole calculus — one of them **removes
|
||
the SMPL licensing hard-stop the plan was built around.**
|
||
|
||
## Selection criteria (PLAN order)
|
||
1. permissive code license (must ship in commercial games)
|
||
2. world-grounded global trajectory (not pose-in-place)
|
||
3. runs on Apple Silicon (m3ultra — MPS or acceptable CPU; no NVIDIA on the fleet)
|
||
4. SMPL-dependency status (the legal rail)
|
||
|
||
## Candidate matrix
|
||
|
||
| Model | World-grounded | Code lic. | Weights lic. | SMPL? | Apple Silicon | Output | Commercial-ship |
|
||
|---|---|---|---|---|---|---|---|
|
||
| **MoCapAnything V2** ⭐ | trajectories (global vs camera **unverified**) | MIT | **MIT** | **No** | standard PyTorch, no custom CUDA ops → MPS-portable *with effort* | **BVH rotations onto an arbitrary target rig** (feed it mixamorig) + hands | **YES (MIT/MIT)** |
|
||
| GVHMR | Yes — SOTA, beats WHAM 18–28% WA-MPJPE | **research-only** | needs SMPL | Yes | plain PyTorch, MPS-portable w/ effort | SMPL params (rot) + global traj (.pt) | **NO** — code is non-commercial *and* SMPL-gated |
|
||
| GEM-X (NVIDIA) | Yes | Apache-2.0 | NVIDIA Open Model Lic. (NVIDIA states commercial-ready) | **No** (SOMA body model) | **CUDA 12.1+ only, NVIDIA arch** — will not run on the fleet | 77-joint axis-angle + world root (m), incl. hands | Yes, but **off-fleet** |
|
||
| WHAM | Yes (autoregressive; GVHMR's predecessor) | research/NC (not re-verified) | needs SMPL | Yes | PyTorch, MPS-portable | SMPL + traj | NO (gate) — dominated by GVHMR |
|
||
| TRAM | Yes (SLAM-based camera+human) | mixed; +DROID-SLAM | needs SMPL | Yes | heavier (SLAM stage) | SMPL + traj | NO (gate) — heaviest, not pursued |
|
||
|
||
Sources verified this session: GVHMR repo LICENSE (custom research-only, "prohibited for
|
||
commercial use", contact xwzhou@zju.edu.cn); GEM-X model card (Apache code / NVIDIA Open
|
||
Model License weights / SOMA / CUDA-only); MoCapAnything V2 weights card
|
||
(`kehong/MoCapAnythingV2-weights`, **License: mit**, author Kehong Gong = paper author) and
|
||
code repo `github.com/phongdaot/MocapAnything` (MIT). WHAM/TRAM licenses not re-verified —
|
||
both are SMPL-gated and dominated by GVHMR on the world-grounded metric, so not installed.
|
||
|
||
## The two findings that matter
|
||
|
||
**1. GVHMR (the accuracy leader) is double-blocked for us.** Its own code license is
|
||
non-commercial — that's a wall *independent of* SMPL, and the joints-only Path-B from the
|
||
PLAN can't get around a non-commercial *code* license. John's clips ship in commercial
|
||
games. GVHMR is therefore a research/accuracy *reference* only, unless John negotiates a
|
||
paid ZJU commercial license. It is **not** a v1 shipping engine.
|
||
|
||
**2. MoCapAnything V2 dissolves the SMPL gate.** MIT code + MIT weights, commercial use
|
||
permitted, **no SMPL files anywhere** in the pipeline or dependency list (torch/opencv/
|
||
trimesh only). It is category-agnostic: you give it a monocular video **plus the target
|
||
rig as a prompt**, and it emits a rotation-based BVH that drives *that rig* directly, via
|
||
constraint-aware IK. Cross-species too (it retargets across heterogeneous rigs — a free
|
||
preview of the Lane E4 centaur work).
|
||
|
||
### Why #2 is bigger than "gate cleared"
|
||
If we prompt MoCapAnything with the **mixamorig** skeleton (`spec/mixamorig_bones.json` /
|
||
`female_game.glb`), it emits **mixamorig BVH directly**. That means:
|
||
- `spec/smpl_to_mixamorig.json` (B2 task 2) is **unnecessary** — there is no SMPL
|
||
intermediate to map.
|
||
- Lane C's core constraint-retarget (bvh→mixamorig) **largely collapses** — the BVH is
|
||
already on-skeleton. Lane C shrinks to: import → foot-contact fix → jitter → trim →
|
||
export FBX.
|
||
|
||
The fresh search didn't just clear the legal hold-up; it deleted a spec file and most of a
|
||
lane. `# ponytail win: don't build the SMPL→mixamorig mapper or the Lane C retarget rig
|
||
until we've confirmed MoCapAnything can't hit mixamorig directly.`
|
||
|
||
**Caveat (honest):** getting it to emit *exactly* the treaty's mixamorig (joint order,
|
||
Y-up, faces −Z, bone-roll) will still need a rig-prompt config and a convention
|
||
reconciliation pass. "Collapses" = "greatly simplifies," not "free."
|
||
|
||
## Recommendation
|
||
|
||
**Primary: MoCapAnything V2.** Only candidate that is commercial-clean AND on-fleet AND
|
||
BVH-direct. Accept two risks, both verifiable cheaply at inference (below).
|
||
**Reference (optional): GVHMR** under its research license, purely to benchmark how much
|
||
accuracy we're leaving on the table — informs whether a paid ZJU license is ever worth it.
|
||
**Shelved: GEM-X** — revisit only if the fleet ever gains an NVIDIA GPU (it's the cleanest
|
||
option the day CUDA is available; zero-cloud rules out renting it).
|
||
|
||
## Must-verify at the inference eyeball (the other half of B1)
|
||
These are why the lane isn't ticked. Cheap to settle — MoCapAnything ships a ~160 MB sample
|
||
dataset and an HF demo Space, so we don't even need John's footage to answer them:
|
||
1. **Global trajectory?** Treaty requires world-space root motion. MoCapAnything says
|
||
"temporally coherent trajectories" — confirm the character *travels through the world*,
|
||
not runs-in-place. If camera-space only, we need a grounding pass (or fall back to GVHMR
|
||
for locomotion clips). **This is the #1 risk.**
|
||
2. **MPS reality.** Standard PyTorch predicts yes, but the PLAN's rule is "check MPS early —
|
||
it decides everything." Expect some ops to need CPU fallback; time it (<2× realtime is
|
||
the bar).
|
||
3. **Quality.** The linked code is a re-implementation ("use as reference, not
|
||
reproduction"). Eyeball walk / dance / pick-up-object in Blender before trusting it.
|
||
|
||
## Preconditions before the install/inference session
|
||
- **m3ultra disk: 29 GB free of 926 GB (97% full).** Marginal for one torch env + weights;
|
||
too tight for two. Clear space first (see `DECISIONS.md`).
|
||
- **Test footage:** inbox is empty. Sample data covers the smoke test; John's walk / dance /
|
||
pick-up-object clips are needed for the real quality call and B2/B3/acceptance.
|
||
- **Engine decision:** confirm MoCapAnything-primary (vs. paying for GVHMR-grade). See
|
||
`DECISIONS.md`.
|
||
|
||
## Sources
|
||
- [GVHMR](https://github.com/zju3dv/GVHMR) · [GVHMR paper](https://arxiv.org/html/2409.06662v1)
|
||
- [NVIDIA GEM-X](https://huggingface.co/nvidia/GEM-X)
|
||
- [MoCapAnything project](https://animotionlab.github.io/MoCapAnything/) ·
|
||
[V2 weights (MIT)](https://huggingface.co/kehong/MoCapAnythingV2-weights) ·
|
||
[code](https://github.com/phongdaot/MocapAnything) ·
|
||
[paper](https://arxiv.org/abs/2512.10881)
|
||
- [TRAM](https://github.com/yufu-wang/tram)
|