Lane B / B1: pose-model bake-off + SMPL decision gate
Fresh SOTA survey (per PLAN — distrust the snapshot). Findings: - MoCapAnything V2 (MIT code+weights, no SMPL, BVH-direct onto arbitrary rig) is the pick — sidesteps the SMPL commercial-license hard-stop and collapses the SMPL->mixamorig mapper + most of Lane C retarget. - GVHMR: best accuracy but non-commercial code license -> reference only. - GEM-X: commercial-clean + perfect output but CUDA-only -> off-fleet. Stopped at the John-decision gate (docs/DECISIONS.md): confirm engine, clear m3ultra disk (97% full), supply test footage. Inference eyeball, B2 service-ize, B3 jitter remain. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
468f35be7d
commit
7819e12a5a
12
PROGRESS.md
12
PROGRESS.md
@ -6,13 +6,23 @@ One lane per session. Tick on completion. See PLAN.md for acceptance checks.
|
|||||||
spec/mixamorig_bones.json (65 bones, `mixamorig:` colon-prefixed). — 2026-07-17
|
spec/mixamorig_bones.json (65 bones, `mixamorig:` colon-prefixed). — 2026-07-17
|
||||||
- [x] **Lane A** — capture + ingest. docs/CAPTURE.md protocol, ingest.py
|
- [x] **Lane A** — capture + ingest. docs/CAPTURE.md protocol, ingest.py
|
||||||
(probe → 30fps normalize → rename → rsync to m3ultra queue → LEDGER.tsv). — 2026-07-17
|
(probe → 30fps normalize → rename → rsync to m3ultra queue → LEDGER.tsv). — 2026-07-17
|
||||||
- [ ] **Lane B** — pose engine (m3ultra). Model bake-off, service-ize, jitter pass.
|
- [~] **Lane B** — pose engine (m3ultra). B1 research + selection DONE (2026-07-17):
|
||||||
|
picked **MoCapAnything V2** (MIT/MIT, no SMPL, BVH-direct onto mixamorig) — see
|
||||||
|
`docs/B1_BAKEOFF.md`. **Stopped at the John-decision gate** (`docs/DECISIONS.md`):
|
||||||
|
confirm engine + clear m3ultra disk (97% full) + supply test footage. Inference
|
||||||
|
eyeball, B2 (service-ize), B3 (jitter) remain.
|
||||||
- [ ] **Lane C** — retarget + cleanup (Blender). bvh → mixamorig FBX + QC turntable.
|
- [ ] **Lane C** — retarget + cleanup (Blender). bvh → mixamorig FBX + QC turntable.
|
||||||
- [ ] **Lane D** — clip factory. `mocap` CLI end-to-end; games + Unreal integration.
|
- [ ] **Lane D** — clip factory. `mocap` CLI end-to-end; games + Unreal integration.
|
||||||
- [ ] Lane E — ML upgrades (post-v1, gated).
|
- [ ] Lane E — ML upgrades (post-v1, gated).
|
||||||
- [ ] Lane F — UX (optional).
|
- [ ] Lane F — UX (optional).
|
||||||
|
|
||||||
## Notes for the next lane
|
## Notes for the next lane
|
||||||
|
- **MoCapAnything is BVH-direct onto an arbitrary rig.** If it hits mixamorig directly, do
|
||||||
|
NOT build `spec/smpl_to_mixamorig.json` (B2 task 2) or Lane C's constraint-retarget rig —
|
||||||
|
both are obsoleted. Verify the direct-to-mixamorig path first; only build a mapper if it
|
||||||
|
can't. (`# ponytail: don't write the retarget machinery a BVH-direct model deletes.`)
|
||||||
|
- **Check MPS + global-trajectory FIRST next session** (the two unproven risks in B1_BAKEOFF).
|
||||||
|
MoCapAnything ships a ~160MB sample + HF demo — smoke-test without waiting on John's footage.
|
||||||
- Ground-truth skeleton naming is **`mixamorig:`** (colon), armature object named `Armature`.
|
- Ground-truth skeleton naming is **`mixamorig:`** (colon), armature object named `Armature`.
|
||||||
Lane C's name-map must handle colon vs underscore variants (PLAN warned about this).
|
Lane C's name-map must handle colon vs underscore variants (PLAN warned about this).
|
||||||
- Blender headless entrypoint: `/Applications/Blender.app/Contents/MacOS/Blender --background --python <script>`.
|
- Blender headless entrypoint: `/Applications/Blender.app/Contents/MacOS/Blender --background --python <script>`.
|
||||||
|
|||||||
103
docs/B1_BAKEOFF.md
Normal file
103
docs/B1_BAKEOFF.md
Normal file
@ -0,0 +1,103 @@
|
|||||||
|
# B1 — pose-model bake-off
|
||||||
|
|
||||||
|
**Status:** research + selection done (2026-07-17). Inference eyeball NOT yet run —
|
||||||
|
blocked on 3 preconditions (see bottom + `DECISIONS.md`). This lane does **not** tick until
|
||||||
|
the chosen model produces a right-way-up BVH on a real clip.
|
||||||
|
|
||||||
|
The PLAN told this lane to distrust the July-2024 snapshot and search fresh. It paid off:
|
||||||
|
two models that post-date the snapshot change the whole calculus — one of them **removes
|
||||||
|
the SMPL licensing hard-stop the plan was built around.**
|
||||||
|
|
||||||
|
## Selection criteria (PLAN order)
|
||||||
|
1. permissive code license (must ship in commercial games)
|
||||||
|
2. world-grounded global trajectory (not pose-in-place)
|
||||||
|
3. runs on Apple Silicon (m3ultra — MPS or acceptable CPU; no NVIDIA on the fleet)
|
||||||
|
4. SMPL-dependency status (the legal rail)
|
||||||
|
|
||||||
|
## Candidate matrix
|
||||||
|
|
||||||
|
| Model | World-grounded | Code lic. | Weights lic. | SMPL? | Apple Silicon | Output | Commercial-ship |
|
||||||
|
|---|---|---|---|---|---|---|---|
|
||||||
|
| **MoCapAnything V2** ⭐ | trajectories (global vs camera **unverified**) | MIT | **MIT** | **No** | standard PyTorch, no custom CUDA ops → MPS-portable *with effort* | **BVH rotations onto an arbitrary target rig** (feed it mixamorig) + hands | **YES (MIT/MIT)** |
|
||||||
|
| GVHMR | Yes — SOTA, beats WHAM 18–28% WA-MPJPE | **research-only** | needs SMPL | Yes | plain PyTorch, MPS-portable w/ effort | SMPL params (rot) + global traj (.pt) | **NO** — code is non-commercial *and* SMPL-gated |
|
||||||
|
| GEM-X (NVIDIA) | Yes | Apache-2.0 | NVIDIA Open Model Lic. (NVIDIA states commercial-ready) | **No** (SOMA body model) | **CUDA 12.1+ only, NVIDIA arch** — will not run on the fleet | 77-joint axis-angle + world root (m), incl. hands | Yes, but **off-fleet** |
|
||||||
|
| WHAM | Yes (autoregressive; GVHMR's predecessor) | research/NC (not re-verified) | needs SMPL | Yes | PyTorch, MPS-portable | SMPL + traj | NO (gate) — dominated by GVHMR |
|
||||||
|
| TRAM | Yes (SLAM-based camera+human) | mixed; +DROID-SLAM | needs SMPL | Yes | heavier (SLAM stage) | SMPL + traj | NO (gate) — heaviest, not pursued |
|
||||||
|
|
||||||
|
Sources verified this session: GVHMR repo LICENSE (custom research-only, "prohibited for
|
||||||
|
commercial use", contact xwzhou@zju.edu.cn); GEM-X model card (Apache code / NVIDIA Open
|
||||||
|
Model License weights / SOMA / CUDA-only); MoCapAnything V2 weights card
|
||||||
|
(`kehong/MoCapAnythingV2-weights`, **License: mit**, author Kehong Gong = paper author) and
|
||||||
|
code repo `github.com/phongdaot/MocapAnything` (MIT). WHAM/TRAM licenses not re-verified —
|
||||||
|
both are SMPL-gated and dominated by GVHMR on the world-grounded metric, so not installed.
|
||||||
|
|
||||||
|
## The two findings that matter
|
||||||
|
|
||||||
|
**1. GVHMR (the accuracy leader) is double-blocked for us.** Its own code license is
|
||||||
|
non-commercial — that's a wall *independent of* SMPL, and the joints-only Path-B from the
|
||||||
|
PLAN can't get around a non-commercial *code* license. John's clips ship in commercial
|
||||||
|
games. GVHMR is therefore a research/accuracy *reference* only, unless John negotiates a
|
||||||
|
paid ZJU commercial license. It is **not** a v1 shipping engine.
|
||||||
|
|
||||||
|
**2. MoCapAnything V2 dissolves the SMPL gate.** MIT code + MIT weights, commercial use
|
||||||
|
permitted, **no SMPL files anywhere** in the pipeline or dependency list (torch/opencv/
|
||||||
|
trimesh only). It is category-agnostic: you give it a monocular video **plus the target
|
||||||
|
rig as a prompt**, and it emits a rotation-based BVH that drives *that rig* directly, via
|
||||||
|
constraint-aware IK. Cross-species too (it retargets across heterogeneous rigs — a free
|
||||||
|
preview of the Lane E4 centaur work).
|
||||||
|
|
||||||
|
### Why #2 is bigger than "gate cleared"
|
||||||
|
If we prompt MoCapAnything with the **mixamorig** skeleton (`spec/mixamorig_bones.json` /
|
||||||
|
`female_game.glb`), it emits **mixamorig BVH directly**. That means:
|
||||||
|
- `spec/smpl_to_mixamorig.json` (B2 task 2) is **unnecessary** — there is no SMPL
|
||||||
|
intermediate to map.
|
||||||
|
- Lane C's core constraint-retarget (bvh→mixamorig) **largely collapses** — the BVH is
|
||||||
|
already on-skeleton. Lane C shrinks to: import → foot-contact fix → jitter → trim →
|
||||||
|
export FBX.
|
||||||
|
|
||||||
|
The fresh search didn't just clear the legal hold-up; it deleted a spec file and most of a
|
||||||
|
lane. `# ponytail win: don't build the SMPL→mixamorig mapper or the Lane C retarget rig
|
||||||
|
until we've confirmed MoCapAnything can't hit mixamorig directly.`
|
||||||
|
|
||||||
|
**Caveat (honest):** getting it to emit *exactly* the treaty's mixamorig (joint order,
|
||||||
|
Y-up, faces −Z, bone-roll) will still need a rig-prompt config and a convention
|
||||||
|
reconciliation pass. "Collapses" = "greatly simplifies," not "free."
|
||||||
|
|
||||||
|
## Recommendation
|
||||||
|
|
||||||
|
**Primary: MoCapAnything V2.** Only candidate that is commercial-clean AND on-fleet AND
|
||||||
|
BVH-direct. Accept two risks, both verifiable cheaply at inference (below).
|
||||||
|
**Reference (optional): GVHMR** under its research license, purely to benchmark how much
|
||||||
|
accuracy we're leaving on the table — informs whether a paid ZJU license is ever worth it.
|
||||||
|
**Shelved: GEM-X** — revisit only if the fleet ever gains an NVIDIA GPU (it's the cleanest
|
||||||
|
option the day CUDA is available; zero-cloud rules out renting it).
|
||||||
|
|
||||||
|
## Must-verify at the inference eyeball (the other half of B1)
|
||||||
|
These are why the lane isn't ticked. Cheap to settle — MoCapAnything ships a ~160 MB sample
|
||||||
|
dataset and an HF demo Space, so we don't even need John's footage to answer them:
|
||||||
|
1. **Global trajectory?** Treaty requires world-space root motion. MoCapAnything says
|
||||||
|
"temporally coherent trajectories" — confirm the character *travels through the world*,
|
||||||
|
not runs-in-place. If camera-space only, we need a grounding pass (or fall back to GVHMR
|
||||||
|
for locomotion clips). **This is the #1 risk.**
|
||||||
|
2. **MPS reality.** Standard PyTorch predicts yes, but the PLAN's rule is "check MPS early —
|
||||||
|
it decides everything." Expect some ops to need CPU fallback; time it (<2× realtime is
|
||||||
|
the bar).
|
||||||
|
3. **Quality.** The linked code is a re-implementation ("use as reference, not
|
||||||
|
reproduction"). Eyeball walk / dance / pick-up-object in Blender before trusting it.
|
||||||
|
|
||||||
|
## Preconditions before the install/inference session
|
||||||
|
- **m3ultra disk: 29 GB free of 926 GB (97% full).** Marginal for one torch env + weights;
|
||||||
|
too tight for two. Clear space first (see `DECISIONS.md`).
|
||||||
|
- **Test footage:** inbox is empty. Sample data covers the smoke test; John's walk / dance /
|
||||||
|
pick-up-object clips are needed for the real quality call and B2/B3/acceptance.
|
||||||
|
- **Engine decision:** confirm MoCapAnything-primary (vs. paying for GVHMR-grade). See
|
||||||
|
`DECISIONS.md`.
|
||||||
|
|
||||||
|
## Sources
|
||||||
|
- [GVHMR](https://github.com/zju3dv/GVHMR) · [GVHMR paper](https://arxiv.org/html/2409.06662v1)
|
||||||
|
- [NVIDIA GEM-X](https://huggingface.co/nvidia/GEM-X)
|
||||||
|
- [MoCapAnything project](https://animotionlab.github.io/MoCapAnything/) ·
|
||||||
|
[V2 weights (MIT)](https://huggingface.co/kehong/MoCapAnythingV2-weights) ·
|
||||||
|
[code](https://github.com/phongdaot/MocapAnything) ·
|
||||||
|
[paper](https://arxiv.org/abs/2512.10881)
|
||||||
|
- [TRAM](https://github.com/yufu-wang/tram)
|
||||||
47
docs/DECISIONS.md
Normal file
47
docs/DECISIONS.md
Normal file
@ -0,0 +1,47 @@
|
|||||||
|
# Decisions needing John
|
||||||
|
|
||||||
|
Lane B stops here per the PLAN's John-decision protocol. Full analysis in `B1_BAKEOFF.md`.
|
||||||
|
|
||||||
|
## Headline: the SMPL hard-stop is (probably) avoided
|
||||||
|
|
||||||
|
The gate the plan feared — "SOTA model needs SMPL, which is research-licensed, so a
|
||||||
|
commercial license decision blocks everything" — **does not fire for the recommended
|
||||||
|
engine.** The fresh search turned up **MoCapAnything V2**: MIT code + MIT weights,
|
||||||
|
commercial use permitted, **no SMPL anywhere**, and it emits BVH straight onto our
|
||||||
|
mixamorig rig. So the remaining decision is lighter and strategic, not a legal wall.
|
||||||
|
|
||||||
|
## Decision 1 — pose engine (the real choice)
|
||||||
|
|
||||||
|
| Option | Pros | Cons |
|
||||||
|
|---|---|---|
|
||||||
|
| **A. MoCapAnything V2** (recommended) | commercial-clean (MIT/MIT), no SMPL, BVH-direct onto mixamorig (kills a spec file + most of Lane C), runs on the fleet, cross-species bonus for centaurs | newer/less battle-tested; linked code is a re-implementation; global-trajectory + MPS not yet proven (cheap to verify) |
|
||||||
|
| B. Pay for GVHMR-grade | best measured accuracy (beats WHAM 18–28%) | GVHMR code is **non-commercial** → must negotiate a paid ZJU license *and* separately clear SMPL. Two negotiations for one engine. |
|
||||||
|
| C. GEM-X | cleanest license + best output format | **CUDA-only** — needs an NVIDIA box the fleet doesn't have; renting cloud CUDA breaks the zero-cloud rule |
|
||||||
|
|
||||||
|
**Recommendation: A.** Go MoCapAnything V2. It's the only option that is commercial-clean,
|
||||||
|
on-fleet, and BVH-direct at once. Verify its two unknowns (global trajectory, MPS) on the
|
||||||
|
sample data next session before committing B2 code to it. Keep GVHMR as a research-only
|
||||||
|
accuracy yardstick if you ever want to know what B would buy.
|
||||||
|
|
||||||
|
→ **John: confirm "go MoCapAnything," or say if you want the paid-GVHMR route explored.**
|
||||||
|
|
||||||
|
## Decision 2 — clear m3ultra disk (blocker)
|
||||||
|
|
||||||
|
`~` on m3ultra is **97% full — 29 GB free of 926 GB.** Enough for one torch env + weights,
|
||||||
|
too tight to be comfortable, and MODELBEAST shares the box. Please free some space (or point
|
||||||
|
me at what's safe to prune) before the install session.
|
||||||
|
|
||||||
|
→ **John: free space on m3ultra, or greenlight me to find prunable caches.**
|
||||||
|
|
||||||
|
## Decision 3 — test footage (blocker for the real quality call)
|
||||||
|
|
||||||
|
`capture/inbox/` is empty. I can smoke-test MoCapAnything on its bundled sample, but the
|
||||||
|
honest walk/dance/pick-up-object quality verdict — and all of B2/B3 + the acceptance check —
|
||||||
|
needs your clips. Film the PLAN's three per `docs/CAPTURE.md` (T-pose bookends, feet in
|
||||||
|
frame) and drop them in the inbox, then `python3 ingest.py`.
|
||||||
|
|
||||||
|
→ **John: film + ingest walk / dance / pick-up-object when convenient.**
|
||||||
|
|
||||||
|
---
|
||||||
|
Once Decision 1 is confirmed and 2–3 are unblocked, the next session installs MoCapAnything
|
||||||
|
on m3ultra, proves MPS + global trajectory, and proceeds to B2 (service-ize) / B3 (jitter).
|
||||||
@ -8,8 +8,10 @@ the pose-model rows, Lane E fills any training-dataset rows. See PLAN.md §Groun
|
|||||||
|
|
||||||
| Component | Version | License | Commercial OK? | Notes | Checked |
|
| Component | Version | License | Commercial OK? | Notes | Checked |
|
||||||
|---|---|---|---|---|---|
|
|---|---|---|---|---|---|
|
||||||
| _pose model (Lane B1 winner)_ | — | — | — | fill after bake-off | — |
|
| **MoCapAnything** (B1 pick) | V2 | code MIT + weights MIT | **YES** | `kehong/MoCapAnythingV2-weights` states `License: mit`; code `github.com/phongdaot/MocapAnything` MIT. No SMPL. BVH-direct onto arbitrary rig. Re-verify no SMPL/AMASS-derived weights taint at install. | 2026-07-17 |
|
||||||
| SMPL / SMPL-X body model | — | research (Meshcapade) | **NO** by default | commercial needs Meshcapade license; or use joints-only path (PLAN §5) | — |
|
| GVHMR (rejected — NC) | siga24 | custom research-only | **NO** | "prohibited for commercial use"; commercial via xwzhou@zju.edu.cn. Also SMPL-gated. Reference/accuracy yardstick only. | 2026-07-17 |
|
||||||
|
| GEM-X (shelved — CUDA) | X | code Apache-2.0 + weights NVIDIA Open Model Lic. | yes, but off-fleet | Uses NVIDIA **SOMA** body model, not SMPL; NVIDIA states commercial-ready. CUDA-only → won't run on Apple Silicon. | 2026-07-17 |
|
||||||
|
| SMPL / SMPL-X body model | — | research (Meshcapade) | **NO** by default | **Avoided** by the MoCapAnything path — no SMPL files touched. Gate only returns if we ever switch to a SMPL-based engine (GVHMR/WHAM/TRAM). | 2026-07-17 |
|
||||||
| AMASS dataset | — | academic | **verify** before any shipped training | Lane E only | — |
|
| AMASS dataset | — | academic | **verify** before any shipped training | Lane E only | — |
|
||||||
| Mixamo clips | — | Adobe ToS | ship-in-game OK; **not** ML training data | — | — |
|
| Mixamo clips | — | Adobe ToS | ship-in-game OK; **not** ML training data | — | — |
|
||||||
|
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user