Lane B: resolve decision gate -> Path B (local HSMR/SKEL, accept NC risk)
B1's MoCapAnything pick reversed (animal model, wrong for human capture); license analysis recorded in LICENSES.md. John chose Path B: run HSMR->SKEL locally on ultra (MPS), ship rig-motion only, accept the non-commercial licence risk as interim until a commercial SKEL licence request goes out (Path A, the clean upgrade). Engine installed under .engine/ (gitignored): uv venv py3.10, torch 2.13 MPS-green, ungated weights, smoke_test.py with Apple-Silicon bypasses (detectron2 stubbed with full-frame bbox, headless pyrender stubbed -> QC in Blender) -- see docs/B_ENGINE_SETUP.md. Remaining before B ships: John's gated SKEL v1.1.1 download, then the forward-pass smoke test (MPS end-to-end, timing, pose sanity). Known caveat: HSMR is per-frame so root is camera-space cam_t, NOT world-grounded -> Lane C derives root from foot-contact. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
fefb1f71ff
commit
4cc67b90fa
1
.gitignore
vendored
1
.gitignore
vendored
@ -1,5 +1,6 @@
|
||||
# weights + runtime data stay out of git
|
||||
weights/
|
||||
.engine/
|
||||
capture/
|
||||
queue/
|
||||
out/
|
||||
|
||||
41
PROGRESS.md
41
PROGRESS.md
@ -6,23 +6,42 @@ One lane per session. Tick on completion. See PLAN.md for acceptance checks.
|
||||
spec/mixamorig_bones.json (65 bones, `mixamorig:` colon-prefixed). — 2026-07-17
|
||||
- [x] **Lane A** — capture + ingest. docs/CAPTURE.md protocol, ingest.py
|
||||
(probe → 30fps normalize → rename → rsync to m3ultra queue → LEDGER.tsv). — 2026-07-17
|
||||
- [~] **Lane B** — pose engine (m3ultra). B1 research + selection DONE (2026-07-17):
|
||||
picked **MoCapAnything V2** (MIT/MIT, no SMPL, BVH-direct onto mixamorig) — see
|
||||
`docs/B1_BAKEOFF.md`. **Stopped at the John-decision gate** (`docs/DECISIONS.md`):
|
||||
confirm engine + clear m3ultra disk (97% full) + supply test footage. Inference
|
||||
eyeball, B2 (service-ize), B3 (jitter) remain.
|
||||
- [ ] **Lane B** — pose engine. **Decision made 2026-07-17 (session 3): Path B** — engine =
|
||||
**HSMR → SKEL**, run local, ship rig-motion only, John accepts NC-licence risk
|
||||
(`docs/DECISIONS.md` Decision 1 RESOLVED). Env is 90% up in `.engine/HSMR` (uv venv,
|
||||
torch 2.13 **MPS verified green**, ungated weights pulled). **Still open before B ships:**
|
||||
(1) John downloads gated **SKEL v1.1.1** → `.engine/HSMR/data_inputs/body_models/skel/`;
|
||||
(2) fix the person-detector (detectron2 + chumpy failed to build — see gotchas);
|
||||
(3) real forward-pass smoke test on an HSMR demo image; then B2 (service-ize → BVH) + B3
|
||||
(jitter). Lanes C and D remain blocked on B.
|
||||
- [ ] **Lane C** — retarget + cleanup (Blender). bvh → mixamorig FBX + QC turntable.
|
||||
- [ ] **Lane D** — clip factory. `mocap` CLI end-to-end; games + Unreal integration.
|
||||
- [ ] Lane E — ML upgrades (post-v1, gated).
|
||||
- [ ] Lane F — UX (optional).
|
||||
|
||||
## Notes for the next lane
|
||||
- **MoCapAnything is BVH-direct onto an arbitrary rig.** If it hits mixamorig directly, do
|
||||
NOT build `spec/smpl_to_mixamorig.json` (B2 task 2) or Lane C's constraint-retarget rig —
|
||||
both are obsoleted. Verify the direct-to-mixamorig path first; only build a mapper if it
|
||||
can't. (`# ponytail: don't write the retarget machinery a BVH-direct model deletes.`)
|
||||
- **Check MPS + global-trajectory FIRST next session** (the two unproven risks in B1_BAKEOFF).
|
||||
MoCapAnything ships a ~160MB sample + HF demo — smoke-test without waiting on John's footage.
|
||||
- **Lane D can't be started until B + C ship + test footage exists** — it's the CLI that wraps
|
||||
ingest → pose_engine.py → retarget.py, and the latter two don't exist yet. Session 3 (the
|
||||
"Lane D" session) correctly built nothing; it locked Decision 1 and de-risked MPS instead.
|
||||
- **SKEL version trap:** HSMR needs **v1.1.1** exactly (`changelog_v1.1.1.txt`); older = silent
|
||||
incompat. Source: `.engine/HSMR/docs/SETUP.md` (register skel.is.tue.mpg.de → download.php).
|
||||
- **Engine wiring + ARM bypasses live in `docs/B_ENGINE_SETUP.md`** — `smoke_test.py` already
|
||||
stubs the un-buildable detectron2 (full-frame bbox) and headless pyrender (QC → Blender), so
|
||||
the detector swap + real QC are B2 polish, **not** blockers for a centered demo clip.
|
||||
- **World-grounded gap (important):** HSMR is per-frame → root is camera-space `cam_t`, NOT the
|
||||
treaty's world-grounded trajectory. Fine for in-place DJ moves; walking/locomotion needs Lane C
|
||||
to derive root from foot-contact (or a grounding pass). Carry into Lane C design.
|
||||
- **HSMR is installed on ultra, not m3ultra** (PLAN Ground-Rule 1 puts the pose engine on
|
||||
m3ultra). Likely because m3ultra's disk reclaim waits on the reboot (Decision 2). Next B
|
||||
session decides: move it to m3ultra, or consciously host inference on ultra.
|
||||
- **VOID** (was: "MoCapAnything is BVH-direct, skip the retarget rig"). Reversed — the model
|
||||
is animal-only and not our engine. Whatever human model we pick emits **SMPL or SKEL, not
|
||||
mixamorig**, so **Lane C's constraint-retarget IS required** and
|
||||
`spec/{smpl,skel}_to_mixamorig.json` is back on the menu. HSMR's SKEL output (a biomechanical
|
||||
skeleton) is the likely source format if John goes SKEL.
|
||||
- **Check MPS + global-trajectory FIRST** for whichever human model John's decision unlocks
|
||||
(HSMR has an HF Space + demo to smoke-test before John's footage). These two risks are
|
||||
unchanged by the engine swap.
|
||||
- Ground-truth skeleton naming is **`mixamorig:`** (colon), armature object named `Armature`.
|
||||
Lane C's name-map must handle colon vs underscore variants (PLAN warned about this).
|
||||
- Blender headless entrypoint: `/Applications/Blender.app/Contents/MacOS/Blender --background --python <script>`.
|
||||
|
||||
@ -1,8 +1,12 @@
|
||||
# B1 — pose-model bake-off
|
||||
|
||||
**Status:** research + selection done (2026-07-17). Inference eyeball NOT yet run —
|
||||
blocked on 3 preconditions (see bottom + `DECISIONS.md`). This lane does **not** tick until
|
||||
the chosen model produces a right-way-up BVH on a real clip.
|
||||
**Status: SUPERSEDED 2026-07-17 (session 2).** The selection below (MoCapAnything V2) was a
|
||||
**category error** — MoCapAnything is an **animal** mocap model (71 animal targets, no
|
||||
human/biped/mixamorig, no fingers), so it cannot be MOCAPGOD's human v1 engine. Its "SMPL gate
|
||||
dissolved" conclusion is therefore wrong; the gate is **LIVE**. See `docs/DECISIONS.md` for the
|
||||
corrected, license-verified decision. The candidate research below is kept as history — its
|
||||
*human*-model findings (GVHMR non-commercial, GEM-X CUDA-only) still stand; its recommendation
|
||||
does not.
|
||||
|
||||
The PLAN told this lane to distrust the July-2024 snapshot and search fresh. It paid off:
|
||||
two models that post-date the snapshot change the whole calculus — one of them **removes
|
||||
|
||||
50
docs/B_ENGINE_SETUP.md
Normal file
50
docs/B_ENGINE_SETUP.md
Normal file
@ -0,0 +1,50 @@
|
||||
# Lane B engine — HSMR install & smoke-test state (session 3, 2026-07-17)
|
||||
|
||||
Engine per Decision 1 = **Path B: HSMR → SKEL**, run local on **ultra**, MPS. Installed under
|
||||
`.engine/HSMR` (gitignored). This doc = how it's wired + how to run once SKEL lands.
|
||||
|
||||
## State
|
||||
- ✅ `uv venv` py3.10 at `.engine/HSMR/.venv`; torch **2.13.0**, `mps.is_available() == True`.
|
||||
- ✅ Ungated weights in `.engine/HSMR/data_inputs/`: HSMR-ViTH ckpt
|
||||
(`released_models/HSMR-ViTH-r1d1/checkpoints/hsmr.ckpt`), ViTPose backbone, 3 regressors.
|
||||
- ✅ Smoke-test runner `.engine/HSMR/smoke_test.py` — **validated to the exact SKEL gate**:
|
||||
imports + detector stub + patch-crop all run; it fails only on the missing `skel_male.pkl`.
|
||||
- 🚧 **Blocked on the gated SKEL download (John's manual step, below).**
|
||||
- Install script (re-runnable): `.engine/setup_hsmr.sh`.
|
||||
|
||||
## Two Apple-Silicon bypasses (non-obvious — next session, inherit these)
|
||||
- **detectron2 will NOT build on ARM Mac** (uv build-isolation hides the installed torch from
|
||||
its `setup.py`; a `uv pip install --no-build-isolation` retry may fix it if a real detector is
|
||||
wanted). The runner **sidesteps it**: stubs the `vitdet` import and injects a **full-frame
|
||||
bbox** (fine for centered single-subject demo clips). *B2 decision:* swap detectron2 for a
|
||||
light torchvision/YOLO person-cropper rather than fight the from-source build.
|
||||
- **pyrender is dead headless on macOS** — `py_renderer/__init__.py` forces
|
||||
`PYOPENGL_PLATFORM=egl` and there is no EGL on Mac. The runner permissively stubs
|
||||
`pyrender`/`OpenGL` and does **visual QC in Blender** (fleet renderer + Lane C's tool). It
|
||||
dumps `mesh_qc.npz` (skin/skel verts+faces for Blender) and `params.npz` (SKEL
|
||||
poses/betas/cam_t — the Path-B payload we keep).
|
||||
- **chumpy** also failed to install. Likely irrelevant for SKEL (its `.pkl` is modern, not
|
||||
chumpy-pickled); only matters if the SMPL model (eval-only) is ever loaded. If SKEL load
|
||||
throws on a chumpy import, `uv pip install --no-build-isolation chumpy` + a numpy-alias patch.
|
||||
|
||||
## John's one step — drop the SKEL files (login + licence, so it's yours to do)
|
||||
Register **skel.is.tue.mpg.de** → Download → **"SKEL and BSM models"** → `skel_models_v1.1.zip`
|
||||
(**must be v1.1.1** — check `changelog_v1.1.1.txt` inside; older = silent incompatibility).
|
||||
Unzip, then:
|
||||
```
|
||||
mv /path/to/skel_models_v1.1 ~/Documents/MOCAPGOD/.engine/HSMR/data_inputs/body_models/skel
|
||||
```
|
||||
so that `…/data_inputs/body_models/skel/skel_male.pkl` exists.
|
||||
|
||||
## Run the smoke test (once SKEL is in place)
|
||||
```
|
||||
cd ~/Documents/MOCAPGOD/.engine/HSMR && source .venv/bin/activate
|
||||
PYTHONPATH=. python smoke_test.py -i data_inputs/demo/example_videos/gymnasts.mp4 # or .../example_imgs
|
||||
```
|
||||
Outputs → `data_outputs/smoke/`: `params.npz` (rotations), `mesh_qc.npz` (Blender QC), and
|
||||
console device/fps/NaN. Answers the Lane B smoke questions: **MPS forward OK? timing (<2×
|
||||
realtime)? pose sane?**
|
||||
|
||||
⚠️ HSMR is a **per-frame** recoverer → root is **camera-space `cam_t`, not world-grounded**.
|
||||
Fine for in-place DJ moves; locomotion (walk) needs Lane C's foot-contact root derivation or a
|
||||
grounding pass. This is the treaty's global-trajectory caveat, carried into Path B.
|
||||
@ -1,29 +1,68 @@
|
||||
# Decisions needing John
|
||||
|
||||
Lane B stops here per the PLAN's John-decision protocol. Full analysis in `B1_BAKEOFF.md`.
|
||||
**Status 2026-07-17 (session 2): Lane B1's engine pick is REVERSED** (Decision 1). The disk
|
||||
work (Decision 2) from the concurrent session is preserved intact. Corrected, license-verified
|
||||
analysis below + `LICENSES.md`. Prior "SMPL hard-stop avoided / go MoCapAnything" memo
|
||||
superseded (git @ 7819e12).
|
||||
|
||||
## Headline: the SMPL hard-stop is (probably) avoided
|
||||
## Decision 1 — pose engine: B1 picked an ANIMAL model for a HUMAN job; the gate is LIVE
|
||||
|
||||
The gate the plan feared — "SOTA model needs SMPL, which is research-licensed, so a
|
||||
commercial license decision blocks everything" — **does not fire for the recommended
|
||||
engine.** The fresh search turned up **MoCapAnything V2**: MIT code + MIT weights,
|
||||
commercial use permitted, **no SMPL anywhere**, and it emits BVH straight onto our
|
||||
mixamorig rig. So the remaining decision is lighter and strategic, not a legal wall.
|
||||
> **RESOLVED 2026-07-17 — John chose Path B (run local, accept NC risk).** Engine = **HSMR
|
||||
> → SKEL** on the fleet. Ship only the retargeted mixamorig motion (no Max-Planck file
|
||||
> redistributed); John explicitly accepts the judgment call that authoring game clips from an
|
||||
> NC-licensed model is his risk. Path A (email MPI for a commercial licence) stays on the shelf
|
||||
> as the clean upgrade if that risk ever needs retiring. **MPS verified green on ultra this
|
||||
> session** (torch 2.13.0, `mps_available=True`, `mps_built=True`). **Sole remaining blocker to
|
||||
> a first forward pass: the gated SKEL download — John's manual step** (register at
|
||||
> skel.is.tue.mpg.de → download.php → `skel_models_v1.1.zip`, **must be v1.1.1**, unzip into
|
||||
> `.engine/HSMR/data_inputs/body_models/skel/` per HSMR `docs/SETUP.md`). Behind a login +
|
||||
> NC-licence agreement, so it is John's to do — I can't create the account or accept the terms.
|
||||
> Analysis below is retained for the record.
|
||||
|
||||
## Decision 1 — pose engine (the real choice)
|
||||
B1 chose **MoCapAnything V2** for being MIT/MIT with no SMPL, and concluded "the SMPL hard-stop
|
||||
is avoided." Reading the actual repo this session shows why that's wrong: MoCapAnything is an
|
||||
**animal** motion-capture system — **71 animal retarget targets** (Alligator…Tyranno), trained
|
||||
on `zoo1030` (animals) + `obj1k` (objects), **no human/biped/mixamorig reference and no
|
||||
fingers**. MOCAPGOD's v1 job is capturing **John (a human)** onto mixamorig (PLAN line 1; the
|
||||
Lane B brief says "monocular **human** motion recovery"). MoCapAnything didn't dissolve the SMPL
|
||||
gate — it swapped in a task the gate doesn't cover. **Wrong tool for the human path.** (It's a
|
||||
clean, strong tool for *creature* NPCs — option C.)
|
||||
|
||||
| Option | Pros | Cons |
|
||||
|---|---|---|
|
||||
| **A. MoCapAnything V2** (recommended) | commercial-clean (MIT/MIT), no SMPL, BVH-direct onto mixamorig (kills a spec file + most of Lane C), runs on the fleet, cross-species bonus for centaurs | newer/less battle-tested; linked code is a re-implementation; global-trajectory + MPS not yet proven (cheap to verify) |
|
||||
| B. Pay for GVHMR-grade | best measured accuracy (beats WHAM 18–28%) | GVHMR code is **non-commercial** → must negotiate a paid ZJU license *and* separately clear SMPL. Two negotiations for one engine. |
|
||||
| C. GEM-X | cleanest license + best output format | **CUDA-only** — needs an NVIDIA box the fleet doesn't have; renting cloud CUDA breaks the zero-cloud rule |
|
||||
**Verified from license text this session (links/dates in `LICENSES.md`):**
|
||||
|
||||
**Recommendation: A.** Go MoCapAnything V2. It's the only option that is commercial-clean,
|
||||
on-fleet, and BVH-direct at once. Verify its two unknowns (global trajectory, MPS) on the
|
||||
sample data next session before committing B2 code to it. Keep GVHMR as a research-only
|
||||
accuracy yardstick if you ever want to know what B would buy.
|
||||
| Human-capable option | Code license | Body model | Body-model license | On the fleet? |
|
||||
|---|---|---|---|---|
|
||||
| GVHMR / WHAM / TRAM / OnlineHMR (accuracy leaders) | research/NC (GVHMR NC confirmed) | **SMPL** | NC — commercial via Meshcapade (quote) | PyTorch→MPS ✓ |
|
||||
| **HSMR** (CVPR25 oral) / SKEL-CF (Nov 2025) | **MIT** (HSMR) | **SKEL** | NC — commercial via MPI `ps-license@tue.mpg.de` | PyTorch→MPS ✓ (unverified) |
|
||||
| GEM-X (NVIDIA) | Apache | SOMA (not SMPL) | commercial-ready | **CUDA-only — OFF fleet** ✗ |
|
||||
| MoCapAnything V2 | MIT | none | — (clean) | ✓ but **animal-only** ✗ for humans |
|
||||
|
||||
→ **John: confirm "go MoCapAnything," or say if you want the paid-GVHMR route explored.**
|
||||
**No free-lunch, commercial-clean, on-fleet, world-grounded *human* model exists.** SMPL and
|
||||
SKEL are both Max-Planck research licenses with an "email for commercial" path; the one
|
||||
clean-licensed human model (GEM-X) needs NVIDIA hardware the fleet lacks. Pick a direction:
|
||||
|
||||
**A. License a body model, use a proven human model — RECOMMENDED.** Email for an indie
|
||||
commercial license. Primary: **SKEL → HSMR** (code is MIT, outputs a biomechanical **skeleton**
|
||||
ideal for the mixamorig retarget) via `ps-license@tue.mpg.de`. Fallback: **SMPL** via
|
||||
`sales@meshcapade.com` (unlocks WHAM/TRAM, though their *code* licenses also need clearing).
|
||||
One email + likely a modest fee + a few days. Clean and shippable. *I can draft the email; I
|
||||
can't send it.*
|
||||
|
||||
**B. SMPL-free-outputs path (PLAN Path B) — run locally, ship only your rig's motion.** HSMR's
|
||||
code is MIT: run on the fleet, extract joint **rotations**, retarget onto YOUR mixamorig, ship
|
||||
only that — distribute no Max-Planck file. Zero cost, unblocks today. **Risk:** the NC licenses
|
||||
forbid "production of artefacts for commercial purposes"; whether authoring game clips counts is
|
||||
a legal judgment call. The clean mitigation (email MPI to ask) collapses into A. John-accepts-risk.
|
||||
|
||||
**C. Reconsider scope — creatures now, defer the human license.** MOCAPGOD's downstream is
|
||||
stylized NPCs + a centaur RPG; some targets aren't human. If the near-term need is creature
|
||||
NPCs, **MoCapAnything (MIT, clean, on-fleet) does that TODAY**; defer the human-capture license
|
||||
until you need your own DJ clips.
|
||||
|
||||
→ **Recommendation A.** Honest, cheap, converts B's grey into a clear yes. In parallel I keep
|
||||
MoCapAnything as the sanctioned *creature* engine (free upside, blocks nothing) and verify
|
||||
HSMR's MPS-runnability so we're ready when a license lands. I will **not** build the human v1
|
||||
pipeline on an NC model without your explicit go (that's B).
|
||||
|
||||
## Decision 2 — clear m3ultra disk (RESOLVED pending reboot, 2026-07-17)
|
||||
|
||||
@ -40,13 +79,22 @@ ample for Lane B). Not doing this myself — it's your workstation. Kept the FLU
|
||||
|
||||
## Decision 3 — test footage (blocker for the real quality call)
|
||||
|
||||
`capture/inbox/` is empty. I can smoke-test MoCapAnything on its bundled sample, but the
|
||||
honest walk/dance/pick-up-object quality verdict — and all of B2/B3 + the acceptance check —
|
||||
needs your clips. Film the PLAN's three per `docs/CAPTURE.md` (T-pose bookends, feet in
|
||||
frame) and drop them in the inbox, then `python3 ingest.py`.
|
||||
`capture/inbox/` is empty. Whichever human engine Decision 1 unlocks, the honest
|
||||
walk/dance/pick-up-object quality verdict — and all of B2/B3 + the acceptance check — needs your
|
||||
clips. (HSMR has an HF Space + demo images for the MPS/trajectory smoke-test before your
|
||||
footage.) Film the PLAN's three per `docs/CAPTURE.md` (T-pose bookends, feet in frame), drop
|
||||
them in the inbox, then `python3 ingest.py`.
|
||||
|
||||
→ **John: film + ingest walk / dance / pick-up-object when convenient.**
|
||||
|
||||
## Blast radius of the Decision-1 reversal (next session must know)
|
||||
- **Lane B1 is NOT done** — wrong engine selected. Re-opened; PROGRESS un-ticked.
|
||||
- **VOID the old note "MoCapAnything is BVH-direct, skip the retarget rig."** Any human model
|
||||
emits **SMPL or SKEL, not mixamorig** → **Lane C's constraint-retarget IS required again**,
|
||||
and `spec/{smpl,skel}_to_mixamorig.json` is back on the menu.
|
||||
- Preconditions: disk resolves on the m3ultra reboot (Decision 2); the MPS/global-trajectory
|
||||
smoke-test still stands for whichever engine John's pick unlocks.
|
||||
|
||||
---
|
||||
Once Decision 1 is confirmed and 2–3 are unblocked, the next session installs MoCapAnything
|
||||
on m3ultra, proves MPS + global trajectory, and proceeds to B2 (service-ize) / B3 (jitter).
|
||||
Once Decision 1 is picked and 2–3 clear, the next session installs the chosen human engine,
|
||||
proves MPS + global trajectory, and proceeds to B2 (service-ize) / B3 (jitter).
|
||||
|
||||
@ -1,26 +1,42 @@
|
||||
# Licenses & legal findings
|
||||
|
||||
Every third-party model/dataset/tool that touches the pipeline gets a row here with
|
||||
its license, commercial status, link, and the date checked. Skeleton now; Lane B fills
|
||||
the pose-model rows, Lane E fills any training-dataset rows. See PLAN.md §Ground rules.
|
||||
Every third-party model/dataset/tool that touches the pipeline gets a row here with its
|
||||
license, commercial status, link, and the date checked. See PLAN.md §Ground rules.
|
||||
|
||||
> **2026-07-17 correction (session 2):** B1's original row billed MoCapAnything as the human
|
||||
> engine — it is an **animal** model (see `DECISIONS.md`). The SMPL/SKEL gate is **LIVE**, not
|
||||
> avoided. Rows below corrected and license-verified.
|
||||
|
||||
## Components
|
||||
|
||||
| Component | Version | License | Commercial OK? | Notes | Checked |
|
||||
|---|---|---|---|---|---|
|
||||
| **MoCapAnything** (B1 pick) | V2 | code MIT + weights MIT | **YES** | `kehong/MoCapAnythingV2-weights` states `License: mit`; code `github.com/phongdaot/MocapAnything` MIT. No SMPL. BVH-direct onto arbitrary rig. Re-verify no SMPL/AMASS-derived weights taint at install. | 2026-07-17 |
|
||||
| GVHMR (rejected — NC) | siga24 | custom research-only | **NO** | "prohibited for commercial use"; commercial via xwzhou@zju.edu.cn. Also SMPL-gated. Reference/accuracy yardstick only. | 2026-07-17 |
|
||||
| GEM-X (shelved — CUDA) | X | code Apache-2.0 + weights NVIDIA Open Model Lic. | yes, but off-fleet | Uses NVIDIA **SOMA** body model, not SMPL; NVIDIA states commercial-ready. CUDA-only → won't run on Apple Silicon. | 2026-07-17 |
|
||||
| SMPL / SMPL-X body model | — | research (Meshcapade) | **NO** by default | **Avoided** by the MoCapAnything path — no SMPL files touched. Gate only returns if we ever switch to a SMPL-based engine (GVHMR/WHAM/TRAM). | 2026-07-17 |
|
||||
| AMASS dataset | — | academic | **verify** before any shipped training | Lane E only | — |
|
||||
| Mixamo clips | — | Adobe ToS | ship-in-game OK; **not** ML training data | — | — |
|
||||
| Component | License (code / body model) | Commercial OK? | Notes | Checked |
|
||||
|---|---|---|---|---|
|
||||
| **HSMR** (human — candidate engine) | code **MIT** / body **SKEL** | code yes / **model NO** | `IsshikiHugh/HSMR`, CVPR25 oral. Predicts **SKEL** params (biomechanical skeleton, great for retarget). Needs SKEL files (NC) at inference. MPS-runnability unverified. | 2026-07-17 |
|
||||
| SKEL-CF (human — candidate) | code (verify) / body **SKEL** | **model NO** | Coarse-to-fine skeleton+mesh, Nov 2025. Same SKEL gate. | 2026-07-17 |
|
||||
| **SKEL body model** | MPG research | **NO** (default) | "any use for commercial… purposes is prohibited"; commercial via **ps-license@tue.mpg.de**. Same family as SMPL. | 2026-07-17 |
|
||||
| **SMPL / SMPL-X body model** | MPG research (Meshcapade sublicenses) | **NO** (default) | Commercial via **sales@meshcapade.com** (quote-based; incl. SMPL+H/-X/STAR). Gate for all SMPL-based human models. | 2026-07-17 |
|
||||
| GVHMR / WHAM / TRAM / OnlineHMR (human, SMPL) | research/NC (GVHMR NC confirmed) / SMPL | **NO** | Accuracy leaders but **double gate**: NC *code* AND SMPL. GVHMR commercial via xwzhou@zju.edu.cn. Reference/yardstick unless licensed. | 2026-07-17 |
|
||||
| GEM-X (human, shelved) | Apache / SOMA (not SMPL) | yes, but **off-fleet** | NVIDIA commercial-ready; **CUDA-only** → won't run on Apple Silicon. Revisit only if the fleet gains NVIDIA. | 2026-07-17 |
|
||||
| MoCapAnything V2 (**animal** — creature NPCs only) | MIT / none | **YES** (creatures) | `kehong/MoCapAnythingV2-weights` MIT, no SMPL. **71 animal targets, no human/biped/fingers** → NOT the human engine. Keep for creature/centaur (Lane E4) work. | 2026-07-17 |
|
||||
| AMASS dataset | academic | **verify** before shipped training | Lane E only | — |
|
||||
| Mixamo clips | Adobe ToS | ship-in-game OK; **not** ML training data | — | — |
|
||||
|
||||
## The SMPL gate (hard stop — John decision)
|
||||
## The SMPL/SKEL gate (hard stop — LIVE, John decision)
|
||||
|
||||
Most SOTA monocular mocap models emit SMPL parameters. Before any commercial use:
|
||||
- **Path A:** buy a Meshcapade commercial license (cheap tiers exist).
|
||||
- **Path B (SMPL-free):** use only the model's joint positions/rotations, never the SMPL
|
||||
mesh/shape files, AND confirm the inference repo's own code license permits commercial
|
||||
use of outputs.
|
||||
Every accurate world-grounded **human** model depends on a Max-Planck body model — **SMPL**
|
||||
(GVHMR/WHAM/TRAM/OnlineHMR) or **SKEL** (HSMR/SKEL-CF) — both research-licensed. Two paths:
|
||||
- **Path A — license it.** Email MPI (SKEL, `ps-license@tue.mpg.de`) or Meshcapade (SMPL,
|
||||
`sales@meshcapade.com`) for an indie commercial license. Clean; quote-based.
|
||||
- **Path B — SMPL-free outputs.** Run an MIT-code model (HSMR) locally, keep only joint
|
||||
rotations, retarget onto our own mixamorig, distribute no Max-Planck file. Grey area: the NC
|
||||
license bars "production of artefacts for commercial purposes." John-accepts-risk.
|
||||
|
||||
Lane B writes the decision to `docs/DECISIONS.md` and stops there if unresolved.
|
||||
Decision lives in `docs/DECISIONS.md`. The human path does not proceed until John picks.
|
||||
|
||||
## Sources (session 2, 2026-07-17)
|
||||
- SKEL license (commercial prohibited): https://skel.is.tue.mpg.de/license.html
|
||||
- HSMR (MIT code, SKEL-based): https://github.com/IsshikiHugh/HSMR ·
|
||||
https://isshikihugh.github.io/HSMR/ · paper https://arxiv.org/html/2503.21751v1
|
||||
- SKEL-CF: https://arxiv.org/html/2511.20157
|
||||
- SMPL commercial licensing (Meshcapade): https://meshcapade.com/infopages/licensing.html
|
||||
- GVHMR (NC): https://github.com/zju3dv/GVHMR · OnlineHMR: https://github.com/Tsukasane/Video-OnlineHMR
|
||||
|
||||
Loading…
Reference in New Issue
Block a user