c918ac3dcd
First generated-footage clip proven through the full chain: Wan2.1 T2V (m3ultra, prompt-engineered to CAPTURE.md rules) -> ./mocap -> wan_walk_t1.fbx -> merge_anims bank (49 tracks). PROGRESS.md bug closed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
57 lines
3.6 KiB
Markdown
57 lines
3.6 KiB
Markdown
# MOCAPGOD runbook — film → clip, one command
|
||
|
||
The whole v1 pipeline on one page. Everything runs on **ultra**.
|
||
|
||
## The one command
|
||
```
|
||
cd ~/Documents/MOCAPGOD
|
||
./mocap capture/inbox/<vid>.mp4 --name dj_scratch_01 # female rig, in-place
|
||
./mocap <vid>.mov --name spin --character male --travel # male rig, keep world travel
|
||
```
|
||
Options: `--character female|male` (default female), `--travel` (keep root translation; default
|
||
in-place), `--no-qc`, `--gender` (override pose gender), `--selfcheck`.
|
||
|
||
**Output:** `clips_out/<name>.fbx` (anim-only, Mixamo "Without Skin") + `clips_out/<name>_qc.mp4`.
|
||
**House rule:** watch the QC mp4 before the clip enters any bank. Not soup = good.
|
||
|
||
## What it chains (all proven; `mocap` is just the glue)
|
||
| Lane | Script | Does | Emits | Runs in |
|
||
|---|---|---|---|---|
|
||
| A | `ingest.py` | probe → 30fps h264 → rename → queue. **Moves** the source to `capture/done/`. | `queue/<date>_<name>.mp4` | system python3 |
|
||
| B | `pose_engine.py` | HSMR/SKEL monocular recovery + SKEL→BVH + jitter | `out/<stem>/motion.bvh` + `motion.json` + `params.npz` | `.engine/HSMR/.venv` (MPS) |
|
||
| C | `retarget.py` | BVH → mixamorig (constraint + bake) → FBX + turntable QC | `clips_out/<name>.fbx` + `_qc.mp4` | headless Blender 5 |
|
||
|
||
Timing: pose engine dominates (~1.6× realtime on ultra/MPS); a 10s clip ≈ a couple of minutes.
|
||
|
||
## Prerequisites (one-time)
|
||
- **SKEL v1.1.1** at `.engine/HSMR/data_inputs/body_models/skel/skel_male.pkl` (+ female). Gated
|
||
download — see `docs/B_ENGINE_SETUP.md`. This is the only hard blocker if the engine is fresh.
|
||
- HSMR env built: `.engine/setup_hsmr.sh --patch` (idempotent). Blender at `/Applications/Blender.app`.
|
||
- Filming protocol: `docs/CAPTURE.md` (landscape, **feet in frame**, T-pose bookends, one move/take).
|
||
|
||
## Known-good vs. rough edges (honest)
|
||
- **In-place clips are first-class.** HSMR is per-frame → root is camera-space, not world-grounded
|
||
(`motion.json: root_space:"camera"`). DJ moves / dancing-on-the-spot come out clean.
|
||
- **Travel (`--travel`) is best-effort** until a foot-contact grounding pass exists — walking clips
|
||
may drift/slide. Foot-fix is a logged stub today (`retarget.py foot_fix`); tune it against a real
|
||
walk clip (Lane C deferred).
|
||
- **Fingers are Lane E** — the body clip has static hands. Fine for v1 NPC/DJ body moves.
|
||
- Male rig (`--character male` → `hum_character.glb`) is **unverified** until a male clip is run;
|
||
female (`female_game.glb`) is the proven path.
|
||
|
||
## Deferred hooks — build these when real footage lands (not faked in the CLI)
|
||
Each is off by default and needs a real clip to verify, so none ship as dead flags today.
|
||
1. **Clip-bank GLB merge** (`--merge`, → 90sDJsim / three.js games). Games load GLB, so this is the
|
||
"lands in-game" step. Needs a small merge script first: `character_kit/scripts/merge_anims.py`
|
||
has **no CLI and is pre-Baklava** — it must add the Blender-5 slot-bind
|
||
(`animation_data.action_slot = act.slots[0]`, verified in Lane C) or the merged GLB won't animate.
|
||
Then merge `clips_out/<name>.fbx` onto the character's bank GLB and export.
|
||
2. **`--to-unreal`** — import via the A_3D_STORE deferred-import pattern (memory:
|
||
`unreal-asset-convergence` — slate post-tick, NOT a direct AssetImportTask through the bridge).
|
||
3. **`--publish-3god`** — depot pattern from procity `publish.py`.
|
||
|
||
## John's homework (blocks the real quality verdict)
|
||
Film **walk / dance / pick-up-object** per `docs/CAPTURE.md`, drop in `capture/inbox/`, run `mocap`.
|
||
Until then the pipeline is proven on the HSMR `gymnasts` demo only — machinery works; the honest
|
||
"is it good enough for my DJ NPCs" call needs footage of you.
|