mocapgod/PLAN.md

# MOCAPGOD — private video-to-mocap pipeline

**Mission:** film John with a phone → extract 3D skeleton motion → retarget onto the
character_kit mixamorig rigs → clean → export FBX/GLB clips → feed the existing clip-bank
pipeline (merge_anims.py → GLB → three.js games + Unreal + 3GOD depot).
100% local, 100% owned, zero cloud, zero subscriptions.

**Written:** 2026-07-17 by Fable, for execution by Opus 4.8 sessions, one lane at a time.
Each lane is a self-contained work order with acceptance checks. Lanes A→D are the v1
critical path. Lanes E–F are follow-ons. Do NOT start E/F before D ships.

---

## Ground rules (read before any lane)

1. **Fleet roles**
   - **m3ultra** (`m3ultra@100.89.131.57`, 256GB) — POSE ENGINE. Model inference, any
     training. Python via conda/uv there. MODELBEAST lives here; mocap becomes a sibling
     service, same patterns (job dirs, batch keys — see `MESHGOD/scripts/mb_*` for the
     client convention).
     - **v1 deviation (session 5):** HSMR is installed and MPS-verified on **ultra**, not
       m3ultra, so v1 runs inference on ultra and `ingest.py` defaults to a **local** `queue/`.
       `# ponytail: v1 inference on ultra; migrate to m3ultra when a real throughput need appears`
       (`ingest.py --remote-queue` already ships to m3ultra when that day comes).
   - **ultra** (this Mac, M1 Ultra 128GB) — GLUE + DCC. Blender headless retarget/cleanup/
     export, orchestration, Unreal verification. Owns the canonical repo.
   - **JING5** (M5 MBP) — capture companion + portable inference later. Not in v1 critical path.
2. **Repo:** `~/Documents/MOCAPGOD` on ultra is canonical. Mirror what's needed to m3ultra
   under `~/Documents/MOCAPGOD` there too; keep model weights OUT of git (`weights/`,
   `.gitignore` from day one). Init git in Lane 0.
3. **Long jobs** follow the heartbeat convention (`~/.claude/skills/jobs/SKILL.md`, status
   in `~/.jobs/`). Any script expected >10 min gets a heartbeat. Scheduled jobs touching
   ~/Documents use `/opt/homebrew/bin/python3` (TCC — see memory: cron python trap).
4. **Verbose output** (installs, model downloads, test runs) pipes through `| lm -l 2`.
5. **Legal rails (hard):**
   - NO reverse-engineering Cascadeur or extracting its weights. Ever.
   - **SMPL/SMPL-X body model is research-licensed.** Most SOTA video-mocap models emit
     SMPL params. Using SMPL-family model files in a commercial pipeline needs a
     Meshcapade commercial license. Two allowed paths — pick per Lane B findings:
     (a) obtain the license (cheap tiers exist; John decides), or
     (b) SMPL-free path: use the model's **joint positions/rotations only**, never the
         SMPL mesh/shape model files, and verify the chosen inference repo's own license
         permits commercial use of outputs (many are MIT/Apache code with the SMPL file
         as a separate user-supplied download — the *outputs* are generally ours, but
         VERIFY per repo and record findings in `docs/LICENSES.md`).
   - AMASS dataset (if Lane E trains anything): academic license by default — check
     commercial terms before training anything shipped. Record in `docs/LICENSES.md`.
   - Mixamo clips: fine to ship in games per Adobe ToS; do NOT use as ML training data.
   - Everything decided goes in `docs/LICENSES.md` with links + dates.
6. **The treaty (interchange contract)** — every lane speaks this or doesn't ship:
   - Skeleton: **mixamorig 65-joint** as defined by `~/Documents/character_kit`
     (`female_game.glb` / male base). Canonical bone list generated in Lane 0.
   - Intermediate motion format: **BVH at 30fps** + sidecar JSON
     (`{fps, joint_order, root_space, root_trajectory, source_video, model, version}`).
     **`root_space` ∈ {`"camera"`, `"world"`} is REQUIRED** (session-5 amendment): HSMR is a
     per-frame recoverer so it emits `"camera"`; Lane C reads this and, for locomotion clips,
     derives a world root from foot contacts. BVH because Blender imports it natively and it's
     human-diffable.
   - Units metres, Y-up, character faces −Z (three.js convention; Unreal yaw-fix happens
     at UE import, as established in the A_3D_STORE experiments).
   - Final artifact: FBX (anim-only, Mixamo-style "Without Skin") dropped into
     `clips_out/`, then merged to multi-clip GLB via character_kit `merge_anims.py`.

---

## Architecture

```
[iPhone/JING5 video]                       LANE A
      │  (AirDrop/Tailscale to ultra: capture/inbox/)
      ▼
[ingest.py — probe, trim, rename, ship to m3ultra]     LANE A
      │
      ▼  ssh/rsync
[m3ultra POSE ENGINE: video → 3D joints]                LANE B
  monocular HMR model (GVHMR/WHAM/TRAM-class, chosen in B1)
  + hand pass later (E)
      │  emits motion.bvh + motion.json
      ▼  rsync back
[ultra: Blender headless RETARGET]                      LANE C
  bvh → mixamorig, foot-contact fix, jitter filter, QC render (turntable mp4)
      │  emits clip.fbx + qc.mp4
      ▼
[CLIP FACTORY]                                          LANE D
  merge_anims.py → multi-clip GLB → games / 3GOD depot / Unreal import
```

One CLI to rule it: `mocap <video.mp4> --name dj_scratch_01` runs A→D end-to-end.

---

## LANE 0 — bootstrap (30 min, ultra)

Deliverables:
- git init MOCAPGOD, `.gitignore` (weights/, capture/, *.mp4, out dirs), `CLAUDE.md`
  (fleet roles, treaty summary, pointers to this PLAN).
- `docs/LICENSES.md` skeleton.
- `spec/mixamorig_bones.json`: generate by opening
  `character_kit/female/female_game.glb` in headless Blender and dumping the armature
  bone names + rest pose. This file is the retarget ground truth.
- Acceptance: `python3 -c "import json; json.load(open('spec/mixamorig_bones.json'))"`
  lists 65 bones; git log shows initial commit.

## LANE A — capture + ingest (half day, ultra)

Goal: friction-free path from "John films a move" to "video queued for the pose engine."

Tasks:
1. `docs/CAPTURE.md` — the recording protocol (keep to one page):
   - Phone landscape on tripod/lean, 1080p60 (4K unnecessary for pose), full body in
     frame INCLUDING FEET at all times, 2–3m distance, even light, plain-ish background,
     fitted clothing (baggy = joint error), start/end in T-pose for 1s (calibration +
     clip trimming anchor), one move per take, 5–30s takes.
   - Transfer: AirDrop to ultra → `~/Documents/MOCAPGOD/capture/inbox/`.
2. `ingest.py` (ultra): watches/scans inbox; per video: ffprobe sanity (fps, duration,
   resolution), normalize to 30fps h264 via ffmpeg (`-r 30`), rename
   `YYYYMMDD_<name>.mp4`, rsync to `m3ultra:~/Documents/MOCAPGOD/queue/`, log to
   `capture/LEDGER.tsv`.
3. Acceptance: drop any phone video in inbox → one command → appears in m3ultra queue,
   ledger row written, malformed/portrait/no-audio files handled with clear errors.

## LANE B — pose engine (1–2 days, m3ultra) ← the heart

Goal: `queue/video.mp4` → `out/<name>/motion.bvh` + `motion.json`, world-grounded
(root trajectory included, not just pose-in-place).

Tasks:
1. **B1 — model bake-off (do first, half day).** WebSearch for current-best open
   monocular human motion recovery ("world-grounded HMR", successors to
   GVHMR / WHAM / TRAM / 4D-Humans — the field moves fast; July 2026 SOTA may be newer).
   Selection criteria, in order: (1) permissive code license, (2) world-grounded global
   trajectory, (3) runs on Apple Silicon (MPS or CPU acceptably; CUDA-only repos often
   port with `map_location` + MPS fallbacks — budget for this), (4) SMPL-dependency
   status per the legal rails. Install top 2 candidates in separate uv envs, run both on
   3 test videos (walk, dance move, pick-up-object), eyeball in Blender. Record the
   decision + benchmarks in `docs/B1_BAKEOFF.md`.
2. **B2 — service-ize.** `pose_engine.py`: consumes queue dir, runs chosen model,
   converts output (SMPL rotations or joints) → mixamorig-ordered BVH @30fps + sidecar
   JSON per the treaty. Heartbeat convention. The SMPL→mixamorig joint mapping table
   lives in `spec/smpl_to_mixamorig.json` (write once, verify visually).
3. **B3 — jitter pass.** One-Euro or Savitzky–Golay filter on rotations, configurable;
   root trajectory smoothed separately (preserve intentional locomotion).
4. Acceptance: 10s test video → BVH that imports into Blender with a recognizable,
   right-way-up, non-sliding-through-floor human motion; runtime < 2× realtime on
   m3ultra; survives a queue of 5 videos unattended (heartbeat proves it).

Gotchas from experience: check MPS support early — it decides everything; if the model
needs SMPL files, STOP and do the licensing decision before proceeding (present both
paths to John); keep every model in its own uv env (their dependency trees fight).

## LANE C — retarget + cleanup (1 day, ultra)

Goal: `motion.bvh` → clean `clip.fbx` on the character_kit rig + a QC video proof.

Tasks:
1. `retarget.py` (headless Blender, extends the house style of
   `character_kit/scripts/merge_anims.py`):
   - Import BVH → constraint-map bones to mixamorig armature from
     `spec/mixamorig_bones.json` (name-map table in `spec/`, handles `mixamorig:` vs
     `mixamorig_` variants — this bit us before), bake to keyframes.
   - Foot-contact fix: detect frames where foot velocity ≈ 0 & height < threshold → IK
     pin. `# ponytail: threshold heuristic, upgrade to learned contact detection in Lane E if it annoys`.
   - Trim the T-pose bookends (Lane A protocol guarantees them). Root motion is driven by the
     sidecar **`root_space`** (session-5 treaty): `"camera"` (HSMR) → in-place by default, or
     derive a world root from foot contacts for traveling clips; a `--in-place`/`--traveling`
     flag overrides. v1 quality bar: **in-place clips are first-class; traveling is best-effort**
     until a proper grounding pass exists.
   - Export anim-only FBX (Mixamo "Without Skin" style — merge_anims.py already eats these).
2. QC render: 3-second turntable mp4 of the clip on the female_game body, written next
   to the FBX. **Verify with own eyes** (the rule) — every clip gets its mp4 looked at
   before entering the bank.
3. Acceptance: full chain A→C on one video of John walking produces an FBX that
   `merge_anims.py` merges into `female_game.glb` without errors, and the QC mp4 shows
   walking, not soup.

## LANE D — clip factory + integration (half day, ultra)

Goal: one command end-to-end; clips land everywhere they should.

Tasks:
1. `mocap` CLI (single script): `mocap capture/inbox/vid.mp4 --name dj_scratch_01
   [--character female|male|both] [--in-place]` → runs A ingest → waits on B (ssh) → C →
   drops `clips_out/dj_scratch_01.fbx` + QC mp4, optionally re-merges the character's
   clip-bank GLB.
2. Publish hooks (flags, off by default): `--publish-3god` (depot pattern from procity
   `publish.py`), `--to-unreal` (import via the established A_3D_STORE deferred-import
   pattern; see memory: unreal-asset-convergence — slate post-tick, NOT direct
   AssetImportTask through the bridge).
3. `docs/RUNBOOK.md`: the whole flow on one page, written for future-John.
4. Acceptance: film a 10s move → ONE command → new clip playable in 90sDJsim dev build
   AND visible on a character in Unreal. That's the v1 finish line. Take the win, stop.

## LANE E — ML upgrades (post-v1, m3ultra, MLX territory; each item independent)

- E1: hand/finger pass (MediaPipe Hands or successor) blended onto body clips —
  DJ work needs fingers eventually.
- E2: custom autoposer — MLP (6 control points → full pose) trained in MLX on
  licensed/open mocap; our own baby Cascadeur-autopose for procedural NPC posing in
  games at runtime. Check AMASS commercial terms FIRST (docs/LICENSES.md).
- E3: inbetweening/motion generation (current open motion-diffusion lineage) for
  clip variation + transitions.
- E4: quadruped/centaur retarget experiments (front-half human map + back-half horse
  chain) — feeds the centaur RPG. Cascadeur Pro handles the authored side; this is the
  generated side.
- E5: style filter — train small nets on OUR OWN authored/captured clip library
  (which we fully own) to stylize mocap output.

## LANE F — UX (optional, whenever it annoys)

- Drop-folder daemon (launchd on ultra; remember TCC python path) so AirDropped videos
  auto-process; notification with QC mp4 when done.
- Tiny web dashboard on tailnet listing clips + QC videos (stdlib http.server, house style).

---

## Effort + risk summary

| Lane | Effort | Risk | Mitigation |
|---|---|---|---|
| 0+A | ~1 day | trivial | — |
| B | 1–2 days | MPS compat; SMPL licensing | bake-off first; legal rails decision gate |
| C | 1 day | retarget quality (feet, drift) | contact-fix pass + human QC gate |
| D | half day | none new | reuses proven pieces |
| E | open-ended | dataset licenses | per-item gates |

v1 total: **~3–4 focused days of agent work.** Quality expectation: solid for stylized
NPC/DJ clips; not Vicon; fingers come in E1. Cascadeur (bought separately) remains the
authoring/polish tool — its "animation unbaking" even cleans OUR mocap output, fully
within its license.

## Session protocol for Opus

One lane per session. Read this file + `CLAUDE.md` + the lane's acceptance checks first.
On completion: tick the lane in `PROGRESS.md`, commit, update `docs/` with anything
learned, leave gotchas for the next lane. Do not gold-plate; every lane's acceptance
check is the definition of done. When blocked on a John-decision (licenses, purchases),
write the decision memo in `docs/DECISIONS.md` and stop the lane there.