mocapgod/PLAN.md

MOCAPGOD — private video-to-mocap pipeline

Mission: film John with a phone → extract 3D skeleton motion → retarget onto the character_kit mixamorig rigs → clean → export FBX/GLB clips → feed the existing clip-bank pipeline (merge_anims.py → GLB → three.js games + Unreal + 3GOD depot). 100% local, 100% owned, zero cloud, zero subscriptions.

Written: 2026-07-17 by Fable, for execution by Opus 4.8 sessions, one lane at a time. Each lane is a self-contained work order with acceptance checks. Lanes A→D are the v1 critical path. Lanes E–F are follow-ons. Do NOT start E/F before D ships.

---

Ground rules (read before any lane)

1. Fleet roles
   • m3ultra (m3ultra@100.89.131.57, 256GB) — POSE ENGINE. Model inference, any training. Python via conda/uv there. MODELBEAST lives here; mocap becomes a sibling service, same patterns (job dirs, batch keys — see MESHGOD/scripts/mb_* for the client convention).
   • ultra (this Mac, M1 Ultra 128GB) — GLUE + DCC. Blender headless retarget/cleanup/ export, orchestration, Unreal verification. Owns the canonical repo.
   • JING5 (M5 MBP) — capture companion + portable inference later. Not in v1 critical path.
2. Repo: ~/Documents/MOCAPGOD on ultra is canonical. Mirror what's needed to m3ultra under ~/Documents/MOCAPGOD there too; keep model weights OUT of git (weights/, .gitignore from day one). Init git in Lane 0.
3. Long jobs follow the heartbeat convention (~/.claude/skills/jobs/SKILL.md, status in ~/.jobs/). Any script expected >10 min gets a heartbeat. Scheduled jobs touching ~/Documents use /opt/homebrew/bin/python3 (TCC — see memory: cron python trap).
4. Verbose output (installs, model downloads, test runs) pipes through | lm -l 2.
5. Legal rails (hard):
   • NO reverse-engineering Cascadeur or extracting its weights. Ever.
   • SMPL/SMPL-X body model is research-licensed. Most SOTA video-mocap models emit SMPL params. Using SMPL-family model files in a commercial pipeline needs a Meshcapade commercial license. Two allowed paths — pick per Lane B findings: (a) obtain the license (cheap tiers exist; John decides), or (b) SMPL-free path: use the model's joint positions/rotations only, never the SMPL mesh/shape model files, and verify the chosen inference repo's own license permits commercial use of outputs (many are MIT/Apache code with the SMPL file as a separate user-supplied download — the outputs are generally ours, but VERIFY per repo and record findings in docs/LICENSES.md).
   • AMASS dataset (if Lane E trains anything): academic license by default — check commercial terms before training anything shipped. Record in docs/LICENSES.md.
   • Mixamo clips: fine to ship in games per Adobe ToS; do NOT use as ML training data.
   • Everything decided goes in docs/LICENSES.md with links + dates.
6. The treaty (interchange contract) — every lane speaks this or doesn't ship:
   • Skeleton: mixamorig 65-joint as defined by ~/Documents/character_kit (female_game.glb / male base). Canonical bone list generated in Lane 0.
   • Intermediate motion format: BVH at 30fps + sidecar JSON ({fps, joint_order, root_trajectory, source_video, model, version}). BVH because Blender imports it natively and it's human-diffable.
   • Units metres, Y-up, character faces −Z (three.js convention; Unreal yaw-fix happens at UE import, as established in the A_3D_STORE experiments).
   • Final artifact: FBX (anim-only, Mixamo-style "Without Skin") dropped into clips_out/, then merged to multi-clip GLB via character_kit merge_anims.py.

---

Architecture

[iPhone/JING5 video]                       LANE A
      │  (AirDrop/Tailscale to ultra: capture/inbox/)
      ▼
[ingest.py — probe, trim, rename, ship to m3ultra]     LANE A
      │
      ▼  ssh/rsync
[m3ultra POSE ENGINE: video → 3D joints]                LANE B
  monocular HMR model (GVHMR/WHAM/TRAM-class, chosen in B1)
  + hand pass later (E)
      │  emits motion.bvh + motion.json
      ▼  rsync back
[ultra: Blender headless RETARGET]                      LANE C
  bvh → mixamorig, foot-contact fix, jitter filter, QC render (turntable mp4)
      │  emits clip.fbx + qc.mp4
      ▼
[CLIP FACTORY]                                          LANE D
  merge_anims.py → multi-clip GLB → games / 3GOD depot / Unreal import

One CLI to rule it: mocap <video.mp4> --name dj_scratch_01 runs A→D end-to-end.

---

LANE 0 — bootstrap (30 min, ultra)

Deliverables:

• git init MOCAPGOD, .gitignore (weights/, capture/, *.mp4, out dirs), CLAUDE.md (fleet roles, treaty summary, pointers to this PLAN).
• docs/LICENSES.md skeleton.
• spec/mixamorig_bones.json: generate by opening character_kit/female/female_game.glb in headless Blender and dumping the armature bone names + rest pose. This file is the retarget ground truth.
• Acceptance: python3 -c "import json; json.load(open('spec/mixamorig_bones.json'))" lists 65 bones; git log shows initial commit.

LANE A — capture + ingest (half day, ultra)

Goal: friction-free path from "John films a move" to "video queued for the pose engine."

Tasks:

1. docs/CAPTURE.md — the recording protocol (keep to one page):
   • Phone landscape on tripod/lean, 1080p60 (4K unnecessary for pose), full body in frame INCLUDING FEET at all times, 2–3m distance, even light, plain-ish background, fitted clothing (baggy = joint error), start/end in T-pose for 1s (calibration + clip trimming anchor), one move per take, 5–30s takes.
   • Transfer: AirDrop to ultra → ~/Documents/MOCAPGOD/capture/inbox/.
2. ingest.py (ultra): watches/scans inbox; per video: ffprobe sanity (fps, duration, resolution), normalize to 30fps h264 via ffmpeg (-r 30), rename YYYYMMDD_<name>.mp4, rsync to m3ultra:~/Documents/MOCAPGOD/queue/, log to capture/LEDGER.tsv.
3. Acceptance: drop any phone video in inbox → one command → appears in m3ultra queue, ledger row written, malformed/portrait/no-audio files handled with clear errors.

LANE B — pose engine (1–2 days, m3ultra) ← the heart

Goal: queue/video.mp4 → out/<name>/motion.bvh + motion.json, world-grounded (root trajectory included, not just pose-in-place).

Tasks:

1. B1 — model bake-off (do first, half day). WebSearch for current-best open monocular human motion recovery ("world-grounded HMR", successors to GVHMR / WHAM / TRAM / 4D-Humans — the field moves fast; July 2026 SOTA may be newer). Selection criteria, in order: (1) permissive code license, (2) world-grounded global trajectory, (3) runs on Apple Silicon (MPS or CPU acceptably; CUDA-only repos often port with map_location + MPS fallbacks — budget for this), (4) SMPL-dependency status per the legal rails. Install top 2 candidates in separate uv envs, run both on 3 test videos (walk, dance move, pick-up-object), eyeball in Blender. Record the decision + benchmarks in docs/B1_BAKEOFF.md.
2. B2 — service-ize. pose_engine.py: consumes queue dir, runs chosen model, converts output (SMPL rotations or joints) → mixamorig-ordered BVH @30fps + sidecar JSON per the treaty. Heartbeat convention. The SMPL→mixamorig joint mapping table lives in spec/smpl_to_mixamorig.json (write once, verify visually).
3. B3 — jitter pass. One-Euro or Savitzky–Golay filter on rotations, configurable; root trajectory smoothed separately (preserve intentional locomotion).
4. Acceptance: 10s test video → BVH that imports into Blender with a recognizable, right-way-up, non-sliding-through-floor human motion; runtime < 2× realtime on m3ultra; survives a queue of 5 videos unattended (heartbeat proves it).

Gotchas from experience: check MPS support early — it decides everything; if the model needs SMPL files, STOP and do the licensing decision before proceeding (present both paths to John); keep every model in its own uv env (their dependency trees fight).

LANE C — retarget + cleanup (1 day, ultra)

Goal: motion.bvh → clean clip.fbx on the character_kit rig + a QC video proof.

Tasks:

1. retarget.py (headless Blender, extends the house style of character_kit/scripts/merge_anims.py):
   • Import BVH → constraint-map bones to mixamorig armature from spec/mixamorig_bones.json (name-map table in spec/, handles mixamorig: vs mixamorig_ variants — this bit us before), bake to keyframes.
   • Foot-contact fix: detect frames where foot velocity ≈ 0 & height < threshold → IK pin. # ponytail: threshold heuristic, upgrade to learned contact detection in Lane E if it annoys.
   • Trim the T-pose bookends (Lane A protocol guarantees them), root-motion option (in-place vs traveling) as a flag.
   • Export anim-only FBX (Mixamo "Without Skin" style — merge_anims.py already eats these).
2. QC render: 3-second turntable mp4 of the clip on the female_game body, written next to the FBX. Verify with own eyes (the rule) — every clip gets its mp4 looked at before entering the bank.
3. Acceptance: full chain A→C on one video of John walking produces an FBX that merge_anims.py merges into female_game.glb without errors, and the QC mp4 shows walking, not soup.

LANE D — clip factory + integration (half day, ultra)

Goal: one command end-to-end; clips land everywhere they should.

Tasks:

1. mocap CLI (single script): mocap capture/inbox/vid.mp4 --name dj_scratch_01 [--character female|male|both] [--in-place] → runs A ingest → waits on B (ssh) → C → drops clips_out/dj_scratch_01.fbx + QC mp4, optionally re-merges the character's clip-bank GLB.
2. Publish hooks (flags, off by default): --publish-3god (depot pattern from procity publish.py), --to-unreal (import via the established A_3D_STORE deferred-import pattern; see memory: unreal-asset-convergence — slate post-tick, NOT direct AssetImportTask through the bridge).
3. docs/RUNBOOK.md: the whole flow on one page, written for future-John.
4. Acceptance: film a 10s move → ONE command → new clip playable in 90sDJsim dev build AND visible on a character in Unreal. That's the v1 finish line. Take the win, stop.

LANE E — ML upgrades (post-v1, m3ultra, MLX territory; each item independent)

• E1: hand/finger pass (MediaPipe Hands or successor) blended onto body clips — DJ work needs fingers eventually.
• E2: custom autoposer — MLP (6 control points → full pose) trained in MLX on licensed/open mocap; our own baby Cascadeur-autopose for procedural NPC posing in games at runtime. Check AMASS commercial terms FIRST (docs/LICENSES.md).
• E3: inbetweening/motion generation (current open motion-diffusion lineage) for clip variation + transitions.
• E4: quadruped/centaur retarget experiments (front-half human map + back-half horse chain) — feeds the centaur RPG. Cascadeur Pro handles the authored side; this is the generated side.
• E5: style filter — train small nets on OUR OWN authored/captured clip library (which we fully own) to stylize mocap output.

LANE F — UX (optional, whenever it annoys)

• Drop-folder daemon (launchd on ultra; remember TCC python path) so AirDropped videos auto-process; notification with QC mp4 when done.
• Tiny web dashboard on tailnet listing clips + QC videos (stdlib http.server, house style).

---

Effort + risk summary

Lane	Effort	Risk	Mitigation
0+A	~1 day	trivial	—
B	1–2 days	MPS compat; SMPL licensing	bake-off first; legal rails decision gate
C	1 day	retarget quality (feet, drift)	contact-fix pass + human QC gate
D	half day	none new	reuses proven pieces
E	open-ended	dataset licenses	per-item gates

v1 total: ~3–4 focused days of agent work. Quality expectation: solid for stylized NPC/DJ clips; not Vicon; fingers come in E1. Cascadeur (bought separately) remains the authoring/polish tool — its "animation unbaking" even cleans OUR mocap output, fully within its license.

Session protocol for Opus

One lane per session. Read this file + CLAUDE.md + the lane's acceptance checks first. On completion: tick the lane in PROGRESS.md, commit, update docs/ with anything learned, leave gotchas for the next lane. Do not gold-plate; every lane's acceptance check is the definition of done. When blocked on a John-decision (licenses, purchases), write the decision memo in docs/DECISIONS.md and stop the lane there.