commit 619d35cc79fb7a3c5690b0f6e98f3f9ec63b3ad3 Author: type-two Date: Fri Jul 17 00:40:26 2026 +1000 Lane 0: bootstrap MOCAPGOD git init + .gitignore (weights/capture/queue/out/media excluded), docs/LICENSES.md skeleton with the SMPL gate, PROGRESS.md ledger, and spec/mixamorig_bones.json — 65 mixamorig bones (colon-prefixed) dumped from character_kit/female/female_game.glb via headless Blender. This bone file is the retarget ground truth for Lane C. Co-Authored-By: Claude Opus 4.8 diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..39d6033 --- /dev/null +++ b/.gitignore @@ -0,0 +1,10 @@ +# weights + runtime data stay out of git +weights/ +capture/ +queue/ +out/ +clips_out/ +*.mp4 +*.bvh +__pycache__/ +.DS_Store diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..be494fd --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,16 @@ +# MOCAPGOD + +Private video→mocap→clip pipeline. **Read PLAN.md before doing anything** — it is the +work order; lanes execute one per session, acceptance checks are the definition of done. + +Facts: +- Fleet: m3ultra (`m3ultra@100.89.131.57`) = pose engine (inference/training); + ultra (this Mac) = Blender glue + orchestration + canonical repo; JING5 = capture. +- Skeleton canon: mixamorig 65-joint from `~/Documents/character_kit` (see + `spec/mixamorig_bones.json` once Lane 0 runs). +- Interchange: BVH 30fps + JSON sidecar → anim-only FBX → `merge_anims.py` → GLB banks. +- Legal rails are in PLAN.md §Ground rules — SMPL licensing gate in Lane B is a hard + stop for a John-decision; no Cascadeur reverse engineering; no Mixamo-as-training-data. +- Long jobs: heartbeat convention (`~/.jobs/`); scheduled jobs use /opt/homebrew/bin/python3 + (TCC); bulk output through `| lm -l 2`. +- Progress ledger: PROGRESS.md (create on Lane 0). diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 0000000..075791a --- /dev/null +++ b/PLAN.md @@ -0,0 +1,227 @@ +# MOCAPGOD — private video-to-mocap pipeline + +**Mission:** film John with a phone → extract 3D skeleton motion → retarget onto the +character_kit mixamorig rigs → clean → export FBX/GLB clips → feed the existing clip-bank +pipeline (merge_anims.py → GLB → three.js games + Unreal + 3GOD depot). +100% local, 100% owned, zero cloud, zero subscriptions. + +**Written:** 2026-07-17 by Fable, for execution by Opus 4.8 sessions, one lane at a time. +Each lane is a self-contained work order with acceptance checks. Lanes A→D are the v1 +critical path. Lanes E–F are follow-ons. Do NOT start E/F before D ships. + +--- + +## Ground rules (read before any lane) + +1. **Fleet roles** + - **m3ultra** (`m3ultra@100.89.131.57`, 256GB) — POSE ENGINE. Model inference, any + training. Python via conda/uv there. MODELBEAST lives here; mocap becomes a sibling + service, same patterns (job dirs, batch keys — see `MESHGOD/scripts/mb_*` for the + client convention). + - **ultra** (this Mac, M1 Ultra 128GB) — GLUE + DCC. Blender headless retarget/cleanup/ + export, orchestration, Unreal verification. Owns the canonical repo. + - **JING5** (M5 MBP) — capture companion + portable inference later. Not in v1 critical path. +2. **Repo:** `~/Documents/MOCAPGOD` on ultra is canonical. Mirror what's needed to m3ultra + under `~/Documents/MOCAPGOD` there too; keep model weights OUT of git (`weights/`, + `.gitignore` from day one). Init git in Lane 0. +3. **Long jobs** follow the heartbeat convention (`~/.claude/skills/jobs/SKILL.md`, status + in `~/.jobs/`). Any script expected >10 min gets a heartbeat. Scheduled jobs touching + ~/Documents use `/opt/homebrew/bin/python3` (TCC — see memory: cron python trap). +4. **Verbose output** (installs, model downloads, test runs) pipes through `| lm -l 2`. +5. **Legal rails (hard):** + - NO reverse-engineering Cascadeur or extracting its weights. Ever. + - **SMPL/SMPL-X body model is research-licensed.** Most SOTA video-mocap models emit + SMPL params. Using SMPL-family model files in a commercial pipeline needs a + Meshcapade commercial license. Two allowed paths — pick per Lane B findings: + (a) obtain the license (cheap tiers exist; John decides), or + (b) SMPL-free path: use the model's **joint positions/rotations only**, never the + SMPL mesh/shape model files, and verify the chosen inference repo's own license + permits commercial use of outputs (many are MIT/Apache code with the SMPL file + as a separate user-supplied download — the *outputs* are generally ours, but + VERIFY per repo and record findings in `docs/LICENSES.md`). + - AMASS dataset (if Lane E trains anything): academic license by default — check + commercial terms before training anything shipped. Record in `docs/LICENSES.md`. + - Mixamo clips: fine to ship in games per Adobe ToS; do NOT use as ML training data. + - Everything decided goes in `docs/LICENSES.md` with links + dates. +6. **The treaty (interchange contract)** — every lane speaks this or doesn't ship: + - Skeleton: **mixamorig 65-joint** as defined by `~/Documents/character_kit` + (`female_game.glb` / male base). Canonical bone list generated in Lane 0. + - Intermediate motion format: **BVH at 30fps** + sidecar JSON + (`{fps, joint_order, root_trajectory, source_video, model, version}`). + BVH because Blender imports it natively and it's human-diffable. + - Units metres, Y-up, character faces −Z (three.js convention; Unreal yaw-fix happens + at UE import, as established in the A_3D_STORE experiments). + - Final artifact: FBX (anim-only, Mixamo-style "Without Skin") dropped into + `clips_out/`, then merged to multi-clip GLB via character_kit `merge_anims.py`. + +--- + +## Architecture + +``` +[iPhone/JING5 video] LANE A + │ (AirDrop/Tailscale to ultra: capture/inbox/) + ▼ +[ingest.py — probe, trim, rename, ship to m3ultra] LANE A + │ + ▼ ssh/rsync +[m3ultra POSE ENGINE: video → 3D joints] LANE B + monocular HMR model (GVHMR/WHAM/TRAM-class, chosen in B1) + + hand pass later (E) + │ emits motion.bvh + motion.json + ▼ rsync back +[ultra: Blender headless RETARGET] LANE C + bvh → mixamorig, foot-contact fix, jitter filter, QC render (turntable mp4) + │ emits clip.fbx + qc.mp4 + ▼ +[CLIP FACTORY] LANE D + merge_anims.py → multi-clip GLB → games / 3GOD depot / Unreal import +``` + +One CLI to rule it: `mocap --name dj_scratch_01` runs A→D end-to-end. + +--- + +## LANE 0 — bootstrap (30 min, ultra) + +Deliverables: +- git init MOCAPGOD, `.gitignore` (weights/, capture/, *.mp4, out dirs), `CLAUDE.md` + (fleet roles, treaty summary, pointers to this PLAN). +- `docs/LICENSES.md` skeleton. +- `spec/mixamorig_bones.json`: generate by opening + `character_kit/female/female_game.glb` in headless Blender and dumping the armature + bone names + rest pose. This file is the retarget ground truth. +- Acceptance: `python3 -c "import json; json.load(open('spec/mixamorig_bones.json'))"` + lists 65 bones; git log shows initial commit. + +## LANE A — capture + ingest (half day, ultra) + +Goal: friction-free path from "John films a move" to "video queued for the pose engine." + +Tasks: +1. `docs/CAPTURE.md` — the recording protocol (keep to one page): + - Phone landscape on tripod/lean, 1080p60 (4K unnecessary for pose), full body in + frame INCLUDING FEET at all times, 2–3m distance, even light, plain-ish background, + fitted clothing (baggy = joint error), start/end in T-pose for 1s (calibration + + clip trimming anchor), one move per take, 5–30s takes. + - Transfer: AirDrop to ultra → `~/Documents/MOCAPGOD/capture/inbox/`. +2. `ingest.py` (ultra): watches/scans inbox; per video: ffprobe sanity (fps, duration, + resolution), normalize to 30fps h264 via ffmpeg (`-r 30`), rename + `YYYYMMDD_.mp4`, rsync to `m3ultra:~/Documents/MOCAPGOD/queue/`, log to + `capture/LEDGER.tsv`. +3. Acceptance: drop any phone video in inbox → one command → appears in m3ultra queue, + ledger row written, malformed/portrait/no-audio files handled with clear errors. + +## LANE B — pose engine (1–2 days, m3ultra) ← the heart + +Goal: `queue/video.mp4` → `out//motion.bvh` + `motion.json`, world-grounded +(root trajectory included, not just pose-in-place). + +Tasks: +1. **B1 — model bake-off (do first, half day).** WebSearch for current-best open + monocular human motion recovery ("world-grounded HMR", successors to + GVHMR / WHAM / TRAM / 4D-Humans — the field moves fast; July 2026 SOTA may be newer). + Selection criteria, in order: (1) permissive code license, (2) world-grounded global + trajectory, (3) runs on Apple Silicon (MPS or CPU acceptably; CUDA-only repos often + port with `map_location` + MPS fallbacks — budget for this), (4) SMPL-dependency + status per the legal rails. Install top 2 candidates in separate uv envs, run both on + 3 test videos (walk, dance move, pick-up-object), eyeball in Blender. Record the + decision + benchmarks in `docs/B1_BAKEOFF.md`. +2. **B2 — service-ize.** `pose_engine.py`: consumes queue dir, runs chosen model, + converts output (SMPL rotations or joints) → mixamorig-ordered BVH @30fps + sidecar + JSON per the treaty. Heartbeat convention. The SMPL→mixamorig joint mapping table + lives in `spec/smpl_to_mixamorig.json` (write once, verify visually). +3. **B3 — jitter pass.** One-Euro or Savitzky–Golay filter on rotations, configurable; + root trajectory smoothed separately (preserve intentional locomotion). +4. Acceptance: 10s test video → BVH that imports into Blender with a recognizable, + right-way-up, non-sliding-through-floor human motion; runtime < 2× realtime on + m3ultra; survives a queue of 5 videos unattended (heartbeat proves it). + +Gotchas from experience: check MPS support early — it decides everything; if the model +needs SMPL files, STOP and do the licensing decision before proceeding (present both +paths to John); keep every model in its own uv env (their dependency trees fight). + +## LANE C — retarget + cleanup (1 day, ultra) + +Goal: `motion.bvh` → clean `clip.fbx` on the character_kit rig + a QC video proof. + +Tasks: +1. `retarget.py` (headless Blender, extends the house style of + `character_kit/scripts/merge_anims.py`): + - Import BVH → constraint-map bones to mixamorig armature from + `spec/mixamorig_bones.json` (name-map table in `spec/`, handles `mixamorig:` vs + `mixamorig_` variants — this bit us before), bake to keyframes. + - Foot-contact fix: detect frames where foot velocity ≈ 0 & height < threshold → IK + pin. `# ponytail: threshold heuristic, upgrade to learned contact detection in Lane E if it annoys`. + - Trim the T-pose bookends (Lane A protocol guarantees them), root-motion option + (in-place vs traveling) as a flag. + - Export anim-only FBX (Mixamo "Without Skin" style — merge_anims.py already eats these). +2. QC render: 3-second turntable mp4 of the clip on the female_game body, written next + to the FBX. **Verify with own eyes** (the rule) — every clip gets its mp4 looked at + before entering the bank. +3. Acceptance: full chain A→C on one video of John walking produces an FBX that + `merge_anims.py` merges into `female_game.glb` without errors, and the QC mp4 shows + walking, not soup. + +## LANE D — clip factory + integration (half day, ultra) + +Goal: one command end-to-end; clips land everywhere they should. + +Tasks: +1. `mocap` CLI (single script): `mocap capture/inbox/vid.mp4 --name dj_scratch_01 + [--character female|male|both] [--in-place]` → runs A ingest → waits on B (ssh) → C → + drops `clips_out/dj_scratch_01.fbx` + QC mp4, optionally re-merges the character's + clip-bank GLB. +2. Publish hooks (flags, off by default): `--publish-3god` (depot pattern from procity + `publish.py`), `--to-unreal` (import via the established A_3D_STORE deferred-import + pattern; see memory: unreal-asset-convergence — slate post-tick, NOT direct + AssetImportTask through the bridge). +3. `docs/RUNBOOK.md`: the whole flow on one page, written for future-John. +4. Acceptance: film a 10s move → ONE command → new clip playable in 90sDJsim dev build + AND visible on a character in Unreal. That's the v1 finish line. Take the win, stop. + +## LANE E — ML upgrades (post-v1, m3ultra, MLX territory; each item independent) + +- E1: hand/finger pass (MediaPipe Hands or successor) blended onto body clips — + DJ work needs fingers eventually. +- E2: custom autoposer — MLP (6 control points → full pose) trained in MLX on + licensed/open mocap; our own baby Cascadeur-autopose for procedural NPC posing in + games at runtime. Check AMASS commercial terms FIRST (docs/LICENSES.md). +- E3: inbetweening/motion generation (current open motion-diffusion lineage) for + clip variation + transitions. +- E4: quadruped/centaur retarget experiments (front-half human map + back-half horse + chain) — feeds the centaur RPG. Cascadeur Pro handles the authored side; this is the + generated side. +- E5: style filter — train small nets on OUR OWN authored/captured clip library + (which we fully own) to stylize mocap output. + +## LANE F — UX (optional, whenever it annoys) + +- Drop-folder daemon (launchd on ultra; remember TCC python path) so AirDropped videos + auto-process; notification with QC mp4 when done. +- Tiny web dashboard on tailnet listing clips + QC videos (stdlib http.server, house style). + +--- + +## Effort + risk summary + +| Lane | Effort | Risk | Mitigation | +|---|---|---|---| +| 0+A | ~1 day | trivial | — | +| B | 1–2 days | MPS compat; SMPL licensing | bake-off first; legal rails decision gate | +| C | 1 day | retarget quality (feet, drift) | contact-fix pass + human QC gate | +| D | half day | none new | reuses proven pieces | +| E | open-ended | dataset licenses | per-item gates | + +v1 total: **~3–4 focused days of agent work.** Quality expectation: solid for stylized +NPC/DJ clips; not Vicon; fingers come in E1. Cascadeur (bought separately) remains the +authoring/polish tool — its "animation unbaking" even cleans OUR mocap output, fully +within its license. + +## Session protocol for Opus + +One lane per session. Read this file + `CLAUDE.md` + the lane's acceptance checks first. +On completion: tick the lane in `PROGRESS.md`, commit, update `docs/` with anything +learned, leave gotchas for the next lane. Do not gold-plate; every lane's acceptance +check is the definition of done. When blocked on a John-decision (licenses, purchases), +write the decision memo in `docs/DECISIONS.md` and stop the lane there. diff --git a/PROGRESS.md b/PROGRESS.md new file mode 100644 index 0000000..ad2219f --- /dev/null +++ b/PROGRESS.md @@ -0,0 +1,20 @@ +# MOCAPGOD progress ledger + +One lane per session. Tick on completion. See PLAN.md for acceptance checks. + +- [x] **Lane 0** — bootstrap. git + .gitignore, docs/LICENSES.md skeleton, + spec/mixamorig_bones.json (65 bones, `mixamorig:` colon-prefixed). — 2026-07-17 +- [x] **Lane A** — capture + ingest. docs/CAPTURE.md protocol, ingest.py + (probe → 30fps normalize → rename → rsync to m3ultra queue → LEDGER.tsv). — 2026-07-17 +- [ ] **Lane B** — pose engine (m3ultra). Model bake-off, service-ize, jitter pass. +- [ ] **Lane C** — retarget + cleanup (Blender). bvh → mixamorig FBX + QC turntable. +- [ ] **Lane D** — clip factory. `mocap` CLI end-to-end; games + Unreal integration. +- [ ] Lane E — ML upgrades (post-v1, gated). +- [ ] Lane F — UX (optional). + +## Notes for the next lane +- Ground-truth skeleton naming is **`mixamorig:`** (colon), armature object named `Armature`. + Lane C's name-map must handle colon vs underscore variants (PLAN warned about this). +- Blender headless entrypoint: `/Applications/Blender.app/Contents/MacOS/Blender --background --python