From e29edc808b77cd97ef4d36f0f650127b32f29327 Mon Sep 17 00:00:00 2001 From: type-two Date: Sat, 18 Jul 2026 17:44:07 +1000 Subject: [PATCH] plan: SCENEGOD master plan + 3-lane instructions + log scaffolding Co-Authored-By: Claude Fable 5 --- .gitignore | 7 ++ CLAUDE.md | 61 +++++++++++++ PLAN.md | 208 ++++++++++++++++++++++++++++++++++++++++++++ lanes/A-stage.md | 100 +++++++++++++++++++++ lanes/B-timeline.md | 81 +++++++++++++++++ lanes/C-server.md | 89 +++++++++++++++++++ logs/laneA.md | 11 +++ logs/laneB.md | 11 +++ logs/laneC.md | 11 +++ 9 files changed, 579 insertions(+) create mode 100644 .gitignore create mode 100644 CLAUDE.md create mode 100644 PLAN.md create mode 100644 lanes/A-stage.md create mode 100644 lanes/B-timeline.md create mode 100644 lanes/C-server.md create mode 100644 logs/laneA.md create mode 100644 logs/laneB.md create mode 100644 logs/laneC.md diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..472d5f3 --- /dev/null +++ b/.gitignore @@ -0,0 +1,7 @@ +.env +assets/ +scenes/ +renders/ +__pycache__/ +*.pyc +.DS_Store diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..656ae4d --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,61 @@ +# SCENEGOD — machinima director's sandbox (iClone/SFM UX, GODVERSE stack) + +Drop characters + animation clips + image backdrops on a 3D stage, keyframe +transforms/cameras/lights on a timeline, render mp4. Web app: FastAPI + +three.js, same conventions as MESHGOD (`~/Documents/MESHGOD`). + +## You are one of THREE parallel lane agents +1. **Read, in order:** this file → `PLAN.md` → your lane file in `lanes/` → + your log in `logs/`. Your lane file is your source of truth — the + orchestrator (Fable) updates it between your sessions. Re-read it at the + start of EVERY session; instructions may have changed since your last one. +2. **Only touch files your lane owns** (ownership map in PLAN.md §3). If you + need a change in another lane's file, write the request in your log under + BLOCKED/REQUESTS — do not edit it yourself. +3. **Log protocol** — append (never rewrite) to `logs/lane.md` at the end + of every session, and mid-session when you make a design decision: + + ``` + ## 2026-07-18 session N + DONE: what shipped, with file:line pointers + DECISIONS: choices made + why (schema tweaks, deps, algorithms) + BLOCKED/REQUESTS: what you need from another lane or the orchestrator + NEXT: what you'd do next session + ``` +4. **Commit early, commit often**, on `main`, prefixed `[laneA]`/`[laneB]`/ + `[laneC]`. `git pull --rebase` before every commit — other lanes commit to + the same branch. File ownership makes conflicts near-zero; if you hit one + anyway, resolve only your own files and log it. +5. **Don't exceed your lane's current milestone.** Milestones and sync points + are in PLAN.md §5. Finishing early = polish + tests + log, not starting + the next milestone uninvited. + +## Stack rules +- Python 3.11 stdlib + `fastapi` + `uvicorn` only (requirements.txt). No + other server deps without logging a REQUEST first. +- Frontend: vanilla ES modules + three.js via importmap (copy the pattern + from `/Users/johnking/Documents/MESHGOD/meshgod/web/rigroom.html`). No + bundler, no npm, no framework. +- Server: `uvicorn scenegod.server:app --port 8020`, binds 127.0.0.1 by + default (local tool on ultra; VPS deploy is a later problem). +- Secrets/env in `.env` (gitignored). Never commit tokens. +- Reference code you may READ (never edit): the MESHGOD repo, especially + `meshgod/web/rigroom.html` (retarget bake), `meshgod/library.py` (folder + scanning), `meshgod/ops.py` (MODELBEAST submit contract), + `scripts/glb2fbx.py` (headless Blender pattern). + +## Gotchas inherited from MESHGOD (learned the hard way — do not relearn) +- trimesh/some GLBs omit `metallicFactor` → glTF default 1.0 → renders black. + Viewer must use an IBL environment (`RoomEnvironment` + PMREM), not just + lights. +- Serve HTML **no-cache** — a stale cache once hid UI buttons for a day. +- Mixamo rig quirks (all already solved in rigroom's `bakeRetarget` — port, + don't reinvent): namespaces differ per download (`mixamorig4Hips`), FBX + files carry duplicate hierarchies, quaternion sign continuity must be + enforced, Blender-exported GLB rest poses differ from FBX rests → retarget + = world-space rotation-delta bake at 30fps, never raw track copy. +- Rigged GLBs must NEVER be sent through any Blender join/decimate "finish" + path — it spawns phantom joint meshes and mauls the armature. +- Path traversal: every endpoint that takes a path must resolve + verify it + stays inside its root (ship-check rule). +- Verbose commands (installs, test runs, ffmpeg): pipe through `| lm -l 2`. diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 0000000..a105c98 --- /dev/null +++ b/PLAN.md @@ -0,0 +1,208 @@ +# SCENEGOD master plan + +Owner: John. Orchestrator: Fable (reviews `logs/`, updates `lanes/`). +Executors: three Opus 4.8 lane agents working in parallel on `main`. + +## 1. What we're building + +A browser page where John can: +- drag characters (rigged GLB/FBX), props (MeshGod gallery GLBs), and image + backdrops onto a lit 3D stage; +- drop Mixamo-style animation clips onto characters (client-side retarget, + ported from MESHGOD rigroom); +- keyframe entity transforms, camera moves + cuts, and lighting on a + timeline with a scrubber; +- save/load scenes as JSON; +- render the timeline to mp4 (draft: MediaRecorder webm; final: per-frame + PNG → server ffmpeg); +- later: audio tracks, MODELBEAST tts/voice-clone/lipsync, viseme lip sync + for blendshape characters, and a headless-Blender limb-graft script. + +Non-goals (v1): multiplayer, physics, in-browser modeling, VPS deploy, +game-engine export. It renders video, it is not a game engine. + +## 2. Repo layout (create as you go — no empty scaffolding) + +``` +scenegod/ + server.py # FastAPI app (Lane C) + web/ + index.html # page skeleton + importmap + mount divs (Lane A) + style.css # (Lane A; B may append a clearly-marked timeline block) + room3d.js # loaders + retarget, ported from rigroom (Lane A) + stage.js # Stage class: scene/entities/gizmos (Lane A) + dock.js # asset dock UI (Lane A) + timeline.js # Timeline class: clock, tracks, evaluation (Lane B) + tlui.js # timeline DOM/canvas UI (Lane B) + render.js # draft+final render client (Lane C) +scripts/ + graft_limb.py # headless Blender limb graft (Lane C, M4) + test_server.py # endpoint smoke tests (Lane C) +requirements.txt # fastapi, uvicorn (Lane C) +lanes/ logs/ # orchestrator + agents +``` + +## 3. File ownership (hard rule) + +| Lane | Owns | May read everything, edits ONLY its own files | +|---|---|---| +| A — Stage | `web/index.html`, `web/style.css`, `web/room3d.js`, `web/stage.js`, `web/dock.js` | | +| B — Timeline | `web/timeline.js`, `web/tlui.js` | appends to `style.css` only inside `/* === LANE B === */` block | +| C — Server | `scenegod/server.py`, `web/render.js`, `scripts/*`, `requirements.txt`, `.gitignore` | | + +`PLAN.md` + `lanes/*` = orchestrator only. `logs/lane.md` = its agent only. +Schema changes (§4): propose in your log, orchestrator amends PLAN.md, then +everyone follows the new version. + +## 4. Contracts (the reason three lanes can run in parallel) + +### 4.1 Scene JSON — schema v1 + +```jsonc +{ + "version": 1, + "name": "street-scene", + "fps": 30, + "duration": 20.0, // seconds + "entities": [ + { + "id": "e1", // unique, stable + "kind": "character", // character|prop|backdrop|camera|light + "label": "lady", + "source": { "type": "assets", // assets|upload (upload = client-side only, warn on save) + "path": "characters/pack01/lady.glb" }, + "params": {}, // kind-specific, see below + "transform": { "pos": [0,0,0], "rot": [0,0,0], "scale": 1 }, // rest pose (used when no keys) + "tracks": { + "transform": [ // sorted by t; euler rad; ease: linear|in|out|inout|step + { "t": 0, "pos": [0,0,0], "rot": [0,0,0], "scale": 1, "ease": "linear" } + ], + "params": [ // keyable scalars/colors, e.g. camera fov, light intensity + { "t": 0, "key": "fov", "value": 45, "ease": "inout" } + ], + "clips": [ // characters only; non-overlapping except fade regions + { "path": "animations/pack01/walk.fbx", "clipIndex": 0, + "start": 1.0, // timeline seconds where block begins + "in": 0.0, "out": 2.4, // trim within source clip + "loop": 2, // repeat count (1 = once) + "fade": 0.25 } // crossfade seconds into NEXT block + ] + } + } + ], + "cameraCuts": [ { "t": 0, "camera": "e7" } ], // active camera entity id from t onward + "audio": [ { "path": "audio/vo1.wav", "start": 0.0, "gain": 1.0 } ] // M4 +} +``` + +`params` by kind — `camera`: `{fov}`; `light`: `{type:"key"|"ambient", color:"#fff", +intensity, castShadow}`; `backdrop`: `{mode:"plane"|"corner"|"dome", image:"backdrops/street.jpg", +width}`. Exactly one `light` entity of type `ambient` per scene (hemisphere); +`key` lights are directional, multiple allowed. + +### 4.2 JS module contracts + +`room3d.js` (Lane A) exports: +```js +parseAny(arrayBuffer, filename) -> Promise<{root, anims}> // glb|fbx|obj|bvh +canon(name) -> string +boneMap(root) -> {canonName: Bone} +captureRest(root) -> {rest: Map, order: Object3D[]} +bakeRetarget(clip, srcRoot, srcRest, tgtRoot, tgtRest) -> AnimationClip // parameterized target (rigroom closes over a global — fix that in the port) +disposeRoot(root); stats(root) -> {tris, meshes, bones} +``` + +`stage.js` (Lane A) exports `class Stage`: +```js +new Stage(viewportEl) // renderer, scene, IBL env, grid, orbit "director cam" +async addEntity(desc) -> entity // desc = scene-JSON entity; loads asset, builds object +removeEntity(id); getEntity(id); entities() -> entity[] +select(id|null); onSelect(cb); onChange(cb) // gizmo edits fire onChange(entity) +entityTransform(id) -> {pos,rot,scale} // current gizmo state (for "capture keyframe") +setTransform(id, {pos,rot,scale}) // timeline drives this every tick +setParam(id, key, value) // fov, light color/intensity, ... +prepareClip(id, path, clipIndex) -> Promise // fetch+parse+retarget, CACHED +entityMixer(id) -> AnimationMixer +setActiveCamera(id|null) // null = director orbit cam +renderActiveCamera(canvasOrNull) // PiP preview / final-render target +captureState() -> sceneJsonEntities; async applyState(sceneJson) +``` + +`timeline.js` (Lane B) exports `class Timeline`: +```js +new Timeline(stage) // talks to Stage ONLY via the API above +load(sceneJson); toJSON() -> sceneJson // owns duration/fps/tracks/cameraCuts +time; playing; play(); pause(); seek(t); onTick(cb(t)) +evaluate(t) // interpolate transform+param keys → stage.setTransform/… + // activate clip blocks → mixer weights/time, crossfades + // camera cuts → stage.setActiveCamera +step(frame) // deterministic: seek(frame/fps)+evaluate — used by final render +``` +Until Lane A ships Stage (M1), Lane B develops against `stagestub.js` +(Lane B writes it, same API, logs to console) — delete at integration. + +### 4.3 HTTP endpoints (Lane C) + +``` +GET / stageroom page (no-cache) + all /web statics +GET /assets/tree {characters:[...], animations:[...], props:[...], backdrops:[...], audio:[...]} + scans SCENEGOD_ASSETS folder tree live (rigroom-library style: + dir+stem grouping, no index file); props may also merge the + MESHGOD gallery if MESHGOD_GALLERY_URL is set (later, M3) +GET /assets/file?path= streams one asset (path-traversal-guarded) +GET /scenes list; GET /scenes/{name} one +POST /scenes/{name} save scene JSON (validated: version, ids unique, sorted tracks) +POST /render/begin {fps,width,height,name} -> {renderId} +POST /render/{id}/frame/{n} raw PNG body +POST /render/{id}/end {audio?: [...]}: ffmpeg PNG seq (+audio M4) -> mp4 +GET /render/{id}/status ; GET /render/{id}/out.mp4 +``` +Env: `SCENEGOD_ASSETS` (default `./assets`), `SCENEGOD_SCENES` (default +`./scenes`), `SCENEGOD_PORT` (8020). John symlinks/rsyncs his existing +banks into `assets/` — the server never writes into it except `/render`. + +## 5. Milestones & sync points + +**M1 — walking skeleton** (all lanes, independent): +- A: index.html + Stage with IBL viewer, load a GLB character + an image + backdrop from the dock, gizmo move/rotate/scale, retarget port proven + (load fbx clip onto character, plays once via a temp button). +- B: Timeline UI against stagestub — tracks list, scrubber, add/drag/delete + transform keyframes, play/pause with interpolation visible in console. +- C: server serving the page + /assets/tree + /assets/file + /scenes CRUD, + with scripts/test_server.py passing. +- **SYNC 1** (orchestrator): wire B to real Stage, delete stub. Definition of + done: load scene JSON → character walks between two keyframed positions + in front of a street backdrop, scrubbed by the timeline. + +**M2 — direction**: A: multiple cameras + PiP + light entities + param +plumbing. B: clip blocks with trim/loop/crossfade, params track, camera-cut +track, snapping, undo (simple command stack). C: scene validation hardening, +asset thumbnails endpoint. **SYNC 2**: the "lady + man + street + 2 cameras + +sunset" demo scene plays end to end. + +**M3 — render**: C: render session endpoints + ffmpeg + render.js client +(draft MediaRecorder + final frame-step using `timeline.step`). A/B: whatever +SYNC 2 revealed + polish pass from orchestrator notes. **SYNC 3**: 10-second +1080p mp4 rendered deterministically. + +**M4 — sound & flesh**: C: audio in scene JSON + ffmpeg mux + MODELBEAST +tts/voice-clone submit (mb contract from meshgod/ops.py, token from +`~/Documents/backnforth/.env`, NEVER logged) + `scripts/graft_limb.py`. +B: audio tracks on the timeline UI, Web Audio sync. A: viseme lane — detect +morph targets, Rhubarb-JSON → morph keyframes, greyed out when absent. + +## 6. How John launches the lanes + +Three Claude Code sessions (Opus 4.8), cwd `~/Documents/SCENEGOD`, one prompt +each — see README-LANES section at the bottom of each lane file, or just: + +> You are Lane A (or B / C). Read CLAUDE.md, PLAN.md, lanes/A-stage.md and +> logs/laneA.md, then continue the work from where the log leaves off. Obey +> the file-ownership map and the log protocol. Stop at your current +> milestone boundary. + +Orchestrator loop (Fable, this session or any MESHGOD/SCENEGOD session): +read `logs/*.md` → answer BLOCKED/REQUESTS by editing `lanes/*.md` (append a +dated **ORCHESTRATOR UPDATE** section) → run the SYNC integration when all +three logs report their milestone DONE. diff --git a/lanes/A-stage.md b/lanes/A-stage.md new file mode 100644 index 0000000..f8bcb68 --- /dev/null +++ b/lanes/A-stage.md @@ -0,0 +1,100 @@ +# Lane A — Stage: viewport, assets on stage, retarget port + +You own: `scenegod/web/index.html`, `style.css`, `room3d.js`, `stage.js`, `dock.js`. +Milestone: **M1** (do not start M2 items until the orchestrator flips this line). + +## A1. index.html skeleton (first commit) +- Layout: full-viewport CSS grid — left dock `#dock` (280px), center `#view`, + bottom `#timeline` (260px, empty div — Lane B mounts into it), right + `#inspector` (260px). Dark UI, system font; steal the visual language of + MESHGOD's rigroom (read `/Users/johnking/Documents/MESHGOD/meshgod/web/rigroom.html`). +- Importmap: copy rigroom's exactly (same three.js version + addons paths) so + behavior matches the code you're porting. +- Script entry: `