# SCENEGOD master plan

Owner: John. Orchestrator: Fable (reviews `logs/`, updates `lanes/`).
Executors: three Opus 4.8 lane agents working in parallel on `main`.

## 1. What we're building

A browser page where John can:
- drag characters (rigged GLB/FBX), props (MeshGod gallery GLBs), and image
  backdrops onto a lit 3D stage;
- drop Mixamo-style animation clips onto characters (client-side retarget,
  ported from MESHGOD rigroom);
- keyframe entity transforms, camera moves + cuts, and lighting on a
  timeline with a scrubber;
- save/load scenes as JSON;
- render the timeline to mp4 (draft: MediaRecorder webm; final: per-frame
  PNG → server ffmpeg);
- later: audio tracks, MODELBEAST tts/voice-clone/lipsync, viseme lip sync
  for blendshape characters, and a headless-Blender limb-graft script.

Non-goals (v1): multiplayer, physics, in-browser modeling, VPS deploy,
game-engine export. It renders video, it is not a game engine.

## 2. Repo layout (create as you go — no empty scaffolding)

```
scenegod/
  server.py            # FastAPI app (Lane C)
  web/
    index.html         # page skeleton + importmap + mount divs (Lane A)
    style.css          # (Lane A; B may append a clearly-marked timeline block)
    room3d.js          # loaders + retarget, ported from rigroom (Lane A)
    stage.js           # Stage class: scene/entities/gizmos (Lane A)
    dock.js            # asset dock UI (Lane A)
    timeline.js        # Timeline class: clock, tracks, evaluation (Lane B)
    tlui.js            # timeline DOM/canvas UI (Lane B)
    render.js          # draft+final render client (Lane C)
scripts/
  graft_limb.py        # headless Blender limb graft (Lane C, M4)
  test_server.py       # endpoint smoke tests (Lane C)
requirements.txt       # fastapi, uvicorn (Lane C)
lanes/  logs/          # orchestrator + agents
```

## 3. File ownership (hard rule)

| Lane | Owns | May read everything, edits ONLY its own files |
|---|---|---|
| A — Stage | `web/index.html`, `web/style.css`, `web/room3d.js`, `web/stage.js`, `web/dock.js` | |
| B — Timeline | `web/timeline.js`, `web/tlui.js` | appends to `style.css` only inside `/* === LANE B === */` block |
| C — Server | `scenegod/server.py`, `web/render.js`, `scripts/*`, `requirements.txt`, `.gitignore` | |

`PLAN.md` + `lanes/*` = orchestrator only. `logs/lane<X>.md` = its agent only.
Schema changes (§4): propose in your log, orchestrator amends PLAN.md, then
everyone follows the new version.

## 4. Contracts (the reason three lanes can run in parallel)

### 4.1 Scene JSON — schema v1

```jsonc
{
  "version": 1,
  "name": "street-scene",
  "fps": 30,
  "duration": 20.0,                       // seconds
  "entities": [
    {
      "id": "e1",                          // unique, stable
      "kind": "character",                 // character|prop|backdrop|camera|light
      "label": "lady",
      "source": { "type": "assets",        // assets|upload|none (upload = client-side only, warn on
                  "path": "characters/pack01/lady.glb" },  //   save; none = needs no bytes: camera/light)
      "params": {},                        // kind-specific, see below
      "transform": { "pos": [0,0,0], "rot": [0,0,0], "scale": 1 },  // rest pose (used when no keys)
      "tracks": {
        "transform": [                     // sorted by t; euler rad; ease: linear|in|out|inout|step
          { "t": 0, "pos": [0,0,0], "rot": [0,0,0], "scale": 1, "ease": "linear" }
        ],
        "params": [                        // keyable scalars/colors, e.g. camera fov, light intensity
          { "t": 0, "key": "fov", "value": 45, "ease": "inout" }
        ],
        "clips": [                         // characters only; non-overlapping except fade regions
          { "path": "animations/pack01/walk.fbx", "clipIndex": 0,
            "start": 1.0,                  // timeline seconds where block begins
            "in": 0.0, "out": 2.4,         // trim within source clip
            "loop": 2,                     // repeat count (1 = once)
            "fade": 0.25 }                 // crossfade seconds into NEXT block
        ]
      }
    }
  ],
  "cameraCuts": [ { "t": 0, "camera": "e7" } ],   // active camera entity id from t onward
  "audio": [ { "path": "audio/vo1.wav", "start": 0.0, "gain": 1.0 } ]  // M4
}
```

`params` by kind — `camera`: `{fov}`; `light`: `{type:"key"|"ambient", color:"#fff",
intensity, castShadow}`; `backdrop`: `{mode:"plane"|"corner"|"dome", image:"backdrops/street.jpg",
width}`. Exactly one `light` entity of type `ambient` per scene (hemisphere);
`key` lights are directional, multiple allowed.

### 4.2 JS module contracts

`room3d.js` (Lane A) exports:
```js
parseAny(arrayBuffer, filename) -> Promise<{root, anims}>   // glb|fbx|obj|bvh
canon(name) -> string
boneMap(root) -> {canonName: Bone}
captureRest(root) -> {rest: Map<Object3D,{wq,parent}>, order: Object3D[]}
bakeRetarget(clip, srcRoot, srcRest, tgtRoot, tgtRest) -> AnimationClip  // parameterized target (rigroom closes over a global — fix that in the port)
disposeRoot(root); stats(root) -> {tris, meshes, bones}
```

`stage.js` (Lane A) exports `class Stage`:
```js
new Stage(viewportEl)                  // renderer, scene, IBL env, grid, orbit "director cam"
async addEntity(desc) -> entity        // desc = scene-JSON entity; loads asset, builds object
removeEntity(id); getEntity(id); entities() -> entity[]
select(id|null); onSelect(cb); onChange(cb)          // gizmo edits fire onChange(entity)
entityTransform(id) -> {pos,rot,scale}               // current gizmo state (for "capture keyframe")
setTransform(id, {pos,rot,scale})                    // timeline drives this every tick
setParam(id, key, value)                             // fov, light color/intensity, ...
prepareClip(id, path, clipIndex) -> Promise<AnimationClip>   // fetch+parse+retarget, CACHED
entityMixer(id) -> AnimationMixer
setActiveCamera(id|null)               // null = director orbit cam
renderActiveCamera(canvasOrNull)       // PiP preview / final-render target
captureState() -> sceneJsonEntities;  async applyState(sceneJson)
```

`timeline.js` (Lane B) exports `class Timeline`:
```js
new Timeline(stage)                    // talks to Stage ONLY via the API above
load(sceneJson); toJSON() -> sceneJson // owns duration/fps/tracks/cameraCuts
time; playing; play(); pause(); seek(t); onTick(cb(t))
evaluate(t)                            // interpolate transform+param keys → stage.setTransform/…
                                       //   activate clip blocks → mixer weights/time, crossfades
                                       //   camera cuts → stage.setActiveCamera
step(frame)                            // deterministic: seek(frame/fps)+evaluate — used by final render
```
Until Lane A ships Stage (M1), Lane B develops against `stagestub.js`
(Lane B writes it, same API, logs to console) — delete at integration.

### 4.3 HTTP endpoints (Lane C)

```
GET  /                      stageroom page (no-cache) + all /web statics
GET  /assets/tree           {characters:[...], animations:[...], props:[...], backdrops:[...], audio:[...]}
                            scans SCENEGOD_ASSETS folder tree live (rigroom-library style:
                            dir+stem grouping, no index file); props may also merge the
                            MESHGOD gallery if MESHGOD_GALLERY_URL is set (later, M3)
GET  /assets/file?path=     streams one asset (path-traversal-guarded)
GET  /scenes                list;  GET /scenes/{name}  one
POST /scenes/{name}         save scene JSON (validated: version, ids unique, sorted tracks)
POST /render/begin          {fps,width,height,name} -> {renderId}
POST /render/{id}/frame/{n} raw PNG body
POST /render/{id}/end       {audio?: [...]}: ffmpeg PNG seq (+audio M4) -> mp4
GET  /render/{id}/status    ; GET /render/{id}/out.mp4
```
Env: `SCENEGOD_ASSETS` (default `./assets`), `SCENEGOD_SCENES` (default
`./scenes`), `SCENEGOD_PORT` (8020). John symlinks/rsyncs his existing
banks into `assets/` — the server never writes into it except `/render`.

## 5. Milestones & sync points

**M1 — walking skeleton** (all lanes, independent):
- A: index.html + Stage with IBL viewer, load a GLB character + an image
  backdrop from the dock, gizmo move/rotate/scale, retarget port proven
  (load fbx clip onto character, plays once via a temp button).
- B: Timeline UI against stagestub — tracks list, scrubber, add/drag/delete
  transform keyframes, play/pause with interpolation visible in console.
- C: server serving the page + /assets/tree + /assets/file + /scenes CRUD,
  with scripts/test_server.py passing.
- **SYNC 1** (orchestrator): wire B to real Stage, delete stub. Definition of
  done: load scene JSON → character walks between two keyframed positions
  in front of a street backdrop, scrubbed by the timeline.

**M2 — direction**: A: multiple cameras + PiP + light entities + param
plumbing. B: clip blocks with trim/loop/crossfade, params track, camera-cut
track, snapping, undo (simple command stack). C: scene validation hardening,
asset thumbnails endpoint. **SYNC 2**: the "lady + man + street + 2 cameras +
sunset" demo scene plays end to end.

**M3 — render**: C: render session endpoints + ffmpeg + render.js client
(draft MediaRecorder + final frame-step using `timeline.step`). A/B: whatever
SYNC 2 revealed + polish pass from orchestrator notes. **SYNC 3**: 10-second
1080p mp4 rendered deterministically.

**M4 — sound & flesh**: C: audio in scene JSON + ffmpeg mux + MODELBEAST
tts/voice-clone submit (mb contract from meshgod/ops.py, token from
`~/Documents/backnforth/.env`, NEVER logged) + `scripts/graft_limb.py`.
B: audio tracks on the timeline UI, Web Audio sync. A: viseme lane — detect
morph targets, Rhubarb-JSON → morph keyframes, greyed out when absent.

**M5 — DIRECTOR MODE (ease of use)**: classic cinema grammar as one-click
presets, LLM strictly optional on top.
- Shared vocabulary in `web/grammar.js` (Lane A owns, data only): shot sizes
  ECU/CU/MCU/MS/MWS/WS/EWS (target vertical span as a fraction of subject
  height + frame-center height), angles (eye/low/high/dutch/OTS-L/OTS-R),
  focal lengths 18/24/35/50/85mm (vertical fov = 2·atan(12/f), full-frame),
  lighting presets (day, golden-hour, night, noir, horror, neon, overcast),
  blocking marks (center, two-shot L/R, walk-to-mark, face-camera, face-other).
- Lane A `web/presets.js`: `frameSubject(camId, subjectId, shot, angle,
  focalmm)` — deterministic solver from the subject's bbox (no LLM), light/
  blocking preset appliers, and the DIRECT panel UI (shot buttons on the
  selected subject, light preset buttons, mark buttons).
- Lane B: preset actions land as normal keys/cuts AT THE PLAYHEAD through
  existing mutators (undo-able); "walk to mark" = 2 transform keys.
- Lane C: `POST /director` — optional NL box ("medium two-shot, golden hour,
  lady walks to camera") → tailnet Ollama (OpenAI-compat, env
  `SCENEGOD_LLM_URL` + `SCENEGOD_LLM_MODEL`, 503 when unset) → strict JSON
  ops validated server-side → client applies via the SAME preset functions.
  Buttons must never depend on the LLM lane.

## 6. How John launches the lanes

Three Claude Code sessions (Opus 4.8), cwd `~/Documents/SCENEGOD`, one prompt
each — see README-LANES section at the bottom of each lane file, or just:

> You are Lane A (or B / C). Read CLAUDE.md, PLAN.md, lanes/A-stage.md and
> logs/laneA.md, then continue the work from where the log leaves off. Obey
> the file-ownership map and the log protocol. Stop at your current
> milestone boundary.

Orchestrator loop (Fable, this session or any MESHGOD/SCENEGOD session):
read `logs/*.md` → answer BLOCKED/REQUESTS by editing `lanes/*.md` (append a
dated **ORCHESTRATOR UPDATE** section) → run the SYNC integration when all
three logs report their milestone DONE.