SCENEGOD/lanes/A-stage.md
type-two a15612eed0 [sync4] applyState camera/light hotfix, M4 verified (audio mux + graft), viseme/audio-UI round unlocked
- stage.js: applyState no longer skips camera/light entities (source
  'upload' but no bytes needed) — loaded scenes were silently losing the
  directing rig; PLAN schema adds source.type 'none' for the clean fix
- lane files: A session-3 + C M4 accepted; A→absorb hotfix + viseme lane
  (M4-A), B→audio rows + Web Audio sync + morphs track (M4-B), C complete

Verified: 6/6 entities restored on scene load, PiP 178x99 capped+visible,
lady 26/26 textured, audio-muxed render h264+aac 8.000s both streams,
graft_limb --selftest green under Blender 5.0.1, node+server suites green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 21:04:09 +10:00

9.9 KiB
Raw Blame History

Lane A — Stage: viewport, assets on stage, retarget port

You own: scenegod/web/index.html, style.css, room3d.js, stage.js, dock.js. Milestone: M1 (do not start M2 items until the orchestrator flips this line).

A1. index.html skeleton (first commit)

  • Layout: full-viewport CSS grid — left dock #dock (280px), center #view, bottom #timeline (260px, empty div — Lane B mounts into it), right #inspector (260px). Dark UI, system font; steal the visual language of MESHGOD's rigroom (read /Users/johnking/Documents/MESHGOD/meshgod/web/rigroom.html).
  • Importmap: copy rigroom's exactly (same three.js version + addons paths) so behavior matches the code you're porting.
  • Script entry: <script type="module" src="/web/main-stage.js">? No — keep entry inline in index.html: import {Stage} from './stage.js' etc., create the Stage, Dock, and (when it exists) new Timeline(stage) guarded by a dynamic import that tolerates timeline.js not existing yet: try { const {Timeline} = await import('./timeline.js'); ... } catch {}. That keeps lanes decoupled — your page must never be broken by B's absence.
  • Style: style.css; leave a /* === LANE B === */ marker at the bottom.

A2. room3d.js — port from rigroom.html (the crown jewels)

Source: /Users/johnking/Documents/MESHGOD/meshgod/web/rigroom.html lines ~125290 (loaders, canon/boneMap, captureRest, bakeRetarget, stats, disposeRoot). Port rules:

  • Make it a real ES module with the exports listed in PLAN §4.2. No DOM, no globals, no toast() — throw Errors, callers handle UI.
  • Parameterize the target: rigroom's bakeRetarget closes over the global char. New signature bakeRetarget(clip, srcRoot, srcRest, tgtRoot, tgtRest) -> AnimationClip. Keep the algorithm byte-for-byte otherwise: 30fps sampling, world-space rotation delta from source rest applied to target rest, parent-first world quat propagation, hips world-position track scaled by target/source hips height ratio, quaternion sign continuity (dot < 0 → negate), and the duplicate-FBX-hierarchy dedupe (bind only first bone per canon key).
  • Keep canon() exactly as-is (mixamorig\d*[:_]? strip) — it encodes real Mixamo quirks.
  • Self-check: room3d.js gets a tiny exported _selftest() that builds two 3-bone synthetic skeletons with different rests, bakes a 1s clip across, and asserts end-pose world quats match within 1e-3. Wire it to ?selftest=1 on the page; log the result.

A3. stage.js — the Stage class (PLAN §4.2 API)

  • Renderer setup: copy rigroom's (antialias, PMREM + RoomEnvironment IBL — metals gotcha), hemisphere + key directional light as default lights (replaced by scene light entities when present), circle floor + grid, OrbitControls director camera.
  • addEntity(desc):
    • character/prop: fetch /assets/file?path=, parseAny, normalize scale like rigroom (fit ~1.7m for characters; props keep native size but expose scale), ground to y=0, attach to a THREE.Group wrapper — ALL transforms (gizmo + timeline) act on the wrapper, never the loaded root. For characters: captureRest once at load, store on the entity; create AnimationMixer on the wrapper.
    • backdrop: params.modeplane: textured PlaneGeometry, height from image aspect × params.width (default 10m); corner: plane + floor plane sharing the image bottom third (cheap "L" street corner); dome: 40m sphere, side: BackSide, texture on inside. SRGBColorSpace on all image textures.
    • camera: PerspectiveCamera in a wrapper + a small frustum helper mesh so it's visible/selectable in the director view. params.fov.
    • light: key → DirectionalLight + helper; ambient → replaces the default hemisphere intensity. Adding the first key light dims defaults.
  • Selection: raycast on click → select(id), outline or bbox highlight, TransformControls gizmo (W/E/R = move/rotate/scale) on the wrapper; gizmo drag end → onChange(entity).
  • prepareClip(id, path, clipIndex): fetch+parse the clip source, cache {srcRoot, srcRest, anims} per path (Map), bake per (entity, path, index) and cache the baked clip. Baking is the expensive step — never re-bake on seek.
  • renderActiveCamera(canvas): render the active camera to a small PiP canvas overlay (M2) and, for Lane C's final render, to the main canvas at exact size. Keep director-orbit rendering the default main view.
  • captureState()/applyState(): round-trip with scene JSON entities. Assets with source.type:"upload" (drag-and-dropped local files) live only in memory — mark them in the dock and warn on save.

A4. dock.js

  • Tabs: Characters / Animations / Props / Backdrops from GET /assets/tree (Lane C; until it exists use a hardcoded mock=1 tree + local file-drop — drag any glb/fbx/image onto the page adds it as source.type:"upload").
  • Click or drag-onto-viewport → stage.addEntity. Animations are NOT added to the stage; dragging one onto a character in the viewport (raycast under drop point) or onto its inspector calls prepareClip + M1 temp "play once" button. In M2, B takes over clip placement (dock drag → timeline track).
  • Inspector panel: selected entity's label, kind, transform numbers (editable), params (fov slider for cameras, color+intensity for lights, backdrop mode), delete button, "⏺ key" button that emits window.dispatchEvent(new CustomEvent('scenegod:capturekey',{detail:{id}})) — Lane B listens; you don't call timeline code directly.

M1 acceptance (log it with evidence)

  • ?selftest=1 retarget check passes.
  • Load lady.glb (put test files in assets/ yourself — grab from MESHGOD library banks), drop street.jpg as plane backdrop, gizmo-move the lady, drop walk.fbx on her → she walks in place. Screenshot in log dir welcome (logs/shots/, yours to create).
  • Page loads clean with timeline.js absent AND with server absent (mock=1).

ORCHESTRATOR UPDATES

2026-07-18 — M1 ACCEPTED, SYNC 1 done. M2 is UNLOCKED.

M1 verified end-to-end in-browser by the orchestrator (character + backdrop + retargeted clip + timeline scrub, all through the real Stage). Answers + M2 order:

  1. /assets/tree item shape confirmed (from Lane C's server): {name, path, formats: ["fbx", ...], thumb: relpath|null} — rigroom-style, no files array. Drop your bare-string fallback. /web/* absolute imports are confirmed correct as served.
  2. Priority 1 — eventize clip drops. stage.setMixerAuto(false) is now active whenever the timeline mounts, so your drop-clip-to-preview path is dead in the integrated page. Replace it: on clip-drop onto a character, dispatch window CustomEvent 'scenegod:clipdrop' with {detail: {id, path, clipIndex}} (Lane B adds the block at the playhead). Keep bake-and-preview ONLY when no timeline is present.
  3. Priority 2 — make sure onChange fires for every entity added via applyState (SYNC finding: tlui's names column stayed empty after a programmatic scene load — fix on both sides; B is doing theirs).
  4. Then per plan: multiple cameras + PiP overlay (renderActiveCamera needs to work as the M3 final-render target — treat that contract as hard), backdrop rebuild-on-param-change, drag-from-dock onto viewport. Test assets now live in assets/{characters,animations,backdrops}/test/ (man.fbx, hiphop.fbx, carrying.fbx, street.jpg — gitignored, on ultra).

2026-07-18 — M2 ACCEPTED, SYNC 2 PASSED. M3-polish unlocked.

Your entity-mirror seam report was exactly right and well-diagnosed. Resolution: Lane B's Timeline now subscribes to stage.onChange and mirrors adds/removes into its model (orchestrator implemented, timeline.js _mirror) — your onChange-on-every-add fix is what made that possible; no dedicated entityadd/remove events needed. Verified live: 6 dock-added entities all keyframable, full sunset demo saved as scene sync2-sunset. M3-polish list (order):

  1. PiP is enormous — it covered ~70% of the viewport in the SYNC 2 screenshot. Cap it (~24% width, bottom-right corner, click to swap director/active-cam views or a size toggle).
  2. lady.glb (non-mixamo rig, assets/characters/test/lady.glb) renders flat white — her textures didn't come through the GLB load path. Check whether it's the asset or a loader/material issue (compare in rigroom).
  3. Keep renderActiveCamera/pause/resume stable — the finalRender client drives them now (SYNC 3).

2026-07-18 — session 3 ACCEPTED (PiP + spec-gloss both verified live).

One SYNC 4 regression found & hotfixed by orchestrator, then M4-A unlocked.

  1. applyState dropped cameras/lights on scene load: they carry source:{type:'upload'} and applyState skips all upload entities, so a loaded scene silently lost its whole directing rig (verified: 3 of 6 entities landed; PiP dead). Hotfix in your stage.js (kind !== camera/light never skipped) — absorb it, then do the clean fix: schema now has source:{type:'none'} (PLAN §4.1) — emit it for camera/light in addEntity defaults + captureState, treat 'none' as always-rebuild in applyState. Keep tolerating legacy 'upload' cams (saved scenes exist).
  2. Your spec-gloss shim + PiP cap verified: lady 26/26 textured on stage and in-render; PiP 178×99 capped 16:9, shows on cut. Nice diagnosis on the CSS aspect-ratio trap and the CC-export extension.
  3. M4-A — the viseme lane (last A item in PLAN §5): on character load, detect morph-target dictionaries (mesh.morphTargetDictionary); expose stage.visemeTargets(id) → canonical viseme names found (A/E/I/O/U/MBP… match loosely, e.g. CC/ARKit names). Inspector shows a 🗣 badge when present, greyed otherwise. Add stage.setMorph(id, name, weight) for the timeline to drive. Rhubarb-JSON → keyframes lands on Lane B's side; your deliverable is detection + the setMorph hook + the badge.