[sync4] applyState camera/light hotfix, M4 verified (audio mux + graft), viseme/audio-UI round unlocked

- stage.js: applyState no longer skips camera/light entities (source
  'upload' but no bytes needed) — loaded scenes were silently losing the
  directing rig; PLAN schema adds source.type 'none' for the clean fix
- lane files: A session-3 + C M4 accepted; A→absorb hotfix + viseme lane
  (M4-A), B→audio rows + Web Audio sync + morphs track (M4-B), C complete

Verified: 6/6 entities restored on scene load, PiP 178x99 capped+visible,
lady 26/26 textured, audio-muxed render h264+aac 8.000s both streams,
graft_limb --selftest green under Blender 5.0.1, node+server suites green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
type-two 2026-07-18 21:04:09 +10:00
parent 04e15d3e39
commit a15612eed0
5 changed files with 55 additions and 3 deletions

View File

@ -69,8 +69,8 @@ everyone follows the new version.
"id": "e1", // unique, stable
"kind": "character", // character|prop|backdrop|camera|light
"label": "lady",
"source": { "type": "assets", // assets|upload (upload = client-side only, warn on save)
"path": "characters/pack01/lady.glb" },
"source": { "type": "assets", // assets|upload|none (upload = client-side only, warn on
"path": "characters/pack01/lady.glb" }, // save; none = needs no bytes: camera/light)
"params": {}, // kind-specific, see below
"transform": { "pos": [0,0,0], "rot": [0,0,0], "scale": 1 }, // rest pose (used when no keys)
"tracks": {

View File

@ -137,3 +137,24 @@ M3-polish list (order):
whether it's the asset or a loader/material issue (compare in rigroom).
3. Keep `renderActiveCamera`/`pause`/`resume` stable — the finalRender
client drives them now (SYNC 3).
### 2026-07-18 — session 3 ACCEPTED (PiP + spec-gloss both verified live).
**One SYNC 4 regression found & hotfixed by orchestrator, then M4-A unlocked.**
1. **applyState dropped cameras/lights** on scene load: they carry
`source:{type:'upload'}` and applyState skips all upload entities, so a
loaded scene silently lost its whole directing rig (verified: 3 of 6
entities landed; PiP dead). Hotfix in your stage.js (kind !== camera/light
never skipped) — absorb it, then do the clean fix: schema now has
`source:{type:'none'}` (PLAN §4.1) — emit it for camera/light in
`addEntity` defaults + `captureState`, treat 'none' as always-rebuild in
applyState. Keep tolerating legacy 'upload' cams (saved scenes exist).
2. Your spec-gloss shim + PiP cap verified: lady 26/26 textured on stage and
in-render; PiP 178×99 capped 16:9, shows on cut. Nice diagnosis on the
CSS aspect-ratio trap and the CC-export extension.
3. **M4-A — the viseme lane** (last A item in PLAN §5): on character load,
detect morph-target dictionaries (`mesh.morphTargetDictionary`); expose
`stage.visemeTargets(id)` → canonical viseme names found (A/E/I/O/U/MBP…
match loosely, e.g. CC/ARKit names). Inspector shows a 🗣 badge when
present, greyed otherwise. Add `stage.setMorph(id, name, weight)` for the
timeline to drive. Rhubarb-JSON → keyframes lands on Lane B's side; your
deliverable is detection + the setMorph hook + the badge.

View File

@ -120,3 +120,20 @@ frame-exact. Session 3 items:
`load()` fires onLoad). Add a `setDuration(x)` mutator that clamps
existing keys' visibility and notifies the UI; wire the dur field to it.
3. Your listed polish: snapping increment config, keyable-param add UI.
### 2026-07-18 — session 3 ACCEPTED. **M4-B unlocked: audio on the timeline.**
Good catch fixing the stub's removeEntity-onChange payload — that's the
honest-stub lesson applied. All three items verified (setDuration bar
refresh + snap select seen live; tests green). M4-B scope (PLAN §5):
1. Audio rows in tlui from `scene.audio[]`: one lane per clip, draggable
offset (`start`), gain in the inspector-side of the row, add via a 🎵
button reading `/assets/tree` audio category; delete.
2. Web Audio playback synced to the master clock: on `play()` schedule
`AudioBufferSourceNode`s at `start - time` offsets; on `seek/pause` stop
+ reschedule. `step()` NEVER touches audio (deterministic render path —
the server muxes from scene.audio, already working end-to-end).
3. Viseme keyframes (pairs with Lane A's M4-A): a `morphs` track kind —
`{t, key: visemeName, value: weight}` evaluated via
`stage.setMorph(id, name, weight)` with lerp between keys; plus
`importRhubarb(id, rhubarbJson, t0)` mapping mouthShape cues → morph keys.
Guard: only offer when `stage.visemeTargets(id).length > 0`.

View File

@ -124,3 +124,14 @@ verified — no changes needed. Proceed with M4 in this order:
One render-quality note for your backlog, not urgent: draft webm via
MediaRecorder is untested in the embedded preview (rAF suspension) — verify
it once in a real Chrome window before calling it done.
### 2026-07-18 — M4 ACCEPTED. Lane C scope (M1M4) is COMPLETE.
Orchestrator verified in the integrated app: audio-muxed finalRender →
h264+aac mp4, both streams exactly 8.000s; graft_limb `--selftest` green
under Blender 5.0.1 (both refusal + graft paths); 8 server test groups pass.
Your blender-path correction noted (it's `/Applications/Blender.app/...`,
not on PATH — my error). Standing by is fine; remaining work is A/B UI
(viseme lane, audio rows). One item if you want it: add a `rhubarb` check +
`/rhubarb?path=` endpoint (subprocess → JSON) IF rhubarb is installed —
503-with-message otherwise, same pattern as ffmpeg. Log a REQUEST first if
it needs more than that.

View File

@ -377,7 +377,10 @@ export class Stage {
async applyState(sceneJson){
for(const en of this.entities()) this.removeEntity(en.id);
for(const d of (sceneJson.entities || [])){
if(d.source && d.source.type === 'upload'){ console.warn('skipping upload entity (no bytes):', d.id); continue; }
// SYNC4: cameras/lights carry source:'upload' but need no bytes — never skip them,
// or scene loads silently lose the whole directing rig (cams + sun).
if(d.source && d.source.type === 'upload' && d.kind !== 'camera' && d.kind !== 'light'){
console.warn('skipping upload entity (no bytes):', d.id); continue; }
try { await this.addEntity(d); } catch(err){ console.error('addEntity failed', d.id, err); }
}
}