[sync4] applyState camera/light hotfix, M4 verified (audio mux + graft), viseme/audio-UI round unlocked
- stage.js: applyState no longer skips camera/light entities (source 'upload' but no bytes needed) — loaded scenes were silently losing the directing rig; PLAN schema adds source.type 'none' for the clean fix - lane files: A session-3 + C M4 accepted; A→absorb hotfix + viseme lane (M4-A), B→audio rows + Web Audio sync + morphs track (M4-B), C complete Verified: 6/6 entities restored on scene load, PiP 178x99 capped+visible, lady 26/26 textured, audio-muxed render h264+aac 8.000s both streams, graft_limb --selftest green under Blender 5.0.1, node+server suites green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
04e15d3e39
commit
a15612eed0
4
PLAN.md
4
PLAN.md
@ -69,8 +69,8 @@ everyone follows the new version.
|
|||||||
"id": "e1", // unique, stable
|
"id": "e1", // unique, stable
|
||||||
"kind": "character", // character|prop|backdrop|camera|light
|
"kind": "character", // character|prop|backdrop|camera|light
|
||||||
"label": "lady",
|
"label": "lady",
|
||||||
"source": { "type": "assets", // assets|upload (upload = client-side only, warn on save)
|
"source": { "type": "assets", // assets|upload|none (upload = client-side only, warn on
|
||||||
"path": "characters/pack01/lady.glb" },
|
"path": "characters/pack01/lady.glb" }, // save; none = needs no bytes: camera/light)
|
||||||
"params": {}, // kind-specific, see below
|
"params": {}, // kind-specific, see below
|
||||||
"transform": { "pos": [0,0,0], "rot": [0,0,0], "scale": 1 }, // rest pose (used when no keys)
|
"transform": { "pos": [0,0,0], "rot": [0,0,0], "scale": 1 }, // rest pose (used when no keys)
|
||||||
"tracks": {
|
"tracks": {
|
||||||
|
|||||||
@ -137,3 +137,24 @@ M3-polish list (order):
|
|||||||
whether it's the asset or a loader/material issue (compare in rigroom).
|
whether it's the asset or a loader/material issue (compare in rigroom).
|
||||||
3. Keep `renderActiveCamera`/`pause`/`resume` stable — the finalRender
|
3. Keep `renderActiveCamera`/`pause`/`resume` stable — the finalRender
|
||||||
client drives them now (SYNC 3).
|
client drives them now (SYNC 3).
|
||||||
|
|
||||||
|
### 2026-07-18 — session 3 ACCEPTED (PiP + spec-gloss both verified live).
|
||||||
|
**One SYNC 4 regression found & hotfixed by orchestrator, then M4-A unlocked.**
|
||||||
|
1. **applyState dropped cameras/lights** on scene load: they carry
|
||||||
|
`source:{type:'upload'}` and applyState skips all upload entities, so a
|
||||||
|
loaded scene silently lost its whole directing rig (verified: 3 of 6
|
||||||
|
entities landed; PiP dead). Hotfix in your stage.js (kind !== camera/light
|
||||||
|
never skipped) — absorb it, then do the clean fix: schema now has
|
||||||
|
`source:{type:'none'}` (PLAN §4.1) — emit it for camera/light in
|
||||||
|
`addEntity` defaults + `captureState`, treat 'none' as always-rebuild in
|
||||||
|
applyState. Keep tolerating legacy 'upload' cams (saved scenes exist).
|
||||||
|
2. Your spec-gloss shim + PiP cap verified: lady 26/26 textured on stage and
|
||||||
|
in-render; PiP 178×99 capped 16:9, shows on cut. Nice diagnosis on the
|
||||||
|
CSS aspect-ratio trap and the CC-export extension.
|
||||||
|
3. **M4-A — the viseme lane** (last A item in PLAN §5): on character load,
|
||||||
|
detect morph-target dictionaries (`mesh.morphTargetDictionary`); expose
|
||||||
|
`stage.visemeTargets(id)` → canonical viseme names found (A/E/I/O/U/MBP…
|
||||||
|
match loosely, e.g. CC/ARKit names). Inspector shows a 🗣 badge when
|
||||||
|
present, greyed otherwise. Add `stage.setMorph(id, name, weight)` for the
|
||||||
|
timeline to drive. Rhubarb-JSON → keyframes lands on Lane B's side; your
|
||||||
|
deliverable is detection + the setMorph hook + the badge.
|
||||||
|
|||||||
@ -120,3 +120,20 @@ frame-exact. Session 3 items:
|
|||||||
`load()` fires onLoad). Add a `setDuration(x)` mutator that clamps
|
`load()` fires onLoad). Add a `setDuration(x)` mutator that clamps
|
||||||
existing keys' visibility and notifies the UI; wire the dur field to it.
|
existing keys' visibility and notifies the UI; wire the dur field to it.
|
||||||
3. Your listed polish: snapping increment config, keyable-param add UI.
|
3. Your listed polish: snapping increment config, keyable-param add UI.
|
||||||
|
|
||||||
|
### 2026-07-18 — session 3 ACCEPTED. **M4-B unlocked: audio on the timeline.**
|
||||||
|
Good catch fixing the stub's removeEntity-onChange payload — that's the
|
||||||
|
honest-stub lesson applied. All three items verified (setDuration bar
|
||||||
|
refresh + snap select seen live; tests green). M4-B scope (PLAN §5):
|
||||||
|
1. Audio rows in tlui from `scene.audio[]`: one lane per clip, draggable
|
||||||
|
offset (`start`), gain in the inspector-side of the row, add via a 🎵
|
||||||
|
button reading `/assets/tree` audio category; delete.
|
||||||
|
2. Web Audio playback synced to the master clock: on `play()` schedule
|
||||||
|
`AudioBufferSourceNode`s at `start - time` offsets; on `seek/pause` stop
|
||||||
|
+ reschedule. `step()` NEVER touches audio (deterministic render path —
|
||||||
|
the server muxes from scene.audio, already working end-to-end).
|
||||||
|
3. Viseme keyframes (pairs with Lane A's M4-A): a `morphs` track kind —
|
||||||
|
`{t, key: visemeName, value: weight}` evaluated via
|
||||||
|
`stage.setMorph(id, name, weight)` with lerp between keys; plus
|
||||||
|
`importRhubarb(id, rhubarbJson, t0)` mapping mouthShape cues → morph keys.
|
||||||
|
Guard: only offer when `stage.visemeTargets(id).length > 0`.
|
||||||
|
|||||||
@ -124,3 +124,14 @@ verified — no changes needed. Proceed with M4 in this order:
|
|||||||
One render-quality note for your backlog, not urgent: draft webm via
|
One render-quality note for your backlog, not urgent: draft webm via
|
||||||
MediaRecorder is untested in the embedded preview (rAF suspension) — verify
|
MediaRecorder is untested in the embedded preview (rAF suspension) — verify
|
||||||
it once in a real Chrome window before calling it done.
|
it once in a real Chrome window before calling it done.
|
||||||
|
|
||||||
|
### 2026-07-18 — M4 ACCEPTED. Lane C scope (M1–M4) is COMPLETE.
|
||||||
|
Orchestrator verified in the integrated app: audio-muxed finalRender →
|
||||||
|
h264+aac mp4, both streams exactly 8.000s; graft_limb `--selftest` green
|
||||||
|
under Blender 5.0.1 (both refusal + graft paths); 8 server test groups pass.
|
||||||
|
Your blender-path correction noted (it's `/Applications/Blender.app/...`,
|
||||||
|
not on PATH — my error). Standing by is fine; remaining work is A/B UI
|
||||||
|
(viseme lane, audio rows). One item if you want it: add a `rhubarb` check +
|
||||||
|
`/rhubarb?path=` endpoint (subprocess → JSON) IF rhubarb is installed —
|
||||||
|
503-with-message otherwise, same pattern as ffmpeg. Log a REQUEST first if
|
||||||
|
it needs more than that.
|
||||||
|
|||||||
@ -377,7 +377,10 @@ export class Stage {
|
|||||||
async applyState(sceneJson){
|
async applyState(sceneJson){
|
||||||
for(const en of this.entities()) this.removeEntity(en.id);
|
for(const en of this.entities()) this.removeEntity(en.id);
|
||||||
for(const d of (sceneJson.entities || [])){
|
for(const d of (sceneJson.entities || [])){
|
||||||
if(d.source && d.source.type === 'upload'){ console.warn('skipping upload entity (no bytes):', d.id); continue; }
|
// SYNC4: cameras/lights carry source:'upload' but need no bytes — never skip them,
|
||||||
|
// or scene loads silently lose the whole directing rig (cams + sun).
|
||||||
|
if(d.source && d.source.type === 'upload' && d.kind !== 'camera' && d.kind !== 'light'){
|
||||||
|
console.warn('skipping upload entity (no bytes):', d.id); continue; }
|
||||||
try { await this.addEntity(d); } catch(err){ console.error('addEntity failed', d.id, err); }
|
try { await this.addEntity(d); } catch(err){ console.error('addEntity failed', d.id, err); }
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user