[m5+m6] DIRECTOR MODE + FLOW VIDEO PLATES — all lanes + sync6 seam fixes
Lane A: grammar.js (shot/angle/focal/light/mark data), presets.js (bbox shot solver, light/mark appliers, DIRECT bar UI, NL box), stage video plates (VideoTexture plane/corner/dome, screen kind, syncVideos), dock video cards. Lane B: scenegod:direct → grouped-undo keys/cuts, 🗣 lip-sync from audio via /rhubarb, Cmd+wheel zoom / wheel scroll, audio trim, video clock-follow via nullable entityVideo seam. Lane C: POST /director (LLM proxy, server-side op validation vs vocab+scene, mock-tested), render.js awaits stage.syncVideos. Sync6 (orchestrator): VIDEO_EXTS in server tree/media types (video primary, img poster as thumb), stage.entityVideo accessor, syncVideos forces videoTex.needsUpdate (rVFC never fires in hidden tabs → black plates). Verified: 9 server groups + timeline + presets suites green; browser SYNC: one-click MCU framing lands camera+fov keys+cut (single undo), golden_hour relight keys, plate scrub-follow frame-exact, finalRender mp4 shows the Flow golden-hour street plate behind an animated character. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
type-two
parent
86b76f6bdc
commit
20af1b20c2
@ -206,3 +206,14 @@ as keys via B (scrub proves it) and is undo-able.
|
||||
### 2026-07-19 — M6 FLOW VIDEO PLATES queued behind M5 (full spec PLAN §5 M6).
|
||||
Do M5 first; M6 is unlocked the moment your M5 acceptance is logged. Test
|
||||
media will be arriving in assets/backdrops/video/ via scripts/flow_intake.py.
|
||||
|
||||
### 2026-07-20 — M5+M6 ACCEPTED at SYNC 6 (all lanes ran as orchestrator subagents).
|
||||
Verified live: DIRECT bar CU/low/frame → camera+keys+cut, ONE undo reverts;
|
||||
golden_hour keys; video plate follows the clock frame-exact; finalRender shows
|
||||
the Flow plate behind an animating character. Orchestrator seam fixes to absorb:
|
||||
(1) server: VIDEO_EXTS in tree/media-types, video beats poster as primary;
|
||||
(2) stage.entityVideo(id) added (Lane B seam); (3) syncVideos forces
|
||||
videoTex.needsUpdate — r160 VideoTexture rides requestVideoFrameCallback which
|
||||
never fires in hidden tabs → black plates in renders. Backlog carried: honor
|
||||
audio in/out at mux (C), walk-to-mark prevPos pre-apply (A), live viseme asset
|
||||
still owed by John.
|
||||
|
||||
@ -166,3 +166,14 @@ discover the panel scrolls — consider wheel = scroll, Cmd/Ctrl+wheel = zoom.
|
||||
### 2026-07-19 — M6 FLOW VIDEO PLATES queued behind M5 (full spec PLAN §5 M6).
|
||||
Do M5 first; M6 is unlocked the moment your M5 acceptance is logged. Test
|
||||
media will be arriving in assets/backdrops/video/ via scripts/flow_intake.py.
|
||||
|
||||
### 2026-07-20 — M5+M6 ACCEPTED at SYNC 6 (all lanes ran as orchestrator subagents).
|
||||
Verified live: DIRECT bar CU/low/frame → camera+keys+cut, ONE undo reverts;
|
||||
golden_hour keys; video plate follows the clock frame-exact; finalRender shows
|
||||
the Flow plate behind an animating character. Orchestrator seam fixes to absorb:
|
||||
(1) server: VIDEO_EXTS in tree/media-types, video beats poster as primary;
|
||||
(2) stage.entityVideo(id) added (Lane B seam); (3) syncVideos forces
|
||||
videoTex.needsUpdate — r160 VideoTexture rides requestVideoFrameCallback which
|
||||
never fires in hidden tabs → black plates in renders. Backlog carried: honor
|
||||
audio in/out at mux (C), walk-to-mark prevPos pre-apply (A), live viseme asset
|
||||
still owed by John.
|
||||
|
||||
@ -161,3 +161,14 @@ it needs more than that.
|
||||
### 2026-07-19 — M6 FLOW VIDEO PLATES queued behind M5 (full spec PLAN §5 M6).
|
||||
Do M5 first; M6 is unlocked the moment your M5 acceptance is logged. Test
|
||||
media will be arriving in assets/backdrops/video/ via scripts/flow_intake.py.
|
||||
|
||||
### 2026-07-20 — M5+M6 ACCEPTED at SYNC 6 (all lanes ran as orchestrator subagents).
|
||||
Verified live: DIRECT bar CU/low/frame → camera+keys+cut, ONE undo reverts;
|
||||
golden_hour keys; video plate follows the clock frame-exact; finalRender shows
|
||||
the Flow plate behind an animating character. Orchestrator seam fixes to absorb:
|
||||
(1) server: VIDEO_EXTS in tree/media-types, video beats poster as primary;
|
||||
(2) stage.entityVideo(id) added (Lane B seam); (3) syncVideos forces
|
||||
videoTex.needsUpdate — r160 VideoTexture rides requestVideoFrameCallback which
|
||||
never fires in hidden tabs → black plates in renders. Backlog carried: honor
|
||||
audio in/out at mux (C), walk-to-mark prevPos pre-apply (A), live viseme asset
|
||||
still owed by John.
|
||||
|
||||
110
logs/laneA.md
110
logs/laneA.md
@ -185,3 +185,113 @@ BLOCKED/REQUESTS:
|
||||
I'll confirm live and Lane B can validate Rhubarb end-to-end.
|
||||
|
||||
NEXT: hold at M4-A boundary. Available for a live viseme asset check and any SYNC findings.
|
||||
|
||||
## 2026-07-20 session 5 (M5 DIRECTOR MODE + M6 FLOW VIDEO PLATES)
|
||||
DONE — both milestones, headless-verified (no browser/servers touched per orchestrator order):
|
||||
|
||||
M5:
|
||||
- `web/grammar.js` (NEW, pure data, zero imports): SHOTS (exact spans/centers from the lane
|
||||
spec), ANGLES (eye/low −18°/high +22°/dutch 8° roll/ots_L/ots_R), FOCALS [18,24,35,50,85],
|
||||
LIGHTS (day/golden_hour/night/noir/horror-underlight/neon/overcast — {sun{color,intensity,
|
||||
elev,azim}, ambient{color,intensity}, bg}), MARKS (center/two_shot_L/R/face_camera/face_other).
|
||||
- `web/presets.js` (NEW, NO three.js import — node-runnable): fovForFocal (presets.js:11),
|
||||
lookAtEuler w/ dutch roll + three-matching XYZ euler extraction (:14), solveShot bbox→
|
||||
pos/rot/fov/dist deterministic solver incl. OTS over-the-shoulder placement (:37),
|
||||
frameSubject (:62, bbox via new stage.entityBBox), applyLight (:75 — ensures/reuses one
|
||||
sun+ambient pair, writes grammar values verbatim, aims a 20m sun boom via lookAtEuler,
|
||||
background rides the AMBIENT entity as param `bg` so it's keyable AND survives save/load),
|
||||
applyMark (:99, pos marks keep y; face marks yaw toward active cam / nearest character),
|
||||
applyOps (:139) — ONE op runner shared by panel buttons and /director NL, fires
|
||||
`scenegod:direct {op, ...values, entityIds}` per action (B keys them; I never touch the
|
||||
timeline), mountDirectPanel (:174) — shot/angle/lens segments + 🎥 frame, light buttons,
|
||||
mark buttons, NL box.
|
||||
- NL box: POST /director {text, selected, entities:[{id,kind,label}]} → expects {ops:[...]},
|
||||
applied via applyOps. Hidden by default; probe `GET /director` shows it on 405/200 (FastAPI
|
||||
405 = POST route exists); hides itself again when a POST returns 503.
|
||||
- index.html:14 `#direct` bar + guarded dynamic import (:49) — page survives presets.js absent.
|
||||
style.css:85 DIRECT styles (before the LANE B marker).
|
||||
- stage.js additions for M5: entityBBox (stage.js:490, plain arrays — keeps presets three-free),
|
||||
cameraWorldPos (:497), light param `bg` → scene.background live (:393) + restored at
|
||||
_buildLight (:335) so a loaded scene gets its sky back.
|
||||
|
||||
M6:
|
||||
- `_buildBackdrop` rework (stage.js:225): params.mode gains "video" (renders as plane), and
|
||||
plane/corner/dome accept .mp4/.webm/.mov sources; THREE.VideoTexture muted+loop+playsInline,
|
||||
SRGB; poster jpg (server /assets/thumb same-stem sidecar) shown until first play/seek then
|
||||
swapped in via _swapOnPlay (:309). Standalone page (no timeline → _mixerAuto) auto-plays as
|
||||
ambient preview; with a timeline mounted videos stay paused at poster for B's clock.
|
||||
- NEW kind `screen` (addEntity :165, _buildScreen :272): video plane, params {video,width,
|
||||
emissive}, MeshStandard + emissiveMap boost (default 0.6) so club screens read "on" under
|
||||
noir/night. ponytail: CRT curve skipped until a real TV asset asks.
|
||||
- `stage.syncVideos(t)` (:506): seeks every video to ((t−videoStart) mod duration), pauses,
|
||||
awaits 'seeked' per video (1.5s timeout guard so a stuck decoder can't hang a render), skips
|
||||
sub-4ms deltas. render.js already awaits it per frame (Lane C landed the hook). Plus
|
||||
videos() (:501) + setVideoPlaying(on) (:503) for Lane B's clock-follow, and params.videoStart
|
||||
plumbed (inspector fields, dock.js:214/220).
|
||||
- dock.js: _videoPath (:86, derives <stem>.<videoext> from formats), 🎞 badge + 📺
|
||||
add-as-screen button on video backdrop cards (:106), click/drag adds mode:'video' plates
|
||||
(:132 and _viewDrop), local .mp4 drop → upload video plate, screen/backdrop inspector fields
|
||||
(width/emissive/vid start; mode select gains 'video').
|
||||
- Lifecycle: removeEntity/_rebuild release the <video> (_dropVideo :316); applyState treats
|
||||
upload screens as needs-bytes (skip+warn) like backdrops.
|
||||
|
||||
VERIFICATION (headless):
|
||||
- `web/presets_test.mjs` (NEW): fov-from-focal exact for 18/35/85mm (+35mm≈37.849°); CU dist <
|
||||
MS dist on a 1.7m subject + exact span-formula match + CU aims at .87H; dutch sets rot z =
|
||||
8° with zero stray pitch/yaw; low angle puts camera below aim pitched up;
|
||||
applyLight('golden_hour') returns grammar values verbatim (deepEqual) + applies
|
||||
intensity/bg to a stub stage + reuses the sun/ambient pair on a second preset.
|
||||
→ `node scenegod/web/presets_test.mjs` = ALL PASS.
|
||||
- Scratchpad ops smoke (not committed): applyOps shot-by-LABEL auto-creates 'shotcam', sets
|
||||
85mm fov 16.07°, activates cam, low-angle below aim; ots_R auto-resolves the other
|
||||
character; two_shot_L pos; face_camera/face_other yaw; unresolvable subject reports error
|
||||
without throwing. ALL PASS.
|
||||
- ESM syntax-check (node --check via scratchpad .mjs copies): stage/dock/presets/grammar/room3d
|
||||
all clean. Retarget code and every existing Stage API untouched.
|
||||
|
||||
DECISIONS:
|
||||
- presets.js takes bboxes as plain arrays (new stage.entityBBox) instead of importing three —
|
||||
the whole solver runs under node, same trick as Lane B's timeline.
|
||||
- ONE op path: panel buttons call applyOps exactly like /director responses do, so button and
|
||||
LLM behavior can never drift; every applied op fires one scenegod:direct event.
|
||||
- Background color is a param (`bg`) on the ambient light entity rather than new scene-level
|
||||
schema — keyable through B's existing params track, persists via captureState, zero schema
|
||||
change. Orchestrator: if you'd rather have scene-level bg in PLAN §4.1, say so and I'll move it.
|
||||
- Poster-until-play = swap material maps on first 'playing' OR 'seeked' — so a final render
|
||||
that never plays (only seeks) still gets real video frames, and an idle stage shows the jpg.
|
||||
- OTS shoulder offset is a fixed 0.35m slide + 0.1m rise along the partner line — deterministic,
|
||||
no LLM; eyeball-tune at SYNC if the shoulder isn't reading.
|
||||
|
||||
BLOCKED/REQUESTS:
|
||||
- **@Lane C (blocking M6 dock visibility): mp4 is invisible to the server.** server.py
|
||||
ASSET_EXTS/CATEGORIES: backdrops primary = IMG_EXTS only and mp4 ∉ ASSET_EXTS, so
|
||||
assets/backdrops/video/scenegod_plates/suburb_strip_goldenhour.mp4 never appears in
|
||||
/assets/tree (its poster jpg shows up as a plain image backdrop instead). Please add
|
||||
VIDEO_EXTS = {"mp4","webm","mov"} to ASSET_EXTS and to the backdrops primary set. My dock
|
||||
already handles the resulting shape (finds the video ext in `formats`, derives
|
||||
<stem>.<ext>). Also add MEDIA_TYPES entries ("mp4":"video/mp4","webm":"video/webm",
|
||||
"mov":"video/quicktime") — /assets/file currently serves octet-stream and <video> can be
|
||||
picky about that (and note Range support matters for long plates).
|
||||
Until then, video plates ARE testable via console: `stage.addEntity({kind:'backdrop',
|
||||
label:'plate', source:{type:'assets', path:'backdrops/video/scenegod_plates/
|
||||
suburb_strip_goldenhour.mp4'}, params:{mode:'video', image:'backdrops/video/
|
||||
scenegod_plates/suburb_strip_goldenhour.mp4', width:12}})` (file streams fine, just untyped).
|
||||
- @Lane C: /director endpoint per PLAN M5 — my probe expects GET→405 (or 200) when it exists,
|
||||
and I hide the box on POST 503. Request contract above; response {ops:[{op:'shot'|'light'|
|
||||
'mark', ...}]} — shapes documented at presets.js:136.
|
||||
- @Lane B: your M5/M6 hooks are ready on my side — `scenegod:direct` events carry full values
|
||||
for keying; for video clock-follow use stage.videos() [{id,video,start}] +
|
||||
stage.setVideoPlaying(on) + drift-check video.currentTime yourself per PLAN (>50ms), or just
|
||||
await stage.syncVideos(t) on hard seeks.
|
||||
|
||||
NEXT / orchestrator SYNC checklist (browser):
|
||||
1. Select man → CU low 35mm → 🎥 frame: camera frames his face from below in one click; PiP
|
||||
shows it (frame auto-activates the shot cam).
|
||||
2. golden_hour button: sun+ambient recolor + orange sky in one click; scrub proves B keyed it
|
||||
(needs B's scenegod:direct consumer — not landed yet as of this session).
|
||||
3. Suburb plate: console addEntity above → poster shows, then video motion (auto-play only
|
||||
when timeline absent; with timeline mounted it waits for B's clock). 📺 dock button appears
|
||||
once Lane C ships the tree change.
|
||||
4. finalRender a 2s scene with the plate — frames must show advancing video (syncVideos is
|
||||
awaited by render.js already).
|
||||
5. NL box stays hidden until /director exists (probe is GET → expect 405 to reveal).
|
||||
|
||||
@ -179,3 +179,73 @@ NEXT: M4-B complete. Open items if a session 5 lands: audio block trim/clip to
|
||||
C has a /rhubarb endpoint — could wire a "lip-sync from audio" button that
|
||||
POSTs the clip and feeds the result to importRhubarb). Holding for SYNC/next
|
||||
orchestrator update.
|
||||
|
||||
## 2026-07-20 session 5 (M5 director-mode B + backlog, M6 video-follow)
|
||||
DONE: All three M5 items + the M6 item, headless-verified.
|
||||
1. `scenegod:direct` listener (timeline.js ctor) → `applyDirect(detail)`:
|
||||
shot → transform+fov keys on the camera at playhead + addCut; light →
|
||||
param keys; mark → transform key (walkTo → current/prevPos key at
|
||||
playhead + mark key at playhead+2s). New `group(fn)` collapses every
|
||||
mutator push inside one op into ONE undo entry — one Ctrl+Z reverts a
|
||||
whole preset. `importRhubarb` is now grouped too.
|
||||
**Payload coordination**: presets.js landed mid-session (untracked, Lane
|
||||
A in flight), so I read it — Lane A's REAL payloads differ from the
|
||||
PLAN-stated contract: values are spread at TOP level (`{op:'shot', camId,
|
||||
pos, rot, fov,…}` — no `transform:{}` wrapper), and light is
|
||||
`{sun:{color,intensity}, ambient:{…}, bg, sunId, ambientId}` (per-entity
|
||||
values, not a flat `params` map; bg keys onto the ambient entity;
|
||||
applyDirect ALSO captures the sun-boom transform from the stage since the
|
||||
payload lacks sunPos rot). applyDirect accepts BOTH shapes (contract +
|
||||
live); tests cover both.
|
||||
2. 🗣 button on every audio row (tlui) → `GET /rhubarb?path=` →
|
||||
`importRhubarb(charId, json, clip.start)`. Character picker (`_choose`
|
||||
promise-menu) when >1 viseme-bearing character, guard toast when 0,
|
||||
`_toast` shows the 503 install hint from FastAPI `detail`. Viseme lane
|
||||
resyncs after import.
|
||||
3. Backlog: (a) wheel fix — plain wheel is NOT captured (page scrolls);
|
||||
Cmd/Ctrl+wheel (incl. trackpad pinch) zooms 1–64× around the cursor,
|
||||
horizontal wheel pans when zoomed; all geometry through _t2x/_x2t so
|
||||
keys/blocks/playhead/hitboxes just work. (b) audio trim: drag the block's
|
||||
right edge → `trimAudio(clip, {in,out})` (source-relative, like clip
|
||||
blocks; absent = untrimmed so the {path,start,gain} schema is unchanged
|
||||
for untouched clips); draft Web Audio playback honors in/out; ✂ badge on
|
||||
trimmed blocks; undo restores pristine (fields deleted, not undefined).
|
||||
4. M6: `_evalVideo` in evaluate for backdrop/screen entities via new nullable
|
||||
stage seam `entityVideo(id)`: `currentTime = max(0, t-videoStart) %
|
||||
duration` ONLY when drift >50ms; play/pause follows the clock (pause()
|
||||
now re-evaluates so videos freeze); play() promise .catch()ed, never
|
||||
awaited — step() stays sync (render determinism is stage.syncVideos,
|
||||
Lane A/C).
|
||||
Files: timeline.js, tlui.js, stagestub.js (StubVideo records currentTime
|
||||
sets; entityVideo null for image backdrops — honest-nullable),
|
||||
timeline_test.mjs. `node scenegod/web/timeline_test.mjs` → all green
|
||||
(shot-via-real-EventTarget-dispatch + one-undo revert, generic + Lane A
|
||||
light shapes, walkTo 2 keys, Lane A mark shape, rhubarb group undo, trim
|
||||
undo, video drift-gate/modulo/videoStart-clamp/play-pause, nullable video).
|
||||
DECISIONS:
|
||||
- group() splices the undo stack rather than a begin/end transaction API —
|
||||
mutators stay dumb, nesting works, zero cost when fn pushes ≤1 entry.
|
||||
- Sun transform key on light presets comes from stage.entityTransform at
|
||||
event time (Lane A setTransforms before firing) — payload has sunPos but
|
||||
no rot; reading the stage gets both without a payload change.
|
||||
- Video seek target clamps negative (t < videoStart → hold frame 0); wrap
|
||||
uses plain % (t-videoStart ≥ 0 after clamp).
|
||||
- StubVideo lives in a stage-side Map, NOT on the entity object, so
|
||||
captureState/clone stays clean.
|
||||
BLOCKED/REQUESTS:
|
||||
- **Lane A**: walk-to-mark can't work from the current payload — applyMark
|
||||
setTransforms the subject BEFORE firing, so "current pos" at event time is
|
||||
already the mark. When you add a walk mark, either include `prevPos` (+
|
||||
`walkTo:true`) in the detail (applyDirect already honors both) or skip the
|
||||
pre-apply for walk marks. Also: no walk mark exists in grammar.js MARKS yet.
|
||||
- **Lane C**: audio trim adds optional `in`/`out` (source-relative seconds)
|
||||
to scene.audio clips — mux should honor them (ffmpeg -ss/-t per clip);
|
||||
absent = whole file (schema unchanged for untrimmed clips). Client draft
|
||||
playback already honors them.
|
||||
- **Orchestrator**: PLAN §4.1 audio schema + §4.2 Stage API could gain the
|
||||
two seams now in live use: audio {in?, out?} and `entityVideo(id) ->
|
||||
HTMLVideoElement|null` (Lane A implements on real Stage for M6-A).
|
||||
NEXT: SYNC — browser-verify DIRECT panel button → keys+cut land + single
|
||||
undo; 🗣 on real vo.wav → viseme keys on a blendshape character (and the
|
||||
503 toast with rhubarb absent); Cmd+wheel zoom feel; video backdrop
|
||||
scrub-follow once Lane A ships entityVideo. Holding at M6-B boundary.
|
||||
|
||||
@ -111,3 +111,52 @@ untested until `rhubarb` is on PATH. Flagged same as MediaRecorder backlog.
|
||||
BLOCKED/REQUESTS: none. To fully verify: install rhubarb-lip-sync on ultra,
|
||||
then GET /rhubarb?path=audio/<some>.wav.
|
||||
NEXT: nothing outstanding — Lane C M1-M4 + rhubarb shipped. Standing by.
|
||||
|
||||
## 2026-07-20 session 4
|
||||
DONE: M5 `/director` + M6 render.js sliver.
|
||||
- `scenegod/server.py`: `POST /director` {text, scene:{entities,time}} ->
|
||||
OpenAI-compat chat (`SCENEGOD_LLM_URL` + `/v1/chat/completions`, model
|
||||
`SCENEGOD_LLM_MODEL` default `qwen2.5:7b`, temp 0.1, response_format
|
||||
json_object, stdlib urllib, 120s timeout). 503 + env hint when URL unset;
|
||||
502 on LLM error/unreachable/unparseable output — raw model text is NEVER
|
||||
forwarded. `_validate_ops`: ops shot/light/mark checked against vocab +
|
||||
scene entity ids; violations dropped into `rejected` [{op, reason}];
|
||||
response {ops, rejected}. Shot defaults angle="eye", focal=35 filled
|
||||
server-side.
|
||||
- Vocab (SHOTS/ANGLES/FOCALS/LIGHTS/MARKS) defined ONCE at top of the
|
||||
director section, mirroring PLAN §5 M5.
|
||||
- `web/render.js` finalRender: after `timeline.step(f)`, before capture:
|
||||
`if (stage.syncVideos) await stage.syncVideos(timeline.time)` — guarded
|
||||
no-op until Lane A lands it (M6 determinism hook).
|
||||
- `scripts/test_server.py`: +director group (in-process http.server LLM mock,
|
||||
MB-mock pattern): 503 when env unset, 3 valid ops pass through verbatim,
|
||||
unknown op / unknown entity id / unknown preset land in `rejected` with
|
||||
reasons, prompt carries 10 vocab words + roster ids + playhead, model name
|
||||
+ low temp asserted. `OK: 9 test groups passed` (rhubarb-503 group now
|
||||
self-skips — rhubarb is installed on ultra, so that group's absent-path
|
||||
doesn't run; happy path was orchestrator-verified live).
|
||||
DECISIONS:
|
||||
- `web/grammar.js` does NOT exist yet (no presets.js either) — vocab lives in
|
||||
server.py per spec fallback. **AT SYNC: re-point the SHOTS/ANGLES/FOCALS/
|
||||
LIGHTS/MARKS lists + DIRECTOR_SYS at Lane A's grammar.js (read at boot)**
|
||||
so server prompt and client presets can't drift. Op shapes I defined
|
||||
(scenegod:direct): {op:"shot",subject,shot,angle,focal} /
|
||||
{op:"light",preset} / {op:"mark",subject,mark,target?} — Lane A's
|
||||
presets.js appliers must accept exactly these.
|
||||
- Model may return `{"ops":[...]}` or a bare array — both accepted
|
||||
(json_object mode forces an object wrapper; bare array tolerated for
|
||||
servers that ignore response_format).
|
||||
- Env read at request time (matches _mb_token pattern) so John can set
|
||||
SCENEGOD_LLM_URL without a restart-order footgun; test env explicitly
|
||||
pops it for the 503 assert.
|
||||
- focal coerced via int() before vocab check (7b models love "35" strings).
|
||||
BLOCKED/REQUESTS:
|
||||
- Lane A: need `web/grammar.js` (data only) + presets.js appliers matching
|
||||
the op shapes above; flag any shape disagreement at SYNC and I'll conform.
|
||||
- Live end-to-end against real qwen2.5:7b on m4pro untested this session
|
||||
(SCENEGOD_LLM_URL unset locally by design) — orchestrator: one smoke call
|
||||
at SYNC with env set is the acceptance step.
|
||||
NEXT: M6 FLOW VIDEO PLATES server side (unlocked on this acceptance):
|
||||
flow_intake.py poster-jpg intake is already in scripts/; verify video exts
|
||||
in /assets/tree (mp4 not in ASSET_EXTS yet — backdrops video category will
|
||||
need it) + deterministic render check once Lane A's stage.syncVideos lands.
|
||||
|
||||
@ -32,14 +32,16 @@ RENDERS = Path(os.environ.get("SCENEGOD_RENDERS", ROOT / "renders")).resolve()
|
||||
MODEL_EXTS = {"glb", "gltf", "fbx", "obj", "bvh"}
|
||||
IMG_EXTS = {"jpg", "jpeg", "png", "webp"}
|
||||
AUDIO_EXTS = {"wav", "mp3"}
|
||||
ASSET_EXTS = MODEL_EXTS | IMG_EXTS | AUDIO_EXTS
|
||||
VIDEO_EXTS = {"mp4", "webm", "mov"} # M6: Flow video plates
|
||||
ASSET_EXTS = MODEL_EXTS | IMG_EXTS | AUDIO_EXTS | VIDEO_EXTS
|
||||
# primary ext per category (others sharing a stem are thumbnails)
|
||||
CATEGORIES = {"characters": MODEL_EXTS, "animations": MODEL_EXTS, "props": MODEL_EXTS,
|
||||
"backdrops": IMG_EXTS, "audio": AUDIO_EXTS}
|
||||
"backdrops": IMG_EXTS | VIDEO_EXTS, "audio": AUDIO_EXTS}
|
||||
MEDIA_TYPES = {
|
||||
"glb": "model/gltf-binary", "gltf": "model/gltf+json", "fbx": "application/octet-stream",
|
||||
"obj": "text/plain", "bvh": "text/plain", "jpg": "image/jpeg", "jpeg": "image/jpeg",
|
||||
"png": "image/png", "webp": "image/webp", "wav": "audio/wav", "mp3": "audio/mpeg",
|
||||
"mp4": "video/mp4", "webm": "video/webm", "mov": "video/quicktime",
|
||||
}
|
||||
|
||||
SCENES.mkdir(parents=True, exist_ok=True)
|
||||
@ -117,8 +119,11 @@ def scan_assets() -> dict:
|
||||
g["thumb"] = rel
|
||||
continue
|
||||
g["formats"].append(ext)
|
||||
if g["path"] is None or ext == "glb": # prefer glb, else first
|
||||
# prefer glb (models) / video (backdrops: mp4 beats its jpg poster), else first
|
||||
if g["path"] is None or ext == "glb" or ext in VIDEO_EXTS:
|
||||
g["path"] = rel
|
||||
if ext in IMG_EXTS and g["thumb"] is None: # primary img doubles as poster/thumb
|
||||
g["thumb"] = rel
|
||||
entries = [g for g in groups.values() if g["formats"]] # thumb-only stems aren't assets
|
||||
out[cat] = sorted(entries, key=lambda g: g["name"].lower())
|
||||
return out
|
||||
@ -365,6 +370,123 @@ def rhubarb(path: str):
|
||||
return json.loads(p.stdout)
|
||||
|
||||
|
||||
# ---- DIRECTOR MODE (M5) — NL text -> validated preset ops via tailnet LLM ----
|
||||
# Vocab mirrors PLAN §5 M5. NOTE: Lane A's web/grammar.js hasn't landed yet —
|
||||
# when it does, re-point these lists at that file (read at boot) so the server
|
||||
# prompt and the client presets never drift. Single source of truth until then.
|
||||
SHOTS = ["ECU", "CU", "MCU", "MS", "MWS", "WS", "EWS"]
|
||||
ANGLES = ["eye", "low", "high", "dutch", "OTS-L", "OTS-R"]
|
||||
FOCALS = [18, 24, 35, 50, 85]
|
||||
LIGHTS = ["day", "golden-hour", "night", "noir", "horror", "neon", "overcast"]
|
||||
MARKS = ["center", "two-shot-L", "two-shot-R", "walk-to-mark", "face-camera", "face-other"]
|
||||
|
||||
DIRECTOR_SYS = f"""You are a film director's assistant. Translate the direction into JSON: {{"ops": [...]}} and NOTHING else. Allowed ops (scenegod:direct contract):
|
||||
- {{"op": "shot", "subject": "<entity id>", "shot": <shot>, "angle": <angle>, "focal": <mm>}} — frame the subject. shot sizes: {SHOTS}. angles: {ANGLES} (default "eye"). focal lengths mm: {FOCALS} (default 35).
|
||||
- {{"op": "light", "preset": <preset>}} — lighting presets: {LIGHTS}.
|
||||
- {{"op": "mark", "subject": "<entity id>", "mark": <mark>}} — blocking marks: {MARKS}. "face-other" and two-shots may add "target": "<entity id>".
|
||||
Use ONLY entity ids from the scene roster. Skip anything you cannot express with these ops. At most 6 ops."""
|
||||
|
||||
|
||||
def _validate_ops(ops, ids: set) -> tuple[list, list]:
|
||||
"""Enforce the scenegod:direct contract server-side: unknown op names,
|
||||
vocab values, or entity ids not in the scene -> `rejected` (UI toasts it).
|
||||
Never forwards raw model output."""
|
||||
good, rejected = [], []
|
||||
|
||||
def rej(o, why):
|
||||
rejected.append({"op": o, "reason": why})
|
||||
|
||||
if not isinstance(ops, list):
|
||||
return [], [{"op": ops, "reason": "model output is not an op list"}]
|
||||
for o in ops:
|
||||
if not isinstance(o, dict):
|
||||
rej(o, "op is not an object")
|
||||
continue
|
||||
kind = o.get("op")
|
||||
if kind == "shot":
|
||||
o = {"op": "shot", "subject": o.get("subject"), "shot": o.get("shot"),
|
||||
"angle": o.get("angle", "eye"), "focal": o.get("focal", 35)}
|
||||
try:
|
||||
o["focal"] = int(o["focal"])
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
for field, allowed in (("shot", SHOTS), ("angle", ANGLES), ("focal", FOCALS)):
|
||||
if o[field] not in allowed:
|
||||
rej(o, f"unknown {field} {o[field]!r}")
|
||||
break
|
||||
else:
|
||||
if o["subject"] not in ids:
|
||||
rej(o, f"unknown subject {o['subject']!r}")
|
||||
else:
|
||||
good.append(o)
|
||||
elif kind == "light":
|
||||
o = {"op": "light", "preset": o.get("preset")}
|
||||
if o["preset"] not in LIGHTS:
|
||||
rej(o, f"unknown preset {o['preset']!r}")
|
||||
else:
|
||||
good.append(o)
|
||||
elif kind == "mark":
|
||||
keep = {"op": "mark", "subject": o.get("subject"), "mark": o.get("mark")}
|
||||
if "target" in o:
|
||||
keep["target"] = o["target"]
|
||||
if keep["mark"] not in MARKS:
|
||||
rej(keep, f"unknown mark {keep['mark']!r}")
|
||||
elif keep["subject"] not in ids:
|
||||
rej(keep, f"unknown subject {keep['subject']!r}")
|
||||
elif "target" in keep and keep["target"] not in ids:
|
||||
rej(keep, f"unknown target {keep['target']!r}")
|
||||
else:
|
||||
good.append(keep)
|
||||
else:
|
||||
rej(o, f"unknown op {kind!r}")
|
||||
return good, rejected
|
||||
|
||||
|
||||
@app.post("/director")
|
||||
async def director(request: Request):
|
||||
"""{text, scene:{entities:[{id,kind,label}], time}} -> {ops, rejected}.
|
||||
OpenAI-compat chat call (tailnet Ollama); ops validated before returning."""
|
||||
url = os.environ.get("SCENEGOD_LLM_URL", "").rstrip("/")
|
||||
if not url:
|
||||
raise HTTPException(503, "no LLM configured (set SCENEGOD_LLM_URL, e.g. "
|
||||
"http://100.69.21.128:11434, and SCENEGOD_LLM_MODEL)")
|
||||
body = await request.json()
|
||||
text = (body.get("text") or "").strip()
|
||||
if not text:
|
||||
raise HTTPException(400, "text required")
|
||||
scene = body.get("scene") or {}
|
||||
ents = scene.get("entities") or []
|
||||
ids = {e.get("id") for e in ents}
|
||||
roster = "\n".join(f"- id={e.get('id')!r} kind={e.get('kind')} label={e.get('label')!r}"
|
||||
for e in ents) or "(empty scene)"
|
||||
payload = {
|
||||
"model": os.environ.get("SCENEGOD_LLM_MODEL", "qwen2.5:7b"),
|
||||
"messages": [
|
||||
{"role": "system", "content": DIRECTOR_SYS},
|
||||
{"role": "user", "content": f"Scene roster:\n{roster}\n"
|
||||
f"playhead: {scene.get('time', 0)}s\n\nDirection: {text}"},
|
||||
],
|
||||
"temperature": 0.1,
|
||||
"response_format": {"type": "json_object"}, # ignored by servers without it
|
||||
}
|
||||
req = urllib.request.Request(url + "/v1/chat/completions", method="POST",
|
||||
data=json.dumps(payload).encode(), headers={"Content-Type": "application/json"})
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=120) as resp: # 7b on m4pro can be slow
|
||||
data = json.load(resp)
|
||||
except urllib.error.HTTPError as e:
|
||||
raise HTTPException(502, f"LLM: {e.code} {e.read()[:200].decode('utf-8', 'replace')}")
|
||||
except OSError as e:
|
||||
raise HTTPException(502, f"LLM unreachable: {e.__class__.__name__}")
|
||||
try:
|
||||
parsed = json.loads(data["choices"][0]["message"]["content"])
|
||||
except (KeyError, IndexError, TypeError, json.JSONDecodeError):
|
||||
raise HTTPException(502, "LLM returned unparseable output") # never forward raw text
|
||||
ops = parsed.get("ops") if isinstance(parsed, dict) else parsed # accept bare array too
|
||||
good, rejected = _validate_ops(ops, ids)
|
||||
return {"ops": good, "rejected": rejected}
|
||||
|
||||
|
||||
# ---- MODELBEAST proxy (M4, gated) ----
|
||||
def _mb_post(path: str, data: bytes, content_type: str) -> dict:
|
||||
req = urllib.request.Request(MB_HOST + path, data=data, method="POST",
|
||||
|
||||
@ -61,7 +61,7 @@ export class Dock {
|
||||
const raw = e.dataTransfer.getData('application/scenegod');
|
||||
if(!raw) return; // a local file drop — let the window handler take it
|
||||
e.preventDefault(); e.stopPropagation();
|
||||
const { tab, path, name } = JSON.parse(raw);
|
||||
const { tab, path, name, video } = JSON.parse(raw);
|
||||
try {
|
||||
if(tab === 'animations'){
|
||||
const id = this.stage.entityAt(e.clientX, e.clientY, 'character');
|
||||
@ -69,8 +69,9 @@ export class Dock {
|
||||
return this._applyClip(id, path, 0);
|
||||
}
|
||||
const kind = tab === 'characters' ? 'character' : tab === 'props' ? 'prop' : 'backdrop';
|
||||
const desc = { kind, label: name, source:{ type:'assets', path } };
|
||||
if(kind === 'backdrop') desc.params = { mode:'plane', image:path, width:10 };
|
||||
const desc = { kind, label: name, source:{ type:'assets', path: video || path } };
|
||||
if(kind === 'backdrop') desc.params = video
|
||||
? { mode:'video', image:video, width:12 } : { mode:'plane', image:path, width:10 };
|
||||
else desc.transform = { pos: this.stage.screenToFloor(e.clientX, e.clientY), rot:[0,0,0], scale:1 };
|
||||
const en = await this.stage.addEntity(desc);
|
||||
this.stage.select(en.id);
|
||||
@ -81,6 +82,11 @@ export class Dock {
|
||||
// /assets/tree item shape (confirmed by Lane C): {name, path, formats:[...], thumb: relpath|null}
|
||||
_path(it){ return it.path; }
|
||||
_name(it){ return it.name || (it.path || '').split('/').pop(); }
|
||||
// M6 video plates: same-stem grouping guarantees <stem>.<videoExt> when formats carries one
|
||||
_videoPath(it){
|
||||
const vf = (it.formats || []).find(f => ['mp4','webm','mov'].includes(f));
|
||||
return vf ? (it.path || '').replace(/\.\w+$/, '.' + vf) : null;
|
||||
}
|
||||
|
||||
renderBrowser(){
|
||||
this.list.innerHTML = '';
|
||||
@ -97,10 +103,17 @@ export class Dock {
|
||||
const th = it.thumb
|
||||
? `<img class="th" loading="lazy" src="/assets/file?path=${encodeURIComponent(it.thumb)}">`
|
||||
: `<div class="th">${icon}</div>`;
|
||||
c.innerHTML = `${th}<div class="nm">${this._name(it)}</div>`;
|
||||
const vp = this.tab === 'backdrops' ? this._videoPath(it) : null;
|
||||
c.innerHTML = `${th}<div class="nm">${vp ? '🎞 ' : ''}${this._name(it)}</div>`;
|
||||
c.onclick = () => this._addFromLibrary(it);
|
||||
c.ondragstart = ev => ev.dataTransfer.setData('application/scenegod',
|
||||
JSON.stringify({ tab: this.tab, path: this._path(it), name: this._name(it) }));
|
||||
JSON.stringify({ tab: this.tab, path: this._path(it), name: this._name(it), video: vp }));
|
||||
if(vp){ // second affordance on video plates: add as an in-set screen (TV/club/jumbotron)
|
||||
const tv = document.createElement('button'); tv.className = 'tv'; tv.textContent = '📺';
|
||||
tv.title = 'add as in-set screen';
|
||||
tv.onclick = ev => { ev.stopPropagation(); this._addScreen(vp, this._name(it)); };
|
||||
c.appendChild(tv);
|
||||
}
|
||||
this.list.appendChild(c);
|
||||
}
|
||||
}
|
||||
@ -115,7 +128,11 @@ export class Dock {
|
||||
}
|
||||
const kind = this.tab === 'characters' ? 'character' : this.tab === 'props' ? 'prop' : 'backdrop';
|
||||
const desc = { kind, label: this._name(it), source: { type:'assets', path } };
|
||||
if(kind === 'backdrop') desc.params = { mode:'plane', image:path, width:10 };
|
||||
if(kind === 'backdrop'){
|
||||
const vp = this._videoPath(it); // video plate → video plane; else image plane
|
||||
desc.params = vp ? { mode:'video', image:vp, width:12 } : { mode:'plane', image:path, width:10 };
|
||||
if(vp) desc.source = { type:'assets', path: vp };
|
||||
}
|
||||
this._toast('loading ' + this._name(it) + '…');
|
||||
const en = await this.stage.addEntity(desc);
|
||||
this.stage.select(en.id);
|
||||
@ -136,6 +153,16 @@ export class Dock {
|
||||
this._toast('▶ ' + path.split('/').pop());
|
||||
}
|
||||
|
||||
// M6: video plate → `screen` entity (emissive in-set display, PLAN §4.1 params)
|
||||
async _addScreen(path, name){
|
||||
try {
|
||||
this._toast('loading ' + name + '…');
|
||||
const en = await this.stage.addEntity({ kind:'screen', label:name,
|
||||
source:{ type:'assets', path }, params:{ video:path, width:3, emissive:0.6 } });
|
||||
this.stage.select(en.id); this._toast('');
|
||||
} catch(e){ this._toast('failed: ' + String(e).slice(0, 100)); }
|
||||
}
|
||||
|
||||
// cameras and lights have no asset file — spawn them from the toolbar
|
||||
async _addRig(what){
|
||||
const D = {
|
||||
@ -181,9 +208,18 @@ export class Dock {
|
||||
el.appendChild(this._num('intensity', en.params.intensity ?? 1.5, v => this.stage.setParam(en.id, 'intensity', v)));
|
||||
el.appendChild(this._color('color', en.params.color || '#ffffff', v => this.stage.setParam(en.id, 'color', v)));
|
||||
}
|
||||
if(en.kind === 'backdrop')
|
||||
el.appendChild(this._sel('mode', ['plane','corner','dome'], en.params.mode || 'plane',
|
||||
if(en.kind === 'backdrop'){
|
||||
el.appendChild(this._sel('mode', ['plane','corner','dome','video'], en.params.mode || 'plane',
|
||||
v => this.stage.setParam(en.id, 'mode', v))); // stage rebuilds the mesh on change
|
||||
if(en.video) el.appendChild(this._num('vid start', en.params.videoStart ?? 0,
|
||||
v => this.stage.setParam(en.id, 'videoStart', v)));
|
||||
}
|
||||
if(en.kind === 'screen'){
|
||||
el.appendChild(this._num('width', en.params.width ?? 3, v => this.stage.setParam(en.id, 'width', v)));
|
||||
el.appendChild(this._num('emissive', en.params.emissive ?? 0.6, v => this.stage.setParam(en.id, 'emissive', v)));
|
||||
el.appendChild(this._num('vid start', en.params.videoStart ?? 0,
|
||||
v => this.stage.setParam(en.id, 'videoStart', v)));
|
||||
}
|
||||
|
||||
const row = document.createElement('div'); row.className = 'ibtns';
|
||||
const key = document.createElement('button'); key.textContent = '⏺ key'; key.title = 'capture keyframe';
|
||||
@ -223,6 +259,11 @@ export class Dock {
|
||||
const en = await this.stage.addEntity({ kind:'backdrop', label:f.name,
|
||||
source:{ type:'upload', path:f.name, _url:url }, params:{ mode:'plane' } });
|
||||
this.stage.select(en.id);
|
||||
} else if(/\.(mp4|webm|mov)$/.test(nm)){ // M6: dropped video → video plate (upload, warn on save)
|
||||
const url = URL.createObjectURL(f);
|
||||
const en = await this.stage.addEntity({ kind:'backdrop', label:f.name,
|
||||
source:{ type:'upload', path:f.name, _url:url }, params:{ mode:'video', width:12 } });
|
||||
this.stage.select(en.id);
|
||||
} else if(/\.(glb|gltf|fbx|obj|bvh)$/.test(nm)){
|
||||
const buf = await f.arrayBuffer();
|
||||
// clip drop onto selected character → bake; else load as a character
|
||||
|
||||
57
scenegod/web/grammar.js
Normal file
57
scenegod/web/grammar.js
Normal file
@ -0,0 +1,57 @@
|
||||
// grammar.js — classic cinema vocabulary as PURE DATA (Lane A owns; every lane reads).
|
||||
// No imports, no DOM, no three — safe under plain node. Presets/LLM ops both resolve here,
|
||||
// so the button path and the /director path can never drift apart.
|
||||
|
||||
// Shot sizes: vertical span the framing should cover, as a fraction of subject height,
|
||||
// @ where the frame centers on the body (fraction of subject height from the feet).
|
||||
export const SHOTS = {
|
||||
ECU: { span: 0.12, center: 0.93, label: 'extreme close-up' },
|
||||
CU: { span: 0.25, center: 0.87, label: 'close-up' },
|
||||
MCU: { span: 0.45, center: 0.75, label: 'medium close-up' },
|
||||
MS: { span: 0.65, center: 0.65, label: 'medium shot' },
|
||||
MWS: { span: 0.85, center: 0.55, label: 'medium wide' },
|
||||
WS: { span: 1.2, center: 0.5, label: 'wide shot' },
|
||||
EWS: { span: 2.5, center: 0.5, label: 'extreme wide' },
|
||||
};
|
||||
|
||||
// Camera attitude relative to the aim point. elev = degrees above (+) / below (−) the
|
||||
// aim; roll = dutch tilt in degrees; ots = over-the-shoulder of the OTHER subject.
|
||||
export const ANGLES = {
|
||||
eye: { elev: 0, roll: 0 },
|
||||
low: { elev: -18, roll: 0 },
|
||||
high: { elev: 22, roll: 0 },
|
||||
dutch: { elev: 0, roll: 8 },
|
||||
ots_L: { elev: 0, roll: 0, ots: 'L' },
|
||||
ots_R: { elev: 0, roll: 0, ots: 'R' },
|
||||
};
|
||||
|
||||
// Full-frame focal lengths (mm). Vertical fov = 2·atan(12/f) — 24mm-tall sensor.
|
||||
export const FOCALS = [18, 24, 35, 50, 85];
|
||||
|
||||
// Lighting looks: key ("sun") color/intensity + boom direction (degrees), hemisphere
|
||||
// ambient, and a background clear color. Values are what applyLight writes verbatim.
|
||||
export const LIGHTS = {
|
||||
day: { sun: { color: '#fff6e4', intensity: 3.0, elev: 55, azim: 35 },
|
||||
ambient: { color: '#dfe9ff', intensity: 1.15 }, bg: '#8fb0d6' },
|
||||
golden_hour: { sun: { color: '#ffb469', intensity: 2.6, elev: 9, azim: -55 },
|
||||
ambient: { color: '#ffd9b0', intensity: 0.55 }, bg: '#d98a52' },
|
||||
night: { sun: { color: '#7e93d6', intensity: 0.6, elev: 40, azim: 150 },
|
||||
ambient: { color: '#31405e', intensity: 0.4 }, bg: '#0a0e1a' },
|
||||
noir: { sun: { color: '#ffffff', intensity: 3.4, elev: 52, azim: 75 },
|
||||
ambient: { color: '#1c1c24', intensity: 0.15 }, bg: '#08080a' },
|
||||
horror: { sun: { color: '#b9d6a4', intensity: 1.7, elev: -30, azim: 10 }, // underlight
|
||||
ambient: { color: '#25302b', intensity: 0.25 }, bg: '#05070a' },
|
||||
neon: { sun: { color: '#ff4fd8', intensity: 1.9, elev: 25, azim: 120 },
|
||||
ambient: { color: '#2de2e6', intensity: 0.9 }, bg: '#150420' },
|
||||
overcast: { sun: { color: '#dfe4ea', intensity: 1.2, elev: 70, azim: 0 },
|
||||
ambient: { color: '#c7cdd6', intensity: 1.4 }, bg: '#9aa3ad' },
|
||||
};
|
||||
|
||||
// Blocking marks, stage-relative. pos marks move (keep current y); face marks rotate.
|
||||
export const MARKS = {
|
||||
center: { pos: [0, 0, 0] },
|
||||
two_shot_L: { pos: [-0.7, 0, 0] },
|
||||
two_shot_R: { pos: [0.7, 0, 0] },
|
||||
face_camera: { face: 'camera' },
|
||||
face_other: { face: 'other' },
|
||||
};
|
||||
@ -11,6 +11,7 @@
|
||||
<h1>🎬 SCENEGOD</h1><span class="tag">machinima stage</span>
|
||||
<span id="selftest"></span>
|
||||
</header>
|
||||
<div id="direct"></div>
|
||||
<div id="app">
|
||||
<aside id="dock"></aside>
|
||||
<div id="view"></div>
|
||||
@ -45,6 +46,12 @@ const dock = new Dock({
|
||||
viewEl: document.getElementById('view'),
|
||||
});
|
||||
|
||||
// DIRECT panel (M5 director mode) — guarded like the timeline: page survives its absence
|
||||
try {
|
||||
const { mountDirectPanel } = await import('/web/presets.js');
|
||||
mountDirectPanel(stage, document.getElementById('direct'));
|
||||
} catch(e){ console.warn('direct panel not mounted:', e); }
|
||||
|
||||
// Timeline is optional — the page must never break when Lane B's file is absent
|
||||
try {
|
||||
const { Timeline } = await import('/web/timeline.js');
|
||||
|
||||
249
scenegod/web/presets.js
Normal file
249
scenegod/web/presets.js
Normal file
@ -0,0 +1,249 @@
|
||||
// presets.js — DIRECTOR MODE (M5): deterministic cinema-grammar solvers + the DIRECT panel.
|
||||
// NO three.js import — bbox comes from stage.entityBBox as plain arrays, all vector math is
|
||||
// inlined, so the solvers run headless under `node` (presets_test.mjs). Pure functions RETURN
|
||||
// the values they apply — Lane B keys those values at the playhead off `scenegod:direct`
|
||||
// events; this file never touches the timeline.
|
||||
import { SHOTS, ANGLES, FOCALS, LIGHTS, MARKS } from './grammar.js';
|
||||
|
||||
const D2R = Math.PI / 180;
|
||||
|
||||
// full-frame vertical fov (degrees) for a focal length in mm (24mm-tall sensor)
|
||||
export function fovForFocal(mm){ return 2 * Math.atan(12 / mm) / D2R; }
|
||||
|
||||
// XYZ euler (radians) that points a camera (looks down −Z) at `aim`, plus dutch roll.
|
||||
export function lookAtEuler(pos, aim, rollDeg = 0){
|
||||
let z = [pos[0]-aim[0], pos[1]-aim[1], pos[2]-aim[2]]; // camera +Z = away from aim
|
||||
const zn = Math.hypot(z[0], z[1], z[2]) || 1; z = z.map(v => v/zn);
|
||||
let x = [z[2], 0, -z[0]]; // up × z
|
||||
const xn = Math.hypot(x[0], x[1], x[2]);
|
||||
x = xn < 1e-6 ? [1, 0, 0] : x.map(v => v/xn); // looking straight up/down
|
||||
let y = [ z[1]*x[2]-z[2]*x[1], z[2]*x[0]-z[0]*x[2], z[0]*x[1]-z[1]*x[0] ]; // z × x
|
||||
if(rollDeg){
|
||||
const c = Math.cos(rollDeg*D2R), s = Math.sin(rollDeg*D2R);
|
||||
const rx = x.map((v,i) => v*c + y[i]*s), ry = y.map((v,i) => v*c - x[i]*s);
|
||||
x = rx; y = ry;
|
||||
}
|
||||
// extraction matches three's Euler.setFromRotationMatrix('XYZ') on the column basis [x y z]
|
||||
const m13 = z[0], ey = Math.asin(Math.max(-1, Math.min(1, m13)));
|
||||
let ex, ez;
|
||||
if(Math.abs(m13) < 0.9999999){ ex = Math.atan2(-z[1], z[2]); ez = Math.atan2(-y[0], x[0]); }
|
||||
else { ex = Math.atan2(y[2], y[1]); ez = 0; }
|
||||
return [ex, ey, ez];
|
||||
}
|
||||
|
||||
// Deterministic shot solver: subject bbox (world, plain arrays) → camera pos/rot/fov so the
|
||||
// shot's vertical span fills the frame. yaw = subject facing (rot.y); OTS aims down the line
|
||||
// from the OTHER subject instead, offset over their shoulder.
|
||||
export function solveShot({ bboxMin, bboxMax, yaw = 0, shot = 'MS', angle = 'eye', focal = 35, otherPos = null }){
|
||||
const S = SHOTS[shot] || SHOTS.MS, A = ANGLES[angle] || ANGLES.eye;
|
||||
const H = Math.max(0.01, bboxMax[1] - bboxMin[1]);
|
||||
const aim = [ (bboxMin[0]+bboxMax[0])/2, bboxMin[1] + S.center*H, (bboxMin[2]+bboxMax[2])/2 ];
|
||||
const fov = fovForFocal(focal);
|
||||
const dist = (S.span * H / 2) / Math.tan(fov/2 * D2R);
|
||||
let dir; // subject → camera, horizontal
|
||||
if(A.ots && otherPos){
|
||||
dir = [otherPos[0]-aim[0], 0, otherPos[2]-aim[2]];
|
||||
const n = Math.hypot(dir[0], dir[2]) || 1; dir = [dir[0]/n, 0, dir[2]/n];
|
||||
} else dir = [Math.sin(yaw), 0, Math.cos(yaw)]; // characters face +Z at yaw 0
|
||||
const el = A.elev * D2R, hd = dist * Math.cos(el);
|
||||
const pos = [ aim[0] + dir[0]*hd, Math.max(0.05, aim[1] + dist*Math.sin(el)), aim[2] + dir[2]*hd ];
|
||||
if(A.ots){ // slide over the near shoulder
|
||||
const side = A.ots === 'L' ? -1 : 1;
|
||||
pos[0] += side * 0.35 * dir[2]; pos[2] -= side * 0.35 * dir[0]; pos[1] += 0.1;
|
||||
}
|
||||
return { pos, rot: lookAtEuler(pos, aim, A.roll), fov, dist, aim };
|
||||
}
|
||||
|
||||
const _fire = detail => { if(typeof dispatchEvent !== 'undefined')
|
||||
dispatchEvent(new CustomEvent('scenegod:direct', { detail })); };
|
||||
|
||||
// ---- appliers: apply to the stage AND return the values (Lane B keys them) ----
|
||||
|
||||
export function frameSubject(stage, camId, subjectId, { shot = 'MS', angle = 'eye', focal = 35, otherId = null } = {}){
|
||||
const bb = stage.entityBBox(subjectId);
|
||||
if(!bb) throw new Error('frameSubject: no bbox for ' + subjectId);
|
||||
const yaw = (stage.entityTransform(subjectId)?.rot || [0,0,0])[1];
|
||||
const otherPos = otherId ? stage.entityTransform(otherId)?.pos : null;
|
||||
const solved = solveShot({ bboxMin: bb.min, bboxMax: bb.max, yaw, shot, angle, focal, otherPos });
|
||||
stage.setTransform(camId, { pos: solved.pos, rot: solved.rot, scale: 1 });
|
||||
stage.setParam(camId, 'fov', solved.fov);
|
||||
return { camId, subjectId, otherId, shot, angle, focal, ...solved };
|
||||
}
|
||||
|
||||
// Relight the whole scene from one named look. Ensures a sun (key) + ambient entity exist,
|
||||
// writes LIGHTS[name] verbatim (returned for keying), aims the sun boom at the stage center.
|
||||
export async function applyLight(stage, name){
|
||||
const L = LIGHTS[name];
|
||||
if(!L) throw new Error('unknown light preset: ' + name);
|
||||
const ents = stage.entities();
|
||||
let sun = ents.find(e => e.kind === 'light' && e.params.type !== 'ambient');
|
||||
let amb = ents.find(e => e.kind === 'light' && e.params.type === 'ambient');
|
||||
if(!sun) sun = await stage.addEntity({ kind:'light', label:'sun',
|
||||
params:{ type:'key', color:L.sun.color, intensity:L.sun.intensity, castShadow:true } });
|
||||
if(!amb) amb = await stage.addEntity({ kind:'light', label:'ambient',
|
||||
params:{ type:'ambient', color:L.ambient.color, intensity:L.ambient.intensity } });
|
||||
stage.setParam(sun.id, 'color', L.sun.color);
|
||||
stage.setParam(sun.id, 'intensity', L.sun.intensity);
|
||||
stage.setParam(amb.id, 'color', L.ambient.color);
|
||||
stage.setParam(amb.id, 'intensity', L.ambient.intensity);
|
||||
stage.setParam(amb.id, 'bg', L.bg); // background rides the ambient entity → keyable + persists
|
||||
const e = L.sun.elev*D2R, a = L.sun.azim*D2R, R = 20; // 20m boom aimed at chest height
|
||||
const pos = [R*Math.cos(e)*Math.sin(a), R*Math.sin(e), R*Math.cos(e)*Math.cos(a)];
|
||||
stage.setTransform(sun.id, { pos, rot: lookAtEuler(pos, [0, 1, 0]), scale: 1 });
|
||||
return { preset: name, sun: { ...L.sun }, ambient: { ...L.ambient }, bg: L.bg,
|
||||
sunId: sun.id, ambientId: amb.id, sunPos: pos };
|
||||
}
|
||||
|
||||
// Blocking marks: pos marks move (stage-relative, keep current y), face marks yaw the subject
|
||||
// toward the active camera / the nearest other character.
|
||||
export function applyMark(stage, id, mark){
|
||||
const M = MARKS[mark];
|
||||
if(!M) throw new Error('unknown mark: ' + mark);
|
||||
const t = stage.entityTransform(id);
|
||||
if(!t) throw new Error('applyMark: no entity ' + id);
|
||||
const out = { mark, id };
|
||||
if(M.pos){
|
||||
const pos = [M.pos[0], t.pos[1], M.pos[2]];
|
||||
stage.setTransform(id, { pos }); out.pos = pos;
|
||||
}
|
||||
if(M.face){
|
||||
let target = null;
|
||||
if(M.face === 'camera') target = stage.cameraWorldPos();
|
||||
else {
|
||||
let best = Infinity;
|
||||
for(const e of stage.entities()){
|
||||
if(e.id === id || e.kind !== 'character') continue;
|
||||
const p = stage.entityTransform(e.id).pos;
|
||||
const d = Math.hypot(p[0]-t.pos[0], p[2]-t.pos[2]);
|
||||
if(d < best){ best = d; target = p; }
|
||||
}
|
||||
}
|
||||
if(target){
|
||||
const rot = [0, Math.atan2(target[0]-t.pos[0], target[2]-t.pos[2]), 0]; // face +Z toward target
|
||||
stage.setTransform(id, { rot }); out.rot = rot;
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// ---- op runner: ONE path for panel buttons and /director NL ops ----
|
||||
// shapes: {op:'shot', subject, shot, angle, focal, other?} {op:'light', preset}
|
||||
// {op:'mark', subject, mark} — subject/other resolve by entity id or label.
|
||||
function _resolve(stage, ref){
|
||||
if(ref == null) return null;
|
||||
if(stage.getEntity(ref)) return ref;
|
||||
const s = String(ref).toLowerCase();
|
||||
const en = stage.entities().find(e => (e.label || '').toLowerCase() === s);
|
||||
return en ? en.id : null;
|
||||
}
|
||||
export async function applyOps(stage, ops, selectedId = null){
|
||||
const applied = [];
|
||||
for(const o of (ops || [])){
|
||||
try {
|
||||
if(o.op === 'light'){
|
||||
const v = await applyLight(stage, o.preset);
|
||||
_fire({ op:'light', ...v, entityIds:[v.sunId, v.ambientId] });
|
||||
applied.push(v);
|
||||
} else if(o.op === 'mark'){
|
||||
const id = _resolve(stage, o.subject) ?? selectedId;
|
||||
if(!id) throw new Error('mark: no subject');
|
||||
const v = applyMark(stage, id, o.mark);
|
||||
_fire({ op:'mark', ...v, entityIds:[id] });
|
||||
applied.push(v);
|
||||
} else if(o.op === 'shot'){
|
||||
const subjectId = _resolve(stage, o.subject) ?? selectedId;
|
||||
if(!subjectId) throw new Error('shot: no subject');
|
||||
const A = ANGLES[o.angle];
|
||||
const otherId = _resolve(stage, o.other)
|
||||
?? ((A && A.ots) ? stage.entities().find(e => e.kind === 'character' && e.id !== subjectId)?.id : null);
|
||||
let cam = stage.entities().find(e => e.kind === 'camera' && e.label === 'shotcam')
|
||||
|| stage.entities().find(e => e.kind === 'camera');
|
||||
if(!cam) cam = await stage.addEntity({ kind:'camera', label:'shotcam', params:{ fov:45 } });
|
||||
const v = frameSubject(stage, cam.id, subjectId,
|
||||
{ shot:o.shot, angle:o.angle, focal:o.focal, otherId });
|
||||
stage.setActiveCamera(cam.id);
|
||||
_fire({ op:'shot', ...v, entityIds:[cam.id, subjectId] });
|
||||
applied.push(v);
|
||||
} else throw new Error('unknown op: ' + o.op);
|
||||
} catch(err){ applied.push({ error: String(err), op: o.op }); }
|
||||
}
|
||||
return applied;
|
||||
}
|
||||
|
||||
// ---- DIRECT panel UI (browser only — never imported by the tests) ----
|
||||
export function mountDirectPanel(stage, el){
|
||||
const st = { shot:'MS', angle:'eye', focal:35, sel:null };
|
||||
stage.onSelect(en => { st.sel = en ? en.id : null; });
|
||||
|
||||
el.innerHTML = '';
|
||||
const seg = (cls, title) => { const d = document.createElement('div');
|
||||
d.className = 'dseg ' + cls; if(title){ const s = document.createElement('span');
|
||||
s.className = 'dlab'; s.textContent = title; d.appendChild(s); } el.appendChild(d); return d; };
|
||||
const btn = (parent, text, cb, title) => { const b = document.createElement('button');
|
||||
b.textContent = text; if(title) b.title = title; b.onclick = cb; parent.appendChild(b); return b; };
|
||||
const note = m => { info.textContent = m; clearTimeout(note._t);
|
||||
if(m) note._t = setTimeout(() => info.textContent = '', 4000); };
|
||||
|
||||
const shots = seg('shots', 'shot');
|
||||
for(const k of Object.keys(SHOTS)) btn(shots, k, e => {
|
||||
st.shot = k; [...shots.querySelectorAll('button')].forEach(b => b.classList.toggle('on', b === e.target));
|
||||
}, SHOTS[k].label).classList.toggle('on', k === st.shot);
|
||||
|
||||
const angles = seg('angles', 'angle');
|
||||
for(const k of Object.keys(ANGLES)) btn(angles, k.replace('_',' '), e => {
|
||||
st.angle = k; [...angles.querySelectorAll('button')].forEach(b => b.classList.toggle('on', b === e.target));
|
||||
}).classList.toggle('on', k === st.angle);
|
||||
|
||||
const lens = seg('lens', 'lens');
|
||||
const sel = document.createElement('select');
|
||||
for(const f of FOCALS){ const o = document.createElement('option');
|
||||
o.value = f; o.textContent = f + 'mm'; if(f === st.focal) o.selected = true; sel.appendChild(o); }
|
||||
sel.onchange = () => st.focal = +sel.value; lens.appendChild(sel);
|
||||
btn(lens, '🎥 frame', async () => {
|
||||
const en = st.sel && stage.getEntity(st.sel);
|
||||
if(!en || (en.kind !== 'character' && en.kind !== 'prop')) return note('select a character/prop first');
|
||||
const [r] = await applyOps(stage, [{ op:'shot', subject:en.id, shot:st.shot, angle:st.angle, focal:st.focal }]);
|
||||
note(r.error ? String(r.error) : `${st.shot} ${st.angle} ${st.focal}mm`);
|
||||
}, 'solve + move the shot camera');
|
||||
|
||||
const lights = seg('lights', 'light');
|
||||
for(const k of Object.keys(LIGHTS)) btn(lights, k.replace('_',' '), async () => {
|
||||
const [r] = await applyOps(stage, [{ op:'light', preset:k }]);
|
||||
note(r.error ? String(r.error) : 'light: ' + k);
|
||||
});
|
||||
|
||||
const marks = seg('marks', 'mark');
|
||||
for(const k of Object.keys(MARKS)) btn(marks, k.replace(/_/g,' '), async () => {
|
||||
if(!st.sel) return note('select an entity first');
|
||||
const [r] = await applyOps(stage, [{ op:'mark', subject:st.sel, mark:k }], st.sel);
|
||||
note(r.error ? String(r.error) : 'mark: ' + k);
|
||||
});
|
||||
|
||||
// NL box → POST /director (Lane C, LLM-optional). Hidden until the endpoint exists;
|
||||
// hides itself again on 503 (LLM unset). Buttons above never depend on this lane.
|
||||
const nl = seg('nl', null); nl.style.display = 'none';
|
||||
const inp = document.createElement('input');
|
||||
inp.placeholder = '“medium two-shot, golden hour…”'; nl.appendChild(inp);
|
||||
const send = async () => {
|
||||
const text = inp.value.trim(); if(!text) return;
|
||||
note('directing…');
|
||||
try {
|
||||
const r = await fetch('/director', { method:'POST', headers:{ 'Content-Type':'application/json' },
|
||||
body: JSON.stringify({ text, selected: st.sel,
|
||||
entities: stage.entities().map(e => ({ id:e.id, kind:e.kind, label:e.label })) }) });
|
||||
if(r.status === 503){ nl.style.display = 'none'; return note('director LLM offline'); }
|
||||
if(!r.ok) return note('director: HTTP ' + r.status);
|
||||
const { ops } = await r.json();
|
||||
const res = await applyOps(stage, ops, st.sel);
|
||||
const bad = res.filter(x => x.error);
|
||||
note(bad.length ? bad[0].error : `applied ${res.length} op${res.length === 1 ? '' : 's'}`);
|
||||
if(!bad.length) inp.value = '';
|
||||
} catch(e){ note('director: ' + String(e).slice(0, 80)); }
|
||||
};
|
||||
btn(nl, '➤', send); inp.addEventListener('keydown', e => { if(e.key === 'Enter') send(); });
|
||||
// probe: FastAPI answers 405 on GET when only POST exists → endpoint present, show the box
|
||||
fetch('/director').then(r => { if(r.status === 405 || r.ok) nl.style.display = ''; }).catch(() => {});
|
||||
|
||||
const info = document.createElement('span'); info.className = 'dinfo'; el.appendChild(info);
|
||||
return { state: st };
|
||||
}
|
||||
61
scenegod/web/presets_test.mjs
Normal file
61
scenegod/web/presets_test.mjs
Normal file
@ -0,0 +1,61 @@
|
||||
// presets_test.mjs — Lane A M5 self-check. Runs under plain `node`, no deps, no three.js
|
||||
// (presets.js takes bboxes as plain arrays; expected values below are computed inline).
|
||||
// node scenegod/web/presets_test.mjs
|
||||
import assert from 'node:assert/strict';
|
||||
import { LIGHTS, SHOTS } from './grammar.js';
|
||||
import { fovForFocal, solveShot, applyLight } from './presets.js';
|
||||
|
||||
const near = (a, b, eps = 1e-9) => Math.abs(a - b) <= eps;
|
||||
const R2D = 180 / Math.PI;
|
||||
|
||||
// ---- fov from focal: 2·atan(12/f) full-frame vertical ----
|
||||
for(const [mm, want] of [[18, 2*Math.atan(12/18)*R2D], [35, 2*Math.atan(12/35)*R2D], [85, 2*Math.atan(12/85)*R2D]])
|
||||
assert.ok(near(fovForFocal(mm), want, 1e-9), `fov(${mm}mm) = ${fovForFocal(mm)} != ${want}`);
|
||||
assert.ok(near(fovForFocal(35), 37.849, 0.001), '35mm ≈ 37.849° vertical');
|
||||
|
||||
// ---- shot distance: CU frames tighter than MS on a 1.7m subject → shorter camera distance ----
|
||||
const bbox = { bboxMin: [-0.3, 0, -0.2], bboxMax: [0.3, 1.7, 0.2] };
|
||||
const cu = solveShot({ ...bbox, shot: 'CU', focal: 35 });
|
||||
const ms = solveShot({ ...bbox, shot: 'MS', focal: 35 });
|
||||
assert.ok(cu.dist < ms.dist, `CU dist ${cu.dist} !< MS dist ${ms.dist}`);
|
||||
// exact: dist = (span·H/2) / tan(fovV/2)
|
||||
const expDist = s => (SHOTS[s].span * 1.7 / 2) / Math.tan(Math.atan(12/35));
|
||||
assert.ok(near(cu.dist, expDist('CU'), 1e-9) && near(ms.dist, expDist('MS'), 1e-9), 'dist formula');
|
||||
// aim height sits at the shot's center fraction of subject height
|
||||
assert.ok(near(cu.aim[1], 0.87 * 1.7, 1e-9), 'CU centers at .87H');
|
||||
|
||||
// ---- dutch angle sets roll (euler z), straight-on shot keeps yaw/pitch 0 ----
|
||||
const dutch = solveShot({ ...bbox, shot: 'MS', angle: 'dutch', focal: 35 });
|
||||
assert.ok(near(dutch.rot[2], 8 * Math.PI/180, 1e-9), `dutch roll ${dutch.rot[2]*R2D}° != 8°`);
|
||||
assert.ok(near(dutch.rot[0], 0, 1e-9) && near(dutch.rot[1], 0, 1e-9), 'dutch: no stray pitch/yaw');
|
||||
const eye = solveShot({ ...bbox, shot: 'MS', angle: 'eye', focal: 35 });
|
||||
assert.ok(near(eye.rot[2], 0, 1e-9), 'eye level has no roll');
|
||||
// low angle puts the camera below the aim point, looking up (positive pitch)
|
||||
const low = solveShot({ ...bbox, shot: 'MS', angle: 'low', focal: 35 });
|
||||
assert.ok(low.pos[1] < low.aim[1] && low.rot[0] > 0, 'low angle: camera below aim, pitched up');
|
||||
|
||||
// ---- applyLight('golden_hour') returns the grammar.js values verbatim ----
|
||||
const calls = [];
|
||||
const stub = {
|
||||
_ents: [],
|
||||
entities(){ return this._ents; },
|
||||
async addEntity(d){ const en = { id: 'e' + (this._ents.length + 1), kind: d.kind,
|
||||
label: d.label, params: { ...d.params } }; this._ents.push(en); return en; },
|
||||
setParam(id, k, v){ calls.push(['param', id, k, v]); },
|
||||
setTransform(id, t){ calls.push(['tf', id, t]); },
|
||||
};
|
||||
const g = LIGHTS.golden_hour;
|
||||
const v = await applyLight(stub, 'golden_hour');
|
||||
assert.deepEqual(v.sun, g.sun, 'sun values from grammar');
|
||||
assert.deepEqual(v.ambient, g.ambient, 'ambient values from grammar');
|
||||
assert.equal(v.bg, g.bg, 'bg from grammar');
|
||||
assert.ok(v.sunId && v.ambientId, 'sun + ambient entities created');
|
||||
assert.ok(calls.some(c => c[0] === 'param' && c[1] === v.sunId && c[2] === 'intensity' && c[3] === g.sun.intensity),
|
||||
'sun intensity applied to stage');
|
||||
assert.ok(calls.some(c => c[0] === 'param' && c[1] === v.ambientId && c[2] === 'bg' && c[3] === g.bg),
|
||||
'bg applied via ambient entity');
|
||||
// second call reuses the same entities (no light spam)
|
||||
await applyLight(stub, 'noir');
|
||||
assert.equal(stub._ents.length, 2, 'presets reuse the sun/ambient pair');
|
||||
|
||||
console.log('presets_test: ALL PASS');
|
||||
@ -39,6 +39,7 @@ export async function finalRender(stage, timeline, opts = {}) {
|
||||
const inflight = new Set();
|
||||
for (let f = 0; f < total; f++) {
|
||||
timeline.step(f); // seek(f/fps)+evaluate — never wall-clock
|
||||
if (stage.syncVideos) await stage.syncVideos(timeline.time); // M6 video plates: seek + wait 'seeked'; no-op until Lane A lands it
|
||||
stage.renderActiveCamera(null); // draw active cam (or director cam) to main canvas
|
||||
const blob = await new Promise(res => canvas.toBlob(res, 'image/png'));
|
||||
while (inflight.size >= 4) await Promise.race(inflight); // backpressure: <=4 in flight
|
||||
|
||||
@ -151,7 +151,7 @@ export class Stage {
|
||||
const noAsset = desc.kind === 'camera' || desc.kind === 'light'; // rig entities have no file
|
||||
const en = { id, kind: desc.kind, label: desc.label || desc.kind,
|
||||
source: desc.source || (noAsset ? {type:'none'} : null), params: {...(desc.params||{})},
|
||||
wrapper, root:null, mixer:null, rest:null, cam:null, light:null, morphs:null, visemes:null };
|
||||
wrapper, root:null, mixer:null, rest:null, cam:null, light:null, morphs:null, visemes:null, video:null };
|
||||
|
||||
if(desc.kind === 'character' || desc.kind === 'prop'){
|
||||
const { root } = await this._loadAsset(desc);
|
||||
@ -162,6 +162,8 @@ export class Stage {
|
||||
const v = detectVisemes(root); en.morphs = v.morphs; en.visemes = v.visemes; }
|
||||
} else if(desc.kind === 'backdrop'){
|
||||
en.root = await this._buildBackdrop(en); wrapper.add(en.root);
|
||||
} else if(desc.kind === 'screen'){
|
||||
en.root = await this._buildScreen(en); wrapper.add(en.root);
|
||||
} else if(desc.kind === 'camera'){
|
||||
const fov = en.params.fov || 45;
|
||||
en.cam = new THREE.PerspectiveCamera(fov, 16/9, 0.05, 3000);
|
||||
@ -183,6 +185,7 @@ export class Stage {
|
||||
removeEntity(id){
|
||||
const en = this._entities.get(id); if(!en) return;
|
||||
if(this._selected === id) this.select(null);
|
||||
this._dropVideo(en);
|
||||
if(en.root) disposeRoot(en.root);
|
||||
if(en.light) en.light.parent && en.light.parent.remove(en.light);
|
||||
this.scene.remove(en.wrapper);
|
||||
@ -221,39 +224,106 @@ export class Stage {
|
||||
root.position.y -= b.min.y; // feet on the floor
|
||||
}
|
||||
|
||||
// Backdrop plate — image OR video (M6). params.mode: plane|corner|dome|video ('video' = a
|
||||
// plane; the other modes also accept .mp4 sources). Video plates show their poster jpg (the
|
||||
// server's same-stem sidecar) until first play/seek, then swap to the live VideoTexture.
|
||||
async _buildBackdrop(en){
|
||||
const p = en.params;
|
||||
const url = en.source
|
||||
? (en.source.type === 'upload' ? en.source._url : ASSET_URL(p.image || en.source.path))
|
||||
: ASSET_URL(p.image);
|
||||
const tex = await new THREE.TextureLoader().loadAsync(url);
|
||||
tex.colorSpace = THREE.SRGBColorSpace;
|
||||
const aspect = (tex.image.width / tex.image.height) || 1;
|
||||
const w = p.width || 10, h = w / aspect;
|
||||
const mat = new THREE.MeshStandardMaterial({map:tex, roughness:1, metalness:0});
|
||||
const grp = new THREE.Group();
|
||||
if(p.mode === 'dome'){
|
||||
const dome = new THREE.Mesh(new THREE.SphereGeometry(40, 40, 24),
|
||||
new THREE.MeshBasicMaterial({map:tex, side:THREE.BackSide}));
|
||||
grp.add(dome);
|
||||
const src = p.image || (en.source && en.source.path) || '';
|
||||
const upload = en.source && en.source.type === 'upload';
|
||||
const url = upload ? en.source._url : ASSET_URL(src);
|
||||
const isVideo = p.mode === 'video' || /\.(mp4|webm|mov)$/i.test(src);
|
||||
let tex, aspect, vidTex = null, poster = null;
|
||||
if(isVideo){
|
||||
const { video, texture } = await this._makeVideo(url, src);
|
||||
en.video = video; en.videoTex = vidTex = texture;
|
||||
aspect = (video.videoWidth / video.videoHeight) || 16/9;
|
||||
poster = upload ? null : await this._posterTex(src);
|
||||
tex = poster || vidTex;
|
||||
} else {
|
||||
tex = await new THREE.TextureLoader().loadAsync(url);
|
||||
tex.colorSpace = THREE.SRGBColorSpace;
|
||||
aspect = (tex.image.width / tex.image.height) || 1;
|
||||
}
|
||||
const w = p.width || 10, h = w / aspect;
|
||||
const grp = new THREE.Group(), mats = [];
|
||||
if(p.mode === 'dome'){
|
||||
const dm = new THREE.MeshBasicMaterial({map:tex, side:THREE.BackSide}); mats.push(dm);
|
||||
grp.add(new THREE.Mesh(new THREE.SphereGeometry(40, 40, 24), dm));
|
||||
} else {
|
||||
const mat = new THREE.MeshStandardMaterial({map:tex, roughness:1, metalness:0}); mats.push(mat);
|
||||
const wall = new THREE.Mesh(new THREE.PlaneGeometry(w, h), mat);
|
||||
wall.position.y = h/2; wall.receiveShadow = true; grp.add(wall);
|
||||
if(p.mode === 'corner'){ // cheap "L": duplicate lower third laid flat as ground
|
||||
const gh = h/3;
|
||||
const gm = new THREE.Mesh(new THREE.PlaneGeometry(w, gh), mat.clone());
|
||||
gm.rotation.x = -Math.PI/2; gm.position.z = gh/2; gm.receiveShadow = true; grp.add(gm);
|
||||
const gh = h/3, gm = mat.clone(); mats.push(gm);
|
||||
const g = new THREE.Mesh(new THREE.PlaneGeometry(w, gh), gm);
|
||||
g.rotation.x = -Math.PI/2; g.position.z = gh/2; g.receiveShadow = true; grp.add(g);
|
||||
wall.position.z = -gh/2;
|
||||
}
|
||||
}
|
||||
if(vidTex && poster) this._swapOnPlay(en.video, vidTex, mats);
|
||||
if(en.video && this._mixerAuto) en.video.play().catch(()=>{}); // ambient preview until a timeline owns the clock
|
||||
return grp;
|
||||
}
|
||||
|
||||
// In-set display (M6 `screen` kind): a video plane with an emissive boost so TVs/club
|
||||
// screens/jumbotrons read "on" under any lighting preset. params {video, width, emissive}.
|
||||
// ponytail: CRT curve skipped — flat plane until a real TV asset asks for it.
|
||||
async _buildScreen(en){
|
||||
const p = en.params;
|
||||
const src = p.video || (en.source && en.source.path) || '';
|
||||
const upload = en.source && en.source.type === 'upload';
|
||||
const { video, texture } = await this._makeVideo(upload ? en.source._url : ASSET_URL(src), src);
|
||||
en.video = video; en.videoTex = texture;
|
||||
const aspect = (video.videoWidth / video.videoHeight) || 16/9;
|
||||
const poster = upload ? null : await this._posterTex(src);
|
||||
const tex = poster || texture;
|
||||
const w = p.width || 3, h = w / aspect;
|
||||
const mat = new THREE.MeshStandardMaterial({ map:tex, emissive:0xffffff, emissiveMap:tex,
|
||||
emissiveIntensity: p.emissive ?? 0.6, roughness:0.4, metalness:0 });
|
||||
if(poster) this._swapOnPlay(video, texture, [mat]);
|
||||
if(this._mixerAuto) video.play().catch(()=>{});
|
||||
const mesh = new THREE.Mesh(new THREE.PlaneGeometry(w, h), mat);
|
||||
mesh.position.y = h/2;
|
||||
const grp = new THREE.Group(); grp.add(mesh);
|
||||
return grp;
|
||||
}
|
||||
|
||||
// <video> element + VideoTexture, resolved once metadata (dimensions/duration) is in
|
||||
_makeVideo(url, srcPath){
|
||||
return new Promise((resolve, reject) => {
|
||||
const v = document.createElement('video');
|
||||
v.muted = true; v.loop = true; v.playsInline = true; v.preload = 'auto'; v.crossOrigin = 'anonymous';
|
||||
v.src = url;
|
||||
const texture = new THREE.VideoTexture(v);
|
||||
texture.colorSpace = THREE.SRGBColorSpace;
|
||||
v.addEventListener('loadedmetadata', () => resolve({ video: v, texture }), { once:true });
|
||||
v.addEventListener('error', () => reject(new Error('video load failed: ' + srcPath)), { once:true });
|
||||
});
|
||||
}
|
||||
_posterTex(src){ // same-stem jpg via the server's sidecar route; null when absent
|
||||
return new THREE.TextureLoader().loadAsync('/assets/thumb?path=' + encodeURIComponent(src))
|
||||
.then(t => { t.colorSpace = THREE.SRGBColorSpace; return t; })
|
||||
.catch(() => null);
|
||||
}
|
||||
_swapOnPlay(video, texture, mats){ // poster stays up until the video actually shows a frame
|
||||
const swap = () => {
|
||||
video.removeEventListener('playing', swap); video.removeEventListener('seeked', swap);
|
||||
for(const m of mats){ m.map = texture; if(m.emissiveMap) m.emissiveMap = texture; m.needsUpdate = true; }
|
||||
};
|
||||
video.addEventListener('playing', swap); video.addEventListener('seeked', swap);
|
||||
}
|
||||
_dropVideo(en){
|
||||
if(!en.video) return;
|
||||
en.video.pause(); en.video.removeAttribute('src'); en.video.load(); en.video = null;
|
||||
}
|
||||
|
||||
_buildLight(en){
|
||||
const p = en.params;
|
||||
if(p.type === 'ambient'){
|
||||
en.light = new THREE.HemisphereLight(new THREE.Color(p.color||'#ffffff'), 0x444450, p.intensity ?? 1.0);
|
||||
en.wrapper.add(en.light);
|
||||
if(p.bg) this.scene.background = new THREE.Color(p.bg); // saved lighting preset restores its backdrop color
|
||||
} else {
|
||||
const dir = en.light = new THREE.DirectionalLight(new THREE.Color(p.color||'#ffffff'), p.intensity ?? 1.5);
|
||||
dir.castShadow = p.castShadow ?? true;
|
||||
@ -320,13 +390,22 @@ export class Stage {
|
||||
if(key === 'intensity') en.light.intensity = value;
|
||||
if(key === 'color') en.light.color.set(value);
|
||||
if(key === 'castShadow' && en.light.isDirectionalLight) en.light.castShadow = !!value;
|
||||
if(key === 'bg') this.scene.background = new THREE.Color(value); // bg rides the ambient entity (keyable, persists)
|
||||
}
|
||||
const rebuild = () => this._rebuild(en).catch(e => console.warn('media rebuild failed:', en.id, e));
|
||||
if(en.kind === 'backdrop' && (key === 'mode' || key === 'image' || key === 'width'))
|
||||
this._rebuildBackdrop(en); // geometry depends on these — rebuild the mesh
|
||||
rebuild(); // geometry depends on these — rebuild the mesh
|
||||
if(en.kind === 'screen'){
|
||||
if(key === 'emissive' && en.root)
|
||||
en.root.traverse(o => { if(o.isMesh) o.material.emissiveIntensity = value; });
|
||||
if(key === 'video' || key === 'width') rebuild();
|
||||
}
|
||||
async _rebuildBackdrop(en){
|
||||
// videoStart is data for the timeline/syncVideos — nothing to rebuild
|
||||
}
|
||||
async _rebuild(en){ // backdrop/screen media rebuild (mode/image/video/width changed)
|
||||
const old = en.root;
|
||||
en.root = await this._buildBackdrop(en);
|
||||
this._dropVideo(en);
|
||||
en.root = en.kind === 'screen' ? await this._buildScreen(en) : await this._buildBackdrop(en);
|
||||
en.wrapper.add(en.root);
|
||||
if(old){ en.wrapper.remove(old); disposeRoot(old); }
|
||||
}
|
||||
@ -407,6 +486,44 @@ export class Stage {
|
||||
pause(){ this._paused = true; }
|
||||
resume(){ this._paused = false; this.clock.getDelta(); } // swallow the paused gap so mixers don't jump
|
||||
|
||||
// ---- director-mode helpers (M5 presets.js solvers are three-free — plain arrays) ----
|
||||
entityBBox(id){
|
||||
const en = this._entities.get(id); if(!en) return null;
|
||||
en.wrapper.updateMatrixWorld(true);
|
||||
const box = new THREE.Box3().setFromObject(en.wrapper);
|
||||
return (box.isEmpty() || !isFinite(box.min.x)) ? null : { min: box.min.toArray(), max: box.max.toArray() };
|
||||
}
|
||||
cameraWorldPos(){ // active camera if set, else the director orbit cam
|
||||
const p = new THREE.Vector3(); this._activeCam().getWorldPosition(p); return p.toArray();
|
||||
}
|
||||
|
||||
// ---- video plates (M6): master-clock sync + deterministic render seeks ----
|
||||
videos(){ return this.entities().filter(e => e.video)
|
||||
.map(e => ({ id: e.id, video: e.video, start: e.params.videoStart || 0 })); }
|
||||
entityVideo(id){ return this._entities.get(id)?.video || null; } // SYNC6: Lane B's per-entity seam
|
||||
setVideoPlaying(on){ for(const { video } of this.videos()) on ? video.play().catch(()=>{}) : video.pause(); }
|
||||
// Seek every stage video to timeline time t and resolve once all fire 'seeked' —
|
||||
// render.js awaits this per frame so plates are frame-exact, never mid-decode. Hard contract.
|
||||
async syncVideos(t){
|
||||
const jobs = [];
|
||||
// SYNC6: r160 VideoTexture auto-updates via requestVideoFrameCallback, which never
|
||||
// fires in a hidden/background tab — force the GPU upload or renders show black plates.
|
||||
for(const en of this._entities.values()) if(en.videoTex) en.videoTex.needsUpdate = true;
|
||||
for(const { video, start } of this.videos()){
|
||||
if(video.readyState < 1 || !isFinite(video.duration) || video.duration <= 0) continue;
|
||||
const vt = (((t - start) % video.duration) + video.duration) % video.duration;
|
||||
if(!video.paused) video.pause();
|
||||
if(Math.abs(video.currentTime - vt) < 0.004) continue; // already on this frame
|
||||
jobs.push(new Promise(res => {
|
||||
const done = () => { video.removeEventListener('seeked', done); res(); };
|
||||
video.addEventListener('seeked', done);
|
||||
setTimeout(done, 1500); // ponytail: never hang a render on a stuck decoder
|
||||
video.currentTime = vt;
|
||||
}));
|
||||
}
|
||||
await Promise.all(jobs);
|
||||
}
|
||||
|
||||
// ---- pointer→world helpers (drag-from-dock placement) ----
|
||||
screenToFloor(clientX, clientY){
|
||||
const rect = this.renderer.domElement.getBoundingClientRect();
|
||||
@ -437,9 +554,9 @@ export class Stage {
|
||||
async applyState(sceneJson){
|
||||
for(const en of this.entities()) this.removeEntity(en.id);
|
||||
for(const d of (sceneJson.entities || [])){
|
||||
// skip only what needs asset BYTES we don't have: an 'upload' mesh or backdrop image.
|
||||
// skip only what needs asset BYTES we don't have: an 'upload' mesh, backdrop or screen.
|
||||
// camera/light use source:{type:'none'} (legacy scenes: 'upload') and always rebuild.
|
||||
const needsBytes = d.kind === 'character' || d.kind === 'prop' || d.kind === 'backdrop';
|
||||
const needsBytes = d.kind === 'character' || d.kind === 'prop' || d.kind === 'backdrop' || d.kind === 'screen';
|
||||
if(d.source && d.source.type === 'upload' && needsBytes){
|
||||
console.warn('skipping upload entity (no bytes):', d.id); continue; }
|
||||
try { await this.addEntity(d); } catch(err){ console.error('addEntity failed', d.id, err); }
|
||||
|
||||
@ -17,6 +17,16 @@ class StubAction {
|
||||
reset() { this.time = 0; return this; }
|
||||
}
|
||||
|
||||
// M6 seam: fake HTMLVideoElement for video backdrops/screens. Records every
|
||||
// currentTime assignment so tests can assert "seek only when drift >50ms".
|
||||
class StubVideo {
|
||||
constructor(duration) { this.duration = duration; this.paused = true; this.seeks = []; this._t = 0; }
|
||||
get currentTime() { return this._t; }
|
||||
set currentTime(v) { this._t = v; this.seeks.push(v); }
|
||||
play() { this.paused = false; return Promise.resolve(); } // real video.play() → Promise
|
||||
pause() { this.paused = true; }
|
||||
}
|
||||
|
||||
class StubMixer {
|
||||
constructor(id, stage) { this.id = id; this.stage = stage; this._actions = new Map(); }
|
||||
clipAction(clip) {
|
||||
@ -38,6 +48,7 @@ export class StageStub {
|
||||
this.viewportEl = viewportEl || null;
|
||||
this._entities = new Map();
|
||||
this._mixers = new Map();
|
||||
this._videos = new Map();
|
||||
this._clipCache = new Map();
|
||||
this._changeCbs = [];
|
||||
this._selectCbs = [];
|
||||
@ -51,12 +62,18 @@ export class StageStub {
|
||||
if (!e.transform) e.transform = { pos: [0, 0, 0], rot: [0, 0, 0], scale: 1 };
|
||||
if (!e.params) e.params = {};
|
||||
this._entities.set(e.id, e);
|
||||
// M6: video backdrops/screens get a StubVideo (kept OUT of the entity
|
||||
// object so captureState stays clean). videoDuration is a stub-only knob.
|
||||
if (e.kind === 'screen' || (e.kind === 'backdrop' &&
|
||||
((e.params && e.params.mode === 'video') || /\.mp4$/i.test((e.params && e.params.image) || '')))) {
|
||||
this._videos.set(e.id, new StubVideo((e.params && e.params.videoDuration) || 5));
|
||||
}
|
||||
this._fireChange(e);
|
||||
return e;
|
||||
}
|
||||
removeEntity(id) {
|
||||
const e = this._entities.get(id);
|
||||
this._entities.delete(id); this._mixers.delete(id);
|
||||
this._entities.delete(id); this._mixers.delete(id); this._videos.delete(id);
|
||||
this._fireChange(e || { id }); // fire WITH the removed entity; getEntity(id) now null (real Stage contract)
|
||||
}
|
||||
getEntity(id) { return this._entities.get(id) || null; }
|
||||
@ -101,9 +118,13 @@ export class StageStub {
|
||||
setMorph(id, name, weight) { this.applied.push({ type: 'morph', id, name, weight }); console.debug('stub.setMorph', id, name, weight); }
|
||||
visemeTargets(id) { const e = this._entities.get(id); return (e && e.visemes) || []; }
|
||||
|
||||
// M6 seam (real Stage: Lane A): the HTMLVideoElement behind a video
|
||||
// backdrop/screen, or null (image backdrop, other kinds, not loaded yet).
|
||||
entityVideo(id) { return this._videos.get(id) || null; }
|
||||
|
||||
captureState() { return this.entities().map(clone); }
|
||||
async applyState(scene) {
|
||||
this._entities.clear();
|
||||
this._entities.clear(); this._mixers.clear(); this._videos.clear();
|
||||
for (const e of (scene.entities || [])) await this.addEntity(e);
|
||||
}
|
||||
}
|
||||
|
||||
@ -81,5 +81,27 @@ body.hot::after{content:'drop to load';position:fixed;inset:0;display:flex;align
|
||||
.addbar button:hover{border-color:var(--teal)}
|
||||
.card[draggable]{cursor:grab}
|
||||
|
||||
/* DIRECT panel — cinema-grammar one-click presets (M5) */
|
||||
#direct{flex:0 0 auto;display:flex;align-items:center;gap:14px;padding:6px 12px;
|
||||
border-bottom:1px solid var(--line);background:var(--bg2);overflow-x:auto;white-space:nowrap}
|
||||
#direct:empty{display:none}
|
||||
.dseg{display:flex;align-items:center;gap:4px}
|
||||
.dlab{font-family:var(--mono);font-size:9.5px;color:var(--mut);text-transform:uppercase;letter-spacing:.08em;margin-right:2px}
|
||||
.dseg button{background:var(--panel);border:1px solid var(--line2);color:var(--ink);border-radius:6px;
|
||||
padding:4px 8px;font-family:var(--mono);font-size:10.5px;cursor:pointer}
|
||||
.dseg button:hover{border-color:var(--teal)}
|
||||
.dseg button.on{background:var(--teal);border-color:var(--teal);color:#0c0f12;font-weight:700}
|
||||
.dseg select{background:var(--panel);border:1px solid var(--line2);color:var(--ink);border-radius:6px;
|
||||
padding:4px 6px;font-family:var(--mono);font-size:10.5px}
|
||||
.dseg.nl input{background:var(--panel);border:1px solid var(--line2);color:var(--ink);border-radius:6px;
|
||||
padding:4px 8px;font-family:var(--mono);font-size:11px;width:230px}
|
||||
.dinfo{font-family:var(--mono);font-size:10.5px;color:var(--teal);margin-left:auto;padding-left:10px}
|
||||
|
||||
/* M6 video plates: 📺 add-as-screen button on dock cards */
|
||||
.card{position:relative}
|
||||
.card .tv{position:absolute;top:4px;right:4px;background:rgba(16,19,23,.82);border:1px solid var(--line2);
|
||||
border-radius:6px;font-size:13px;padding:2px 5px;cursor:pointer}
|
||||
.card .tv:hover{border-color:var(--teal)}
|
||||
|
||||
/* === LANE B === */
|
||||
/* Lane B appends timeline styling below this marker */
|
||||
|
||||
@ -98,6 +98,11 @@ export class Timeline {
|
||||
window.addEventListener('scenegod:clipdrop', (e) => {
|
||||
if (e.detail && e.detail.id) this.addClipDrop(e.detail);
|
||||
});
|
||||
// M5 DIRECTOR MODE: Lane A presets dispatch ops → normal keys/cuts at
|
||||
// the playhead (contract: lanes/A §M5.3 / PLAN §5 M5).
|
||||
window.addEventListener('scenegod:direct', (e) => {
|
||||
if (e.detail && e.detail.op) this.applyDirect(e.detail);
|
||||
});
|
||||
}
|
||||
// SYNC2 seam: mirror live stage adds/removes into scene.entities so
|
||||
// dock-added entities can be keyframed without a scene load.
|
||||
@ -208,7 +213,12 @@ export class Timeline {
|
||||
};
|
||||
this._raf = requestAnimationFrame(loop);
|
||||
}
|
||||
pause() { this.playing = false; if (this._raf) cancelAnimationFrame(this._raf); this._raf = null; }
|
||||
pause() {
|
||||
this.playing = false;
|
||||
if (this._raf) cancelAnimationFrame(this._raf);
|
||||
this._raf = null;
|
||||
if (this.stage.entityVideo) this.evaluate(this.time); // M6: videos pause with the clock
|
||||
}
|
||||
seek(t) {
|
||||
this.time = Math.max(0, Math.min(this.duration, t));
|
||||
this.evaluate(this.time);
|
||||
@ -239,10 +249,27 @@ export class Timeline {
|
||||
this._evalParams(e, t);
|
||||
if (e.tracks && e.tracks.morphs) this._evalMorphs(e, t);
|
||||
if (e.tracks && e.tracks.clips) this._evalClips(e, t);
|
||||
if ((e.kind === 'backdrop' || e.kind === 'screen') && this.stage.entityVideo) this._evalVideo(e, t);
|
||||
}
|
||||
this._evalCameraCuts(t);
|
||||
}
|
||||
|
||||
// M6 FLOW VIDEO PLATES: video backdrops/screens follow the master clock.
|
||||
// Cheap scrub-follow — seek only when drift >50ms; play/pause with the
|
||||
// clock. NEVER awaits (step() stays sync — the deterministic render path
|
||||
// awaits stage.syncVideos(t) per frame instead, Lane A/C's seam).
|
||||
_evalVideo(e, t) {
|
||||
const v = this.stage.entityVideo(e.id);
|
||||
if (!v) return; // nullable: image backdrop / not loaded
|
||||
const dur = v.duration;
|
||||
if (Number.isFinite(dur) && dur > 0) {
|
||||
const want = Math.max(0, t - ((e.params && e.params.videoStart) || 0)) % dur;
|
||||
if (Math.abs(v.currentTime - want) > 0.05) v.currentTime = want;
|
||||
}
|
||||
if (this.playing && v.paused) { const p = v.play(); if (p && p.catch) p.catch(() => {}); }
|
||||
else if (!this.playing && !v.paused) v.pause();
|
||||
}
|
||||
|
||||
_evalMorphs(e, t) { // viseme/blendshape weights → stage.setMorph (M4-B)
|
||||
const keys = e.tracks && e.tracks.morphs;
|
||||
if (!keys || !keys.length || !this.stage.setMorph) return;
|
||||
@ -392,6 +419,73 @@ export class Timeline {
|
||||
}
|
||||
undo() { const op = this.undoStack.pop(); if (op) op.undo(); this.evaluate(this.time); }
|
||||
|
||||
// Run fn (which may push many undo entries) and collapse everything it
|
||||
// pushed into ONE undo entry — a multi-key preset op reverts in one Ctrl+Z.
|
||||
group(fn) {
|
||||
const n = this.undoStack.length;
|
||||
const r = fn();
|
||||
if (this.undoStack.length > n + 1) {
|
||||
const ops = this.undoStack.splice(n);
|
||||
this.undoStack.push({ undo: () => { for (let i = ops.length - 1; i >= 0; i--) ops[i].undo(); } });
|
||||
}
|
||||
return r;
|
||||
}
|
||||
|
||||
// ---- M5 DIRECTOR MODE: scenegod:direct ops → keys/cuts at the playhead.
|
||||
// Accepts BOTH Lane A's live presets.js payloads (values spread at top level:
|
||||
// shot {camId, pos, rot, fov, …, entityIds:[camId, subjectId]}
|
||||
// light {sun:{color,intensity}, ambient:{…}, bg, sunId, ambientId}
|
||||
// mark {id, pos?, rot?, entityIds:[id]} )
|
||||
// and the PLAN §5 contract shapes ({transform:{pos,rot}}, {params:{k:v}},
|
||||
// walkTo). One op = ONE undo entry (group()).
|
||||
applyDirect(d) {
|
||||
const t = this.time;
|
||||
const key = (id, k) => this.addKey(id, 'params', { t, key: k[0], value: k[1], ease: 'inout' });
|
||||
this.group(() => {
|
||||
if (d.op === 'shot') {
|
||||
const camId = d.camId ?? (d.entityIds && d.entityIds[0]);
|
||||
const cur = this.stage.entityTransform(camId);
|
||||
const tr = d.transform || d; // Lane A spreads pos/rot at top level
|
||||
this.addKey(camId, 'transform', {
|
||||
t, pos: tr.pos || cur.pos, rot: tr.rot || cur.rot,
|
||||
scale: tr.scale ?? cur.scale ?? 1, ease: 'inout',
|
||||
});
|
||||
if (d.fov != null) key(camId, ['fov', d.fov]);
|
||||
this.addCut(t, camId);
|
||||
} else if (d.op === 'light') {
|
||||
if (d.sunId != null && d.sun) { // Lane A applyLight shape
|
||||
key(d.sunId, ['color', d.sun.color]);
|
||||
key(d.sunId, ['intensity', d.sun.intensity]);
|
||||
// sun boom pos/aim moved per preset — capture it so presets scrub
|
||||
this.addKey(d.sunId, 'transform', { t, ...this.stage.entityTransform(d.sunId), ease: 'inout' });
|
||||
if (d.ambientId != null && d.ambient) {
|
||||
key(d.ambientId, ['color', d.ambient.color]);
|
||||
key(d.ambientId, ['intensity', d.ambient.intensity]);
|
||||
if (d.bg != null) key(d.ambientId, ['bg', d.bg]);
|
||||
}
|
||||
} else { // generic: entityId(s) + params map
|
||||
const ids = d.entityIds || (d.entityId != null ? [d.entityId] : []);
|
||||
for (const id of ids) for (const kv of Object.entries(d.params || {})) key(id, kv);
|
||||
}
|
||||
} else if (d.op === 'mark') {
|
||||
const ids = d.entityIds || (d.entityId != null ? [d.entityId] : (d.id != null ? [d.id] : []));
|
||||
for (const id of ids) {
|
||||
const cur = this.stage.entityTransform(id);
|
||||
const tr = d.transform || d; // Lane A spreads pos/rot at top level
|
||||
const target = { pos: tr.pos || cur.pos, rot: tr.rot || cur.rot, scale: tr.scale ?? cur.scale ?? 1 };
|
||||
if (d.walkTo) { // walk: from prevPos (or current) → mark over 2s
|
||||
const from = d.prevPos ? { ...cur, pos: d.prevPos } : cur;
|
||||
this.addKey(id, 'transform', { t, ...from, ease: 'inout' });
|
||||
this.addKey(id, 'transform', { t: t + 2, ...target, ease: 'inout' });
|
||||
} else {
|
||||
this.addKey(id, 'transform', { t, ...target, ease: 'inout' });
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
this.seek(t); // re-evaluate: cut/keys take effect now
|
||||
}
|
||||
|
||||
// ---- audio track (M4-B). Clips are {path, start, gain}; duration is a
|
||||
// client-side draw concern (decode + cache in tlui). Server muxes from
|
||||
// scene.audio at render, so step()/evaluate never touch audio. ----
|
||||
@ -415,17 +509,32 @@ export class Timeline {
|
||||
this.scene.audio.splice(i, 1);
|
||||
this.undoStack.push({ undo: () => { this.scene.audio.splice(i, 0, clip); } });
|
||||
}
|
||||
// Trim within the source file, like clip blocks: {in, out} seconds. Absent =
|
||||
// untrimmed (schema stays {path,start,gain} for untouched clips). Server-side
|
||||
// mux honoring in/out is a Lane C REQUEST (logged); draft Web Audio playback
|
||||
// honors it client-side already.
|
||||
trimAudio(clip, { in: inV, out: outV } = {}) {
|
||||
const oldIn = clip.in, oldOut = clip.out;
|
||||
if (inV != null) clip.in = Math.max(0, inV);
|
||||
if (outV != null) clip.out = Math.max((clip.in || 0) + 0.05, outV);
|
||||
this.undoStack.push({ undo: () => {
|
||||
if (oldIn === undefined) delete clip.in; else clip.in = oldIn;
|
||||
if (oldOut === undefined) delete clip.out; else clip.out = oldOut;
|
||||
} });
|
||||
}
|
||||
|
||||
// Rhubarb lip-sync JSON → morph keys. { mouthCues: [{start, end, value}] }.
|
||||
// ponytail: per-cue ramp (shape 1 at cue start → 0 at cue end); hold/crossfade
|
||||
// tuning deferred until it looks wrong on a real blendshape character.
|
||||
importRhubarb(id, rhubarbJson, t0 = 0) {
|
||||
const cues = (rhubarbJson && rhubarbJson.mouthCues) || [];
|
||||
this.group(() => { // one undo reverts the whole import
|
||||
for (const c of cues) {
|
||||
if (!c.value) continue;
|
||||
this.addKey(id, 'morphs', { t: t0 + c.start, key: c.value, value: 1, ease: 'linear' });
|
||||
this.addKey(id, 'morphs', { t: t0 + c.end, key: c.value, value: 0, ease: 'linear' });
|
||||
}
|
||||
});
|
||||
return (this._tracks(id).morphs || []).length;
|
||||
}
|
||||
}
|
||||
|
||||
@ -194,4 +194,127 @@ assert.deepEqual(tlA.toJSON().audio, [{ path: 'audio/vo1.wav', start: 3.5, gain:
|
||||
tlA.removeAudio(aclip); assert.equal(tlA.scene.audio.length, 0, 'removeAudio drops it');
|
||||
tlA.undo(); assert.equal(tlA.scene.audio.length, 1, 'undo restores removed audio');
|
||||
|
||||
// ---- M5 backlog: audio trim (in/out within source, like clip blocks) ----
|
||||
const ac = tlA.scene.audio[0];
|
||||
tlA.trimAudio(ac, { out: 2.5 }); assert.equal(ac.out, 2.5, 'trimAudio sets out');
|
||||
tlA.trimAudio(ac, { in: 0.5 }); assert.equal(ac.in, 0.5, 'trimAudio sets in');
|
||||
tlA.undo(); tlA.undo();
|
||||
assert.ok(!('in' in ac) && !('out' in ac), 'undo removes trim fields entirely (schema stays clean)');
|
||||
assert.deepEqual(tlA.toJSON().audio, [{ path: 'audio/vo1.wav', start: 3.5, gain: 0.4 }], 'untrimmed clip round-trips without in/out');
|
||||
|
||||
// ---- M5: importRhubarb is grouped — ONE undo reverts the whole import ----
|
||||
const preLen = tlMo.scene.entities[0].tracks.morphs.length;
|
||||
const preUndo = tlMo.undoStack.length;
|
||||
tlMo.importRhubarb('face', rj, 3.0);
|
||||
assert.equal(tlMo.undoStack.length, preUndo + 1, 'import pushes one grouped undo entry');
|
||||
assert.ok(tlMo.scene.entities[0].tracks.morphs.length > preLen, 'import added keys');
|
||||
tlMo.undo();
|
||||
assert.equal(tlMo.scene.entities[0].tracks.morphs.length, preLen, 'one undo reverts the whole import');
|
||||
|
||||
// ---- M5 DIRECTOR MODE: scenegod:direct ops → grouped keys/cuts at playhead ----
|
||||
// Wire a real EventTarget as `window` so the ctor's listener path is what's
|
||||
// tested (node ≥19 has CustomEvent). Removed again below so later Timelines
|
||||
// stay headless-pure.
|
||||
globalThis.window = new EventTarget();
|
||||
const dStage = new StageStub();
|
||||
const tlDir = new Timeline(dStage); // ctor: listener + _mirror wired
|
||||
await dStage.addEntity({ id: 'cam1', kind: 'camera', params: { fov: 45 } });
|
||||
await dStage.addEntity({ id: 'sun', kind: 'light', params: { type: 'key', color: '#ffffff', intensity: 1 } });
|
||||
await dStage.addEntity({ id: 'lady', kind: 'character', label: 'lady' });
|
||||
tlDir.seek(3);
|
||||
|
||||
// shot: transform+fov keys on the camera at playhead + a cut — via the EVENT,
|
||||
// using Lane A's live payload shape (values spread at top level, presets.js:165)
|
||||
let depth = tlDir.undoStack.length;
|
||||
window.dispatchEvent(new CustomEvent('scenegod:direct', { detail: {
|
||||
op: 'shot', camId: 'cam1', subjectId: 'lady', shot: 'MS', angle: 'eye', focal: 35,
|
||||
pos: [0, 2, 6], rot: [0, 0.4, 0], fov: 35, dist: 4.2, aim: [0, 1.2, 0], entityIds: ['cam1', 'lady'],
|
||||
} }));
|
||||
const camE = tlDir.scene.entities.find((x) => x.id === 'cam1');
|
||||
assert.equal(camE.tracks.transform.length, 1, 'shot lands one transform key');
|
||||
assert.ok(near(camE.tracks.transform[0].t, 3) && near(camE.tracks.transform[0].pos[2], 6), 'shot key at playhead with the preset pos');
|
||||
assert.ok(camE.tracks.params.some((k) => k.key === 'fov' && k.value === 35 && near(k.t, 3)), 'shot lands the fov param key');
|
||||
assert.deepEqual(tlDir.cameraCuts, [{ t: 3, camera: 'cam1' }], 'shot lands a cut to the camera');
|
||||
assert.equal(dStage._active, 'cam1', 'shot re-evaluates: camera is live immediately');
|
||||
assert.equal(tlDir.undoStack.length, depth + 1, 'one shot op = one undo entry');
|
||||
tlDir.undo();
|
||||
assert.equal(camE.tracks.transform.length, 0, 'undo reverts the shot transform key');
|
||||
assert.equal(camE.tracks.params.length, 0, 'undo reverts the fov key');
|
||||
assert.equal(tlDir.cameraCuts.length, 0, 'undo reverts the cut');
|
||||
|
||||
// light: generic contract shape — entityId + params map at the playhead
|
||||
depth = tlDir.undoStack.length;
|
||||
tlDir.applyDirect({ op: 'light', entityId: 'sun', params: { intensity: 6.5, color: '#ffaa66' } });
|
||||
const sunE = tlDir.scene.entities.find((x) => x.id === 'sun');
|
||||
assert.equal(sunE.tracks.params.length, 2, 'light op lands one key per param');
|
||||
assert.ok(sunE.tracks.params.every((k) => near(k.t, 3)), 'light keys at the playhead');
|
||||
assert.equal(tlDir.undoStack.length, depth + 1, 'one light op = one undo entry');
|
||||
tlDir.undo();
|
||||
assert.equal(sunE.tracks.params.length, 0, 'undo reverts all light keys');
|
||||
|
||||
// light: Lane A applyLight shape — sun + ambient + bg (presets.js:145)
|
||||
await dStage.addEntity({ id: 'amb', kind: 'light', params: { type: 'ambient', color: '#111111', intensity: 0.3 } });
|
||||
depth = tlDir.undoStack.length;
|
||||
tlDir.applyDirect({ op: 'light', preset: 'noir',
|
||||
sun: { color: '#aabbcc', intensity: 2, elev: 55, azim: -60 }, ambient: { color: '#112233', intensity: 0.5 },
|
||||
bg: '#000011', sunId: 'sun', ambientId: 'amb', sunPos: [1, 16, 2], entityIds: ['sun', 'amb'] });
|
||||
const ambE = tlDir.scene.entities.find((x) => x.id === 'amb');
|
||||
assert.equal(sunE.tracks.params.length, 2, 'preset lands sun color+intensity keys');
|
||||
assert.equal(sunE.tracks.transform.length, 1, 'preset captures the sun boom transform');
|
||||
assert.equal(ambE.tracks.params.length, 3, 'preset lands ambient color+intensity+bg keys');
|
||||
assert.ok(ambE.tracks.params.some((k) => k.key === 'bg' && k.value === '#000011'), 'bg key rides the ambient entity');
|
||||
assert.equal(tlDir.undoStack.length, depth + 1, 'whole preset = one undo entry');
|
||||
tlDir.undo();
|
||||
assert.equal(sunE.tracks.params.length + sunE.tracks.transform.length + ambE.tracks.params.length, 0,
|
||||
'one undo reverts the entire light preset');
|
||||
|
||||
// mark + walkTo: current-pos key at playhead, mark key at playhead+2s
|
||||
depth = tlDir.undoStack.length;
|
||||
tlDir.applyDirect({ op: 'mark', entityId: 'lady', transform: { pos: [2, 0, 1], rot: [0, 1.57, 0] }, walkTo: true });
|
||||
const ladyE = tlDir.scene.entities.find((x) => x.id === 'lady');
|
||||
assert.equal(ladyE.tracks.transform.length, 2, 'walk-to-mark lands 2 keys');
|
||||
assert.ok(near(ladyE.tracks.transform[0].t, 3) && near(ladyE.tracks.transform[0].pos[0], 0), 'first key = current pos at playhead');
|
||||
assert.ok(near(ladyE.tracks.transform[1].t, 5) && near(ladyE.tracks.transform[1].pos[0], 2), 'second key = mark at playhead+2s');
|
||||
tlDir.undo();
|
||||
assert.equal(ladyE.tracks.transform.length, 0, 'one undo reverts both walk keys');
|
||||
assert.equal(tlDir.undoStack.length, depth, 'walk-to-mark was a single undo entry');
|
||||
|
||||
// mark: Lane A applyMark shape — top-level pos, id field (presets.js:151)
|
||||
tlDir.applyDirect({ op: 'mark', mark: 'two_shot_L', id: 'lady', pos: [-0.7, 0, 0], entityIds: ['lady'] });
|
||||
assert.equal(ladyE.tracks.transform.length, 1, 'plain mark lands one key');
|
||||
assert.ok(near(ladyE.tracks.transform[0].pos[0], -0.7), 'mark key carries the preset pos');
|
||||
tlDir.undo();
|
||||
delete globalThis.window;
|
||||
|
||||
// ---- M6 FLOW VIDEO PLATES: videos follow the clock, seek only on >50ms drift ----
|
||||
const vStage = new StageStub();
|
||||
const tlV = new Timeline(vStage);
|
||||
const vScene = { version: 1, fps: 30, duration: 12, cameraCuts: [], entities: [
|
||||
{ id: 'bg', kind: 'backdrop', params: { mode: 'video', image: 'backdrops/video/street.mp4', videoStart: 1, videoDuration: 4 }, tracks: {} },
|
||||
{ id: 'still', kind: 'backdrop', params: { mode: 'plane', image: 'backdrops/street.jpg' }, tracks: {} },
|
||||
] };
|
||||
await vStage.applyState(vScene);
|
||||
tlV.load(vScene);
|
||||
assert.equal(vStage.entityVideo('still'), null, 'image backdrop has no video (nullable contract)');
|
||||
const vid = vStage.entityVideo('bg');
|
||||
assert.ok(vid, 'video backdrop exposes a video');
|
||||
vid.seeks.length = 0;
|
||||
tlV.step(90); // t=3 → want (3-1)%4 = 2
|
||||
assert.deepEqual(vid.seeks, [2], 'drift >50ms → one seek to (t-videoStart)%duration');
|
||||
tlV.step(90); // same t → drift 0
|
||||
tlV.step(91); // t≈3.033 → drift ~33ms
|
||||
assert.equal(vid.seeks.length, 1, 'no seek while drift <=50ms');
|
||||
tlV.step(94); // t≈3.133 → drift ~133ms
|
||||
assert.equal(vid.seeks.length, 2, 'drift >50ms → seeks again');
|
||||
assert.ok(near(vid.currentTime, 94 / 30 - 1, 1e-6), 'resynced to the clock');
|
||||
tlV.step(180); // t=6 → want (6-1)%4 = 1 (wraps)
|
||||
assert.ok(near(vid.currentTime, 1, 1e-6), 'loops via modulo duration');
|
||||
tlV.step(0); // t=0 < videoStart → clamp to frame 0
|
||||
assert.ok(near(vid.currentTime, 0, 1e-6), 'before videoStart holds frame 0');
|
||||
assert.equal(vid.paused, true, 'video paused while the clock is stopped');
|
||||
tlV.playing = true; tlV.evaluate(tlV.time); // clock running → video plays
|
||||
assert.equal(vid.paused, false, 'video plays with the clock');
|
||||
tlV.pause(); // pause() syncs videos
|
||||
assert.equal(vid.paused, true, 'pause() pauses the video');
|
||||
|
||||
console.log('OK — timeline_test.mjs: all assertions passed');
|
||||
|
||||
@ -3,7 +3,9 @@
|
||||
//
|
||||
// Layout: [scene bar] / [names column | ruler+canvas lanes]. One <canvas> for
|
||||
// all lanes (decision: simpler than canvas-per-lane; hit-testing via a rebuilt
|
||||
// hitbox list each draw). Fit-to-width time axis; zoom/pan is a later problem.
|
||||
// hitbox list each draw). Time axis fits to width at zoom 1; Cmd/Ctrl+wheel
|
||||
// zooms around the cursor, plain wheel is left to the page (scroll), and
|
||||
// trackpad horizontal wheel pans when zoomed in.
|
||||
|
||||
const ROW_H = 28;
|
||||
const RULER_H = 22;
|
||||
@ -30,12 +32,15 @@ const CSS = `
|
||||
#timeline .tl-names .row .k{color:#565f6b;margin-right:6px;font-size:10px}
|
||||
#timeline .tl-names .row.sub{color:#8b949e;padding-left:20px;font-size:11px}
|
||||
#timeline .tl-names .row input.gain{width:38px;margin-left:auto;background:#0d1117;border:1px solid #2b323c;color:#c9d1d9;font:10px ui-monospace;border-radius:3px;padding:1px 2px}
|
||||
#timeline .tl-names .row button.lip{margin-left:4px;background:#21262d;border:1px solid #2b323c;border-radius:3px;color:#c9d1d9;cursor:pointer;font-size:11px;padding:0 4px}
|
||||
#timeline .tl-names .row button.lip:hover{background:#2b323c}
|
||||
#timeline .tl-lanes{flex:1;position:relative;overflow:hidden}
|
||||
#timeline canvas{display:block;width:100%;height:100%}
|
||||
.tl-menu{position:fixed;z-index:9999;background:#12161c;border:1px solid #2b323c;border-radius:4px;max-height:240px;overflow:auto;font:12px ui-monospace;color:#c9d1d9;min-width:160px}
|
||||
.tl-menu .item{padding:4px 10px;cursor:pointer;white-space:nowrap}
|
||||
.tl-menu .item:hover{background:#2b323c}
|
||||
.tl-menu .empty{padding:6px 10px;color:#565f6b}
|
||||
.tl-toast{position:fixed;left:50%;bottom:18px;transform:translateX(-50%);z-index:10000;background:#12161c;border:1px solid #e0af68;color:#e0e6ed;padding:6px 14px;border-radius:4px;font:12px ui-monospace;max-width:70vw}
|
||||
`;
|
||||
|
||||
export class TimelineUI {
|
||||
@ -48,6 +53,8 @@ export class TimelineUI {
|
||||
this._selKeys = []; // box-selected keys: [{id, track, key}]
|
||||
this._snapOn = true;
|
||||
this._snapStep = 'frame'; // 'frame' | seconds string
|
||||
this._zoom = 1; // 1 = fit-to-width; Cmd/Ctrl+wheel zooms
|
||||
this._panT = 0; // left edge of the view, in seconds
|
||||
this._audioBuf = new Map(); // path -> decoded AudioBuffer
|
||||
this._audioSrcs = []; // live BufferSourceNodes during playback
|
||||
this._wasPlaying = false;
|
||||
@ -122,6 +129,7 @@ export class TimelineUI {
|
||||
this.lanesEl = lanes;
|
||||
|
||||
this.canvas.addEventListener('mousedown', (e) => this._down(e));
|
||||
this.canvas.addEventListener('wheel', (e) => this._wheel(e), { passive: false });
|
||||
window.addEventListener('mousemove', (e) => this._move(e));
|
||||
window.addEventListener('mouseup', () => this._up());
|
||||
this.canvas.addEventListener('dblclick', (e) => this._dbl(e));
|
||||
@ -169,6 +177,11 @@ export class TimelineUI {
|
||||
g.value = r.clip.gain ?? 1;
|
||||
g.onchange = () => this.tl.setAudioGain(r.clip, Math.max(0, +g.value || 0));
|
||||
d.appendChild(g);
|
||||
const lip = document.createElement('button'); // M5: lip-sync from this audio
|
||||
lip.className = 'lip'; lip.textContent = '🗣';
|
||||
lip.title = 'lip-sync: rhubarb this audio → viseme keys on a character';
|
||||
lip.onclick = () => this._lipSync(r.clip, lip);
|
||||
d.appendChild(lip);
|
||||
} else d.textContent = r.label;
|
||||
this.$names.appendChild(d);
|
||||
});
|
||||
@ -188,10 +201,31 @@ export class TimelineUI {
|
||||
if (this.canvas.width !== w * dpr || this.canvas.height !== h * dpr) {
|
||||
this.canvas.width = w * dpr; this.canvas.height = h * dpr;
|
||||
}
|
||||
return { w, h, dpr, pps: w / (this.tl.duration || 1) };
|
||||
return { w, h, dpr, pps: (w / (this.tl.duration || 1)) * this._zoom };
|
||||
}
|
||||
_x2t(x) { const { pps } = this._dims(); return Math.max(0, Math.min(this.tl.duration, this._panT + x / pps)); }
|
||||
_t2x(t) { const { pps } = this._dims(); return (t - this._panT) * pps; }
|
||||
// UX (session-5 nit): plain wheel is NOT captured — the page scrolls as users
|
||||
// expect. Cmd/Ctrl+wheel (incl. trackpad pinch → ctrlKey) zooms around the
|
||||
// cursor; horizontal wheel pans when zoomed in.
|
||||
_wheel(e) {
|
||||
const zoomG = e.ctrlKey || e.metaKey;
|
||||
const panG = this._zoom > 1 && Math.abs(e.deltaX) > Math.abs(e.deltaY);
|
||||
if (!zoomG && !panG) return;
|
||||
e.preventDefault();
|
||||
const { x } = this._local(e);
|
||||
const { w, pps } = this._dims();
|
||||
if (zoomG) {
|
||||
const tAt = this._panT + x / pps; // time under the cursor
|
||||
this._zoom = Math.max(1, Math.min(64, this._zoom * Math.pow(1.0015, -e.deltaY)));
|
||||
this._panT = tAt - x / ((w / (this.tl.duration || 1)) * this._zoom); // keep it under the cursor
|
||||
} else {
|
||||
this._panT += e.deltaX / pps;
|
||||
}
|
||||
const view = (this.tl.duration || 1) / this._zoom; // visible seconds
|
||||
this._panT = Math.max(0, Math.min(this.tl.duration - view, this._panT));
|
||||
this.draw();
|
||||
}
|
||||
_x2t(x) { const { pps } = this._dims(); return Math.max(0, Math.min(this.tl.duration, x / pps)); }
|
||||
_t2x(t) { const { pps } = this._dims(); return t * pps; }
|
||||
_rowAtY(y) { const i = Math.floor((y - RULER_H) / ROW_H); return (i >= 0 && i < this.rows.length) ? i : -1; }
|
||||
|
||||
// ---- draw ----
|
||||
@ -314,16 +348,21 @@ export class TimelineUI {
|
||||
}
|
||||
_drawAudio(c, r, y) {
|
||||
const clip = r.clip;
|
||||
const dur = this._audioDur(clip.path);
|
||||
const srcDur = this._audioDur(clip.path);
|
||||
const inT = clip.in || 0; // trim within the source (like clip blocks)
|
||||
const outT = clip.out ?? srcDur;
|
||||
const len = outT != null ? Math.max(0.05, outT - inT) : null;
|
||||
const x0 = this._t2x(clip.start);
|
||||
const x1 = dur != null ? this._t2x(clip.start + dur) : x0 + 60; // fallback width until decoded
|
||||
const x1 = len != null ? this._t2x(clip.start + len) : x0 + 60; // fallback width until decoded
|
||||
const bw = Math.max(3, x1 - x0);
|
||||
c.fillStyle = '#2e7d5b'; c.fillRect(x0, y + 5, bw, ROW_H - 10);
|
||||
c.strokeStyle = '#57c99a'; c.strokeRect(x0 + 0.5, y + 5.5, bw - 1, ROW_H - 11);
|
||||
c.fillStyle = '#dff5ea'; c.font = '10px ui-monospace';
|
||||
c.save(); c.beginPath(); c.rect(x0, y, bw, ROW_H); c.clip();
|
||||
c.fillText((clip.path || '').split('/').pop() + (dur == null ? ' …' : ''), x0 + 4, y + ROW_H / 2 + 3); c.restore();
|
||||
const trimmed = clip.in != null || clip.out != null;
|
||||
c.fillText((trimmed ? '✂ ' : '') + (clip.path || '').split('/').pop() + (srcDur == null ? ' …' : ''), x0 + 4, y + ROW_H / 2 + 3); c.restore();
|
||||
this._hits.push({ kind: 'audioblock', clip, x0, x1, y });
|
||||
if (len != null) this._hits.push({ kind: 'aedge', clip, x: x1, y }); // drag = trim tail
|
||||
}
|
||||
|
||||
// ---- interaction ----
|
||||
@ -336,7 +375,7 @@ export class TimelineUI {
|
||||
_hitAt(x, y) {
|
||||
for (const h of this._hits) {
|
||||
if (h.kind === 'key' || h.kind === 'cut') { if (Math.abs(h.x - x) < 7 && Math.abs(h.y - y) < 9) return h; }
|
||||
else if (h.kind === 'edge') { if (Math.abs(h.x - x) < 5 && y > h.y && y < h.y + ROW_H) return h; }
|
||||
else if (h.kind === 'edge' || h.kind === 'aedge') { if (Math.abs(h.x - x) < 5 && y > h.y && y < h.y + ROW_H) return h; }
|
||||
}
|
||||
for (const h of this._hits) if ((h.kind === 'block' || h.kind === 'audioblock') && x >= h.x0 && x <= h.x1 && y > h.y && y < h.y + ROW_H) return h;
|
||||
return null;
|
||||
@ -359,6 +398,7 @@ export class TimelineUI {
|
||||
this._drag = { kind: 'multikey', grabT: this._x2t(x), orig: this._selKeys.map((s) => ({ ...s, t0: s.key.t })) };
|
||||
} else { this._selKeys = []; this._drag = { kind: 'key', h }; }
|
||||
} else if (h.kind === 'edge') this._drag = { kind: 'trim', h };
|
||||
else if (h.kind === 'aedge') this._drag = { kind: 'atrim', h };
|
||||
else if (h.kind === 'block') this._drag = { kind: 'block', h, grabT: this._x2t(x) - h.block.start };
|
||||
else if (h.kind === 'audioblock') this._drag = { kind: 'audio', h, grabT: this._x2t(x) - h.clip.start };
|
||||
else if (h.kind === 'cut') this._drag = { kind: 'cut', h };
|
||||
@ -374,6 +414,12 @@ export class TimelineUI {
|
||||
else if (d.kind === 'block') { this.tl.moveClipBlock(d.h.id, d.h.block, Math.max(0, this._snap(t - d.grabT, e))); this.draw(); }
|
||||
else if (d.kind === 'audio') { this.tl.moveAudio(d.h.clip, this._snap(t - d.grabT, e)); this.draw(); }
|
||||
else if (d.kind === 'trim') { const b = d.h.block; this.tl.trimClipBlock(d.h.id, b, { out: Math.max(b.in + 0.05, b.in + (t - b.start)) }); this.draw(); }
|
||||
else if (d.kind === 'atrim') {
|
||||
const cl = d.h.clip, srcDur = this._audioDur(cl.path);
|
||||
let out = (cl.in || 0) + (t - cl.start);
|
||||
if (srcDur != null) out = Math.min(out, srcDur);
|
||||
this.tl.trimAudio(cl, { out }); this.draw();
|
||||
}
|
||||
else if (d.kind === 'box') { d.x = x; d.y = y; d.moved = Math.abs(x - d.x0) > 4 || Math.abs(y - d.y0) > 4; this.draw(); }
|
||||
}
|
||||
_up() {
|
||||
@ -462,13 +508,15 @@ export class TimelineUI {
|
||||
const now = ctx.currentTime, t = this.tl.time;
|
||||
for (const clip of (this.tl.scene.audio || [])) {
|
||||
const buf = this._audioBuf.get(clip.path); if (!buf) continue;
|
||||
const offset = t - clip.start; // seconds into the clip at current playhead
|
||||
if (offset >= buf.duration) continue; // already finished
|
||||
const inT = Math.min(clip.in || 0, buf.duration); // trim within source
|
||||
const len = Math.min(clip.out ?? buf.duration, buf.duration) - inT;
|
||||
const offset = Math.max(0, t - clip.start); // seconds into the (trimmed) block at playhead
|
||||
if (len <= 0 || offset >= len) continue; // already finished
|
||||
const src = ctx.createBufferSource(); src.buffer = buf;
|
||||
const g = ctx.createGain(); g.gain.value = clip.gain ?? 1;
|
||||
src.connect(g).connect(ctx.destination);
|
||||
const when = now + Math.max(0, clip.start - t);
|
||||
src.start(when, Math.max(0, offset));
|
||||
src.start(when, inT + offset, len - offset);
|
||||
this._audioSrcs.push(src);
|
||||
}
|
||||
}
|
||||
@ -498,6 +546,51 @@ export class TimelineUI {
|
||||
setTimeout(() => document.addEventListener('mousedown', close), 0);
|
||||
}
|
||||
|
||||
// ---- M5: lip-sync from audio (🗣 on an audio row) ----
|
||||
// GET /rhubarb?path=… → importRhubarb onto a viseme-bearing character.
|
||||
// >1 candidate → picker menu; 503 → toast the install hint from `detail`.
|
||||
async _lipSync(clip, anchorEl) {
|
||||
const vt = (id) => (this.stage.visemeTargets && this.stage.visemeTargets(id)) || [];
|
||||
const chars = this.stage.entities().filter((en) => en.kind === 'character' && vt(en.id).length > 0);
|
||||
if (!chars.length) { this._toast('no character with viseme morph targets in the scene'); return; }
|
||||
const id = chars.length === 1 ? chars[0].id
|
||||
: await this._choose(chars.map((c) => ({ label: c.label || c.id, value: c.id })), anchorEl);
|
||||
if (!id) return;
|
||||
let res;
|
||||
try { res = await fetch('/rhubarb?path=' + encodeURIComponent(clip.path)); }
|
||||
catch { this._toast('server unreachable — /rhubarb needs Lane C running'); return; }
|
||||
if (!res.ok) {
|
||||
let msg = '/rhubarb failed: ' + res.status;
|
||||
try { const j = await res.json(); if (j.detail) msg = j.detail; } catch { /* non-json body */ }
|
||||
this._toast(msg); return; // 503 carries the rhubarb install hint
|
||||
}
|
||||
const json = await res.json();
|
||||
this.tl.importRhubarb(id, json, clip.start || 0); // grouped: one undo reverts the import
|
||||
this._toast(`🗣 ${(json.mouthCues || []).length} cues → viseme keys on ${id}`);
|
||||
this.tl.seek(this.tl.time);
|
||||
this._syncRows(); // viseme lane may be new
|
||||
}
|
||||
_choose(items, anchorEl) { // small promise-menu (resolves value | null)
|
||||
return new Promise((resolve) => {
|
||||
const menu = document.createElement('div'); menu.className = 'tl-menu';
|
||||
for (const it of items) {
|
||||
const el = document.createElement('div'); el.className = 'item'; el.textContent = it.label;
|
||||
el.onclick = () => { menu.remove(); resolve(it.value); };
|
||||
menu.appendChild(el);
|
||||
}
|
||||
document.body.appendChild(menu);
|
||||
const rect = anchorEl.getBoundingClientRect();
|
||||
menu.style.left = rect.left + 'px'; menu.style.top = (rect.bottom + 2) + 'px';
|
||||
const close = (ev) => { if (!menu.contains(ev.target)) { menu.remove(); document.removeEventListener('mousedown', close); resolve(null); } };
|
||||
setTimeout(() => document.addEventListener('mousedown', close), 0);
|
||||
});
|
||||
}
|
||||
_toast(msg) {
|
||||
const d = document.createElement('div'); d.className = 'tl-toast'; d.textContent = msg;
|
||||
document.body.appendChild(d);
|
||||
setTimeout(() => d.remove(), 4000);
|
||||
}
|
||||
|
||||
// ---- persistence (Lane C endpoints; localStorage fallback) ----
|
||||
async save() {
|
||||
const name = (this.$name.value || 'untitled').trim();
|
||||
|
||||
@ -60,6 +60,7 @@ def main():
|
||||
renders = tmp / "renders"
|
||||
env = {**os.environ, "SCENEGOD_ASSETS": str(assets), "SCENEGOD_SCENES": str(scenes),
|
||||
"SCENEGOD_RENDERS": str(renders)}
|
||||
env.pop("SCENEGOD_LLM_URL", None) # main instance must 503 on /director
|
||||
proc = subprocess.Popen(
|
||||
[str(PY), "-m", "uvicorn", "scenegod.server:app", "--port", str(PORT)],
|
||||
cwd=ROOT, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
@ -236,6 +237,75 @@ def main():
|
||||
p2.wait()
|
||||
srv.shutdown()
|
||||
|
||||
# --- /director: 503 unconfigured; mock LLM -> ops validated, garbage rejected ---
|
||||
dbody = json.dumps({"text": "medium shot of lady, golden hour",
|
||||
"scene": {"entities": [{"id": "e1", "kind": "character", "label": "lady"},
|
||||
{"id": "e7", "kind": "camera", "label": "cam A"}],
|
||||
"time": 2.5}})
|
||||
assert req("POST", "/director", dbody)[0] == 503 # env unset -> hint, no crash
|
||||
|
||||
llm_captured = []
|
||||
canned = {"ops": [
|
||||
{"op": "shot", "subject": "e1", "shot": "MS", "angle": "low", "focal": 35},
|
||||
{"op": "light", "preset": "golden-hour"},
|
||||
{"op": "mark", "subject": "e1", "mark": "walk-to-mark"},
|
||||
{"op": "explode", "subject": "e1"}, # unknown op
|
||||
{"op": "shot", "subject": "ghost", "shot": "CU"}, # unknown entity id
|
||||
{"op": "light", "preset": "disco"}, # unknown preset
|
||||
]}
|
||||
|
||||
class LLM(BaseHTTPRequestHandler):
|
||||
def do_POST(self):
|
||||
llm_captured.append(json.loads(
|
||||
self.rfile.read(int(self.headers.get("Content-Length", 0)))))
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.end_headers()
|
||||
self.wfile.write(json.dumps(
|
||||
{"choices": [{"message": {"content": json.dumps(canned)}}]}).encode())
|
||||
|
||||
def log_message(self, *a):
|
||||
pass
|
||||
|
||||
llmport, port3 = PORT + 3, PORT + 4
|
||||
srv3 = HTTPServer(("127.0.0.1", llmport), LLM)
|
||||
threading.Thread(target=srv3.serve_forever, daemon=True).start()
|
||||
env3 = {**env, "SCENEGOD_LLM_URL": f"http://127.0.0.1:{llmport}",
|
||||
"SCENEGOD_LLM_MODEL": "mockmodel"}
|
||||
p3 = subprocess.Popen(
|
||||
[str(PY), "-m", "uvicorn", "scenegod.server:app", "--port", str(port3)],
|
||||
cwd=ROOT, env=env3, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
try:
|
||||
for _ in range(50):
|
||||
try:
|
||||
urllib.request.urlopen(f"http://127.0.0.1:{port3}/scenes")
|
||||
break
|
||||
except urllib.error.URLError:
|
||||
time.sleep(0.2)
|
||||
r = urllib.request.urlopen(urllib.request.Request(
|
||||
f"http://127.0.0.1:{port3}/director", data=dbody.encode(), method="POST",
|
||||
headers={"Content-Type": "application/json"}))
|
||||
out = json.loads(r.read())
|
||||
assert [o["op"] for o in out["ops"]] == ["shot", "light", "mark"], out
|
||||
assert out["ops"][0] == {"op": "shot", "subject": "e1", "shot": "MS",
|
||||
"angle": "low", "focal": 35}, out
|
||||
assert len(out["rejected"]) == 3, out
|
||||
reasons = " | ".join(rj["reason"] for rj in out["rejected"])
|
||||
assert "explode" in reasons and "ghost" in reasons and "disco" in reasons, reasons
|
||||
# prompt carries the grammar vocab + the scene roster, model + low temp set
|
||||
sent = llm_captured[0]
|
||||
prompt = " ".join(m["content"] for m in sent["messages"])
|
||||
for word in ("ECU", "MWS", "dutch", "OTS-L", "85", "golden-hour", "noir",
|
||||
"walk-to-mark", "face-other", "two-shot-L"):
|
||||
assert word in prompt, f"vocab {word!r} missing from prompt"
|
||||
assert "e1" in prompt and "lady" in prompt and "2.5" in prompt, "roster/time missing"
|
||||
assert sent["model"] == "mockmodel" and sent["temperature"] <= 0.3, sent
|
||||
n += 1
|
||||
finally:
|
||||
p3.terminate()
|
||||
p3.wait()
|
||||
srv3.shutdown()
|
||||
|
||||
print(f"OK: {n} test groups passed")
|
||||
finally:
|
||||
proc.terminate()
|
||||
|
||||
Loading…
Reference in New Issue
Block a user