From a8273127ba705d85254d998916a66bedca6b86cf Mon Sep 17 00:00:00 2001 From: type-two Date: Sun, 19 Jul 2026 01:09:30 +1000 Subject: [PATCH] [m5] DIRECTOR MODE round: cinema-grammar presets spec + /director LLM lane; rhubarb live MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - PLAN §5 M5: shot sizes/angles/focals/light presets/blocking marks as one-click deterministic presets (grammar.js data + presets.js solver, Lane A), keyed at the playhead via scenegod:direct events (Lane B), optional NL box -> POST /director -> tailnet Ollama with server-side op validation (Lane C; m4pro serves qwen2.5:7b) - rhubarb 1.14.0 built from John's source zip (JING5), installed arm64 at ~/.local/opt/rhubarb, symlinked /opt/homebrew/bin; happy path verified: /rhubarb on a real spoken wav (assets/audio/test/vo.wav) returns mouthCues - lane B unlocked to wire the lip-sync-from-audio button now Co-Authored-By: Claude Fable 5 --- PLAN.md | 20 ++++++++++++++++++++ lanes/A-stage.md | 29 +++++++++++++++++++++++++++++ lanes/B-timeline.md | 14 ++++++++++++++ lanes/C-server.md | 22 ++++++++++++++++++++++ 4 files changed, 85 insertions(+) diff --git a/PLAN.md b/PLAN.md index c785b3f..753677e 100644 --- a/PLAN.md +++ b/PLAN.md @@ -192,6 +192,26 @@ tts/voice-clone submit (mb contract from meshgod/ops.py, token from B: audio tracks on the timeline UI, Web Audio sync. A: viseme lane — detect morph targets, Rhubarb-JSON → morph keyframes, greyed out when absent. +**M5 — DIRECTOR MODE (ease of use)**: classic cinema grammar as one-click +presets, LLM strictly optional on top. +- Shared vocabulary in `web/grammar.js` (Lane A owns, data only): shot sizes + ECU/CU/MCU/MS/MWS/WS/EWS (target vertical span as a fraction of subject + height + frame-center height), angles (eye/low/high/dutch/OTS-L/OTS-R), + focal lengths 18/24/35/50/85mm (vertical fov = 2·atan(12/f), full-frame), + lighting presets (day, golden-hour, night, noir, horror, neon, overcast), + blocking marks (center, two-shot L/R, walk-to-mark, face-camera, face-other). +- Lane A `web/presets.js`: `frameSubject(camId, subjectId, shot, angle, + focalmm)` — deterministic solver from the subject's bbox (no LLM), light/ + blocking preset appliers, and the DIRECT panel UI (shot buttons on the + selected subject, light preset buttons, mark buttons). +- Lane B: preset actions land as normal keys/cuts AT THE PLAYHEAD through + existing mutators (undo-able); "walk to mark" = 2 transform keys. +- Lane C: `POST /director` — optional NL box ("medium two-shot, golden hour, + lady walks to camera") → tailnet Ollama (OpenAI-compat, env + `SCENEGOD_LLM_URL` + `SCENEGOD_LLM_MODEL`, 503 when unset) → strict JSON + ops validated server-side → client applies via the SAME preset functions. + Buttons must never depend on the LLM lane. + ## 6. How John launches the lanes Three Claude Code sessions (Opus 4.8), cwd `~/Documents/SCENEGOD`, one prompt diff --git a/lanes/A-stage.md b/lanes/A-stage.md index 220f0f5..52f6d41 100644 --- a/lanes/A-stage.md +++ b/lanes/A-stage.md @@ -173,3 +173,32 @@ asserts L === [v_tongue_up] and E includes the stripped smile. Lesson worth keeping: assert group CONTENTS, not just which groups exist. Still owed by John, not you: a viseme-bearing test character for a live badge check. Standing by. + +### 2026-07-19 — **M5 DIRECTOR MODE unlocked** (spec in PLAN §5 M5). Your part: +1. `web/grammar.js` — pure data, you own it, everyone reads it: SHOTS + (ECU .12H@.93H, CU .25H@.87H, MCU .45H@.75H, MS .65H@.65H, MWS .85H@.55H, + WS 1.2H@.5H, EWS 2.5H@.5H — vertical span fraction of subject height @ + frame-center height), ANGLES (eye/low/high/dutch 8°roll/OTS-L/OTS-R), + FOCALS [18,24,35,50,85] (fovV = 2·atan(12/f) in degrees), LIGHTS (day, + golden_hour, night, noir, horror, neon, overcast — each {sunColor, + sunIntensity, sunPos-ish elevation/azimuth, ambient, bg}), MARKS (center, + two_shot_L/R, face_camera, face_other). +2. `web/presets.js` — `frameSubject(stage, camId, subjectId, {shot, angle, + focal})`: bbox of subject → camera pos/rot/fov so the shot's span fills + the frame; OTS places the cam over the OTHER selected entity's shoulder. + `applyLight(stage, presetName)` → setParam on sun/ambient + background. + `applyMark(stage, id, mark)` → transform (marks are stage-relative). + Pure functions returning the values AND applying them — Lane B needs the + values to write keys. +3. DIRECT panel UI (top toolbar or dock tab): with an entity selected — + shot-size buttons, angle chips, focal select, "🎥 frame it" (creates/moves + a camera + fires one `scenegod:direct` CustomEvent with the op payload, + detail below); light preset buttons; mark buttons. Event contract (B + consumes): `scenegod:direct {detail: {op:'shot'|'light'|'mark', ...values, + entityIds}}` — YOU apply nothing to the timeline; B keys it. +4. NL box (small text input in the panel) POSTs to `/director` (Lane C) and + dispatches the returned ops through the same `scenegod:direct` events. + Hidden entirely when /director 503s. +Acceptance: select man → CU low 35mm → camera frames his face from below in +one click; golden_hour relights the scene in one click; every action lands +as keys via B (scrub proves it) and is undo-able. diff --git a/lanes/B-timeline.md b/lanes/B-timeline.md index 6005a0f..5c7be6f 100644 --- a/lanes/B-timeline.md +++ b/lanes/B-timeline.md @@ -148,3 +148,17 @@ button — POST the audio path to Lane C's `/rhubarb`, feed the JSON to importRhubarb (503 → toast the install hint); (2) audio block trim; (3) UX nit found live: mouse wheel over the track canvas zooms, so users may not discover the panel scrolls — consider wheel = scroll, Cmd/Ctrl+wheel = zoom. + +### 2026-07-19 — **M5 DIRECTOR MODE unlocked** (spec in PLAN §5 M5). Your part: +1. Listen for `scenegod:direct` (contract in lanes/A §M5.3): every preset op + becomes NORMAL keys/cuts at the playhead through your existing mutators — + `shot` → transform+fov keys on the camera + a cut to it; `light` → param + keys on sun/ambient (+bg); `mark` → transform key; `walk_to_mark` → key at + playhead-with-current-pos + key at playhead+2s at the mark. All undo-able + (one undo entry per op — group the batch, don't leave 5 half-undoable keys). +2. Wire the "lip-sync from audio" button now that `/rhubarb` is LIVE and + verified (`GET /rhubarb?path=audio/test/vo.wav` returns real mouthCues — + vo.wav is in assets): on an audio row, 🗣 button → fetch cues → + `importRhubarb(characterId, json, audioStart)`; character picker if >1 + viseme-bearing char; toast the 503 install hint if absent. +3. Session-5 backlog if time remains: audio trim, wheel-scroll fix (above). diff --git a/lanes/C-server.md b/lanes/C-server.md index 5234bbd..770fb4b 100644 --- a/lanes/C-server.md +++ b/lanes/C-server.md @@ -125,6 +125,28 @@ One render-quality note for your backlog, not urgent: draft webm via MediaRecorder is untested in the embedded preview (rAF suspension) — verify it once in a real Chrome window before calling it done. +### 2026-07-19 — **M5 DIRECTOR MODE unlocked**: `POST /director` (spec PLAN §5 M5). +- rhubarb is now INSTALLED on ultra (built from source, arm64, v1.14.0, + symlinked /opt/homebrew/bin/rhubarb) — your happy path is verified: + `GET /rhubarb?path=audio/test/vo.wav` → real mouthCues. Nothing to do there. +- `POST /director {text, scene:{entities:[{id,kind,label}], time}}` → chat + request to `SCENEGOD_LLM_URL` (OpenAI-compat; ultra's tailnet Ollama is + `http://100.69.21.128:11434`, model via `SCENEGOD_LLM_MODEL`). Orchestrator + checked /api/tags on that box: it serves `qwen2.5:7b` (+ an embed model) — + default SCENEGOD_LLM_MODEL to `qwen2.5:7b`; John can pull a gemma later + and it's just an env change. 503 with a hint when env unset. stdlib + urllib only. +- System prompt: translate director language into a JSON array of ops + matching Lane A's `scenegod:direct` contract EXACTLY (enumerate the shot/ + angle/focal/light/mark vocabularies from grammar.js in the prompt; read + the file at boot so the vocab never drifts). `format: json` / low temp. +- VALIDATE server-side before returning: unknown op/shot/preset/entity id → + drop the op and include it in a `rejected` list so the UI can toast it. + Never forward raw model text to the client. +- test_server.py: mock the LLM (in-process http.server, same pattern as your + MB mock): assert prompt carries the vocab, valid ops pass, garbage ops land + in `rejected`, 503 when env unset. + ### 2026-07-18 — M4 ACCEPTED. Lane C scope (M1–M4) is COMPLETE. Orchestrator verified in the integrated app: audio-muxed finalRender → h264+aac mp4, both streams exactly 8.000s; graft_limb `--selftest` green