[m5] DIRECTOR MODE round: cinema-grammar presets spec + /director LLM lane; rhubarb live

- PLAN §5 M5: shot sizes/angles/focals/light presets/blocking marks as
  one-click deterministic presets (grammar.js data + presets.js solver,
  Lane A), keyed at the playhead via scenegod:direct events (Lane B),
  optional NL box -> POST /director -> tailnet Ollama with server-side op
  validation (Lane C; m4pro serves qwen2.5:7b)
- rhubarb 1.14.0 built from John's source zip (JING5), installed arm64 at
  ~/.local/opt/rhubarb, symlinked /opt/homebrew/bin; happy path verified:
  /rhubarb on a real spoken wav (assets/audio/test/vo.wav) returns mouthCues
- lane B unlocked to wire the lip-sync-from-audio button now

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
type-two 2026-07-19 01:09:30 +10:00
parent 866551f266
commit a8273127ba
4 changed files with 85 additions and 0 deletions

20
PLAN.md
View File

@ -192,6 +192,26 @@ tts/voice-clone submit (mb contract from meshgod/ops.py, token from
B: audio tracks on the timeline UI, Web Audio sync. A: viseme lane — detect B: audio tracks on the timeline UI, Web Audio sync. A: viseme lane — detect
morph targets, Rhubarb-JSON → morph keyframes, greyed out when absent. morph targets, Rhubarb-JSON → morph keyframes, greyed out when absent.
**M5 — DIRECTOR MODE (ease of use)**: classic cinema grammar as one-click
presets, LLM strictly optional on top.
- Shared vocabulary in `web/grammar.js` (Lane A owns, data only): shot sizes
ECU/CU/MCU/MS/MWS/WS/EWS (target vertical span as a fraction of subject
height + frame-center height), angles (eye/low/high/dutch/OTS-L/OTS-R),
focal lengths 18/24/35/50/85mm (vertical fov = 2·atan(12/f), full-frame),
lighting presets (day, golden-hour, night, noir, horror, neon, overcast),
blocking marks (center, two-shot L/R, walk-to-mark, face-camera, face-other).
- Lane A `web/presets.js`: `frameSubject(camId, subjectId, shot, angle,
focalmm)` — deterministic solver from the subject's bbox (no LLM), light/
blocking preset appliers, and the DIRECT panel UI (shot buttons on the
selected subject, light preset buttons, mark buttons).
- Lane B: preset actions land as normal keys/cuts AT THE PLAYHEAD through
existing mutators (undo-able); "walk to mark" = 2 transform keys.
- Lane C: `POST /director` — optional NL box ("medium two-shot, golden hour,
lady walks to camera") → tailnet Ollama (OpenAI-compat, env
`SCENEGOD_LLM_URL` + `SCENEGOD_LLM_MODEL`, 503 when unset) → strict JSON
ops validated server-side → client applies via the SAME preset functions.
Buttons must never depend on the LLM lane.
## 6. How John launches the lanes ## 6. How John launches the lanes
Three Claude Code sessions (Opus 4.8), cwd `~/Documents/SCENEGOD`, one prompt Three Claude Code sessions (Opus 4.8), cwd `~/Documents/SCENEGOD`, one prompt

View File

@ -173,3 +173,32 @@ asserts L === [v_tongue_up] and E includes the stripped smile. Lesson worth
keeping: assert group CONTENTS, not just which groups exist. keeping: assert group CONTENTS, not just which groups exist.
Still owed by John, not you: a viseme-bearing test character for a live Still owed by John, not you: a viseme-bearing test character for a live
badge check. Standing by. badge check. Standing by.
### 2026-07-19 — **M5 DIRECTOR MODE unlocked** (spec in PLAN §5 M5). Your part:
1. `web/grammar.js` — pure data, you own it, everyone reads it: SHOTS
(ECU .12H@.93H, CU .25H@.87H, MCU .45H@.75H, MS .65H@.65H, MWS .85H@.55H,
WS 1.2H@.5H, EWS 2.5H@.5H — vertical span fraction of subject height @
frame-center height), ANGLES (eye/low/high/dutch 8°roll/OTS-L/OTS-R),
FOCALS [18,24,35,50,85] (fovV = 2·atan(12/f) in degrees), LIGHTS (day,
golden_hour, night, noir, horror, neon, overcast — each {sunColor,
sunIntensity, sunPos-ish elevation/azimuth, ambient, bg}), MARKS (center,
two_shot_L/R, face_camera, face_other).
2. `web/presets.js` — `frameSubject(stage, camId, subjectId, {shot, angle,
focal})`: bbox of subject → camera pos/rot/fov so the shot's span fills
the frame; OTS places the cam over the OTHER selected entity's shoulder.
`applyLight(stage, presetName)` → setParam on sun/ambient + background.
`applyMark(stage, id, mark)` → transform (marks are stage-relative).
Pure functions returning the values AND applying them — Lane B needs the
values to write keys.
3. DIRECT panel UI (top toolbar or dock tab): with an entity selected —
shot-size buttons, angle chips, focal select, "🎥 frame it" (creates/moves
a camera + fires one `scenegod:direct` CustomEvent with the op payload,
detail below); light preset buttons; mark buttons. Event contract (B
consumes): `scenegod:direct {detail: {op:'shot'|'light'|'mark', ...values,
entityIds}}` — YOU apply nothing to the timeline; B keys it.
4. NL box (small text input in the panel) POSTs to `/director` (Lane C) and
dispatches the returned ops through the same `scenegod:direct` events.
Hidden entirely when /director 503s.
Acceptance: select man → CU low 35mm → camera frames his face from below in
one click; golden_hour relights the scene in one click; every action lands
as keys via B (scrub proves it) and is undo-able.

View File

@ -148,3 +148,17 @@ button — POST the audio path to Lane C's `/rhubarb`, feed the JSON to
importRhubarb (503 → toast the install hint); (2) audio block trim; (3) UX importRhubarb (503 → toast the install hint); (2) audio block trim; (3) UX
nit found live: mouse wheel over the track canvas zooms, so users may not nit found live: mouse wheel over the track canvas zooms, so users may not
discover the panel scrolls — consider wheel = scroll, Cmd/Ctrl+wheel = zoom. discover the panel scrolls — consider wheel = scroll, Cmd/Ctrl+wheel = zoom.
### 2026-07-19 — **M5 DIRECTOR MODE unlocked** (spec in PLAN §5 M5). Your part:
1. Listen for `scenegod:direct` (contract in lanes/A §M5.3): every preset op
becomes NORMAL keys/cuts at the playhead through your existing mutators —
`shot` → transform+fov keys on the camera + a cut to it; `light` → param
keys on sun/ambient (+bg); `mark` → transform key; `walk_to_mark` → key at
playhead-with-current-pos + key at playhead+2s at the mark. All undo-able
(one undo entry per op — group the batch, don't leave 5 half-undoable keys).
2. Wire the "lip-sync from audio" button now that `/rhubarb` is LIVE and
verified (`GET /rhubarb?path=audio/test/vo.wav` returns real mouthCues —
vo.wav is in assets): on an audio row, 🗣 button → fetch cues →
`importRhubarb(characterId, json, audioStart)`; character picker if >1
viseme-bearing char; toast the 503 install hint if absent.
3. Session-5 backlog if time remains: audio trim, wheel-scroll fix (above).

View File

@ -125,6 +125,28 @@ One render-quality note for your backlog, not urgent: draft webm via
MediaRecorder is untested in the embedded preview (rAF suspension) — verify MediaRecorder is untested in the embedded preview (rAF suspension) — verify
it once in a real Chrome window before calling it done. it once in a real Chrome window before calling it done.
### 2026-07-19 — **M5 DIRECTOR MODE unlocked**: `POST /director` (spec PLAN §5 M5).
- rhubarb is now INSTALLED on ultra (built from source, arm64, v1.14.0,
symlinked /opt/homebrew/bin/rhubarb) — your happy path is verified:
`GET /rhubarb?path=audio/test/vo.wav` → real mouthCues. Nothing to do there.
- `POST /director {text, scene:{entities:[{id,kind,label}], time}}` → chat
request to `SCENEGOD_LLM_URL` (OpenAI-compat; ultra's tailnet Ollama is
`http://100.69.21.128:11434`, model via `SCENEGOD_LLM_MODEL`). Orchestrator
checked /api/tags on that box: it serves `qwen2.5:7b` (+ an embed model) —
default SCENEGOD_LLM_MODEL to `qwen2.5:7b`; John can pull a gemma later
and it's just an env change. 503 with a hint when env unset. stdlib
urllib only.
- System prompt: translate director language into a JSON array of ops
matching Lane A's `scenegod:direct` contract EXACTLY (enumerate the shot/
angle/focal/light/mark vocabularies from grammar.js in the prompt; read
the file at boot so the vocab never drifts). `format: json` / low temp.
- VALIDATE server-side before returning: unknown op/shot/preset/entity id →
drop the op and include it in a `rejected` list so the UI can toast it.
Never forward raw model text to the client.
- test_server.py: mock the LLM (in-process http.server, same pattern as your
MB mock): assert prompt carries the vocab, valid ops pass, garbage ops land
in `rejected`, 503 when env unset.
### 2026-07-18 — M4 ACCEPTED. Lane C scope (M1M4) is COMPLETE. ### 2026-07-18 — M4 ACCEPTED. Lane C scope (M1M4) is COMPLETE.
Orchestrator verified in the integrated app: audio-muxed finalRender → Orchestrator verified in the integrated app: audio-muxed finalRender →
h264+aac mp4, both streams exactly 8.000s; graft_limb `--selftest` green h264+aac mp4, both streams exactly 8.000s; graft_limb `--selftest` green