[m5] DIRECTOR MODE round: cinema-grammar presets spec + /director LLM lane; rhubarb live
- PLAN §5 M5: shot sizes/angles/focals/light presets/blocking marks as one-click deterministic presets (grammar.js data + presets.js solver, Lane A), keyed at the playhead via scenegod:direct events (Lane B), optional NL box -> POST /director -> tailnet Ollama with server-side op validation (Lane C; m4pro serves qwen2.5:7b) - rhubarb 1.14.0 built from John's source zip (JING5), installed arm64 at ~/.local/opt/rhubarb, symlinked /opt/homebrew/bin; happy path verified: /rhubarb on a real spoken wav (assets/audio/test/vo.wav) returns mouthCues - lane B unlocked to wire the lip-sync-from-audio button now Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
866551f266
commit
a8273127ba
20
PLAN.md
20
PLAN.md
@ -192,6 +192,26 @@ tts/voice-clone submit (mb contract from meshgod/ops.py, token from
|
|||||||
B: audio tracks on the timeline UI, Web Audio sync. A: viseme lane — detect
|
B: audio tracks on the timeline UI, Web Audio sync. A: viseme lane — detect
|
||||||
morph targets, Rhubarb-JSON → morph keyframes, greyed out when absent.
|
morph targets, Rhubarb-JSON → morph keyframes, greyed out when absent.
|
||||||
|
|
||||||
|
**M5 — DIRECTOR MODE (ease of use)**: classic cinema grammar as one-click
|
||||||
|
presets, LLM strictly optional on top.
|
||||||
|
- Shared vocabulary in `web/grammar.js` (Lane A owns, data only): shot sizes
|
||||||
|
ECU/CU/MCU/MS/MWS/WS/EWS (target vertical span as a fraction of subject
|
||||||
|
height + frame-center height), angles (eye/low/high/dutch/OTS-L/OTS-R),
|
||||||
|
focal lengths 18/24/35/50/85mm (vertical fov = 2·atan(12/f), full-frame),
|
||||||
|
lighting presets (day, golden-hour, night, noir, horror, neon, overcast),
|
||||||
|
blocking marks (center, two-shot L/R, walk-to-mark, face-camera, face-other).
|
||||||
|
- Lane A `web/presets.js`: `frameSubject(camId, subjectId, shot, angle,
|
||||||
|
focalmm)` — deterministic solver from the subject's bbox (no LLM), light/
|
||||||
|
blocking preset appliers, and the DIRECT panel UI (shot buttons on the
|
||||||
|
selected subject, light preset buttons, mark buttons).
|
||||||
|
- Lane B: preset actions land as normal keys/cuts AT THE PLAYHEAD through
|
||||||
|
existing mutators (undo-able); "walk to mark" = 2 transform keys.
|
||||||
|
- Lane C: `POST /director` — optional NL box ("medium two-shot, golden hour,
|
||||||
|
lady walks to camera") → tailnet Ollama (OpenAI-compat, env
|
||||||
|
`SCENEGOD_LLM_URL` + `SCENEGOD_LLM_MODEL`, 503 when unset) → strict JSON
|
||||||
|
ops validated server-side → client applies via the SAME preset functions.
|
||||||
|
Buttons must never depend on the LLM lane.
|
||||||
|
|
||||||
## 6. How John launches the lanes
|
## 6. How John launches the lanes
|
||||||
|
|
||||||
Three Claude Code sessions (Opus 4.8), cwd `~/Documents/SCENEGOD`, one prompt
|
Three Claude Code sessions (Opus 4.8), cwd `~/Documents/SCENEGOD`, one prompt
|
||||||
|
|||||||
@ -173,3 +173,32 @@ asserts L === [v_tongue_up] and E includes the stripped smile. Lesson worth
|
|||||||
keeping: assert group CONTENTS, not just which groups exist.
|
keeping: assert group CONTENTS, not just which groups exist.
|
||||||
Still owed by John, not you: a viseme-bearing test character for a live
|
Still owed by John, not you: a viseme-bearing test character for a live
|
||||||
badge check. Standing by.
|
badge check. Standing by.
|
||||||
|
|
||||||
|
### 2026-07-19 — **M5 DIRECTOR MODE unlocked** (spec in PLAN §5 M5). Your part:
|
||||||
|
1. `web/grammar.js` — pure data, you own it, everyone reads it: SHOTS
|
||||||
|
(ECU .12H@.93H, CU .25H@.87H, MCU .45H@.75H, MS .65H@.65H, MWS .85H@.55H,
|
||||||
|
WS 1.2H@.5H, EWS 2.5H@.5H — vertical span fraction of subject height @
|
||||||
|
frame-center height), ANGLES (eye/low/high/dutch 8°roll/OTS-L/OTS-R),
|
||||||
|
FOCALS [18,24,35,50,85] (fovV = 2·atan(12/f) in degrees), LIGHTS (day,
|
||||||
|
golden_hour, night, noir, horror, neon, overcast — each {sunColor,
|
||||||
|
sunIntensity, sunPos-ish elevation/azimuth, ambient, bg}), MARKS (center,
|
||||||
|
two_shot_L/R, face_camera, face_other).
|
||||||
|
2. `web/presets.js` — `frameSubject(stage, camId, subjectId, {shot, angle,
|
||||||
|
focal})`: bbox of subject → camera pos/rot/fov so the shot's span fills
|
||||||
|
the frame; OTS places the cam over the OTHER selected entity's shoulder.
|
||||||
|
`applyLight(stage, presetName)` → setParam on sun/ambient + background.
|
||||||
|
`applyMark(stage, id, mark)` → transform (marks are stage-relative).
|
||||||
|
Pure functions returning the values AND applying them — Lane B needs the
|
||||||
|
values to write keys.
|
||||||
|
3. DIRECT panel UI (top toolbar or dock tab): with an entity selected —
|
||||||
|
shot-size buttons, angle chips, focal select, "🎥 frame it" (creates/moves
|
||||||
|
a camera + fires one `scenegod:direct` CustomEvent with the op payload,
|
||||||
|
detail below); light preset buttons; mark buttons. Event contract (B
|
||||||
|
consumes): `scenegod:direct {detail: {op:'shot'|'light'|'mark', ...values,
|
||||||
|
entityIds}}` — YOU apply nothing to the timeline; B keys it.
|
||||||
|
4. NL box (small text input in the panel) POSTs to `/director` (Lane C) and
|
||||||
|
dispatches the returned ops through the same `scenegod:direct` events.
|
||||||
|
Hidden entirely when /director 503s.
|
||||||
|
Acceptance: select man → CU low 35mm → camera frames his face from below in
|
||||||
|
one click; golden_hour relights the scene in one click; every action lands
|
||||||
|
as keys via B (scrub proves it) and is undo-able.
|
||||||
|
|||||||
@ -148,3 +148,17 @@ button — POST the audio path to Lane C's `/rhubarb`, feed the JSON to
|
|||||||
importRhubarb (503 → toast the install hint); (2) audio block trim; (3) UX
|
importRhubarb (503 → toast the install hint); (2) audio block trim; (3) UX
|
||||||
nit found live: mouse wheel over the track canvas zooms, so users may not
|
nit found live: mouse wheel over the track canvas zooms, so users may not
|
||||||
discover the panel scrolls — consider wheel = scroll, Cmd/Ctrl+wheel = zoom.
|
discover the panel scrolls — consider wheel = scroll, Cmd/Ctrl+wheel = zoom.
|
||||||
|
|
||||||
|
### 2026-07-19 — **M5 DIRECTOR MODE unlocked** (spec in PLAN §5 M5). Your part:
|
||||||
|
1. Listen for `scenegod:direct` (contract in lanes/A §M5.3): every preset op
|
||||||
|
becomes NORMAL keys/cuts at the playhead through your existing mutators —
|
||||||
|
`shot` → transform+fov keys on the camera + a cut to it; `light` → param
|
||||||
|
keys on sun/ambient (+bg); `mark` → transform key; `walk_to_mark` → key at
|
||||||
|
playhead-with-current-pos + key at playhead+2s at the mark. All undo-able
|
||||||
|
(one undo entry per op — group the batch, don't leave 5 half-undoable keys).
|
||||||
|
2. Wire the "lip-sync from audio" button now that `/rhubarb` is LIVE and
|
||||||
|
verified (`GET /rhubarb?path=audio/test/vo.wav` returns real mouthCues —
|
||||||
|
vo.wav is in assets): on an audio row, 🗣 button → fetch cues →
|
||||||
|
`importRhubarb(characterId, json, audioStart)`; character picker if >1
|
||||||
|
viseme-bearing char; toast the 503 install hint if absent.
|
||||||
|
3. Session-5 backlog if time remains: audio trim, wheel-scroll fix (above).
|
||||||
|
|||||||
@ -125,6 +125,28 @@ One render-quality note for your backlog, not urgent: draft webm via
|
|||||||
MediaRecorder is untested in the embedded preview (rAF suspension) — verify
|
MediaRecorder is untested in the embedded preview (rAF suspension) — verify
|
||||||
it once in a real Chrome window before calling it done.
|
it once in a real Chrome window before calling it done.
|
||||||
|
|
||||||
|
### 2026-07-19 — **M5 DIRECTOR MODE unlocked**: `POST /director` (spec PLAN §5 M5).
|
||||||
|
- rhubarb is now INSTALLED on ultra (built from source, arm64, v1.14.0,
|
||||||
|
symlinked /opt/homebrew/bin/rhubarb) — your happy path is verified:
|
||||||
|
`GET /rhubarb?path=audio/test/vo.wav` → real mouthCues. Nothing to do there.
|
||||||
|
- `POST /director {text, scene:{entities:[{id,kind,label}], time}}` → chat
|
||||||
|
request to `SCENEGOD_LLM_URL` (OpenAI-compat; ultra's tailnet Ollama is
|
||||||
|
`http://100.69.21.128:11434`, model via `SCENEGOD_LLM_MODEL`). Orchestrator
|
||||||
|
checked /api/tags on that box: it serves `qwen2.5:7b` (+ an embed model) —
|
||||||
|
default SCENEGOD_LLM_MODEL to `qwen2.5:7b`; John can pull a gemma later
|
||||||
|
and it's just an env change. 503 with a hint when env unset. stdlib
|
||||||
|
urllib only.
|
||||||
|
- System prompt: translate director language into a JSON array of ops
|
||||||
|
matching Lane A's `scenegod:direct` contract EXACTLY (enumerate the shot/
|
||||||
|
angle/focal/light/mark vocabularies from grammar.js in the prompt; read
|
||||||
|
the file at boot so the vocab never drifts). `format: json` / low temp.
|
||||||
|
- VALIDATE server-side before returning: unknown op/shot/preset/entity id →
|
||||||
|
drop the op and include it in a `rejected` list so the UI can toast it.
|
||||||
|
Never forward raw model text to the client.
|
||||||
|
- test_server.py: mock the LLM (in-process http.server, same pattern as your
|
||||||
|
MB mock): assert prompt carries the vocab, valid ops pass, garbage ops land
|
||||||
|
in `rejected`, 503 when env unset.
|
||||||
|
|
||||||
### 2026-07-18 — M4 ACCEPTED. Lane C scope (M1–M4) is COMPLETE.
|
### 2026-07-18 — M4 ACCEPTED. Lane C scope (M1–M4) is COMPLETE.
|
||||||
Orchestrator verified in the integrated app: audio-muxed finalRender →
|
Orchestrator verified in the integrated app: audio-muxed finalRender →
|
||||||
h264+aac mp4, both streams exactly 8.000s; graft_limb `--selftest` green
|
h264+aac mp4, both streams exactly 8.000s; graft_limb `--selftest` green
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user