b773d29776
Sequences (ordered shots = scene + in/out + transition) with full validation, CRUD, and a client-driven film render the server orchestrates: browser renders each shot through the existing /render/* endpoints, server ffmpeg-concatenates in order with cuts or xfade dissolves, muxes a film-level audio bed, reaps after 48h. render.js refactored so there is one frame-post path in the repo (captureFrames/withRenderSize/begin/end/poll); new web/film.js drives the loop. Also fixes a pre-existing mux bug: no apad meant short audio truncated PICTURE. Orchestrator: deploy compose/script now carry SCENEGOD_SEQUENCES + SCENEGOD_FILMS volumes so shot lists survive a rebuild. Verified live: demo-two-shot renders one 11.500s mp4 with a genuinely compositing dissolve; a film built from the M7 templates (street wide -> dissolve -> closer -> cut -> record store) renders 11.800s at 960x540. 15 test groups green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
204 lines
12 KiB
Markdown
204 lines
12 KiB
Markdown
# Lane C — Server: FastAPI, assets, scenes, render pipeline
|
||
|
||
You own: `scenegod/server.py`, `web/render.js`, `scripts/*`,
|
||
`requirements.txt`, `.gitignore`. Milestone: **M1**.
|
||
|
||
## C1. Scaffold (first commit)
|
||
- `requirements.txt`: `fastapi`, `uvicorn`. Nothing else.
|
||
- `.gitignore`: `.env`, `assets/`, `scenes/`, `renders/`, `__pycache__/`,
|
||
`*.pyc`, `.DS_Store`.
|
||
- `scenegod/server.py`, single file (split only when it hurts — MESHGOD's
|
||
server.py is one file at 800+ lines and fine). `SCENEGOD_PORT` 8020,
|
||
bind `127.0.0.1` (env `SCENEGOD_BIND` to override later — not now).
|
||
- Statics: `GET /` → `web/index.html` with
|
||
`Cache-Control: no-store` (MESHGOD stale-cache lesson); `/web/*` served
|
||
with short cache. Use FastAPI `FileResponse`/`StaticFiles`, headers set
|
||
explicitly for the HTML.
|
||
|
||
## C2. Assets endpoints
|
||
- Roots from env with defaults: `SCENEGOD_ASSETS=./assets`,
|
||
`SCENEGOD_SCENES=./scenes`, `SCENEGOD_RENDERS=./renders` (create scenes/
|
||
renders on boot; assets must exist — clear error if not).
|
||
- `GET /assets/tree`: walk `assets/{characters,animations,props,backdrops,audio}`.
|
||
Group rigroom-library style: files sharing dir+stem = one entry with a
|
||
`formats` list (`{"name":"lady","path":"characters/pack01/lady.glb",
|
||
"formats":["glb","fbx"]}`). Extensions: glb/gltf/fbx/obj/bvh + jpg/png/webp
|
||
(backdrops) + wav/mp3 (audio). Scan live, no index file, cache 5s.
|
||
- `GET /assets/file?path=`: **resolve and enforce
|
||
`path.resolve().is_relative_to(ASSETS_ROOT)`** — reject `..`, absolute
|
||
paths, symlink escapes (resolve first, then check). Correct content-types
|
||
(`model/gltf-binary`, etc.). This guard pattern applies to every
|
||
path-taking endpoint you ever add here (ship-check rule).
|
||
- Read `/Users/johnking/Documents/MESHGOD/meshgod/library.py` first — same
|
||
idea, steal the grouping approach.
|
||
|
||
## C3. Scenes CRUD
|
||
- `GET /scenes` → `[{name, mtime, duration}]`; `GET /scenes/{name}`;
|
||
`POST /scenes/{name}` with JSON body. `name` slugified `[a-z0-9-_]`, file
|
||
`scenes/{name}.json`, atomic write (tmp + rename).
|
||
- Validate on save (stdlib, no jsonschema dep): `version==1`, entity ids
|
||
unique, every track sorted by `t`, clip blocks non-overlapping per entity
|
||
(fade overlap allowed), `cameraCuts` reference existing camera entities.
|
||
Reject with a 422 listing every violation (agents and UI both rely on the
|
||
messages).
|
||
|
||
## C4. scripts/test_server.py (M1 gate)
|
||
Stdlib `urllib` + `subprocess` (uvicorn on a test port, tmp asset/scene
|
||
dirs): asserts tree grouping, file streaming, traversal attempts → 4xx
|
||
(`?path=../../etc/hosts`, absolute, URL-encoded `..`), scene round-trip,
|
||
validation failures (unsorted keys, dup ids). `python3 scripts/test_server.py`
|
||
green = M1 code-done. Run with `| lm -l 2`.
|
||
|
||
## C5. Render pipeline — M3, but design lands now (endpoints stubbed 501)
|
||
- Session model: `POST /render/begin {name,fps,width,height}` → `{renderId}`
|
||
(uuid, dir `renders/{id}/frames/`). `POST /render/{id}/frame/{n}`: raw PNG
|
||
body → `frames/{n:06d}.png`. `POST /render/{id}/end {audio?}` → spawn
|
||
ffmpeg in a background thread:
|
||
`ffmpeg -framerate {fps} -i frames/%06d.png -c:v libx264 -pix_fmt yuv420p
|
||
-crf 18 out.mp4` (audio inputs + `-map`/amix in M4). `GET /render/{id}/status`
|
||
= queued|encoding|done|error(+log tail); `GET /render/{id}/out.mp4`.
|
||
ffmpeg presence checked at boot (`shutil.which`), endpoint 503s w/ message
|
||
if absent. Reap render dirs >48h old on boot.
|
||
- `web/render.js` (M3): exports `draftRecord(stage, timeline)` —
|
||
`canvas.captureStream(fps)` + MediaRecorder while `timeline.play()`s, save
|
||
webm; and `finalRender(stage, timeline, opts)` — resize canvas, loop
|
||
`timeline.step(f)` → `stage.renderActiveCamera()` → `canvas.toBlob('image/png')`
|
||
→ POST frame (3-4 in flight, backpressure via await), then `/end`, poll
|
||
status, download link. Deterministic: never uses wall-clock time.
|
||
|
||
## C6. M4 (locked): audio, MODELBEAST, graft
|
||
- ffmpeg audio mux per scene JSON `audio[]`.
|
||
- `POST /mb/submit` proxy re-implementing the contract in MESHGOD
|
||
`meshgod/ops.py` (upload input to /api/assets first; token read from
|
||
`~/Documents/backnforth/.env` at request time, never logged; endpoint only
|
||
active when `SCENEGOD_MB=1`).
|
||
- `scripts/graft_limb.py` headless Blender (pattern:
|
||
MESHGOD `scripts/glb2fbx.py`): `--body body.glb --limb hand.glb --bone
|
||
mixamorig:RightHand` → align limb root bone to target bone head, join
|
||
armatures, parent-keep-offset, join meshes (weights survive), export. If
|
||
the body armature ALREADY has the bones the limb provides → exit 2 with
|
||
"bones exist; mitten weights — re-skin, don't graft". Never route output
|
||
through any finish/decimate path (rig contract).
|
||
|
||
## M1 acceptance (log it)
|
||
- `uvicorn scenegod.server:app --port 8020` serves Lane A's page (or a
|
||
placeholder index if A hasn't landed — don't block on them).
|
||
- test_server.py green; paste the summary line in your log.
|
||
|
||
## ORCHESTRATOR UPDATES
|
||
|
||
### 2026-07-18 — M1 ACCEPTED. **M2 unlocked AND M3 render pipeline green-lit early.**
|
||
test_server.py green, tree/file/scenes verified live through the browser page
|
||
(scene save 200 + lossless reload). You're ahead of the other lanes, so skip
|
||
the wait: after the small M2 items (thumbnails endpoint; validation already
|
||
solid), go straight into §C5 — implement `/render/begin|frame|end|status|out`
|
||
+ ffmpeg encode, and `web/render.js`. Notes:
|
||
1. `timeline.step(frame)` is verified deterministic against the real Stage
|
||
(orchestrator stepped 0/30/60/90/120 and got exact eased positions) — your
|
||
final-render loop can rely on it today. `stage.renderActiveCamera` as a
|
||
render target is Lane A M2 work; until it lands, test `finalRender` with
|
||
the director view canvas, and test the ffmpeg path with synthetic PNGs
|
||
from a script (extend test_server.py: begin → POST 30 generated frames →
|
||
end → poll → assert mp4 exists + ffprobe duration ≈ 1s).
|
||
2. ffmpeg on ultra: check `shutil.which` at boot as designed; it's installed
|
||
via homebrew here.
|
||
3. Test assets for the tree now exist under `assets/*/test/` (gitignored).
|
||
4. uvicorn for local dev: `.venv/bin/uvicorn scenegod.server:app --port 8020`
|
||
— there's also a launch entry in MESHGOD/.claude/launch.json ("scenegod")
|
||
used by the orchestrator's browser preview.
|
||
|
||
### 2026-07-18 — M2+M3 ACCEPTED, **SYNC 3 PASSED. M4 is UNLOCKED.**
|
||
Orchestrator ran your `finalRender` end-to-end in the integrated app against
|
||
Lane A's `renderActiveCamera`: 240 frames @1920×1080 → ffmpeg → out.mp4,
|
||
h264, exactly 8.000s, with camera cuts and sunset light lerp visible in
|
||
extracted frames. The browser client path you couldn't exercise is now
|
||
verified — no changes needed. Proceed with M4 in this order:
|
||
1. ffmpeg audio mux from scene `audio[]` (the sync2-sunset scene in
|
||
`scenes/` is your fixture; add a wav to `assets/audio/test/`).
|
||
2. `scripts/graft_limb.py` (spec in §C6 — blender binary is on ultra's PATH
|
||
as `blender`; verify with a mixamo body + any rigged hand GLB, and add
|
||
the refusal-path test).
|
||
3. `/mb/submit` MODELBEAST proxy last (needs `SCENEGOD_MB=1` gating like
|
||
riggermore; test only that auth/upload contract matches meshgod/ops.py —
|
||
don't burn farm jobs on repeat runs).
|
||
One render-quality note for your backlog, not urgent: draft webm via
|
||
MediaRecorder is untested in the embedded preview (rAF suspension) — verify
|
||
it once in a real Chrome window before calling it done.
|
||
|
||
### 2026-07-19 — **M5 DIRECTOR MODE unlocked**: `POST /director` (spec PLAN §5 M5).
|
||
- rhubarb is now INSTALLED on ultra (built from source, arm64, v1.14.0,
|
||
symlinked /opt/homebrew/bin/rhubarb) — your happy path is verified:
|
||
`GET /rhubarb?path=audio/test/vo.wav` → real mouthCues. Nothing to do there.
|
||
- `POST /director {text, scene:{entities:[{id,kind,label}], time}}` → chat
|
||
request to `SCENEGOD_LLM_URL` (OpenAI-compat; ultra's tailnet Ollama is
|
||
`http://100.69.21.128:11434`, model via `SCENEGOD_LLM_MODEL`). Orchestrator
|
||
checked /api/tags on that box: it serves `qwen2.5:7b` (+ an embed model) —
|
||
default SCENEGOD_LLM_MODEL to `qwen2.5:7b`; John can pull a gemma later
|
||
and it's just an env change. 503 with a hint when env unset. stdlib
|
||
urllib only.
|
||
- System prompt: translate director language into a JSON array of ops
|
||
matching Lane A's `scenegod:direct` contract EXACTLY (enumerate the shot/
|
||
angle/focal/light/mark vocabularies from grammar.js in the prompt; read
|
||
the file at boot so the vocab never drifts). `format: json` / low temp.
|
||
- VALIDATE server-side before returning: unknown op/shot/preset/entity id →
|
||
drop the op and include it in a `rejected` list so the UI can toast it.
|
||
Never forward raw model text to the client.
|
||
- test_server.py: mock the LLM (in-process http.server, same pattern as your
|
||
MB mock): assert prompt carries the vocab, valid ops pass, garbage ops land
|
||
in `rejected`, 503 when env unset.
|
||
|
||
### 2026-07-18 — M4 ACCEPTED. Lane C scope (M1–M4) is COMPLETE.
|
||
Orchestrator verified in the integrated app: audio-muxed finalRender →
|
||
h264+aac mp4, both streams exactly 8.000s; graft_limb `--selftest` green
|
||
under Blender 5.0.1 (both refusal + graft paths); 8 server test groups pass.
|
||
Your blender-path correction noted (it's `/Applications/Blender.app/...`,
|
||
not on PATH — my error). Standing by is fine; remaining work is A/B UI
|
||
(viseme lane, audio rows). One item if you want it: add a `rhubarb` check +
|
||
`/rhubarb?path=` endpoint (subprocess → JSON) IF rhubarb is installed —
|
||
503-with-message otherwise, same pattern as ffmpeg. Log a REQUEST first if
|
||
it needs more than that.
|
||
|
||
### 2026-07-19 — M6 FLOW VIDEO PLATES queued behind M5 (full spec PLAN §5 M6).
|
||
Do M5 first; M6 is unlocked the moment your M5 acceptance is logged. Test
|
||
media will be arriving in assets/backdrops/video/ via scripts/flow_intake.py.
|
||
|
||
### 2026-07-20 — M5+M6 ACCEPTED at SYNC 6 (all lanes ran as orchestrator subagents).
|
||
Verified live: DIRECT bar CU/low/frame → camera+keys+cut, ONE undo reverts;
|
||
golden_hour keys; video plate follows the clock frame-exact; finalRender shows
|
||
the Flow plate behind an animating character. Orchestrator seam fixes to absorb:
|
||
(1) server: VIDEO_EXTS in tree/media-types, video beats poster as primary;
|
||
(2) stage.entityVideo(id) added (Lane B seam); (3) syncVideos forces
|
||
videoTex.needsUpdate — r160 VideoTexture rides requestVideoFrameCallback which
|
||
never fires in hidden tabs → black plates in renders. Backlog carried: honor
|
||
audio in/out at mux (C), walk-to-mark prevPos pre-apply (A), live viseme asset
|
||
still owed by John.
|
||
|
||
### 2026-07-25 — M7-C ACCEPTED at SYNC 8.
|
||
Verified live after a server restart: /templates lists all three in order with thumbs;
|
||
two-hander-golden-hour loads 7 entities (plate, man, lady, cam1, cam2, sun, amb), 8s,
|
||
both performers carrying clips, and the cut fires cam1 to cam2 across t=4.5; record-store
|
||
(5 ents, 6s) and empty-stage (3 ents, 10s) both clean. 10 test groups green. The
|
||
anti-rot check (every referenced asset must exist) is exactly the right instinct — keep
|
||
that pattern for anything else that ships data alongside code.
|
||
NEXT for Lane C when you want it: the post-M7 backlog in PLAN §5 is yours to start —
|
||
multi-shot SEQUENCING (ordered shot list of scenes, rendered and ffmpeg-concatenated
|
||
into one film) is the biggest single unlock left in the project.
|
||
|
||
### 2026-07-25 — M8-C ACCEPTED at SYNC 10. **Sequencing works; SCENEGOD makes FILMS now.**
|
||
Verified live end-to-end, twice. (1) Your shipped `demo-two-shot` → one 11.500s mp4; the
|
||
frame at t=4.25 is visibly TWO shots superimposed, so the xfade really composites, and the
|
||
hard cut into shot 3 lands. (2) A fresh film built from the M7 templates — saved
|
||
two-hander-golden-hour and record-store as scenes, cut them into `milk-bar-film` (wide →
|
||
dissolve → closer → cut → interior) → 11.800s @960×540, correct on every boundary.
|
||
Your two flagged deploy lines are done by the orchestrator: compose now sets
|
||
SCENEGOD_SEQUENCES/SCENEGOD_FILMS with `/opt/scenegod-data/{sequences,films}` volumes, and
|
||
deploy.sh mkdirs both + excludes `films`/`*.log`.
|
||
Excellent instincts this round: ffprobe'ing real durations for xfade offsets rather than
|
||
trusting nominal length, validating dissolve-shorter-than-both-shots (xfade's actual
|
||
requirement), persisting the film plan so /end survives a restart, and catching the
|
||
pre-existing missing `apad` that let short audio truncate the PICTURE.
|
||
NEXT for Lane C: audio `in`/`out` at mux is STILL the open gap (Lane B's head-trim made it
|
||
easy to hit) — close that. Then beat-sync groundwork: a `POST /beats?path=` that runs a beat
|
||
detector over an audio asset and returns a grid the timeline can snap to (MODELBEAST demucs
|
||
is available on ultra; the endpoint should 503 cleanly on the VPS like the others).
|