e05a1d2946
/beats?path= returns bpm, a phase-locked beat grid, downbeats (4/4) and an honest confidence, using only stdlib + ffmpeg: decode to PCM, log-energy onset flux, harmonic-enhanced autocorrelation with an octave check, comb phase search, weighted LS grid refit. 4-min track analyses in ~0.5s, cached after. Audio in/out now honoured at mux in both scene renders and film beds via one resolve_audio() path. Found en route: on this ffmpeg build the obvious atrim/asetpts + amix/apad chain yields an aac stream with ZERO samples, rc=0, no warning - trim moved to the input side; and the test helper would have passed on that empty stream, so it now refuses to answer on a window that decodes nothing. Tempo bias fixed after orchestrator ground-truth check: decoding at 22050 discarded the HF transient, and since the attack's HF share grew across a file the surviving LF onset landed progressively later - a per-beat-varying lateness that reads as tempo error. BEAT_SR now 44100/512 (same 86fps envelope). Second bug found by sweeping: harmonics outside the lag table counted as zero and the parabola was fitted on a slope, so 118 BPM read 114.79. Verified vs independent ground truth: 120.030 vs 120.003 true, 128.020 vs 128.004; worst offset from the true grid 6.0/10.1ms over the whole file (was 45/52ms). Tests now assert whole-file DRIFT, not average BPM, incl. a 200s track and hftick.wav (16kHz tick over a 60Hz body) which pins BEAT_SR. 18 groups green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
531 lines
36 KiB
Markdown
531 lines
36 KiB
Markdown
# Lane C log — append-only, newest at bottom
|
||
|
||
Format per session:
|
||
|
||
## <date> session N
|
||
DONE: …
|
||
DECISIONS: …
|
||
BLOCKED/REQUESTS: …
|
||
NEXT: …
|
||
|
||
## 2026-07-18 session 1
|
||
DONE: M1 code-complete. Commit bb85eef.
|
||
- `requirements.txt` (fastapi, uvicorn only); `.gitignore` += `.venv/`.
|
||
- `scenegod/server.py` single file: statics (`/` no-store placeholder,
|
||
`/web/{path}` short-cache, path-guarded), `/assets/tree` (5s cache,
|
||
per-category grouping), `/assets/file` (traversal-guarded), `/scenes`
|
||
CRUD with atomic write + slugify + `validate_scene`, render endpoints
|
||
(begin/frame accept + save; end/status/out 501 until M3).
|
||
- `scripts/test_server.py`: 5 groups green (tree grouping, file stream,
|
||
4 traversal attempts rejected, scene round-trip w/ slugify, 5 validation
|
||
failures + 1 fade-overlap-allowed). `OK: 5 test groups passed`.
|
||
DECISIONS:
|
||
- System python3 is 3.14 (fastapi present but pydantic missing); made a
|
||
`.venv` on python3.11 per CLAUDE.md stack rule. Tests + server use
|
||
`.venv/bin/python`. `.venv/` gitignored.
|
||
- `web/` is under `scenegod/web/` per PLAN §2 (not repo root) — `WEB`
|
||
points there; Lane A/B statics resolve via `/web/{path}`.
|
||
- Asset grouping: per-category primary ext set (models for chars/anims/
|
||
props, images for backdrops, audio for audio). Non-primary image sharing
|
||
a stem → `thumb`, not a `format`. `path` prefers glb else first model.
|
||
Entries with no primary format dropped (thumb-only stems aren't assets).
|
||
- Clip-overlap check: violation if `next.start < a.end - a.fade`, where
|
||
`a.end = start + (out-in)*loop`. Fade-region overlap allowed.
|
||
BLOCKED/REQUESTS: none.
|
||
NEXT: M2 (when unlocked): scene-validation hardening + asset thumbnails
|
||
endpoint. M3 render pipeline (ffmpeg encode + web/render.js) is designed in
|
||
C-server.md §C5, endpoints already stubbed — implement at SYNC-2/M3.
|
||
|
||
## 2026-07-18 session 2
|
||
DONE: M2 thumbnails + M3 render pipeline (orchestrator green-lit both). Commit b627a0f.
|
||
- `/render/begin` (uuid dir + meta.json, 503 if no ffmpeg), `/frame/{n}`
|
||
(PNG->frames/{n:06d}.png, 400 on empty), `/end` (spawns bg ffmpeg thread:
|
||
libx264 crf18 yuv420p, 400 if no frames), `/status` (queued|encoding|
|
||
done|error+log tail; on-disk out.mp4 fallback after restart), `/out.mp4`.
|
||
- Boot-time `reap_renders()` deletes render dirs >48h.
|
||
- `/assets/thumb?path=` serves same-stem image sidecar (404 if none).
|
||
- `web/render.js`: draftRecord (MediaRecorder webm), finalRender
|
||
(timeline.step -> renderActiveCamera -> toBlob -> POST, <=4 in flight,
|
||
resize+restore renderer), download() helper. node --check clean.
|
||
- `test_server.py`: added render group (ffmpeg lavfi -> 30 PNGs -> begin ->
|
||
post -> end -> poll -> ffprobe duration ~1s) + thumbnail asserts.
|
||
`OK: 6 test groups passed`.
|
||
DECISIONS:
|
||
- Render state in-memory (single-process local tool); status() infers from
|
||
disk after a restart. ponytail-commented.
|
||
- Thumbnails pass-through, no resize — Pillow would be a new dep (stack rule
|
||
says log a REQUEST first); not worth it until dock load is measurably slow.
|
||
- finalRender total frames = round(duration*fps); relies on timeline.step
|
||
being deterministic (orchestrator verified).
|
||
BLOCKED/REQUESTS:
|
||
- finalRender's per-frame path depends on Lane A `stage.renderActiveCamera`
|
||
as a real render target (their M2). Server + ffmpeg path fully tested with
|
||
synthetic frames; the browser client path is unexercised until A lands it.
|
||
NEXT: M4 is locked. When unlocked: ffmpeg audio mux (scene audio[]),
|
||
/mb/submit MODELBEAST proxy, scripts/graft_limb.py. Meanwhile available to
|
||
integration-test finalRender once Lane A's renderActiveCamera lands.
|
||
|
||
## 2026-07-18 session 3
|
||
DONE: M4 complete (unlocked after SYNC 3). Commit be65355.
|
||
- Audio mux: `/render/end` now accepts `{audio:[{path,start,gain}]}`, muxes via
|
||
ffmpeg adelay+volume+amix -> aac -shortest; audio paths path-guarded (400 on
|
||
missing). `render.js finalRender` passes `opts.audio` through.
|
||
- `scripts/graft_limb.py`: headless Blender (`blender -b -P ... -- --body B
|
||
--limb L --bone NAME --out O`). Align limb root head->target bone world head,
|
||
join armatures, parent limb root under target (keep offset), join meshes
|
||
(vgroup weights survive). Refusal: overlapping bones -> GraftRefused ->
|
||
exit 2 "re-skin, don't graft". No finish/decimate. `--selftest` builds
|
||
synthetic body/limb GLBs and verifies refusal + graft paths under Blender.
|
||
- `/mb/submit`: MODELBEAST proxy, SCENEGOD_MB=1 gated (503 otherwise), token
|
||
from ~/Documents/backnforth/.env at request time (never logged; errors
|
||
scrub it), /api/assets multipart -> /api/jobs, mirrors meshgod/ops.py.
|
||
- `test_server.py`: 8 groups green (added audio-stream assert, MB gating,
|
||
MB two-step contract via in-process mock — no farm jobs).
|
||
DECISIONS:
|
||
- Blender 5 glTF importer injects a stray unparented 42-vert icosphere on
|
||
every import; `_import` now keeps only rig-connected meshes (parented to
|
||
armature / has vgroups / armature modifier) and deletes strays. Right filter
|
||
for a graft tool regardless.
|
||
- graft refusal check: provides = limb bones - {limb root}; refuse if
|
||
provides ∩ body bones. Attach/root bone sharing the target name is expected.
|
||
- MB test uses a stdlib http.server mock + a 2nd scenegod instance; asserts
|
||
Bearer token forwarded, multipart upload, asset_id chained into job payload.
|
||
BLOCKED/REQUESTS: none.
|
||
BACKLOG (from orchestrator, not urgent): draft webm via MediaRecorder is
|
||
untested in the embedded preview (rAF suspension). Needs one verification in a
|
||
real Chrome window before calling draftRecord done — client-only, needs Lane A
|
||
stage running live; deferred.
|
||
NEXT: Lane C M1-M4 all shipped. Available for polish / integration support.
|
||
Blender binary here is /Applications/Blender.app/Contents/MacOS/Blender (not
|
||
on PATH as `blender` in this session, contrary to orchestrator note).
|
||
|
||
## 2026-07-18 session 3b
|
||
DONE: optional item from orchestrator — `/rhubarb?path=` endpoint. Commit fb59601.
|
||
- `GET /rhubarb?path=` (path-guarded) -> `rhubarb -f json <wav>` -> returns
|
||
`{metadata, mouthCues:[{start,end,value}]}` for Lane A's viseme lane.
|
||
503 with install hint when the binary is absent (ffmpeg-style gate).
|
||
- test_server.py: +503-when-absent assert (9 groups green).
|
||
DECISIONS: synchronous subprocess (VO clips are short); rhubarb NOT installed
|
||
on ultra, so only the 503 path is exercised here — happy path (JSON parse) is
|
||
untested until `rhubarb` is on PATH. Flagged same as MediaRecorder backlog.
|
||
BLOCKED/REQUESTS: none. To fully verify: install rhubarb-lip-sync on ultra,
|
||
then GET /rhubarb?path=audio/<some>.wav.
|
||
NEXT: nothing outstanding — Lane C M1-M4 + rhubarb shipped. Standing by.
|
||
|
||
## 2026-07-20 session 4
|
||
DONE: M5 `/director` + M6 render.js sliver.
|
||
- `scenegod/server.py`: `POST /director` {text, scene:{entities,time}} ->
|
||
OpenAI-compat chat (`SCENEGOD_LLM_URL` + `/v1/chat/completions`, model
|
||
`SCENEGOD_LLM_MODEL` default `qwen2.5:7b`, temp 0.1, response_format
|
||
json_object, stdlib urllib, 120s timeout). 503 + env hint when URL unset;
|
||
502 on LLM error/unreachable/unparseable output — raw model text is NEVER
|
||
forwarded. `_validate_ops`: ops shot/light/mark checked against vocab +
|
||
scene entity ids; violations dropped into `rejected` [{op, reason}];
|
||
response {ops, rejected}. Shot defaults angle="eye", focal=35 filled
|
||
server-side.
|
||
- Vocab (SHOTS/ANGLES/FOCALS/LIGHTS/MARKS) defined ONCE at top of the
|
||
director section, mirroring PLAN §5 M5.
|
||
- `web/render.js` finalRender: after `timeline.step(f)`, before capture:
|
||
`if (stage.syncVideos) await stage.syncVideos(timeline.time)` — guarded
|
||
no-op until Lane A lands it (M6 determinism hook).
|
||
- `scripts/test_server.py`: +director group (in-process http.server LLM mock,
|
||
MB-mock pattern): 503 when env unset, 3 valid ops pass through verbatim,
|
||
unknown op / unknown entity id / unknown preset land in `rejected` with
|
||
reasons, prompt carries 10 vocab words + roster ids + playhead, model name
|
||
+ low temp asserted. `OK: 9 test groups passed` (rhubarb-503 group now
|
||
self-skips — rhubarb is installed on ultra, so that group's absent-path
|
||
doesn't run; happy path was orchestrator-verified live).
|
||
DECISIONS:
|
||
- `web/grammar.js` does NOT exist yet (no presets.js either) — vocab lives in
|
||
server.py per spec fallback. **AT SYNC: re-point the SHOTS/ANGLES/FOCALS/
|
||
LIGHTS/MARKS lists + DIRECTOR_SYS at Lane A's grammar.js (read at boot)**
|
||
so server prompt and client presets can't drift. Op shapes I defined
|
||
(scenegod:direct): {op:"shot",subject,shot,angle,focal} /
|
||
{op:"light",preset} / {op:"mark",subject,mark,target?} — Lane A's
|
||
presets.js appliers must accept exactly these.
|
||
- Model may return `{"ops":[...]}` or a bare array — both accepted
|
||
(json_object mode forces an object wrapper; bare array tolerated for
|
||
servers that ignore response_format).
|
||
- Env read at request time (matches _mb_token pattern) so John can set
|
||
SCENEGOD_LLM_URL without a restart-order footgun; test env explicitly
|
||
pops it for the 503 assert.
|
||
- focal coerced via int() before vocab check (7b models love "35" strings).
|
||
BLOCKED/REQUESTS:
|
||
- Lane A: need `web/grammar.js` (data only) + presets.js appliers matching
|
||
the op shapes above; flag any shape disagreement at SYNC and I'll conform.
|
||
- Live end-to-end against real qwen2.5:7b on m4pro untested this session
|
||
(SCENEGOD_LLM_URL unset locally by design) — orchestrator: one smoke call
|
||
at SYNC with env set is the acceptance step.
|
||
NEXT: M6 FLOW VIDEO PLATES server side (unlocked on this acceptance):
|
||
flow_intake.py poster-jpg intake is already in scripts/; verify video exts
|
||
in /assets/tree (mp4 not in ASSET_EXTS yet — backdrops video category will
|
||
need it) + deterministic render check once Lane A's stage.syncVideos lands.
|
||
|
||
## 2026-07-25 session 5
|
||
DONE: M7-C starter templates — server side complete, 10 test groups green.
|
||
- `templates/` at repo root (NOT gitignored — ships with the code, unlike
|
||
assets/scenes/renders). Three files, all valid against the live
|
||
`validate_scene`, every referenced path verified present in `assets/`:
|
||
- `templates/two-hander-golden-hour.json` — lady + man facing each other
|
||
(yaw +0.55 / -0.6; characters face +Z at yaw 0 per presets.js), Flow
|
||
golden-hour street plate (`mode:"video"`, `fit:"frustum"`, `groundTint`),
|
||
grammar.js `golden_hour` key+ambient verbatim (#ffb469 @2.6 / #ffd9b0 @0.55,
|
||
bg #d98a52, fog 0.02), cam1 "wide" fov40 @5m + cam2 "closer" fov32 @3.4m,
|
||
cut at t=4.5. 8s. Both carry `carrying.fbx`; lady's block is trimmed
|
||
`in:2,out:10` so the two performances don't move in lockstep.
|
||
- `templates/record-store.json` — record-store interior plate, man +
|
||
carrying.fbx, one camera fov38, warm key + ambient (fog 0.012), 6s.
|
||
- `templates/empty-stage.json` — one camera, `day` key+ambient, 10s, no
|
||
characters, no plate. The clean start.
|
||
- `scenegod/server.py:31` TEMPLATES root (`SCENEGOD_TEMPLATES`, default
|
||
`./templates`); `:261-293` the read-only endpoints: `GET /templates` ->
|
||
`[{name,title,description,thumb?}]`, `GET /templates/{name}` -> scene JSON,
|
||
both through `safe_under` + `slugify` (same guard discipline as /scenes).
|
||
No POST/DELETE by design — load a template, then `POST /scenes/{name}`.
|
||
- `scripts/test_server.py:137` templates group: lists exactly the three,
|
||
order-leads with the two-hander, round-trips each one, rejects
|
||
`../../etc/hosts` / encoded `..` / absolute / unknown (all >=400), write
|
||
verbs 404/405, and — the check that stops a rotten template shipping —
|
||
POSTs every template through the real `validate_scene` and asserts every
|
||
`source.path` / `params.video|image` / `clips[].path` / `audio[].path` /
|
||
thumb exists under `assets/`, plus camera+light `source.type == "none"`.
|
||
`OK: 10 test groups passed`.
|
||
- `.gitignore`: `*.log` (a 65KB `scenegod-8020.log` was sitting untracked at
|
||
repo root, one `git add` away from the repo).
|
||
DECISIONS:
|
||
- Presentation metadata rides INSIDE the scene JSON under a `template` key
|
||
({title, description, thumb, order}) rather than a sidecar or a filename
|
||
convention: `validate_scene` ignores unknown top-level keys and
|
||
`Timeline.load/toJSON` doesn't carry it, so a saved scene comes out as plain
|
||
scene JSON with no template residue. `order` is server-side sort only (not in
|
||
the response) so the picker leads with the wow template, not alphabetically.
|
||
- Verified before shipping, not assumed: lady.glb is a **CC_Base** rig (103
|
||
`CC_Base` strings, 0 mixamo) — safe only because Lane A's M7-A alias layer
|
||
landed; `_normalize` scales both characters to 1.7m with feet at y=0, so the
|
||
camera heights/fovs above frame them for real (wide vspan 3.65m, closer
|
||
1.97m). Both clips probed under headless Blender: carrying.fbx = 10.0s and
|
||
hiphop.fbx = 4.467s, and **both are in-place** (hips travel 10cm / 2cm), so
|
||
no template performer can wander out of frame.
|
||
- Video plates follow the orchestrator shape (`params.video`, no `image`):
|
||
`_buildBackdrop` reads `p.image || en.source.path`, and `source.path` is the
|
||
same mp4, so the fallback carries it. If Lane A ever drops that fallback,
|
||
add `image` alongside `video`.
|
||
- Cameras/lights get `source:{type:"none"}` (sync2-sunset's `"upload"` was the
|
||
bug). Light `rot` is not decorative — `_buildLight` aims a directional down
|
||
the wrapper's -Z, so every key light's euler is `lookAtEuler(pos,[0,1,0])`
|
||
computed with presets.js's own algebra.
|
||
BLOCKED/REQUESTS:
|
||
- @orchestrator: the uvicorn on :8020 is NOT `--reload` — it still 404s
|
||
`/templates`. Restart it before browser-checking.
|
||
- @Lane B (M7-B): the picker contract is `GET /templates` ->
|
||
`[{name,title,description,thumb?}]` (already ordered; `thumb` is
|
||
asset-relative -> `assets/file?path=` or `assets/thumb?path=`), then
|
||
`GET /templates/{name}` -> feed straight to `timeline.load()` /
|
||
`stage.applyState()`, and save with the existing `POST /scenes/{name}`.
|
||
NEXT: nothing outstanding in Lane C. Backlog if wanted: a fourth template
|
||
once a prop/screen asset bank exists, and the still-unverified MediaRecorder
|
||
draft path (carried from session 3).
|
||
|
||
## 2026-07-25 session 6
|
||
DONE: **M8-C MULTI-SHOT SEQUENCING** — the film-level object. 15 test groups green
|
||
(was 10). A scene is one take; a *sequence* is an ordered list of trimmed scene
|
||
windows joined by cuts/dissolves over a film-level audio bed.
|
||
- **Sequence doc + validator** — `scenegod/server.py:443 plan_sequence()` validates
|
||
AND resolves in ONE pass (save-time validation and render-time planning can never
|
||
disagree). Roots at `:32-33` (`SCENEGOD_SEQUENCES` -> ./sequences, `SCENEGOD_FILMS`
|
||
-> ./films, both mkdir'd on boot like scenes/renders). Violations, all 422 with a
|
||
message per offence: version!=1, empty/oversized shots (max 200), unknown scene,
|
||
bad scene name, in/out not numbers, in<0, in>=out, out past the scene's duration,
|
||
unknown transition, transitionDur out of 0..10s, dissolve with dur<=0, dissolve not
|
||
shorter than BOTH shots it joins (ffmpeg xfade requirement), scenes with mixed fps
|
||
("one film, one fps"), audio path traversal / missing file / bad start|gain.
|
||
- **Endpoints** `:563-800`: `GET /sequences` (name,title,mtime,shots,duration,ok),
|
||
`GET|POST|DELETE /sequences/{name}` (slugify + safe_under + capped 1MB + atomic
|
||
write), `POST /sequences/{name}/render` -> `{filmId,fps,width,height,duration,
|
||
warnings,shots[]}`, `POST /films/{id}/shot/{i} {renderId}`, `POST /films/{id}/end`,
|
||
`GET /films/{id}/status` -> `{state,shot,of,log}`, `GET /films/{id}/out.mp4`.
|
||
States: collecting -> queued -> encoding -> done|error.
|
||
- **The film render is a client-driven job** (`:648 _concat_film`): the browser is the
|
||
only thing that can render a scene, so the server hands out the resolved shot plan,
|
||
the client renders each shot through the UNCHANGED `/render/*` session endpoints and
|
||
reports renderId-per-index, and `/end` folds the shot mp4s with ONE ffmpeg
|
||
filter_complex: each input `scale=WxH,setsar=1,fps,format=yuv420p`, then a left fold
|
||
— cut = `concat=n=2:v=1:a=0`, dissolve = `xfade=transition=fade:duration=D:offset=`
|
||
(running accumulated length - D). Film-level `audio[]` -> adelay/volume/amix/apad,
|
||
`-shortest`. Film dirs reaped >48h (`:75 reap_dirs(root)`, now called for RENDERS
|
||
and FILMS).
|
||
- **`scenegod/web/film.js`** (new): `renderSequence(stage, timeline, seqName, opts)`
|
||
drives the whole loop — POST render -> per shot `GET scenes/{scene}` ->
|
||
`timeline.applyScene(json)` (Lane B's single apply path) -> `captureFrames` over
|
||
frames [in*fps, out*fps) -> end+poll the shot -> `POST films/{id}/shot/{i}` ->
|
||
`films/{id}/end` -> poll -> resolve with `films/{id}/out.mp4`. Plus
|
||
list/get/save/deleteSequence helpers so a UI lane never hand-rolls (or absolutises)
|
||
the URLs. Relative URLs only.
|
||
- **`web/render.js` refactored, not duplicated**: the frame loop is now
|
||
`captureFrames(stage,timeline,{renderId,from,to,onProgress})` + `withRenderSize` /
|
||
`beginRender` / `endRender` / `pollRender` / `postJSON` / `getJSON`, all exported;
|
||
`finalRender(stage,timeline,opts)` keeps its exact old signature and is now 6 lines
|
||
built from them. film.js imports them — one frame-post path in the repo.
|
||
- **`sequences/demo-two-shot.json`**: 3 shots, 11.5s — sync2-sunset 0..4.5 (cam1)
|
||
dissolve 0.5s into sync2-sunset 4.5..8.0 (cam2, same scene, other side of its cut),
|
||
cut to sync1-demo 0..4.0. Verified live: `GET /sequences` -> ok:true duration 11.5,
|
||
`POST /sequences/demo-two-shot/render` -> the 3-shot plan, and a full fake-shot
|
||
end-to-end (lavfi mp4s registered through the real endpoints) produced out.mp4 at
|
||
exactly 11.500000s.
|
||
- **`scripts/test_server.py`**: +5 groups (`:225` CRUD round-trip incl. slugify +
|
||
lossless read-back + DELETE 200/404; `:248` 13 validation failures each asserted for
|
||
its own message + traversal/unknown guards on GET and DELETE; `:281` film smoke —
|
||
two lavfi mp4s registered as shots, end, poll, ffprobe duration 5.0s = 2+3, output
|
||
64x64, plus guards: end-before-registration 400, bad index 400, traversal renderId
|
||
400, unknown renderId 400, 32x32 shot -> 400 "one resolution per film", unknown
|
||
filmId 404; `:340` dissolve variant -> 4.5s = 2+3-0.5 with an audio bed muxed;
|
||
`:370` the shipped example can't rot — every shot's scene must exist and every
|
||
`out` must be within that scene's duration). `OK: 15 test groups passed`.
|
||
DECISIONS:
|
||
- **One fps per film, validated** rather than resampling: `timeline.step(frame)` is
|
||
the deterministic seek and it divides by the SCENE's fps, so a film whose fps
|
||
differed from a scene's would silently render the wrong instants. Mixed-fps
|
||
sequences are a clear 422 and the film's fps comes from the scenes, not the client.
|
||
Client passes width/height only.
|
||
- **One resolution per film, enforced at registration** (compare `renders/{rid}/
|
||
meta.json` w/h/fps to the film's) so the failure is a clear 400 the moment it
|
||
happens, not a cryptic ffmpeg filter error after the last shot. The scale/setsar
|
||
filter stays as the belt to that braces (renders with no meta.json).
|
||
- **xfade offsets come from ffprobe'd durations** (`:636 _probe_dur`, nominal length
|
||
as fallback): nominal length can differ from the encoded mp4 by up to a frame, and
|
||
an absolute xfade offset that's off by a frame shows as a stutter at the join.
|
||
- **Shots render silent; the soundtrack is film-level.** Concat with per-shot audio
|
||
needs every shot to have an audio stream (mixed presence breaks the filter), and a
|
||
film's bed rarely respects shot boundaries anyway. Not silent-by-surprise: the
|
||
render-start response carries a `warnings[]` naming any referenced scene whose
|
||
`audio[]` won't be carried, and film.js logs them.
|
||
- Existing `/render/end` mux gained `apad` before `-shortest` (`:326`) — a VO shorter
|
||
than the shot was truncating the PICTURE. Same fix in the film mux; the M4 audio
|
||
assert still passes.
|
||
- `sequences/` ships (like `templates/`) but is user-writable, so `.gitignore` takes
|
||
`sequences/*` + `!sequences/demo-two-shot.json`: the example is tracked, John's own
|
||
sequences stay out of `git status`. `films/` fully ignored.
|
||
- Film plan + shot registrations persist in `films/{id}/meta.json` (atomic tmp+rename),
|
||
not just memory: a film spans minutes of client rendering and `/end` must survive a
|
||
server restart underneath it. Only the encode state/log is in-memory (renders model).
|
||
BLOCKED/REQUESTS:
|
||
- @orchestrator (deploy/, not my lane) — two one-line prod fixes so sequences persist:
|
||
1. `deploy/docker-compose.yml`: add `- SCENEGOD_SEQUENCES=/app/sequences` and
|
||
`- SCENEGOD_FILMS=/app/films` to environment, and
|
||
`- /opt/scenegod-data/sequences:/app/sequences` +
|
||
`- /opt/scenegod-data/films:/app/films` to volumes.
|
||
2. `deploy/deploy.sh`: `mkdir -p /opt/scenegod-data/{assets,scenes,renders,sequences,films}`
|
||
and add `--exclude films` to the rsync.
|
||
Without these the app still boots and works (it mkdirs both dirs inside the
|
||
container) but a rebuild eats every saved sequence. Nothing else new is needed in
|
||
prod: no LLM/MB env, ffmpeg is already in the image.
|
||
- @orchestrator — the :8020 uvicorn is not `--reload`; restart it before browser-driving
|
||
the film (it will 404 `/sequences`). Test instances only ever ran on 8097/8099.
|
||
- @Lane B (whenever it's unlocked) — the UI contract is ready and stable:
|
||
`import { renderSequence, listSequences, getSequence, saveSequence, deleteSequence }
|
||
from './web/film.js'`. A sequence editor is a list of `{scene,in,out,transition,
|
||
transitionDur}` rows over `GET /scenes`; `renderSequence(stage, timeline, name,
|
||
{width, height, onShot, onProgress})` does the rest and resolves with the mp4 URL.
|
||
NOTE it calls `timeline.applyScene()` per shot, so it CLOBBERS the loaded scene —
|
||
the UI should confirm via `timeline.hasContent()` first, same as new-from-template.
|
||
NEXT: nothing outstanding in Lane C. Backlog: per-scene audio carried into a film
|
||
(auto-derive the bed from each shot's scene `audio[]` offset into film time — needs a
|
||
silence-input path for shots with no audio); more transition types (xfade has ~50,
|
||
`wipeleft`/`slideup` are one vocab entry each); still-unverified MediaRecorder draft
|
||
path (carried from session 3).
|
||
|
||
## 2026-07-25 session 7
|
||
DONE: **M9-C1 audio trim at the mux** + **M9-C2 BEAT DETECTION**. 18 test groups green
|
||
(was 15). `OK: 18 test groups passed`, whole suite ~6.5s.
|
||
|
||
**M9-C1 — `in`/`out` honoured in BOTH muxes.**
|
||
- `scenegod/server.py:112 resolve_audio()` — ONE function now turns an
|
||
`audio[]` entry into a muxable dict for both lanes (`/render/{id}/end` at :441
|
||
and `plan_sequence` at :562), so a clip means the same thing in a shot and in a
|
||
film. Validates path (guarded), `start`/`gain`/`in`/`out` types, `in>=0`,
|
||
`in<out`. `_num` moved up to :103 next to `safe_under` for the same reason.
|
||
- `:364 audio_input()` puts the trim on the INPUT (`-ss in -t out-in`) and
|
||
`:381 audio_filters()` keeps the adelay/volume/amix/apad chain EXACTLY as
|
||
shipped. Both muxes call the pair (`:401` render, `:718` film).
|
||
- Absent `in`/`out` = byte-identical ffmpeg command to before (proved by the
|
||
untrimmed A/B in the test).
|
||
- Verified end-to-end, not by eyeballing the command line: `scripts/test_server.py:250`
|
||
renders 3s of picture over `trim.wav` (1s silence, then 1s of 880Hz).
|
||
Trimmed to the tone at `start:0.5` -> `volumedetect` reads -91dB at 0.0-0.4s,
|
||
**-6.1dB at 0.6-1.4s**, -91dB at 1.7-2.7s. The SAME clip untrimmed reads -91dB
|
||
at 0.6-1.4 and -6.1dB at 1.6-2.4 — i.e. the trim moved the sound, and it is not
|
||
a tautological test. `:429` does the same for the film-level bed (silent to
|
||
1.8s, tone at 2.1s) inside the existing dissolve film.
|
||
DECISIONS (M9-C1):
|
||
- **`-ss`/`-t` on the input, NOT `atrim` in the filter graph** — this is a bug
|
||
fix, not taste. `atrim,asetpts,adelay,...,amix,apad` on ffmpeg 8.x
|
||
(homebrew, 62.x libs) emits pad frames at ~INT64_MAX PTS; the muxer drops all
|
||
of them and you get an mp4 with an aac stream and **zero audio samples**, rc=0,
|
||
no warning. Reproduced 4 ways: old chain OK, +atrim broken, +atrim without apad
|
||
OK, +atrim with `apad=whole_dur=N` OK. Input seeking leaves the proven graph
|
||
untouched. (Sample-exact for wav; a few ms at worst for mp3.)
|
||
- `scripts/test_server.py:59 seg_db()` REFUSES to answer when a window decoded no
|
||
samples, instead of returning -99. Otherwise every "assert it's silent here"
|
||
passes on a stream that is silent because it is empty — which is exactly the
|
||
failure above, and it would have shipped green.
|
||
- A trim window past EOF made ffmpeg die mid-mux (`-22 Error muxing a packet` ->
|
||
a mystery failed render), so `resolve_audio` ffprobes the source **only when a
|
||
trim is asked for**: `in` past the end is a 400/422 that says so, `out` past the
|
||
end is just no tail trim (a UI dragging the tail wide shouldn't fail a render).
|
||
|
||
**M9-C2 — `GET /beats?path=` (`server.py:869-1090`).**
|
||
`{path,bpm,confidence,beats[],downbeats[],duration,meter,period,acPeak,onBeatContrast,analyzed}`.
|
||
Chain (all stdlib + one ffmpeg pipe, no new deps):
|
||
1. `:896` ffmpeg -> mono s16 PCM @22050 (`-map 0:a:0`, so mp4/webm work too),
|
||
read with `array`. `-t 900` cap: 15 min analysed, grid extrapolated past that
|
||
with `truncated:true` (a 40MB decode ceiling, not a 2-hour DJ set in RAM).
|
||
2. `:908` per-frame **log energy**, hop 256 (86.13 fps) -> half-wave-rectified
|
||
first difference -> 3-tap smooth -> **local-mean subtraction** (±0.37s).
|
||
3. `:933` normalised autocorrelation over 60..180 BPM, harmonics added
|
||
(`r[l] + .5r[2l] + .5r[l/2] + .3r[3l/2]`), times a log-gaussian prior centred
|
||
on **125 BPM**, peak parabola-interpolated.
|
||
4. `:1029` explicit octave check: score half / self / double with the comb, keep
|
||
everything within 15% of the best, prefer the candidate in 90..180 BPM.
|
||
5. `:968` comb-filter phase search over one period, then `:972` **weighted
|
||
least-squares refit** of period+phase onto the nearest onset peaks (sub-frame,
|
||
parabolic). Without the refit the raw AC lag is ±0.5 frame = seconds of drift
|
||
across a 4-minute track.
|
||
6. Emits an absolute-seconds grid across the whole file; `downbeats` = every 4th
|
||
beat, **4/4 assumed**, coset chosen as the strongest of the four (`meter:4` is
|
||
in the response so a caller knows it was an assumption, not a detection).
|
||
MEASURED ACCURACY (synthetic click tracks, `aevalsrc` with `mod(t,period)` so beat
|
||
k is at EXACTLY k*period): 12 tempos 90/100/110/118/120/124/126/128/132/140/150/174
|
||
-> **worst BPM error 0.03** (spec allowed ±2), **worst beat-time error 3.3 ms**
|
||
over the first 16 beats (spec allowed ~30 ms), confidence 0.85-0.94.
|
||
Non-music: pink noise conf 0.10, digital silence conf 0.0 with `beats:[]` and a
|
||
`note` (no crash, 200). Real 90s house from ~/Music/HOUSE: 127.41 / 126.89 /
|
||
121.20 / 126.95 BPM, and independently re-analysing two disjoint 90s windows of
|
||
each track reproduces the full-track BPM to within 0.15 on 3 of 4 (the 4th is an
|
||
intro/breakdown pair that confidence correctly calls mush, 0.16-0.20).
|
||
SPEED (ultra, measured): 4-min wav **0.41s**, 5-min mp3 0.66s, 7.6-min mp3 0.69s;
|
||
cached re-hit **2-3 ms**. Cache key is (path, mtime_ns, size), bounded at 32.
|
||
DECISIONS (M9-C2):
|
||
- **Energy flux, not spectral flux.** A pure-Python radix-2 FFT is ~20k transforms
|
||
for a 4-min track (tens of seconds of CPU) and the repo forbids numpy; for
|
||
four-to-the-floor the kick dominates broadband energy anyway. The log
|
||
compression + local-mean whitening is what buys back the discrimination — it is
|
||
the difference between conf 0.07 and 0.35 on a dense club mix.
|
||
- **`confidence` = sqrt(acPeak × onBeatContrast)**, both reported separately so a
|
||
caller can see why. acPeak = normalised autocorrelation at the chosen period
|
||
("is the envelope periodic at all"); onBeatContrast = (on-beat mean − all-frame
|
||
mean)/(sum) ("does the EMITTED grid explain the onsets"). Measured bands, in the
|
||
code comment: clicks 0.83-0.93, real house 0.32-0.65, noise 0.10, silence 0.0.
|
||
Read >0.5 as a solid grid, <0.2 as mush.
|
||
- Mean (not sum) comb score, deliberately: a double-tempo grid then pays for the
|
||
empty positions it adds, which is half the octave problem solved for free. The
|
||
half/double check + band preference is the other half — verified on `alt128`
|
||
(strong kick every 2nd beat, autocorrelates at 64) -> 128.01.
|
||
- The grid is CONSTANT tempo. No rubato/drift tracking, said plainly in the
|
||
response contract. For a DJ mix, tempo changes per track — analyse a segment.
|
||
- **The two fixtures in `assets/audio/test/` are not exact**: direct sample-level
|
||
onset timing puts `beat120.wav`'s kicks at 0.5 … 14.53 (28 intervals ->
|
||
**119.74 BPM**, it drifts ~0.3%), and `house128.wav` drifts similarly. My detector
|
||
says 119.62 / 127.62 on them, i.e. it is reading the FILE correctly. That is why
|
||
the test synthesises its own exactly-periodic clicks instead of asserting
|
||
against those two.
|
||
- New test group `scripts/test_server.py:525` boots an instance with
|
||
`PATH=/usr/bin:/bin` (no homebrew) and asserts `/beats` AND `/rhubarb` 503 —
|
||
the ffmpeg gate is now actually exercised, and so is rhubarb's 503 branch,
|
||
which had been unreachable on this box since rhubarb was installed.
|
||
- No client change was needed: `web/render.js:74 endRender` already forwards the
|
||
scene's `audio[]` objects verbatim, so `in`/`out` reach the server as-is.
|
||
BLOCKED/REQUESTS:
|
||
- @Lane B — `/beats` is ready for beat-snapping. Contract:
|
||
`GET beats?path=<asset-relative>` -> `{bpm, confidence, beats:[s,...],
|
||
downbeats:[s,...], duration, meter:4, period, acPeak, onBeatContrast}`; times
|
||
are absolute seconds in the FILE, so snapping a timeline key means
|
||
`beat + audioClip.start - (audioClip.in||0)`. Gate the UI on
|
||
`confidence` (>0.5 solid, <0.2 mush) rather than pretending every grid is good.
|
||
First call on a 5-min track is ~0.7s, after that it is instant — fetch once per
|
||
audio clip and cache client-side.
|
||
- @orchestrator — nothing needed for prod: ffmpeg is already in the image, so
|
||
`/beats` works on the VPS; no new env, no new dep (requirements.txt untouched).
|
||
The :8020 uvicorn still isn't `--reload`, so restart it before browser-driving
|
||
`/beats`. All my testing ran on ports 8097-8104 and 8098.
|
||
NEXT: nothing outstanding in Lane C. Backlog: per-scene audio auto-carried into a
|
||
film bed; more xfade transition types; the still-unverified MediaRecorder draft
|
||
path (carried from session 3); and — if beat-sync gets ambitious — a `?from=&to=`
|
||
window on `/beats` so a DJ mix can be analysed per track instead of assuming one
|
||
tempo for the whole hour.
|
||
|
||
## 2026-07-26 session 8 — M9-C2 tempo bias: root-caused and fixed
|
||
The orchestrator is right and my session-7 claim was wrong. **The fixtures are exact**
|
||
(`beat120.wav` kicks on 0.5000s to the microsecond, `house128.wav` on 0.468735s), and my
|
||
detector was **0.32% slow on both**. Worse, I had measured 120.002 correctly myself early
|
||
in that session and then believed a cruder second scan over it — the mistake was
|
||
disbelieving ground truth to protect the code. Corrected below.
|
||
|
||
**ROOT CAUSE — the onset front-end was measuring the wrong instant.**
|
||
A percussive hit is a high-frequency transient riding a slower low-frequency body.
|
||
`BEAT_SR=22050` resamples away everything above 11 kHz, and in `beat120.wav` the >11k
|
||
share of the attack grows ~20x across the file (peak sample 1402 at beat 1 -> 27711 at
|
||
beat 15). Measured, per beat, on the <11k signal that survived the decode:
|
||
beats 1/5/29 peak at **+0 ms** after the attack, beats 15/25 at **+30 / +50 ms**.
|
||
A lateness that VARIES per beat is a tempo error, not an offset — and the weighted
|
||
least-squares refit then fits those biased points perfectly. It was NOT the refit
|
||
(sub-millisecond over 200s click tracks) and NOT integer-lag AC quantisation
|
||
(the parabolic step already handled that; the AC's own estimate was 119.87, and the refit
|
||
moved it to 119.62 because its INPUT peaks were late).
|
||
**Fix: `BEAT_SR, BEAT_HOP = 44100, 512`** (`server.py:882`) — same 86.13 fps envelope, same
|
||
constants downstream, transient preserved. 172 fps was tried and REJECTED (`server.py:875`
|
||
comment): it breaks octave choice on real records (a 90s window of a house track went to
|
||
63.7 BPM) because the prior/harmonic weights are tuned for 86 fps.
|
||
`server.py:906 _frame_energy()` now decodes+frames in ONE STREAMING pass (Popen, ~4MB
|
||
reads) — 44.1k × 900s is 79MB of PCM and `subprocess.run` would hold it twice; peak memory
|
||
is now a few MB. It also returns the exact sample count, so `duration` is 15.0 not 14.988.
|
||
|
||
**SECOND BUG, found by sweeping tempos after the fix** (`server.py:963 _beat_tempo`):
|
||
a 118 BPM / 60s click track read **114.79**. Two defects compounding: (1) harmonics that
|
||
fall outside the lag table counted as ZERO, which silently penalises the lag that owns them
|
||
(lag 43 kept its 2x at 86, lag 44 lost its 88 — so a hair of score flipped the choice to
|
||
the wrong lag); (2) the parabolic interpolation was then applied to a point on a SLOPE,
|
||
which extrapolates: lag 43 became 44.95. Fixed by scoring only the harmonics that exist
|
||
(weight-normalised), then walking uphill to the local AC maximum before interpolating, and
|
||
clamping the correction to ±0.5 lag.
|
||
|
||
**RE-VERIFIED against the ground truth** (live over HTTP, `GET /beats`):
|
||
| file | truth | detected | err | max |beat − true grid| | conf |
|
||
|---|---|---|---|---|---|
|
||
| `beat120.wav` | 120.003 | **120.030** | +0.027 | **6.0 ms** (last beat +3.3) | 0.92 |
|
||
| `house128.wav` | 128.004 | **128.020** | +0.016 | **10.1 ms** (last beat +7.6) | 0.86 |
|
||
Was −0.387 / −0.388 and 45.1 / 52.5 ms of drift.
|
||
Synthetic sweep, 17 tempos 60…179 BPM on 60s files: **worst BPM error 0.010, worst drift
|
||
5.7 ms across the whole file**. Octave traps still correct (alt-strong-every-2nd 128.01,
|
||
64 BPM stays 64, kick+offbeat-hat 125.97). Real 90s house unchanged: 127.41 / 126.89 /
|
||
121.22 / 126.96, disjoint-window self-consistency within 0.13 BPM on 3 of 4.
|
||
Truncation path exercised (cap forced to 60s on a 200s file): grid extrapolates to 200.0s
|
||
with 1.0 ms of drift at the end. Speed: **4-min wav 0.48s, 5-min mp3 0.84s**, 7.6-min 0.78s,
|
||
cached 2-3 ms (was 0.66s at 22050 — the transient costs ~0.2s and is worth it).
|
||
|
||
**TESTS — this class of error cannot pass again** (`scripts/test_server.py:468`, 18 groups):
|
||
1. Every click case now asserts **whole-file drift** (`max|b − round(b/per)·per| < 20 ms`),
|
||
not just average BPM, and the sweep gained **118 BPM @ 60s** (pins the `_beat_tempo` fix).
|
||
2. **200s click track at 128** — end-of-file drift < 20 ms, and the grid must actually reach
|
||
past 195s (catches "correct tempo, grid stops early").
|
||
3. **`hftick.wav`** — a 16 kHz tick over a slow 60 Hz body, i.e. the failure mechanism
|
||
itself: the first 12 beats must land within **12 ms**. This is the one that pins
|
||
`BEAT_SR`, and it ships in the test (assets/ is gitignored).
|
||
4. **The fixtures as ground truth** when present: 120.003 / 128.004 ± 0.2 BPM and < 20 ms
|
||
drift, copied into the test's asset root, skipped with a printed note when absent.
|
||
Proven to discriminate, by monkeypatching the old code back in:
|
||
`22050/256` -> fixtures FAIL at 45.1 / 52.5 ms drift and hftick FAILS at 26.7 ms;
|
||
old `_beat_tempo` -> 118 BPM case FAILS at 114.79 (253 ms drift). Note the pure 200s click
|
||
track passes under BOTH old versions — a clean click track can never catch a spectral bias,
|
||
which is exactly why session 7's tests missed it.
|
||
DECISIONS:
|
||
- Ground truth wins. When a measurement disagrees with the code, the code is wrong until a
|
||
BETTER measurement says otherwise — and "better" means the one with the sharper
|
||
instrument (sample-level threshold crossing over the whole file), not the newer one.
|
||
- Front-end resolution is a tuning-coupled choice, not a free knob: 44100/512 keeps every
|
||
downstream constant valid; 44100/256 would need the prior and harmonic weights retuned.
|
||
The comment at `server.py:875` says so, so nobody "optimises" the decode rate back down.
|
||
BLOCKED/REQUESTS: none. Everything else from session 7 (trim work, speed, caching,
|
||
confidence, the 503 gate) is unchanged and still green.
|
||
NEXT: unchanged from session 7 (per-scene audio into a film bed, more transitions,
|
||
MediaRecorder draft path, `?from=&to=` windowing on /beats for DJ mixes).
|