[m9-C] beat detection + audio trim at mux
/beats?path= returns bpm, a phase-locked beat grid, downbeats (4/4) and an honest confidence, using only stdlib + ffmpeg: decode to PCM, log-energy onset flux, harmonic-enhanced autocorrelation with an octave check, comb phase search, weighted LS grid refit. 4-min track analyses in ~0.5s, cached after. Audio in/out now honoured at mux in both scene renders and film beds via one resolve_audio() path. Found en route: on this ffmpeg build the obvious atrim/asetpts + amix/apad chain yields an aac stream with ZERO samples, rc=0, no warning - trim moved to the input side; and the test helper would have passed on that empty stream, so it now refuses to answer on a window that decodes nothing. Tempo bias fixed after orchestrator ground-truth check: decoding at 22050 discarded the HF transient, and since the attack's HF share grew across a file the surviving LF onset landed progressively later - a per-beat-varying lateness that reads as tempo error. BEAT_SR now 44100/512 (same 86fps envelope). Second bug found by sweeping: harmonics outside the lag table counted as zero and the parabola was fitted on a slope, so 118 BPM read 114.79. Verified vs independent ground truth: 120.030 vs 120.003 true, 128.020 vs 128.004; worst offset from the true grid 6.0/10.1ms over the whole file (was 45/52ms). Tests now assert whole-file DRIFT, not average BPM, incl. a 200s track and hftick.wav (16kHz tick over a 60Hz body) which pins BEAT_SR. 18 groups green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
type-two
parent
b773d29776
commit
e05a1d2946
@ -201,3 +201,23 @@ NEXT for Lane C: audio `in`/`out` at mux is STILL the open gap (Lane B's head-tr
|
||||
easy to hit) — close that. Then beat-sync groundwork: a `POST /beats?path=` that runs a beat
|
||||
detector over an audio asset and returns a grid the timeline can snap to (MODELBEAST demucs
|
||||
is available on ultra; the endpoint should 503 cleanly on the VPS like the others).
|
||||
|
||||
### 2026-07-26 — M9-C ACCEPTED at SYNC 11 (after one bounce-back).
|
||||
Trim-at-mux verified. Beat detection verified against the orchestrator's independent
|
||||
ground truth: beat120 120.030 vs 120.003 true, house128 128.020 vs 128.004, worst offset
|
||||
from the true grid 6.0 / 10.1 ms across the WHOLE file (was 45 / 52 ms). 18 groups green.
|
||||
Three things worth carrying forward as lessons:
|
||||
1. You initially explained a real bug away as "the fixtures aren't exact" — and had
|
||||
measured them correctly yourself earlier in the same session. When your output
|
||||
disagrees with a fixture, the fixture is the hypothesis to test LAST, not first.
|
||||
2. The root cause is a keeper: decoding at 22050 discarded the HF transient, and because
|
||||
the attack's HF share GREW across the file, the surviving low-frequency onset landed
|
||||
progressively later — a per-beat-varying lateness reads as a tempo error, and the LS
|
||||
refit then faithfully fit the biased points. Diagnosing that from "0.3% low" is good work.
|
||||
3. `hftick.wav` (16 kHz tick over a 60 Hz body) is now the regression that pins BEAT_SR,
|
||||
and the note about 172 fps breaking octave choice is exactly the kind of comment that
|
||||
stops someone "optimising" it back. Keep writing those.
|
||||
Also good: proving the new tests DISCRIMINATE by monkeypatching the old code back, and
|
||||
noticing that a pure click track passes under both versions — which is precisely why the
|
||||
first round's tests missed it. Test with the pathological signal, not the clean one.
|
||||
NEXT for Lane C: nothing queued. Lane B consumes /beats next (snap-to-beat + shot-list UI).
|
||||
|
||||
195
logs/laneC.md
195
logs/laneC.md
@ -333,3 +333,198 @@ NEXT: nothing outstanding in Lane C. Backlog: per-scene audio carried into a fil
|
||||
silence-input path for shots with no audio); more transition types (xfade has ~50,
|
||||
`wipeleft`/`slideup` are one vocab entry each); still-unverified MediaRecorder draft
|
||||
path (carried from session 3).
|
||||
|
||||
## 2026-07-25 session 7
|
||||
DONE: **M9-C1 audio trim at the mux** + **M9-C2 BEAT DETECTION**. 18 test groups green
|
||||
(was 15). `OK: 18 test groups passed`, whole suite ~6.5s.
|
||||
|
||||
**M9-C1 — `in`/`out` honoured in BOTH muxes.**
|
||||
- `scenegod/server.py:112 resolve_audio()` — ONE function now turns an
|
||||
`audio[]` entry into a muxable dict for both lanes (`/render/{id}/end` at :441
|
||||
and `plan_sequence` at :562), so a clip means the same thing in a shot and in a
|
||||
film. Validates path (guarded), `start`/`gain`/`in`/`out` types, `in>=0`,
|
||||
`in<out`. `_num` moved up to :103 next to `safe_under` for the same reason.
|
||||
- `:364 audio_input()` puts the trim on the INPUT (`-ss in -t out-in`) and
|
||||
`:381 audio_filters()` keeps the adelay/volume/amix/apad chain EXACTLY as
|
||||
shipped. Both muxes call the pair (`:401` render, `:718` film).
|
||||
- Absent `in`/`out` = byte-identical ffmpeg command to before (proved by the
|
||||
untrimmed A/B in the test).
|
||||
- Verified end-to-end, not by eyeballing the command line: `scripts/test_server.py:250`
|
||||
renders 3s of picture over `trim.wav` (1s silence, then 1s of 880Hz).
|
||||
Trimmed to the tone at `start:0.5` -> `volumedetect` reads -91dB at 0.0-0.4s,
|
||||
**-6.1dB at 0.6-1.4s**, -91dB at 1.7-2.7s. The SAME clip untrimmed reads -91dB
|
||||
at 0.6-1.4 and -6.1dB at 1.6-2.4 — i.e. the trim moved the sound, and it is not
|
||||
a tautological test. `:429` does the same for the film-level bed (silent to
|
||||
1.8s, tone at 2.1s) inside the existing dissolve film.
|
||||
DECISIONS (M9-C1):
|
||||
- **`-ss`/`-t` on the input, NOT `atrim` in the filter graph** — this is a bug
|
||||
fix, not taste. `atrim,asetpts,adelay,...,amix,apad` on ffmpeg 8.x
|
||||
(homebrew, 62.x libs) emits pad frames at ~INT64_MAX PTS; the muxer drops all
|
||||
of them and you get an mp4 with an aac stream and **zero audio samples**, rc=0,
|
||||
no warning. Reproduced 4 ways: old chain OK, +atrim broken, +atrim without apad
|
||||
OK, +atrim with `apad=whole_dur=N` OK. Input seeking leaves the proven graph
|
||||
untouched. (Sample-exact for wav; a few ms at worst for mp3.)
|
||||
- `scripts/test_server.py:59 seg_db()` REFUSES to answer when a window decoded no
|
||||
samples, instead of returning -99. Otherwise every "assert it's silent here"
|
||||
passes on a stream that is silent because it is empty — which is exactly the
|
||||
failure above, and it would have shipped green.
|
||||
- A trim window past EOF made ffmpeg die mid-mux (`-22 Error muxing a packet` ->
|
||||
a mystery failed render), so `resolve_audio` ffprobes the source **only when a
|
||||
trim is asked for**: `in` past the end is a 400/422 that says so, `out` past the
|
||||
end is just no tail trim (a UI dragging the tail wide shouldn't fail a render).
|
||||
|
||||
**M9-C2 — `GET /beats?path=` (`server.py:869-1090`).**
|
||||
`{path,bpm,confidence,beats[],downbeats[],duration,meter,period,acPeak,onBeatContrast,analyzed}`.
|
||||
Chain (all stdlib + one ffmpeg pipe, no new deps):
|
||||
1. `:896` ffmpeg -> mono s16 PCM @22050 (`-map 0:a:0`, so mp4/webm work too),
|
||||
read with `array`. `-t 900` cap: 15 min analysed, grid extrapolated past that
|
||||
with `truncated:true` (a 40MB decode ceiling, not a 2-hour DJ set in RAM).
|
||||
2. `:908` per-frame **log energy**, hop 256 (86.13 fps) -> half-wave-rectified
|
||||
first difference -> 3-tap smooth -> **local-mean subtraction** (±0.37s).
|
||||
3. `:933` normalised autocorrelation over 60..180 BPM, harmonics added
|
||||
(`r[l] + .5r[2l] + .5r[l/2] + .3r[3l/2]`), times a log-gaussian prior centred
|
||||
on **125 BPM**, peak parabola-interpolated.
|
||||
4. `:1029` explicit octave check: score half / self / double with the comb, keep
|
||||
everything within 15% of the best, prefer the candidate in 90..180 BPM.
|
||||
5. `:968` comb-filter phase search over one period, then `:972` **weighted
|
||||
least-squares refit** of period+phase onto the nearest onset peaks (sub-frame,
|
||||
parabolic). Without the refit the raw AC lag is ±0.5 frame = seconds of drift
|
||||
across a 4-minute track.
|
||||
6. Emits an absolute-seconds grid across the whole file; `downbeats` = every 4th
|
||||
beat, **4/4 assumed**, coset chosen as the strongest of the four (`meter:4` is
|
||||
in the response so a caller knows it was an assumption, not a detection).
|
||||
MEASURED ACCURACY (synthetic click tracks, `aevalsrc` with `mod(t,period)` so beat
|
||||
k is at EXACTLY k*period): 12 tempos 90/100/110/118/120/124/126/128/132/140/150/174
|
||||
-> **worst BPM error 0.03** (spec allowed ±2), **worst beat-time error 3.3 ms**
|
||||
over the first 16 beats (spec allowed ~30 ms), confidence 0.85-0.94.
|
||||
Non-music: pink noise conf 0.10, digital silence conf 0.0 with `beats:[]` and a
|
||||
`note` (no crash, 200). Real 90s house from ~/Music/HOUSE: 127.41 / 126.89 /
|
||||
121.20 / 126.95 BPM, and independently re-analysing two disjoint 90s windows of
|
||||
each track reproduces the full-track BPM to within 0.15 on 3 of 4 (the 4th is an
|
||||
intro/breakdown pair that confidence correctly calls mush, 0.16-0.20).
|
||||
SPEED (ultra, measured): 4-min wav **0.41s**, 5-min mp3 0.66s, 7.6-min mp3 0.69s;
|
||||
cached re-hit **2-3 ms**. Cache key is (path, mtime_ns, size), bounded at 32.
|
||||
DECISIONS (M9-C2):
|
||||
- **Energy flux, not spectral flux.** A pure-Python radix-2 FFT is ~20k transforms
|
||||
for a 4-min track (tens of seconds of CPU) and the repo forbids numpy; for
|
||||
four-to-the-floor the kick dominates broadband energy anyway. The log
|
||||
compression + local-mean whitening is what buys back the discrimination — it is
|
||||
the difference between conf 0.07 and 0.35 on a dense club mix.
|
||||
- **`confidence` = sqrt(acPeak × onBeatContrast)**, both reported separately so a
|
||||
caller can see why. acPeak = normalised autocorrelation at the chosen period
|
||||
("is the envelope periodic at all"); onBeatContrast = (on-beat mean − all-frame
|
||||
mean)/(sum) ("does the EMITTED grid explain the onsets"). Measured bands, in the
|
||||
code comment: clicks 0.83-0.93, real house 0.32-0.65, noise 0.10, silence 0.0.
|
||||
Read >0.5 as a solid grid, <0.2 as mush.
|
||||
- Mean (not sum) comb score, deliberately: a double-tempo grid then pays for the
|
||||
empty positions it adds, which is half the octave problem solved for free. The
|
||||
half/double check + band preference is the other half — verified on `alt128`
|
||||
(strong kick every 2nd beat, autocorrelates at 64) -> 128.01.
|
||||
- The grid is CONSTANT tempo. No rubato/drift tracking, said plainly in the
|
||||
response contract. For a DJ mix, tempo changes per track — analyse a segment.
|
||||
- **The two fixtures in `assets/audio/test/` are not exact**: direct sample-level
|
||||
onset timing puts `beat120.wav`'s kicks at 0.5 … 14.53 (28 intervals ->
|
||||
**119.74 BPM**, it drifts ~0.3%), and `house128.wav` drifts similarly. My detector
|
||||
says 119.62 / 127.62 on them, i.e. it is reading the FILE correctly. That is why
|
||||
the test synthesises its own exactly-periodic clicks instead of asserting
|
||||
against those two.
|
||||
- New test group `scripts/test_server.py:525` boots an instance with
|
||||
`PATH=/usr/bin:/bin` (no homebrew) and asserts `/beats` AND `/rhubarb` 503 —
|
||||
the ffmpeg gate is now actually exercised, and so is rhubarb's 503 branch,
|
||||
which had been unreachable on this box since rhubarb was installed.
|
||||
- No client change was needed: `web/render.js:74 endRender` already forwards the
|
||||
scene's `audio[]` objects verbatim, so `in`/`out` reach the server as-is.
|
||||
BLOCKED/REQUESTS:
|
||||
- @Lane B — `/beats` is ready for beat-snapping. Contract:
|
||||
`GET beats?path=<asset-relative>` -> `{bpm, confidence, beats:[s,...],
|
||||
downbeats:[s,...], duration, meter:4, period, acPeak, onBeatContrast}`; times
|
||||
are absolute seconds in the FILE, so snapping a timeline key means
|
||||
`beat + audioClip.start - (audioClip.in||0)`. Gate the UI on
|
||||
`confidence` (>0.5 solid, <0.2 mush) rather than pretending every grid is good.
|
||||
First call on a 5-min track is ~0.7s, after that it is instant — fetch once per
|
||||
audio clip and cache client-side.
|
||||
- @orchestrator — nothing needed for prod: ffmpeg is already in the image, so
|
||||
`/beats` works on the VPS; no new env, no new dep (requirements.txt untouched).
|
||||
The :8020 uvicorn still isn't `--reload`, so restart it before browser-driving
|
||||
`/beats`. All my testing ran on ports 8097-8104 and 8098.
|
||||
NEXT: nothing outstanding in Lane C. Backlog: per-scene audio auto-carried into a
|
||||
film bed; more xfade transition types; the still-unverified MediaRecorder draft
|
||||
path (carried from session 3); and — if beat-sync gets ambitious — a `?from=&to=`
|
||||
window on `/beats` so a DJ mix can be analysed per track instead of assuming one
|
||||
tempo for the whole hour.
|
||||
|
||||
## 2026-07-26 session 8 — M9-C2 tempo bias: root-caused and fixed
|
||||
The orchestrator is right and my session-7 claim was wrong. **The fixtures are exact**
|
||||
(`beat120.wav` kicks on 0.5000s to the microsecond, `house128.wav` on 0.468735s), and my
|
||||
detector was **0.32% slow on both**. Worse, I had measured 120.002 correctly myself early
|
||||
in that session and then believed a cruder second scan over it — the mistake was
|
||||
disbelieving ground truth to protect the code. Corrected below.
|
||||
|
||||
**ROOT CAUSE — the onset front-end was measuring the wrong instant.**
|
||||
A percussive hit is a high-frequency transient riding a slower low-frequency body.
|
||||
`BEAT_SR=22050` resamples away everything above 11 kHz, and in `beat120.wav` the >11k
|
||||
share of the attack grows ~20x across the file (peak sample 1402 at beat 1 -> 27711 at
|
||||
beat 15). Measured, per beat, on the <11k signal that survived the decode:
|
||||
beats 1/5/29 peak at **+0 ms** after the attack, beats 15/25 at **+30 / +50 ms**.
|
||||
A lateness that VARIES per beat is a tempo error, not an offset — and the weighted
|
||||
least-squares refit then fits those biased points perfectly. It was NOT the refit
|
||||
(sub-millisecond over 200s click tracks) and NOT integer-lag AC quantisation
|
||||
(the parabolic step already handled that; the AC's own estimate was 119.87, and the refit
|
||||
moved it to 119.62 because its INPUT peaks were late).
|
||||
**Fix: `BEAT_SR, BEAT_HOP = 44100, 512`** (`server.py:882`) — same 86.13 fps envelope, same
|
||||
constants downstream, transient preserved. 172 fps was tried and REJECTED (`server.py:875`
|
||||
comment): it breaks octave choice on real records (a 90s window of a house track went to
|
||||
63.7 BPM) because the prior/harmonic weights are tuned for 86 fps.
|
||||
`server.py:906 _frame_energy()` now decodes+frames in ONE STREAMING pass (Popen, ~4MB
|
||||
reads) — 44.1k × 900s is 79MB of PCM and `subprocess.run` would hold it twice; peak memory
|
||||
is now a few MB. It also returns the exact sample count, so `duration` is 15.0 not 14.988.
|
||||
|
||||
**SECOND BUG, found by sweeping tempos after the fix** (`server.py:963 _beat_tempo`):
|
||||
a 118 BPM / 60s click track read **114.79**. Two defects compounding: (1) harmonics that
|
||||
fall outside the lag table counted as ZERO, which silently penalises the lag that owns them
|
||||
(lag 43 kept its 2x at 86, lag 44 lost its 88 — so a hair of score flipped the choice to
|
||||
the wrong lag); (2) the parabolic interpolation was then applied to a point on a SLOPE,
|
||||
which extrapolates: lag 43 became 44.95. Fixed by scoring only the harmonics that exist
|
||||
(weight-normalised), then walking uphill to the local AC maximum before interpolating, and
|
||||
clamping the correction to ±0.5 lag.
|
||||
|
||||
**RE-VERIFIED against the ground truth** (live over HTTP, `GET /beats`):
|
||||
| file | truth | detected | err | max |beat − true grid| | conf |
|
||||
|---|---|---|---|---|---|
|
||||
| `beat120.wav` | 120.003 | **120.030** | +0.027 | **6.0 ms** (last beat +3.3) | 0.92 |
|
||||
| `house128.wav` | 128.004 | **128.020** | +0.016 | **10.1 ms** (last beat +7.6) | 0.86 |
|
||||
Was −0.387 / −0.388 and 45.1 / 52.5 ms of drift.
|
||||
Synthetic sweep, 17 tempos 60…179 BPM on 60s files: **worst BPM error 0.010, worst drift
|
||||
5.7 ms across the whole file**. Octave traps still correct (alt-strong-every-2nd 128.01,
|
||||
64 BPM stays 64, kick+offbeat-hat 125.97). Real 90s house unchanged: 127.41 / 126.89 /
|
||||
121.22 / 126.96, disjoint-window self-consistency within 0.13 BPM on 3 of 4.
|
||||
Truncation path exercised (cap forced to 60s on a 200s file): grid extrapolates to 200.0s
|
||||
with 1.0 ms of drift at the end. Speed: **4-min wav 0.48s, 5-min mp3 0.84s**, 7.6-min 0.78s,
|
||||
cached 2-3 ms (was 0.66s at 22050 — the transient costs ~0.2s and is worth it).
|
||||
|
||||
**TESTS — this class of error cannot pass again** (`scripts/test_server.py:468`, 18 groups):
|
||||
1. Every click case now asserts **whole-file drift** (`max|b − round(b/per)·per| < 20 ms`),
|
||||
not just average BPM, and the sweep gained **118 BPM @ 60s** (pins the `_beat_tempo` fix).
|
||||
2. **200s click track at 128** — end-of-file drift < 20 ms, and the grid must actually reach
|
||||
past 195s (catches "correct tempo, grid stops early").
|
||||
3. **`hftick.wav`** — a 16 kHz tick over a slow 60 Hz body, i.e. the failure mechanism
|
||||
itself: the first 12 beats must land within **12 ms**. This is the one that pins
|
||||
`BEAT_SR`, and it ships in the test (assets/ is gitignored).
|
||||
4. **The fixtures as ground truth** when present: 120.003 / 128.004 ± 0.2 BPM and < 20 ms
|
||||
drift, copied into the test's asset root, skipped with a printed note when absent.
|
||||
Proven to discriminate, by monkeypatching the old code back in:
|
||||
`22050/256` -> fixtures FAIL at 45.1 / 52.5 ms drift and hftick FAILS at 26.7 ms;
|
||||
old `_beat_tempo` -> 118 BPM case FAILS at 114.79 (253 ms drift). Note the pure 200s click
|
||||
track passes under BOTH old versions — a clean click track can never catch a spectral bias,
|
||||
which is exactly why session 7's tests missed it.
|
||||
DECISIONS:
|
||||
- Ground truth wins. When a measurement disagrees with the code, the code is wrong until a
|
||||
BETTER measurement says otherwise — and "better" means the one with the sharper
|
||||
instrument (sample-level threshold crossing over the whole file), not the newer one.
|
||||
- Front-end resolution is a tuning-coupled choice, not a free knob: 44100/512 keeps every
|
||||
downstream constant valid; 44100/256 would need the prior and harmonic weights retuned.
|
||||
The comment at `server.py:875` says so, so nobody "optimises" the decode rate back down.
|
||||
BLOCKED/REQUESTS: none. Everything else from session 7 (trim work, speed, caching,
|
||||
confidence, the 503 gate) is unchanged and still green.
|
||||
NEXT: unchanged from session 7 (per-scene audio into a film bed, more transitions,
|
||||
MediaRecorder draft path, `?from=&to=` windowing on /beats for DJ mixes).
|
||||
|
||||
@ -1587,3 +1587,25 @@ INFO: 127.0.0.1:61531 - "GET /render/4c29712db270417a95697e00c73712e3/status
|
||||
INFO: 127.0.0.1:61531 - "POST /films/67b76370c643433ba4ab9f569cea82f4/shot/2 HTTP/1.1" 200 OK
|
||||
INFO: 127.0.0.1:61531 - "POST /films/67b76370c643433ba4ab9f569cea82f4/end HTTP/1.1" 200 OK
|
||||
INFO: 127.0.0.1:61531 - "GET /films/67b76370c643433ba4ab9f569cea82f4/status HTTP/1.1" 200 OK
|
||||
INFO: 127.0.0.1:54829 - "GET /scenes HTTP/1.1" 200 OK
|
||||
INFO: Shutting down
|
||||
INFO: Waiting for application shutdown.
|
||||
INFO: Application shutdown complete.
|
||||
INFO: Finished server process [99875]
|
||||
INFO: Started server process [14884]
|
||||
INFO: Waiting for application startup.
|
||||
INFO: Application startup complete.
|
||||
INFO: Uvicorn running on http://127.0.0.1:8020 (Press CTRL+C to quit)
|
||||
INFO: 127.0.0.1:55508 - "GET /beats?path=audio/test/house128.wav HTTP/1.1" 200 OK
|
||||
INFO: 127.0.0.1:55509 - "GET /beats?path=audio/test/beat120.wav HTTP/1.1" 200 OK
|
||||
INFO: 127.0.0.1:62063 - "GET /scenes HTTP/1.1" 200 OK
|
||||
INFO: Shutting down
|
||||
INFO: Waiting for application shutdown.
|
||||
INFO: Application shutdown complete.
|
||||
INFO: Finished server process [14884]
|
||||
INFO: Started server process [20352]
|
||||
INFO: Waiting for application startup.
|
||||
INFO: Application startup complete.
|
||||
INFO: Uvicorn running on http://127.0.0.1:8020 (Press CTRL+C to quit)
|
||||
INFO: 127.0.0.1:62801 - "GET /beats?path=audio/test/beat120.wav HTTP/1.1" 200 OK
|
||||
INFO: 127.0.0.1:62802 - "GET /beats?path=audio/test/house128.wav HTTP/1.1" 200 OK
|
||||
|
||||
@ -7,8 +7,11 @@ formats. See PLAN.md §4.3 for the HTTP contract.
|
||||
|
||||
uvicorn scenegod.server:app --port 8020 # binds 127.0.0.1
|
||||
"""
|
||||
import array
|
||||
import base64
|
||||
import json
|
||||
import math
|
||||
import operator
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
@ -97,6 +100,55 @@ def safe_under(root: Path, relpath: str) -> Path:
|
||||
return full
|
||||
|
||||
|
||||
def _num(v, default=None):
|
||||
"""Strict number coercion: None -> default, wrong type -> None (a violation)."""
|
||||
if v is None:
|
||||
return default
|
||||
if isinstance(v, bool) or not isinstance(v, (int, float)):
|
||||
return None
|
||||
return float(v)
|
||||
|
||||
|
||||
def resolve_audio(a: dict) -> tuple[dict | None, str | None]:
|
||||
"""One audio entry (scene `audio[]` / sequence bed) -> a muxable dict, or an
|
||||
error string. Shared by /render/end and plan_sequence so a clip means the
|
||||
same thing in a shot and in a film. `in`/`out` are SOURCE-relative seconds
|
||||
(Lane B's audio trim); `start` is where the trimmed piece lands on the
|
||||
timeline. Path-guarded like every other path-taking input."""
|
||||
if not isinstance(a, dict) or not isinstance(a.get("path"), str):
|
||||
return None, "needs a string path"
|
||||
try:
|
||||
f = safe_under(ASSETS, a["path"])
|
||||
except HTTPException:
|
||||
return None, f"bad path {a['path']!r}"
|
||||
if not f.is_file():
|
||||
return None, f"not found under assets: {a['path']!r}"
|
||||
start, gain = _num(a.get("start"), 0.0), _num(a.get("gain"), 1.0)
|
||||
t_in, t_out = _num(a.get("in"), 0.0), _num(a.get("out"))
|
||||
if start is None or gain is None or t_in is None or \
|
||||
(a.get("out") is not None and t_out is None):
|
||||
return None, "start/gain/in/out must be numbers"
|
||||
if start < 0 or t_in < 0:
|
||||
return None, f"start/in must be >= 0 (start={start}, in={t_in})"
|
||||
if t_out is not None and t_out <= t_in:
|
||||
return None, f"in ({t_in}) must be less than out ({t_out})"
|
||||
e = {"path": a["path"], "file": str(f), "start": start, "gain": gain}
|
||||
if t_in or t_out is not None:
|
||||
# a window entirely past the end of the file makes ffmpeg exit -22 mid-mux
|
||||
# ("Error muxing a packet"), which surfaces as a mystery failed render.
|
||||
src = _probe_dur(f) # only probed when a trim is asked for
|
||||
if src:
|
||||
if t_in >= src:
|
||||
return None, f"in ({t_in}) is past the end of {a['path']!r} ({src:.3f}s)"
|
||||
if t_out is not None and t_out > src:
|
||||
t_out = None # tail trim past EOF = no tail trim
|
||||
if t_in:
|
||||
e["in"] = t_in
|
||||
if t_out is not None:
|
||||
e["out"] = t_out
|
||||
return e, None
|
||||
|
||||
|
||||
# ---- assets ----
|
||||
_tree_cache = {"t": 0.0, "val": None}
|
||||
|
||||
@ -309,22 +361,47 @@ def _render_dir(rid: str) -> Path:
|
||||
return safe_under(RENDERS, rid) # rid is a hex uuid; guard anyway
|
||||
|
||||
|
||||
def audio_input(a: dict) -> list[str]:
|
||||
"""ffmpeg input args for one audio entry, honouring the SOURCE-relative
|
||||
`in`/`out` window (Lane B's audio trim; absent = whole file).
|
||||
|
||||
The trim rides on the INPUT (-ss/-t) rather than an `atrim` in the filter
|
||||
graph, which is not a style choice: `atrim,...,apad` yields a zero-sample
|
||||
audio stream on ffmpeg 8.x (apad emits frames at ~INT64_MAX PTS, the muxer
|
||||
drops the lot — an mp4 with an aac stream and no audio in it). Input seeking
|
||||
keeps the graph byte-identical to the chain that has been shipping."""
|
||||
pre = []
|
||||
if a.get("in"):
|
||||
pre += ["-ss", f"{a['in']:.6f}"]
|
||||
if a.get("out") is not None:
|
||||
pre += ["-t", f"{a['out'] - (a.get('in') or 0.0):.6f}"]
|
||||
return pre + ["-i", a["file"]]
|
||||
|
||||
|
||||
def audio_filters(audio: list[dict], first_input: int) -> list[str]:
|
||||
"""Filter-graph fragments for the audio inputs, ending in [aout]: delay each
|
||||
track to its timeline `start`, apply gain, mix down to one stereo stream."""
|
||||
if not audio:
|
||||
return []
|
||||
parts = [f"[{first_input + k}:a]adelay={int(a['start']*1000)}:all=1,volume={a['gain']}[a{k}]"
|
||||
for k, a in enumerate(audio)]
|
||||
# apad + -shortest: the VIDEO decides the length. Without the pad, audio
|
||||
# shorter than the shot truncates the picture (silent footage loss).
|
||||
parts.append("".join(f"[a{k}]" for k in range(len(audio)))
|
||||
+ f"amix=inputs={len(audio)}:normalize=0,apad[aout]")
|
||||
return parts
|
||||
|
||||
|
||||
def _encode(rid: str, fps: int, audio: list[dict]):
|
||||
"""audio = [{file: abs path, start: sec, gain: float}] (paths pre-resolved)."""
|
||||
"""audio = [{file: abs path, start: sec, gain: float, in?, out?}] (resolved)."""
|
||||
d = _render_dir(rid)
|
||||
_renders[rid] = {"state": "encoding", "log": ""}
|
||||
cmd = ["ffmpeg", "-y", "-framerate", str(fps), "-i", str(d / "frames" / "%06d.png")]
|
||||
for a in audio:
|
||||
cmd += ["-i", a["file"]]
|
||||
cmd += audio_input(a) # in/out trim rides on the input (M9)
|
||||
cmd += ["-c:v", "libx264", "-pix_fmt", "yuv420p", "-crf", "18"]
|
||||
if audio: # delay each track to its start, apply gain, mix down to one stereo stream
|
||||
parts = [f"[{i}:a]adelay={int(a['start']*1000)}:all=1,volume={a['gain']}[a{i}]"
|
||||
for i, a in enumerate(audio, start=1)]
|
||||
# apad + -shortest: the VIDEO decides the length. Without the pad, audio
|
||||
# shorter than the shot truncates the picture (silent footage loss).
|
||||
parts.append("".join(f"[a{i}]" for i in range(1, len(audio) + 1))
|
||||
+ f"amix=inputs={len(audio)}:normalize=0,apad[aout]")
|
||||
cmd += ["-filter_complex", ";".join(parts),
|
||||
if audio: # delay to start, apply gain, mix down to one stereo stream
|
||||
cmd += ["-filter_complex", ";".join(audio_filters(audio, 1)),
|
||||
"-map", "0:v", "-map", "[aout]", "-c:a", "aac", "-shortest"]
|
||||
cmd += [str(d / "out.mp4")]
|
||||
try:
|
||||
@ -384,12 +461,11 @@ async def render_end(rid: str, request: Request):
|
||||
except json.JSONDecodeError:
|
||||
body = {}
|
||||
audio = []
|
||||
for a in body.get("audio") or []: # scene JSON audio[] (M4)
|
||||
f = safe_under(ASSETS, a["path"]) # path-guard every audio input
|
||||
if not f.is_file():
|
||||
raise HTTPException(400, f"audio not found: {a['path']}")
|
||||
audio.append({"file": str(f), "start": float(a.get("start", 0)),
|
||||
"gain": float(a.get("gain", 1.0))})
|
||||
for a in body.get("audio") or []: # scene JSON audio[] (M4, +in/out M9)
|
||||
e, err = resolve_audio(a if isinstance(a, dict) else {})
|
||||
if err:
|
||||
raise HTTPException(400, f"audio: {err}")
|
||||
audio.append(e)
|
||||
threading.Thread(target=_encode, args=(rid, fps, audio), daemon=True).start()
|
||||
return {"ok": True}
|
||||
|
||||
@ -431,15 +507,6 @@ MAX_SHOTS = 200
|
||||
MAX_TRANSITION_S = 10.0
|
||||
|
||||
|
||||
def _num(v, default=None):
|
||||
"""Strict number coercion: None -> default, wrong type -> None (a violation)."""
|
||||
if v is None:
|
||||
return default
|
||||
if isinstance(v, bool) or not isinstance(v, (int, float)):
|
||||
return None
|
||||
return float(v)
|
||||
|
||||
|
||||
def plan_sequence(seq: dict) -> tuple[list[str], dict]:
|
||||
"""Validate a sequence AND resolve it to a concrete shot plan in one pass, so
|
||||
save-time validation and render-time planning can never disagree about what a
|
||||
@ -529,22 +596,11 @@ def plan_sequence(seq: dict) -> tuple[list[str], dict]:
|
||||
errs.append("audio must be a list")
|
||||
aud = []
|
||||
for k, a in enumerate(aud):
|
||||
if not isinstance(a, dict) or not isinstance(a.get("path"), str):
|
||||
errs.append(f"audio[{k}]: needs a string path")
|
||||
continue
|
||||
try:
|
||||
f = safe_under(ASSETS, a["path"]) # path-guard every audio input
|
||||
except HTTPException:
|
||||
errs.append(f"audio[{k}]: bad path {a['path']!r}")
|
||||
continue
|
||||
if not f.is_file():
|
||||
errs.append(f"audio[{k}]: not found under assets: {a['path']!r}")
|
||||
continue
|
||||
start, gain = _num(a.get("start"), 0.0), _num(a.get("gain"), 1.0)
|
||||
if start is None or gain is None or start < 0:
|
||||
errs.append(f"audio[{k}]: start/gain must be numbers (start >= 0)")
|
||||
continue
|
||||
plan["audio"].append({"path": a["path"], "file": str(f), "start": start, "gain": gain})
|
||||
e, err = resolve_audio(a)
|
||||
if err:
|
||||
errs.append(f"audio[{k}]: {err}")
|
||||
else:
|
||||
plan["audio"].append(e)
|
||||
|
||||
plan["duration"] = round(sum(s["len"] for s in ps) - sum(s["transitionDur"] for s in ps[:-1]), 6)
|
||||
plan["warnings"] = list(dict.fromkeys(warnings)) # dedupe, keep order
|
||||
@ -661,7 +717,7 @@ def _concat_film(fid: str):
|
||||
durs = [_probe_dur(f) or s["len"] for f, s in zip(srcs, shots)]
|
||||
|
||||
cmd = ["ffmpeg", "-y"] + [x for f in srcs for x in ("-i", str(f))]
|
||||
cmd += [x for a in meta["audio"] for x in ("-i", a["file"])]
|
||||
cmd += [x for a in meta["audio"] for x in audio_input(a)] # in/out trim (M9)
|
||||
# normalise every shot to the film's grid — mismatches are rejected at
|
||||
# registration, this is the belt to that braces (SAR/fps oddities included).
|
||||
parts = [f"[{i}:v]scale={w}:{h},setsar=1,fps={fps},format=yuv420p[v{i}]" for i in range(len(srcs))]
|
||||
@ -677,11 +733,7 @@ def _concat_film(fid: str):
|
||||
parts.append(f"[{cur}][v{i}]concat=n=2:v=1:a=0[x{i}]")
|
||||
acc += durs[i]
|
||||
cur = f"x{i}"
|
||||
for k, a in enumerate(meta["audio"]):
|
||||
parts.append(f"[{len(srcs)+k}:a]adelay={int(a['start']*1000)}:all=1,volume={a['gain']}[a{k}]")
|
||||
if meta["audio"]:
|
||||
parts.append("".join(f"[a{k}]" for k in range(len(meta["audio"])))
|
||||
+ f"amix=inputs={len(meta['audio'])}:normalize=0,apad[aout]")
|
||||
parts += audio_filters(meta["audio"], len(srcs)) # delay/gain/mix the film-level bed
|
||||
cmd += ["-filter_complex", ";".join(parts), "-map", f"[{cur}]"]
|
||||
if meta["audio"]:
|
||||
cmd += ["-map", "[aout]", "-c:a", "aac", "-shortest"] # apad+shortest: video sets the length
|
||||
@ -814,6 +866,274 @@ def rhubarb(path: str):
|
||||
return json.loads(p.stdout)
|
||||
|
||||
|
||||
# ---- BEAT DETECTION (M9-C) — the grid a music video cuts itself to ----
|
||||
# stdlib + ffmpeg only (repo rule: no librosa/aubio/numpy). The chain:
|
||||
# ffmpeg -> mono s16 PCM @ 44100 (anything ffmpeg reads: wav/mp3/mp4/...)
|
||||
# framing -> per-frame log energy, hop 512 = 86.13 envelope fps
|
||||
# onset -> half-wave-rectified first difference [ENERGY flux]
|
||||
# whitening -> subtract a local mean so a dense mix has a floor near zero
|
||||
# tempo -> normalised autocorrelation 60..180 BPM, harmonics added, then
|
||||
# a log-gaussian prior centred on 125 BPM (John's material is
|
||||
# 90s house), then an explicit half/double check
|
||||
# phase -> comb-filter search over one period
|
||||
# grid -> least-squares refit of period+phase to the nearest onset peaks
|
||||
# ENERGY flux, not spectral flux: a pure-Python radix-2 FFT would be ~20k
|
||||
# transforms for a 4-minute track (tens of seconds of CPU), and on
|
||||
# four-to-the-floor material the kick dominates the broadband energy anyway.
|
||||
# Measured on ultra: 5-minute mp3 = 0.85s wall. 4/4 is ASSUMED — `downbeats` is
|
||||
# every 4th beat, phase chosen as the strongest of the four cosets. Beats are a
|
||||
# CONSTANT-tempo grid (no rubato tracking).
|
||||
#
|
||||
# DECODE AT 44100, NOT 22050 (M9-C fix — do not "optimise" this back down):
|
||||
# a percussive hit is a high-frequency transient over a slower low-frequency
|
||||
# body. Resampling to 22050 throws away everything above 11kHz, and in
|
||||
# assets/audio/test/beat120.wav the >11k share of the attack grows 20x across
|
||||
# the file (1402 -> 27711 peak sample) — so for the middle beats the surviving
|
||||
# <11k energy peaks 30-50ms AFTER the attack while the first and last beats
|
||||
# still peak at +0ms. A lateness that VARIES per beat is a tempo bias, not an
|
||||
# offset: the detector read 119.62 on a file whose kicks are on 0.5000s to the
|
||||
# microsecond, i.e. 0.32% slow = ~0.8s of slip across a 4-minute track. At 44100
|
||||
# the transient survives and the same code reads 120.03. (172 fps was tried and
|
||||
# rejected: it breaks octave choice on real records — a 90s house window went to
|
||||
# 63.7 BPM — because the prior/harmonic weights below are tuned for 86 fps.)
|
||||
BEAT_SR, BEAT_HOP = 44100, 512
|
||||
BEAT_FPS = BEAT_SR / BEAT_HOP # 86.13 onset frames/s
|
||||
BEAT_MIN_BPM, BEAT_MAX_BPM = 60.0, 180.0
|
||||
BEAT_PRIOR_BPM, BEAT_PRIOR_OCT = 125.0, 0.9 # tempo prior: centre, width in octaves
|
||||
BEAT_PREF = (90.0, 180.0) # preferred band for the octave check
|
||||
BEAT_WHITEN_W = 32 # local-mean window, +-0.37s
|
||||
BEAT_MAX_S = 900 # analyse at most 15 min (memory); grid extends
|
||||
_beats_cache: dict[tuple, dict] = {}
|
||||
|
||||
|
||||
def _frame_energy(f: Path) -> tuple[list[float], int]:
|
||||
"""Decode and reduce to per-frame log energy in ONE streaming pass. 15
|
||||
minutes at 44.1k is 79MB of PCM; buffering that twice (subprocess buffer +
|
||||
array) is not a thing to do inside the prod container, and none of it is
|
||||
needed after the framing. stderr stays a pipe for the error message —
|
||||
safe only because `-v error` keeps it far under the pipe buffer."""
|
||||
p = subprocess.Popen(["ffmpeg", "-v", "error", "-nostdin", "-i", str(f), "-map", "0:a:0",
|
||||
"-t", str(BEAT_MAX_S), "-f", "s16le", "-ac", "1",
|
||||
"-ar", str(BEAT_SR), "-"],
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
e, buf, stride, nbytes = [], b"", BEAT_HOP * 2, 0
|
||||
ap = e.append
|
||||
while True:
|
||||
chunk = p.stdout.read(stride * 4096) # ~4MB reads
|
||||
if not chunk:
|
||||
break
|
||||
nbytes += len(chunk)
|
||||
buf = buf + chunk if buf else chunk
|
||||
nfr = len(buf) // stride
|
||||
if not nfr:
|
||||
continue
|
||||
a = array.array("h")
|
||||
a.frombytes(buf[:nfr * stride])
|
||||
buf = buf[nfr * stride:]
|
||||
for i in range(0, len(a), BEAT_HOP):
|
||||
fr = a[i:i + BEAT_HOP]
|
||||
ap(math.log1p(sum(map(operator.mul, fr, fr)) / BEAT_HOP))
|
||||
err = p.stderr.read()
|
||||
if p.wait() != 0:
|
||||
raise HTTPException(400, "cannot decode audio: "
|
||||
+ err[-200:].decode("utf-8", "replace").strip())
|
||||
return e, nbytes // 2 # sample count, so `duration` isn't a frame short
|
||||
|
||||
|
||||
def _onset_envelope(e: list[float]) -> list[float]:
|
||||
"""Per-frame log energy -> rectified first difference -> 3-tap smooth ->
|
||||
local-mean subtraction. The whitening is what makes a busy record's peaks
|
||||
stand out from its floor (and the confidence number mean anything)."""
|
||||
o = [0.0] + [max(0.0, e[i] - e[i - 1]) for i in range(1, len(e))]
|
||||
n = len(o)
|
||||
o = [(o[max(0, i - 1)] + o[i] + o[min(n - 1, i + 1)]) / 3 for i in range(n)]
|
||||
w = BEAT_WHITEN_W
|
||||
if n < 2 * w + 2:
|
||||
return o
|
||||
run, out = sum(o[:w + 1]), []
|
||||
for i in range(n):
|
||||
out.append(max(0.0, o[i] - run / (min(n - 1, i + w) - max(0, i - w) + 1)))
|
||||
if i + 1 + w < n:
|
||||
run += o[i + 1 + w]
|
||||
if i - w >= 0:
|
||||
run -= o[i - w]
|
||||
return out
|
||||
|
||||
|
||||
def _beat_tempo(o: list[float]) -> tuple[float, dict]:
|
||||
"""(period in frames, autocorrelation table). Peak is parabola-interpolated:
|
||||
the integer lag is only good to ~0.5 frame, which is seconds of drift across
|
||||
a 4-minute track before the least-squares refit cleans it up."""
|
||||
n = len(o)
|
||||
m = sum(o) / n
|
||||
oc = [x - m for x in o]
|
||||
e0 = sum(map(operator.mul, oc, oc)) / n or 1e-9
|
||||
lo = int(60 * BEAT_FPS / BEAT_MAX_BPM)
|
||||
hi = int(math.ceil(60 * BEAT_FPS / BEAT_MIN_BPM))
|
||||
r = {}
|
||||
for lag in range(lo, hi + 1):
|
||||
k = n - lag
|
||||
r[lag] = (sum(map(operator.mul, oc[:k], oc[lag:])) / k / e0) if k > 8 else 0.0
|
||||
p0 = 60 * BEAT_FPS / BEAT_PRIOR_BPM
|
||||
|
||||
def score(g):
|
||||
# harmonics that fall OUTSIDE the lag table must not count as zero — that
|
||||
# silently penalises every lag whose 2x is past the table edge (it cost a
|
||||
# 118 BPM track 3 BPM: lag 43 kept its 2x at 86, lag 44 lost its 88).
|
||||
terms = [(1.0, r[g])] + [(w, r[h]) for w, h in
|
||||
((.5, 2 * g), (.5, g // 2), (.3, 3 * g // 2)) if h in r]
|
||||
return (sum(w * v for w, v in terms) / sum(w for w, _ in terms)
|
||||
* math.exp(-.5 * (math.log2(g / p0) / BEAT_PRIOR_OCT) ** 2))
|
||||
|
||||
best = max(r, key=score)
|
||||
# the harmonics and the prior choose WHICH peak; the raw autocorrelation
|
||||
# locates it. Walk uphill first: a parabola fitted to a point on a SLOPE
|
||||
# extrapolates instead of interpolating (that is how lag 43 became 44.95).
|
||||
while best + 1 in r and r[best + 1] > r[best]:
|
||||
best += 1
|
||||
while best - 1 in r and r[best - 1] > r[best]:
|
||||
best -= 1
|
||||
a, b, c = r.get(best - 1, 0.), r[best], r.get(best + 1, 0.)
|
||||
d = a - 2 * b + c
|
||||
return best + max(-0.5, min(0.5, 0.5 * (a - c) / d if d else 0.0)), r
|
||||
|
||||
|
||||
def _comb(o: list[float], period: float, phase: float) -> float:
|
||||
"""Mean onset strength on a beat grid — the score a phase/period is judged by.
|
||||
MEAN (not sum) is deliberate: it makes a double-tempo grid pay for the empty
|
||||
positions it adds, which is half the octave problem solved for free."""
|
||||
s, cnt, i, n = 0.0, 0, phase, len(o)
|
||||
while i < n - 1:
|
||||
s += o[int(i + 0.5)]
|
||||
cnt += 1
|
||||
i += period
|
||||
return s / cnt if cnt else 0.0
|
||||
|
||||
|
||||
def _best_phase(o: list[float], period: float) -> tuple[float, int]:
|
||||
return max((_comb(o, period, ph), ph) for ph in range(max(1, int(period))))
|
||||
|
||||
|
||||
def _fit_grid(o: list[float], period: float, phase: float, iters: int = 3):
|
||||
"""Snap each predicted beat to the nearest onset peak (sub-frame, parabolic)
|
||||
and refit t_k = phase + k*period by onset-weighted least squares."""
|
||||
n = len(o)
|
||||
lo_p, hi_p = 60 * BEAT_FPS / BEAT_MAX_BPM, 60 * BEAT_FPS / BEAT_MIN_BPM
|
||||
for _ in range(iters):
|
||||
win = max(1, int(period * 0.12))
|
||||
xs, ys, ws = [], [], []
|
||||
k, t = 0, phase
|
||||
while t < n - 1:
|
||||
c = int(round(t))
|
||||
a, b = max(1, c - win), min(n - 2, c + win)
|
||||
if b > a:
|
||||
j = max(range(a, b + 1), key=o.__getitem__)
|
||||
p1, p2, p3 = o[j - 1], o[j], o[j + 1]
|
||||
d = p1 - 2 * p2 + p3
|
||||
frac = (0.5 * (p1 - p3) / d) if d else 0.0
|
||||
if abs(frac) > 1:
|
||||
frac = 0.0
|
||||
if p2 > 0:
|
||||
xs.append(k)
|
||||
ys.append(j + frac)
|
||||
ws.append(p2)
|
||||
k += 1
|
||||
t = phase + k * period
|
||||
if len(xs) < 4:
|
||||
break
|
||||
sw = sum(ws)
|
||||
mx = sum(w * x for w, x in zip(ws, xs)) / sw
|
||||
my = sum(w * y for w, y in zip(ws, ys)) / sw
|
||||
den = sum(w * (x - mx) ** 2 for w, x in zip(ws, xs))
|
||||
if den <= 0:
|
||||
break
|
||||
period = min(max(sum(w * (x - mx) * (y - my)
|
||||
for w, x, y in zip(ws, xs, ys)) / den, lo_p), hi_p)
|
||||
phase = my - period * mx
|
||||
return period, phase
|
||||
|
||||
|
||||
def analyse_beats(f: Path) -> dict:
|
||||
frames, nsamples = _frame_energy(f)
|
||||
analyzed = nsamples / BEAT_SR
|
||||
truncated = analyzed >= BEAT_MAX_S - 0.5
|
||||
full = (_probe_dur(f) or analyzed) if truncated else analyzed
|
||||
out = {"bpm": 0.0, "confidence": 0.0, "beats": [], "downbeats": [], "meter": 4,
|
||||
"duration": round(full, 3), "analyzed": round(analyzed, 3),
|
||||
"period": 0.0, "acPeak": 0.0, "onBeatContrast": 0.0}
|
||||
o = _onset_envelope(frames)
|
||||
n = len(o)
|
||||
lo_p, hi_p = 60 * BEAT_FPS / BEAT_MAX_BPM, 60 * BEAT_FPS / BEAT_MIN_BPM
|
||||
if n < 2 * hi_p or max(o, default=0.0) <= 1e-9:
|
||||
out["note"] = "no beat grid: silent, or shorter than two slow bars"
|
||||
return out
|
||||
lag, r = _beat_tempo(o)
|
||||
cands = []
|
||||
for p in (lag / 2, lag, lag * 2): # explicit octave check: half, self, double
|
||||
if lo_p <= p <= hi_p:
|
||||
s, ph = _best_phase(o, p)
|
||||
cands.append((s, p, ph))
|
||||
top = max(c[0] for c in cands)
|
||||
keep = [c for c in cands if c[0] >= 0.85 * top] or cands
|
||||
# when half/double score comparably, take the one in the sane band — a 128 BPM
|
||||
# record whose onset envelope autocorrelates hardest at 64 is the normal case.
|
||||
_, period, phase = max(keep, key=lambda c: (BEAT_PREF[0] <= 60 * BEAT_FPS / c[1]
|
||||
<= BEAT_PREF[1], c[0]))
|
||||
period, phase = _fit_grid(o, period, phase)
|
||||
|
||||
# frame i is the window [i, i+1) hops, so its centre is the honest onset time
|
||||
ph = phase + 0.5
|
||||
ph -= period * math.floor((ph + 0.05 * BEAT_FPS) / period) # into [-50ms, period-50ms):
|
||||
beats_f = [] # a track starting ON the
|
||||
k = 0 # beat gets a beat at ~0
|
||||
while ph + k * period <= full * BEAT_FPS:
|
||||
beats_f.append(ph + k * period)
|
||||
k += 1
|
||||
inwin = [(i, int(round(t))) for i, t in enumerate(beats_f) if 0 <= round(t) < n]
|
||||
onb = [o[j] for _, j in inwin]
|
||||
onm = sum(onb) / len(onb) if onb else 0.0
|
||||
allm = sum(o) / n
|
||||
# confidence = geometric mean of two independent, honest checks:
|
||||
# acPeak how periodic the onset envelope is at this period (0..1)
|
||||
# onBeatContrast how much onset energy lands ON the emitted grid vs the
|
||||
# whole track, (on-all)/(on+all), 0 = no better than chance
|
||||
# Measured: synthetic click tracks 0.83-0.93, real 90s house 0.32-0.51,
|
||||
# pink noise 0.09, silence 0.0. Treat >0.5 as a solid grid, <0.2 as mush.
|
||||
ac = max(0.0, min(1.0, r.get(int(round(period)), 0.0)))
|
||||
contrast = max(0.0, (onm - allm) / (onm + allm)) if onm + allm else 0.0
|
||||
dbc = max(range(4), key=lambda c: sum(o[j] for i, j in inwin if i % 4 == c)
|
||||
/ max(1, sum(1 for i, _ in inwin if i % 4 == c)))
|
||||
out.update(bpm=round(60 * BEAT_FPS / period, 2),
|
||||
period=round(period / BEAT_FPS, 6),
|
||||
confidence=round(math.sqrt(ac * contrast), 3),
|
||||
acPeak=round(ac, 3), onBeatContrast=round(contrast, 3),
|
||||
beats=[round(max(0.0, t / BEAT_FPS), 4) for t in beats_f],
|
||||
downbeats=[round(max(0.0, t / BEAT_FPS), 4) for t in beats_f[dbc::4]])
|
||||
if truncated:
|
||||
out["truncated"] = True # grid extrapolated past the analysed window
|
||||
return out
|
||||
|
||||
|
||||
@app.get("/beats")
|
||||
def beats(path: str):
|
||||
"""Beat grid for an audio asset -> {bpm, confidence, beats[], downbeats[],
|
||||
duration, ...}. Times are absolute seconds from the start of the file; the
|
||||
grid is constant-tempo and 4/4 is assumed (see the section header for the
|
||||
algorithm and what `confidence` means). Cached by (path, mtime, size)."""
|
||||
if not HAVE_FFMPEG:
|
||||
raise HTTPException(503, "ffmpeg not found on server")
|
||||
f = safe_under(ASSETS, path)
|
||||
if not f.is_file():
|
||||
raise HTTPException(404, "not found")
|
||||
st = f.stat()
|
||||
key = (str(f), st.st_mtime_ns, st.st_size)
|
||||
if key not in _beats_cache:
|
||||
if len(_beats_cache) > 32:
|
||||
_beats_cache.clear() # ponytail: tiny bounded cache; clearing beats an LRU here
|
||||
_beats_cache[key] = analyse_beats(f)
|
||||
return {"path": path, **_beats_cache[key]}
|
||||
|
||||
|
||||
# ---- DIRECTOR MODE (M5) — NL text -> validated preset ops via tailnet LLM ----
|
||||
# Vocab mirrors PLAN §5 M5. NOTE: Lane A's web/grammar.js hasn't landed yet —
|
||||
# when it does, re-point these lists at that file (read at boot) so the server
|
||||
|
||||
@ -10,6 +10,7 @@ validation failures.
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
@ -44,6 +45,32 @@ def get_json(path):
|
||||
return json.loads(b)
|
||||
|
||||
|
||||
def ff(*args):
|
||||
subprocess.run(["ffmpeg", "-y", *args], check=True,
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
|
||||
|
||||
def probe_dur(f):
|
||||
return float(subprocess.check_output(
|
||||
["ffprobe", "-v", "error", "-show_entries", "format=duration",
|
||||
"-of", "default=nw=1:nokey=1", str(f)], text=True).strip())
|
||||
|
||||
|
||||
def seg_db(f, ss, dur):
|
||||
"""mean volume (dB) of one window of a media file — digital silence reads
|
||||
about -91 dB, so this is how a test asks 'is the audio actually HERE?'.
|
||||
Refuses to answer when the window decoded NO samples: our muxes apad to the
|
||||
video length, so an empty window means a broken stream, not a quiet one
|
||||
(an atrim+apad graph once produced exactly that — aac stream, zero audio)."""
|
||||
p = subprocess.run(["ffmpeg", "-v", "info", "-ss", str(ss), "-t", str(dur), "-i", str(f),
|
||||
"-af", "volumedetect", "-f", "null", "-"], capture_output=True, text=True)
|
||||
m = re.search(r"mean_volume:\s*(-?[\d.]+) dB", p.stderr)
|
||||
if not m:
|
||||
raise AssertionError(f"no audio samples in {Path(f).name} at {ss}..{ss+dur}s\n"
|
||||
+ p.stderr[-400:])
|
||||
return float(m.group(1))
|
||||
|
||||
|
||||
def main():
|
||||
tmp = Path(tempfile.mkdtemp(prefix="scenegod_test_"))
|
||||
assets, scenes = tmp / "assets", tmp / "scenes"
|
||||
@ -219,6 +246,52 @@ def main():
|
||||
assert req("POST", f"/render/{rid}/end",
|
||||
json.dumps({"audio": [{"path": "audio/nope.wav"}]}))[0] == 400
|
||||
n += 1
|
||||
|
||||
# --- M9-C1: audio in/out honoured at the mux -------------------
|
||||
# trim.wav = 1s of silence, then 1s of 880Hz. Trimming to the tone
|
||||
# and starting at 0.5s must put SOUND at 0.5-1.5s and nothing else;
|
||||
# the same clip untrimmed must still lead with its silent second.
|
||||
ff("-f", "lavfi", "-i", "aevalsrc='if(gte(t,1),0.7*sin(2*PI*880*t),0)':s=44100:d=2",
|
||||
"-c:a", "pcm_s16le", str(assets / "audio" / "test" / "trim.wav"))
|
||||
pngs = sorted(fr.glob("*.png"))
|
||||
|
||||
def render_audio(audio, frames=90): # 3s of picture @30fps
|
||||
r2 = json.loads(req("POST", "/render/begin",
|
||||
json.dumps({"name": "t", "fps": 30}))[1])["renderId"]
|
||||
for i in range(frames):
|
||||
assert req("POST", f"/render/{r2}/frame/{i}",
|
||||
pngs[i % len(pngs)].read_bytes())[0] == 200
|
||||
assert req("POST", f"/render/{r2}/end", json.dumps({"audio": audio}))[0] == 200
|
||||
for _ in range(150):
|
||||
s2 = json.loads(req("GET", f"/render/{r2}/status")[1])["state"]
|
||||
if s2 in ("done", "error"):
|
||||
break
|
||||
time.sleep(0.2)
|
||||
assert s2 == "done", s2
|
||||
return renders / r2 / "out.mp4"
|
||||
|
||||
cut = render_audio([{"path": "audio/test/trim.wav", "start": 0.5,
|
||||
"in": 1.0, "out": 2.0, "gain": 1.0}])
|
||||
assert abs(probe_dur(cut) - 3.0) < 0.2, probe_dur(cut)
|
||||
assert seg_db(cut, 0.0, 0.4) < -50, f"trimmed clip leaks before start: {seg_db(cut,0,0.4)}"
|
||||
assert seg_db(cut, 0.6, 0.8) > -35, f"trimmed tone missing at 0.6-1.4s: {seg_db(cut,0.6,0.8)}"
|
||||
assert seg_db(cut, 1.7, 1.0) < -50, f"trimmed clip runs past out: {seg_db(cut,1.7,1.0)}"
|
||||
whole = render_audio([{"path": "audio/test/trim.wav", "start": 0.5, "gain": 1.0}])
|
||||
assert seg_db(whole, 0.6, 0.8) < -50, "untrimmed clip must still play its silent head"
|
||||
assert seg_db(whole, 1.6, 0.8) > -35, "untrimmed tone should land a second later"
|
||||
# and the trim is validated, not trusted
|
||||
for bad in ({"in": -1}, {"in": 2, "out": 1}, {"in": "x"}, {"start": -1}):
|
||||
assert req("POST", f"/render/{rid}/end", json.dumps(
|
||||
{"audio": [{"path": "audio/test/trim.wav", **bad}]}))[0] == 400, bad
|
||||
# a window past EOF would fail the MUX (ffmpeg -22) — caught up front
|
||||
st, b = req("POST", f"/render/{rid}/end", json.dumps(
|
||||
{"audio": [{"path": "audio/test/trim.wav", "in": 99}]}))
|
||||
assert st == 400 and b"past the end" in b, b
|
||||
# a tail past EOF is just no tail trim: renders, and still plays
|
||||
past = render_audio([{"path": "audio/test/trim.wav", "start": 0.0,
|
||||
"in": 1.0, "out": 60.0, "gain": 1.0}])
|
||||
assert seg_db(past, 0.1, 0.8) > -35, "clamped tail trim lost the audio"
|
||||
n += 1
|
||||
else:
|
||||
print("(render test skipped — ffmpeg/ffprobe not found)")
|
||||
|
||||
@ -353,9 +426,12 @@ def main():
|
||||
"-i", "sine=frequency=440:duration=1", str(tone)],
|
||||
check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
dshots = [dict(shots[0], transition="dissolve", transitionDur=0.5), shots[1]]
|
||||
# M9-C1: the film-level bed is trimmed too — trim.wav's tone (source
|
||||
# 1..2s) placed at 2.0s must be silent before it and present after.
|
||||
film2 = make_film("film-dissolve", {"version": 1, "shots": dshots,
|
||||
"audio": [{"path": "audio/test/tone.wav",
|
||||
"start": 0.25, "gain": 0.8}]})
|
||||
"audio": [{"path": "audio/test/trim.wav",
|
||||
"start": 2.0, "in": 1.0, "out": 2.0,
|
||||
"gain": 0.8}]})
|
||||
assert film2["duration"] == 4.5, film2
|
||||
assert not film2["warnings"], film2 # bed present -> no warning
|
||||
out2 = finish_film(film2["filmId"], [fake_shot_render(2.0), fake_shot_render(3.0, color="blue")])
|
||||
@ -363,6 +439,13 @@ def main():
|
||||
assert abs(dur2 - 4.5) < 0.15, f"dissolve film {dur2}s, expected ~4.5s (2+3-0.5)"
|
||||
codecs = probe(out2, "stream=codec_type").split()
|
||||
assert "audio" in codecs, f"film audio bed not muxed: {codecs}"
|
||||
assert seg_db(out2, 0.0, 1.8) < -50, f"film bed leaks before start: {seg_db(out2,0,1.8)}"
|
||||
assert seg_db(out2, 2.1, 0.8) > -35, f"film bed tone missing: {seg_db(out2,2.1,0.8)}"
|
||||
# a bad trim is a 422 with a message, same as every other violation
|
||||
st, b = req("POST", "/sequences/bad", json.dumps(
|
||||
{"version": 1, "shots": dshots,
|
||||
"audio": [{"path": "audio/test/trim.wav", "in": 3, "out": 1}]}))
|
||||
assert st == 422 and b"less than" in b, b
|
||||
n += 1
|
||||
else:
|
||||
print("(film test skipped — ffmpeg/ffprobe not found)")
|
||||
@ -382,6 +465,138 @@ def main():
|
||||
assert sh.get("out", dur) <= dur + 1e-6, f"{p.name}: shot past end of {sh['scene']}"
|
||||
n += 1
|
||||
|
||||
# --- M9-C2: BEAT DETECTION on click tracks of KNOWN tempo ------------
|
||||
if shutil.which("ffmpeg"):
|
||||
(assets / "audio" / "test").mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def click(name, bpm, dur=20):
|
||||
"""Exactly-periodic click track: mod(t, period) in aevalsrc is
|
||||
double-precision, so beat k is at exactly k*period (unlike a
|
||||
concatenated or gated source, which drifts)."""
|
||||
per = 60.0 / bpm
|
||||
ff("-f", "lavfi", "-i", f"aevalsrc='0.8*sin(2*PI*1200*t)"
|
||||
f"*exp(-45*mod(t,{per:.9f}))':s=44100:d={dur}",
|
||||
"-c:a", "pcm_s16le", str(assets / "audio" / "test" / name))
|
||||
return per
|
||||
|
||||
# 118 @ 60s is not padding: harmonics falling off the end of the lag
|
||||
# table used to penalise their own lag, and the parabola was then
|
||||
# fitted to a point on a slope — together they read this one as 114.8.
|
||||
for name, bpm, dur in (("click120.wav", 120, 20), ("click128.wav", 128, 20),
|
||||
("click118.wav", 118, 60)):
|
||||
per = click(name, bpm, dur=dur)
|
||||
r = get_json("/beats?path=audio/test/" + name)
|
||||
assert abs(r["bpm"] - bpm) <= 2, f"{name}: detected {r['bpm']}, truth {bpm}"
|
||||
assert r["confidence"] > 0.5, f"{name}: confidence {r['confidence']} on a click track"
|
||||
assert abs(r["duration"] - dur) < 0.2 and r["meter"] == 4, r
|
||||
assert len(r["beats"]) >= int(dur / per) - 1, len(r["beats"])
|
||||
for i, b in enumerate(r["beats"][:12]): # grid lands on the real clicks
|
||||
assert abs(b - i * per) < 0.03, f"{name}: beat {i} at {b}s, truth {i*per:.4f}s"
|
||||
d = max(abs(b - round(b / per) * per) for b in r["beats"]) # whole file
|
||||
assert d < 0.020, f"{name}: grid slips {d*1000:.0f}ms somewhere in the file"
|
||||
assert r["downbeats"], r
|
||||
i0 = r["beats"].index(r["downbeats"][0]) # 4/4: every 4th beat
|
||||
assert [r["beats"][i0 + 4 * k] for k in range(3)] == r["downbeats"][:3], r["downbeats"]
|
||||
|
||||
# ACCUMULATED DRIFT, not average tempo. A 0.3% tempo bias is invisible
|
||||
# in a 20s clip and is 0.6s of slip by the end of a 200s one — cuts
|
||||
# snapped late in a track would land visibly off the kick.
|
||||
per = click("long128.wav", 128, dur=200)
|
||||
r = get_json("/beats?path=audio/test/long128.wav")
|
||||
assert abs(r["bpm"] - 128) < 0.1, f"long file: {r['bpm']}"
|
||||
assert r["beats"][-1] > 195, f"grid stops at {r['beats'][-1]}s of a 200s file"
|
||||
drift = max(abs(b - round(b / per) * per) for b in r["beats"])
|
||||
assert drift < 0.020, f"grid slips {drift*1000:.0f}ms across a 200s file"
|
||||
|
||||
# The M9 bug this pins: a percussive attack is a HIGH-frequency
|
||||
# transient riding a slower low-frequency body. Decode at 22050 and
|
||||
# the transient is gone, so the onset lands on the body's energy peak
|
||||
# ~25ms late — and by a DIFFERENT amount per beat once the mix
|
||||
# evolves, which is a tempo error, not an offset. 16kHz tick over a
|
||||
# slow 60Hz body: at 44100 this reads +1ms, at 22050 +27ms.
|
||||
p124 = 60.0 / 124
|
||||
ff("-f", "lavfi", "-i",
|
||||
f"aevalsrc='0.45*sin(2*PI*60*t)*(1-exp(-8*mod(t,{p124:.9f})))"
|
||||
f"*exp(-5*mod(t,{p124:.9f}))"
|
||||
f"+0.5*sin(2*PI*16000*t)*exp(-150*mod(t,{p124:.9f}))':s=44100:d=60",
|
||||
"-c:a", "pcm_s16le", str(assets / "audio" / "test" / "hftick.wav"))
|
||||
r = get_json("/beats?path=audio/test/hftick.wav")
|
||||
assert abs(r["bpm"] - 124) < 0.1, f"hf transient: {r['bpm']}"
|
||||
for i, b in enumerate(r["beats"][:12]):
|
||||
assert abs(b - i * p124) < 0.012, (
|
||||
f"hftick beat {i} at {b}s, truth {i*p124:.4f}s — the decode is "
|
||||
f"throwing away the transient that defines the attack")
|
||||
|
||||
# The two shipped fixtures are GROUND TRUTH: measured straight from
|
||||
# PCM, beat120's onsets sit on 0.5000s multiples to the microsecond
|
||||
# (120.003 BPM) and house128's on 0.468735s (128.004). assets/ is
|
||||
# gitignored, so skip when they are not on this box.
|
||||
for name, truth in (("beat120.wav", 120.003), ("house128.wav", 128.004)):
|
||||
srcf = ROOT / "assets" / "audio" / "test" / name
|
||||
if not srcf.is_file():
|
||||
print(f"(fixture check skipped — no assets/audio/test/{name})")
|
||||
continue
|
||||
shutil.copy(srcf, assets / "audio" / "test" / name)
|
||||
r = get_json("/beats?path=audio/test/" + name)
|
||||
assert abs(r["bpm"] - truth) < 0.2, f"{name}: read {r['bpm']}, truth {truth}"
|
||||
p = 60.0 / truth
|
||||
d = max(abs(b - round(b / p) * p) for b in r["beats"])
|
||||
assert d < 0.020, f"{name}: grid is {d*1000:.0f}ms off the real kick"
|
||||
|
||||
# mush must LOOK like mush — low confidence, no crash, still 200
|
||||
ff("-f", "lavfi", "-i", "anoisesrc=d=10:c=pink:a=0.3", "-c:a", "pcm_s16le",
|
||||
str(assets / "audio" / "test" / "noise.wav"))
|
||||
ff("-f", "lavfi", "-i", "anullsrc=r=44100:cl=mono", "-t", "8", "-c:a", "pcm_s16le",
|
||||
str(assets / "audio" / "test" / "quiet.wav"))
|
||||
for name in ("noise.wav", "quiet.wav"):
|
||||
r = get_json("/beats?path=audio/test/" + name)
|
||||
assert r["confidence"] < 0.35, f"{name}: confidence {r['confidence']} on non-music"
|
||||
assert isinstance(r["beats"], list) and isinstance(r["bpm"], (int, float)), r
|
||||
assert get_json("/beats?path=audio/test/quiet.wav")["beats"] == [], "silence has no grid"
|
||||
|
||||
# guards + cache
|
||||
for bad in ("../../etc/hosts", "..%2f..%2fetc%2fhosts",
|
||||
urllib.parse.quote("/etc/hosts", safe="")):
|
||||
assert req("GET", "/beats?path=" + bad)[0] >= 400, bad
|
||||
assert req("GET", "/beats?path=audio/test/nope.wav")[0] == 404
|
||||
click("cachecheck.wav", 124, dur=40) # never analysed yet: first call is cold
|
||||
t0 = time.time()
|
||||
a1 = get_json("/beats?path=audio/test/cachecheck.wav")
|
||||
t1 = time.time()
|
||||
a2 = get_json("/beats?path=audio/test/cachecheck.wav")
|
||||
t2 = time.time()
|
||||
assert abs(a1["bpm"] - 124) <= 2, a1["bpm"]
|
||||
assert a1 == a2, "cached /beats must return the same grid"
|
||||
assert (t2 - t1) < (t1 - t0) / 2, \
|
||||
f"second /beats call not cached ({t2-t1:.3f}s vs {t1-t0:.3f}s)"
|
||||
n += 1
|
||||
else:
|
||||
print("(beats test skipped — ffmpeg not found)")
|
||||
|
||||
# --- the ffmpeg/rhubarb gates: a box without the binaries 503s cleanly ---
|
||||
port4 = PORT + 5
|
||||
p4 = subprocess.Popen(
|
||||
[str(PY), "-m", "uvicorn", "scenegod.server:app", "--port", str(port4)],
|
||||
cwd=ROOT, env={**env, "PATH": "/usr/bin:/bin"}, # no homebrew -> no ffmpeg/rhubarb
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
try:
|
||||
for _ in range(50):
|
||||
try:
|
||||
urllib.request.urlopen(f"http://127.0.0.1:{port4}/scenes")
|
||||
break
|
||||
except urllib.error.URLError:
|
||||
time.sleep(0.2)
|
||||
for path in ("/beats?path=audio/test/tone.wav", "/rhubarb?path=audio/test/tone.wav"):
|
||||
try:
|
||||
urllib.request.urlopen(f"http://127.0.0.1:{port4}{path}")
|
||||
raise AssertionError(f"{path} should 503 without ffmpeg/rhubarb")
|
||||
except urllib.error.HTTPError as e:
|
||||
assert e.code == 503, (path, e.code)
|
||||
n += 1
|
||||
finally:
|
||||
p4.terminate()
|
||||
p4.wait()
|
||||
|
||||
# --- rhubarb: 503 when the binary is absent (happy path needs rhubarb) ---
|
||||
if not shutil.which("rhubarb"):
|
||||
assert req("GET", "/rhubarb?path=audio/x.wav")[0] == 503
|
||||
|
||||
Loading…
Reference in New Issue
Block a user