uv.lock drifted from pyproject when live capture added python-multipart;
uv run resynced it. launch.json gains festifun-web-dist (uv + http.server
over frontend/dist) for sandboxed sessions where node/npm can't start.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Master clock: the selected soundtrack (new GET /api/audio/{id} —
listening-quality stereo AAC, lazily ffmpeg-extracted and cached in
data/work/audio_hq/) plays through WebAudio, and t_global derives from
AudioContext.currentTime. Every <video> is muted picture-only, so hard
seeks and playbackRate corrections are inaudible (the unmuted video
used to warble on every nudge). Graceful fallback to the old
performance.now() clock + <video> audio when a track can't load, with
an in-place upgrade if the decode lands mid-play. Drift is compensated
in the track's playbackRate; _resync() now recomputes from the live
clock since the soundtrack keeps playing while rAF is throttled.
Spatial audio (🎧 3D audio, shown when poses exist): every enabled
camera's track plays through an HRTF PannerNode at its live
reconstructed pose, gains normalized 1/sqrt(N); the listener follows
the viewer camera each frame (free roam, follow-cam, and M9 paths).
Tracks that finish decoding mid-play join the mix in place; toggling,
seeking, rate and enable changes rebuild the graph at the current
t_global with no timeline jump.
The window.__f4d debug hook is now also available in production builds
via localStorage f4dDebug=1 (headless panes throttle rAF; the pump is
the only way to drive the loop there).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pairwise_offsets now computes each signal's rFFT once at a shared padded
length (2*max_len) and reuses it for every pair: O(K) forward FFTs +
cheap per-pair spectrum products instead of O(K^2) full FFTs. Each
pair's lag search stays capped at +/-(la+lb)/2 — the same window the
per-pair transform used — and gcc_phat() itself is unchanged for
callers (drift windows, tests). Measured 1.8x on 6 cams x 10 min;
the advantage grows with camera count.
New test locks the cached path against direct gcc_phat, including
unequal-length signals where the padded sizes genuinely differ.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Records rather than streams: each device records a clip WITH AUDIO and uploads
it; the normal offline pipeline takes over. Fits the design almost for free —
MediaRecorder's webm/mp4 are already ingestible, and audio-based sync means
devices need no clock sync and needn't start together.
- capture.py: POST /api/capture/upload (streamed, sanitized name, size cap,
partial cleanup), heartbeat/devices registry, GET /capture page.
- static/capture.html: standalone page (no build step) — device picker, live
preview, record->upload, 1fps snapshot heartbeat, 'who's shooting what'
monitor, and secure-context detection that tells you to use
(getUserMedia needs HTTPS — the #1 gotcha for phones).
- OPT-IN via FESTIVAL4D_CAPTURE=1; manifest reports has_capture and the viewer
only then shows a Capture link. Public deploy leaves it off (open upload
endpoint would be unsafe) — noted in DEPLOY.md.
- Adds python-multipart dep. Suite 103 -> 119.
Also fixes test_manifest_shape, which was already RED on main: f12b6e0 added
has_splat to the manifest without updating the exact-set lock. Updated to the
true set (now also has_capture); the lock is what caught both drifts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- GET /api/splat serves data/work/splat.ply; manifest reports has_splat
- scene3d: lazy-load @mkkellogg/gaussian-splats-3d DropInViewer when a splat
exists (rigs/anchors/paths overlay as before); point-cloud fallback intact
- scripts/splat_via_modelbeast.sh: train the splat on the MODELBEAST fleet
(colmap_poses -> brush_train via the mb queue) and drop it in place
- docs/modelbeast-crossover.md: full crossover map (fleet compute, volumetric
capture w/ CorridorKey + audio sync, per-moment splats)
Verified in-browser: 32MB room splat rendered via /api/splat with overlays live.
Frontend: API_BASE env-driven (VITE_API_BASE, default localhost:8000 so dev is
unchanged); vite.config.js base from FESTIVAL4D_BASE. Prod build
'FESTIVAL4D_BASE=/festifun/ VITE_API_BASE=/festifun' serves same-origin under
the prefix. Validated end-to-end via a local proxy mirroring the nginx config:
app boots, media Range 206, 3D + overlays + timeline render under /festifun.
deploy/: nginx location blocks (prefix strip + Range), systemd unit
(uvicorn 127.0.0.1:8000, no keys on the public box), DEPLOY.md. Labels baked
into the shipped DB locally so detect can't spend credits from an anonymous
public endpoint. Backend suite 103 passed; default build unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fixes the sole Round-3 wart: annotating one object from two views now yields
ONE anchor (the single-view fallback is superseded in place by the
triangulation, same id) instead of a sibling pair. Adds DELETE /api/anchors/{id}
(unlinks annotations, 404 on unknown) + a deletable anchor list in the
correction panel. db gains get/update/delete_anchor + get_annotations(event_id).
CR-3 filed (additive). Suite 101 -> 103; frontend build clean; delete UI
verified live (state + DOM + 3D scene stay consistent).
Also: docs/ideas.md (future extensions), plan/ISSUES.md (seeded with the
gemini-2.5-flash retirement, FIXED), plan/status/phase4.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Google retired gemini-2.5-flash for new API keys (404). Verified
gemini-3.1-flash-lite live: inline video + structured output, 3/3 synthetic
candidates classified. FESTIVAL4D_GEMINI_MODEL overrides. Round 3 addendum
documents the local .env (gitignored) with Gemini + OpenRouter creds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Full frontend built against the synthetic API (no dependence on lanes A/B/D):
M4 — synchronized playback: master clock from performance.now() (never a <video>);
per-video correction via requestVideoFrameCallback (exact mediaTime) with a
currentTime+half-frame fallback — hard-seek >150ms / nudge playbackRate +/-5% 20-150ms /
lock <20ms; out-of-range videos pause+dim; single audio source; dev sync-error overlay.
Timebase mirrors config.py (lib/timebase.js). Verified: seek is exact to 0ms; continuous
inter-video desync 13ms mean / 46ms max.
M5 — 3D viewer: PLY point cloud, per-video camera paths + current-pose frusta, OrbitControls,
snap-to-camera (intrinsics->PerspectiveCamera fov) + free roam. All COLMAP->Three.js via the
frozen lib/pose.js; pose interpolation in lib/poseTrack.js (slerp+lerp of contract inputs).
M6 — anchor overlays: letterbox-correct per-video canvas; anchors projected via pose.js;
behind-camera cull. Projection agrees with an independent pinhole to 1.45e-13 px.
Timeline — colored event markers + legend, hover tooltip, click-to-jump; draggable scrubber.
Phase-3 seams (annotate.js M8, camPath.js M9) left as stubs. No frozen files edited; no new
deps; no change requests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fills events_ai.py against the frozen M7 contract:
- detect_candidates: RMS x spectral-flux, local-max over a 10s neighborhood,
>=P90, with an AC-RMS (std) silence/DC floor so flat audio yields no events.
Matches the fixture ground-truth pulses [3,10,17] within ~0.02s (<0.5s).
- GeminiClassifier (native video), ClaudeClassifier (claude-opus-4-8,
messages.parse, adaptive thinking, no temperature), LocalClassifier
(OpenAI-compatible, 6 frames, 1 retry). SDKs imported lazily.
- get_classifier: FESTIVAL4D_CLASSIFIER selection; unconfigured -> None
(candidates-only, logged).
- prepare_inputs: ffmpeg 3s clip (<=720p, audio kept) + 6 JPEGs (<=768px).
- run_events: reference-audio detect -> classify (insert-once in final state,
since db.py has no update-event helper); per-candidate exception isolation;
whole-track replaces machine events (keeps user edits), windowed is additive.
test_events_ai.py: candidate-vs-ground-truth (no ffmpeg/API), stub-classifier
orchestration, exception isolation, degradation, provider selection, silence.
Also updates the now-obsolete api stub test (test_detect_events_endpoint) to the
real M3 {result, events} shape; frozen api.py unchanged (see CR-1). pytest: 90 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fills the foundation stubs in ingest.py and audio_sync.py.
- ingest: ffprobe probe + mono 16 kHz WAV extraction per video (ffmpeg);
idempotent run_ingest reuses existing rows so synthetic -> ingest -> sync
works without unique-constraint collisions.
- sync: PHAT-whitened GCC-PHAT pairwise offsets with sub-sample parabolic
peak refinement, scored by peak-to-second-peak ratio; confidence-weighted
global least-squares solve with cycle-consistency rejection (>50 ms);
windowed drift estimation (ppm) with a 5 ppm deadband; disconnected
sync-graph components -> offset_ms=None (spec pitfall #5).
- Persists offset_ms/drift_ppm/sync_confidence via db.update_video_sync and
exports data/work/sync.json.
Verified on the synthetic fixture: synthetic -> ingest -> sync recovers the
ground-truth offsets (0 / +1370 / -842 ms) to <0.001 ms and drift 0 ppm,
well inside the spec M1 tolerances (+/-10 ms, +/-3 ppm). pytest: 43 passed
(24 foundation + 19 new in test_ingest.py and test_audio_sync.py).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
