Completes M18 on top of the WIP checkpoint (6a703c3). Three lanes (H/J/K) depend only on
what this freezes.
- synthetic.py: base testsrc2 saturation/brightness-muted so its magenta/cyan colour bars
don't swamp lane H's colour detector; the two moving markers stay the only detectable blobs.
- Frontend (step 6): friendTracks.js stub (never-throws, fx.js pattern) wired into main.js;
hidden #btn-friends + #friends-panel; state.tracks loaded when manifest.has_tracks.
- Tests (step 8): test_tracks_foundation.py (20) — DB helpers, /api/tracks routes, solve
degradation, manifest.has_tracks, frozen OOK protocol + encoders, track_truth shape, the
projection round-trip (rendered pixels back-project to 3D truth, worst err <1e-3 → lane H's
median<0.3 reachable by construction), and a real-ffmpeg render test proving marker A is the
dominant magenta blob after base-muting.
- CR-6: the two sanctioned M18 test-lock edits (test_api manifest key-set +has_tracks;
test_capsule bundle paths +api/tracks). Additive; flagged for integration3 ratification.
Suite: 14 pre-existing files sum to 189 (floor held) + 20 new = 209 passed, exit 0.
Frontend `npm run build` clean. Do NOT merge — coordinator reviews + merges (DIRECTIVES R6 #3).
Recovered by the coordinator session after the foundation3 agent's isolation worktree was
removed mid-run, leaving the work uncommitted; every file independently reviewed, full suite +
build re-run green, then committed. No work redone.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Incomplete: step 1 (fixture markers + track_truth.json), step 6 (frontend wiring),
step 8 (tests) not done; suite not yet run. Salvaged from interrupted prior session.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Coordinator-verified: 189 passed independently re-run in the lane worktree;
CR-5 (retire stub-era assertions, shape-lock director test) ratified per CR-1/CR-4 precedent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Conflicts resolved: (a) test_phase5_api.py — with lane G's capsule AND
lane E's features/direct all landed, the CLI stub loop is empty, so
test_cli_dispatches_stubs_gracefully is retired with a pointer comment to
the real dispatch coverage (test_capsule.py / test_audio_features.py /
test_director.py); (b) CHANGE_REQUESTS.md — lane G keeps the ratified
CR-4; lane E's entry renumbered to CR-5 and updated for the merged
reality. Merged tree: 189 passed (169 main + 21 lane E - 1 retired),
frontend build clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Coordinator-verified: ownership clean (3 owned modules + status only);
lane self-verified live in isolated servers (evidence in plan/status/lane-F.md).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
✨ toggle: fx.update(tGlobal) (already called each frame by main.js) tracks
the previous tGlobal and fires on marker crossings in play AND scrub;
backward jumps and >2s forward jumps just reset the tracker. pyro/confetti ->
~1s additive particle burst at the event's resolved anchor (mapped by the
backend's label convention, stage-centroid fallback); bass_drop -> point
cloud/splat scale pulse for ~a beat (60/tempo via GET /api/beats, else 0.5s),
exact scale restored. Scene access only via scene3d.addObject/removeObject.
Hot path: 6 preallocated burst slots (160 particles each), zero per-frame
allocation, whole body behind a breaker so a throw can never kill the app
loop (it disables FX instead).
Live-verified: pyro fired at t=10.00 in play, confetti on scrub crossing 17.0,
no retro-fire on backward jump, pulse 1.16x -> exact 1.0 restore, worst-case
frame body 0.101 ms CPU (~330x inside the 30fps budget), clean toggle-off
(plan/status/lane-F.md).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
📷 / P: pause if playing, scene3d.setHelpersVisible(false) + one scene3d
update so the per-video rigs honor the flag, then render ONE frame at 3840px
wide (aspect preserved, pixelRatio 1) by temporarily resizing the live
renderer, snapshot via canvas.toBlob in the same task, download
festival4d-photo.png, and restore pixelRatio/size/camera.aspect/helpers/play
state (try/finally). Splat: DropInViewer is a scene child and renders with
the same call — no branch; live splat verification deferred to coordinator.
Live-verified: PNG 3840x3093 (aspect 1.2416 preserved), zero helpers in the
captured image, full restore incl. play state (plan/status/lane-F.md).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collapsible #anchor-panel (toggle ⚓ #btn-anchors): every anchor gets a color
dot (recolor via inline color input), label (rename via prompt), one-click
jump-to (scene3d.focusOn), and delete — PATCH/DELETE /api/anchors/{id}, then
state.anchors update + emit("anchors-changed") so scene3d + overlays refresh.
＋ Tag drives the FROZEN M8 annotator without an event (annotate.js is
null-event safe by construction; the backend keeps event-less annotations as
independent anchors): prompt a name -> startAnnotateMode -> user drags a bbox
on any video -> the resolved anchor is PATCHed to the name + a palette color.
Esc / ＋ Tag again cancels. All mutating controls carry .write-ui (capsule
mode keeps the read-only list + jump). CSS injected from the module —
index.html is frozen.
Live-verified: create->rename->recolor->jump->delete roundtrip, labels in 3D
+ both video overlays, reload survival, capsule hiding (plan/status/lane-F.md).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
frontend/src/exportVideo.js fills the foundation2 stub: the ⏺ button
records the scene3d canvas captureStream(30) muxed with the live WebAudio
mix via the transport.captureAudioStream() hook (spatial included, silent
tap when no track — degrade, don't block), MediaRecorder vp9/opus webm
(vp8/webm fallbacks), spanning first -> last camPath keyframe. Playback is
driven through the existing #btn-path flow at forced rate 1 (file duration
must equal the path span), stops at the last keyframe (or early on a
second click / anything that halts the ride), downloads
festival4d-cut.webm, then restores ALL prior transport/UI state (playhead,
rate, play state, path state, followed camera) and releases the audio tap
+ canvas tracks. Button disabled under 2 keyframes by chaining main.js's
camPath.onChange (frozen files untouched).
Requires live-browser verification (focused tab — captureStream delivers
no frames hidden): see plan/status/lane-E.md. Frontend build clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Coordinator-verified: 169 passed independently re-run in the lane worktree;
CR-4 (retire capsule stub assertion) ratified per CR-1 precedent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
generate_path() fills the foundation2 stub with the deterministic v1 rules:
top-N events by confidence (ties -> earlier, then shot chronologically);
per event, the registered camera whose pose track is nearest the event
time; keyframes at t_event - lead_s (clamped >= 0) and t_event + duration;
nearest-beat snapping when beats.json exists (missing/corrupt file -> no
snapping); FOV = 2*atan(h/(2*fy)) like scene3d.snapTo. Every pose goes
through the frozen geometry.colmap_to_threejs and the camPath quaternion
is emitted in Three.js [x,y,z,w] order (pitfall #1 — locked by test).
Degradation: no events / no registered poses -> valid empty camPath with a
note; the response always round-trips through camPath.fromJSON.
frontend/src/director.js wires the pre-wired hidden 🎬 button: POST
/api/director, camPath.fromJSON(response), auto-play via the existing
#btn-path flow; empty path or fetch failure surfaces on the button and
recovers — never a crash.
Acceptance (test_director.py, 14 tests): structure-locked against the
frozen camPath shape (exact key sets, unit quats, monotone times);
top-3 grid == [1,4,8,11,14] incl. shared-keyframe dedupe; conversion
matches colmap_to_threejs + mat_to_quat exactly; snapping puts every key
on the beat grid; unregistered poses excluded; API route + CLI dispatch
real output. Suite 148 passed; frontend build clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
run_features() fills the foundation2 stub: librosa beat/tempo tracking +
onset-strength detection at a 10 ms hop (librosa's default 32 ms grid at
16 kHz quantizes past the +/-50 ms acceptance bound), tempo from the median
inter-beat interval, times mapped to t_global via the frozen timebase
helpers. Degradation per house style: missing videos/WAV -> summary with
note, nothing written; silent/short/beatless audio -> valid-empty
beats.json, never a crash.
Acceptance (test_audio_features.py, 7 tests): synthetic 120 BPM grid
recovered — tempo 120.0 (+/-2 required), worst beat 30 ms off grid
(+/-50 required); the three ground-truth bangs appear as strong onsets;
silence/short/missing all degrade cleanly; CLI dispatch end-to-end.
CR-4: minimal stub-era assertion updates in test_phase5_api.py (CLI stub
loop shrinks to capsule) per the approved CR-1 precedent — see
plan/CHANGE_REQUESTS.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
build_capsule copies frontend/dist (clear error if missing), copies referenced media,
bakes every read-only API response extension-less at the live API's relative paths
(manifest with capsule:true + has_capture:false, events, anchors, poses, beats,
pointcloud, splat, audio via the same lazy extract_audio_hq cache, plus M16 saved
paths — listing baked as api/paths/index.html so per-id files can share the prefix),
injects window.__F4D_API_BASE__="" ahead of the app script, and ships a stdlib
serve.py (206 Range slicing incl. suffix ranges + 416, JSON content types for the
extension-less api/* files). Optional artifacts degrade by omission.
test_capsule.py: bundle structure, baked-JSON deep-equal vs the live route functions,
injection position, degrade/overwrite-guard/no-dist errors, CLI dispatch (CR-4), and
serve.py exercised over real HTTP in a subprocess (200/206/416/404, content types,
mp4+m4a magic bytes). Suite: 169 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Roundtrip fidelity (float precision, 500-keyframe lists, extra keys ride along),
multiple coexisting paths + id-ordered listing, duplicate/unicode/whitespace names,
ISO created_at, 404s, and an 18-case 422 matrix over the frozen camPath contract.
Cleans up after itself so test_phase5_api still sees an empty table.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Unhides #btn-path-save / #path-load-select / #btn-path-del; save prompts a name and
POSTs camPath.toJSON(), load fetches + camPath.fromJSON, delete confirms; dropdown
refreshes after every mutation. Capsule mode: write buttons hide via write-ui, the
load dropdown keeps working against baked api/paths (hides itself if unavailable).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
uv.lock drifted from pyproject when live capture added python-multipart;
uv run resynced it. launch.json gains festifun-web-dist (uv + http.server
over frontend/dist) for sandboxed sessions where node/npm can't start.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Master clock: the selected soundtrack (new GET /api/audio/{id} —
listening-quality stereo AAC, lazily ffmpeg-extracted and cached in
data/work/audio_hq/) plays through WebAudio, and t_global derives from
AudioContext.currentTime. Every <video> is muted picture-only, so hard
seeks and playbackRate corrections are inaudible (the unmuted video
used to warble on every nudge). Graceful fallback to the old
performance.now() clock + <video> audio when a track can't load, with
an in-place upgrade if the decode lands mid-play. Drift is compensated
in the track's playbackRate; _resync() now recomputes from the live
clock since the soundtrack keeps playing while rAF is throttled.
Spatial audio (🎧 3D audio, shown when poses exist): every enabled
camera's track plays through an HRTF PannerNode at its live
reconstructed pose, gains normalized 1/sqrt(N); the listener follows
the viewer camera each frame (free roam, follow-cam, and M9 paths).
Tracks that finish decoding mid-play join the mix in place; toggling,
seeking, rate and enable changes rebuild the graph at the current
t_global with no timeline jump.
The window.__f4d debug hook is now also available in production builds
via localStorage f4dDebug=1 (headless panes throttle rAF; the pump is
the only way to drive the loop there).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pairwise_offsets now computes each signal's rFFT once at a shared padded
length (2*max_len) and reuses it for every pair: O(K) forward FFTs +
cheap per-pair spectrum products instead of O(K^2) full FFTs. Each
pair's lag search stays capped at +/-(la+lb)/2 — the same window the
per-pair transform used — and gcc_phat() itself is unchanged for
callers (drift windows, tests). Measured 1.8x on 6 cams x 10 min;
the advantage grows with camera count.
New test locks the cached path against direct gcc_phat, including
unequal-length signals where the padded sizes genuinely differ.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Records rather than streams: each device records a clip WITH AUDIO and uploads
it; the normal offline pipeline takes over. Fits the design almost for free —
MediaRecorder's webm/mp4 are already ingestible, and audio-based sync means
devices need no clock sync and needn't start together.
- capture.py: POST /api/capture/upload (streamed, sanitized name, size cap,
partial cleanup), heartbeat/devices registry, GET /capture page.
- static/capture.html: standalone page (no build step) — device picker, live
preview, record->upload, 1fps snapshot heartbeat, 'who's shooting what'
monitor, and secure-context detection that tells you to use
(getUserMedia needs HTTPS — the #1 gotcha for phones).
- OPT-IN via FESTIVAL4D_CAPTURE=1; manifest reports has_capture and the viewer
only then shows a Capture link. Public deploy leaves it off (open upload
endpoint would be unsafe) — noted in DEPLOY.md.
- Adds python-multipart dep. Suite 103 -> 119.
Also fixes test_manifest_shape, which was already RED on main: f12b6e0 added
has_splat to the manifest without updating the exact-set lock. Updated to the
true set (now also has_capture); the lock is what caught both drifts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- GET /api/splat serves data/work/splat.ply; manifest reports has_splat
- scene3d: lazy-load @mkkellogg/gaussian-splats-3d DropInViewer when a splat
exists (rigs/anchors/paths overlay as before); point-cloud fallback intact
- scripts/splat_via_modelbeast.sh: train the splat on the MODELBEAST fleet
(colmap_poses -> brush_train via the mb queue) and drop it in place
- docs/modelbeast-crossover.md: full crossover map (fleet compute, volumetric
capture w/ CorridorKey + audio sync, per-moment splats)
Verified in-browser: 32MB room splat rendered via /api/splat with overlays live.
Frontend: API_BASE env-driven (VITE_API_BASE, default localhost:8000 so dev is
unchanged); vite.config.js base from FESTIVAL4D_BASE. Prod build
'FESTIVAL4D_BASE=/festifun/ VITE_API_BASE=/festifun' serves same-origin under
the prefix. Validated end-to-end via a local proxy mirroring the nginx config:
app boots, media Range 206, 3D + overlays + timeline render under /festifun.
deploy/: nginx location blocks (prefix strip + Range), systemd unit
(uvicorn 127.0.0.1:8000, no keys on the public box), DEPLOY.md. Labels baked
into the shipped DB locally so detect can't spend credits from an anonymous
public endpoint. Backend suite 103 passed; default build unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fixes the sole Round-3 wart: annotating one object from two views now yields
ONE anchor (the single-view fallback is superseded in place by the
triangulation, same id) instead of a sibling pair. Adds DELETE /api/anchors/{id}
(unlinks annotations, 404 on unknown) + a deletable anchor list in the
correction panel. db gains get/update/delete_anchor + get_annotations(event_id).
CR-3 filed (additive). Suite 101 -> 103; frontend build clean; delete UI
verified live (state + DOM + 3D scene stay consistent).
Also: docs/ideas.md (future extensions), plan/ISSUES.md (seeded with the
gemini-2.5-flash retirement, FIXED), plan/status/phase4.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Google retired gemini-2.5-flash for new API keys (404). Verified
gemini-3.1-flash-lite live: inline video + structured output, 3/3 synthetic
candidates classified. FESTIVAL4D_GEMINI_MODEL overrides. Round 3 addendum
documents the local .env (gitignored) with Gemini + OpenRouter creds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Full frontend built against the synthetic API (no dependence on lanes A/B/D):
M4 — synchronized playback: master clock from performance.now() (never a <video>);
per-video correction via requestVideoFrameCallback (exact mediaTime) with a
currentTime+half-frame fallback — hard-seek >150ms / nudge playbackRate +/-5% 20-150ms /
lock <20ms; out-of-range videos pause+dim; single audio source; dev sync-error overlay.
Timebase mirrors config.py (lib/timebase.js). Verified: seek is exact to 0ms; continuous
inter-video desync 13ms mean / 46ms max.
M5 — 3D viewer: PLY point cloud, per-video camera paths + current-pose frusta, OrbitControls,
snap-to-camera (intrinsics->PerspectiveCamera fov) + free roam. All COLMAP->Three.js via the
frozen lib/pose.js; pose interpolation in lib/poseTrack.js (slerp+lerp of contract inputs).
M6 — anchor overlays: letterbox-correct per-video canvas; anchors projected via pose.js;
behind-camera cull. Projection agrees with an independent pinhole to 1.45e-13 px.
Timeline — colored event markers + legend, hover tooltip, click-to-jump; draggable scrubber.
Phase-3 seams (annotate.js M8, camPath.js M9) left as stubs. No frozen files edited; no new
deps; no change requests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fills events_ai.py against the frozen M7 contract:
- detect_candidates: RMS x spectral-flux, local-max over a 10s neighborhood,
>=P90, with an AC-RMS (std) silence/DC floor so flat audio yields no events.
Matches the fixture ground-truth pulses [3,10,17] within ~0.02s (<0.5s).
- GeminiClassifier (native video), ClaudeClassifier (claude-opus-4-8,
messages.parse, adaptive thinking, no temperature), LocalClassifier
(OpenAI-compatible, 6 frames, 1 retry). SDKs imported lazily.
- get_classifier: FESTIVAL4D_CLASSIFIER selection; unconfigured -> None
(candidates-only, logged).
- prepare_inputs: ffmpeg 3s clip (<=720p, audio kept) + 6 JPEGs (<=768px).
- run_events: reference-audio detect -> classify (insert-once in final state,
since db.py has no update-event helper); per-candidate exception isolation;
whole-track replaces machine events (keeps user edits), windowed is additive.
test_events_ai.py: candidate-vs-ground-truth (no ffmpeg/API), stub-classifier
orchestration, exception isolation, degradation, provider selection, silence.
Also updates the now-obsolete api stub test (test_detect_events_endpoint) to the
real M3 {result, events} shape; frozen api.py unchanged (see CR-1). pytest: 90 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fills the foundation stubs in ingest.py and audio_sync.py.
- ingest: ffprobe probe + mono 16 kHz WAV extraction per video (ffmpeg);
idempotent run_ingest reuses existing rows so synthetic -> ingest -> sync
works without unique-constraint collisions.
- sync: PHAT-whitened GCC-PHAT pairwise offsets with sub-sample parabolic
peak refinement, scored by peak-to-second-peak ratio; confidence-weighted
global least-squares solve with cycle-consistency rejection (>50 ms);
windowed drift estimation (ppm) with a 5 ppm deadband; disconnected
sync-graph components -> offset_ms=None (spec pitfall #5).
- Persists offset_ms/drift_ppm/sync_confidence via db.update_video_sync and
exports data/work/sync.json.
Verified on the synthetic fixture: synthetic -> ingest -> sync recovers the
ground-truth offsets (0 / +1370 / -842 ms) to <0.001 ms and drift 0 ppm,
well inside the spec M1 tolerances (+/-10 ms, +/-3 ppm). pytest: 43 passed
(24 foundation + 19 new in test_ingest.py and test_audio_sync.py).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
