Completes M18 on top of the WIP checkpoint (6a703c3). Three lanes (H/J/K) depend only on
what this freezes.
- synthetic.py: base testsrc2 saturation/brightness-muted so its magenta/cyan colour bars
don't swamp lane H's colour detector; the two moving markers stay the only detectable blobs.
- Frontend (step 6): friendTracks.js stub (never-throws, fx.js pattern) wired into main.js;
hidden #btn-friends + #friends-panel; state.tracks loaded when manifest.has_tracks.
- Tests (step 8): test_tracks_foundation.py (20) — DB helpers, /api/tracks routes, solve
degradation, manifest.has_tracks, frozen OOK protocol + encoders, track_truth shape, the
projection round-trip (rendered pixels back-project to 3D truth, worst err <1e-3 → lane H's
median<0.3 reachable by construction), and a real-ffmpeg render test proving marker A is the
dominant magenta blob after base-muting.
- CR-6: the two sanctioned M18 test-lock edits (test_api manifest key-set +has_tracks;
test_capsule bundle paths +api/tracks). Additive; flagged for integration3 ratification.
Suite: 14 pre-existing files sum to 189 (floor held) + 20 new = 209 passed, exit 0.
Frontend `npm run build` clean. Do NOT merge — coordinator reviews + merges (DIRECTIVES R6 #3).
Recovered by the coordinator session after the foundation3 agent's isolation worktree was
removed mid-run, leaving the work uncommitted; every file independently reviewed, full suite +
build re-run green, then committed. No work redone.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Incomplete: step 1 (fixture markers + track_truth.json), step 6 (frontend wiring),
step 8 (tests) not done; suite not yet run. Salvaged from interrupted prior session.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Conflicts resolved: (a) test_phase5_api.py — with lane G's capsule AND
lane E's features/direct all landed, the CLI stub loop is empty, so
test_cli_dispatches_stubs_gracefully is retired with a pointer comment to
the real dispatch coverage (test_capsule.py / test_audio_features.py /
test_director.py); (b) CHANGE_REQUESTS.md — lane G keeps the ratified
CR-4; lane E's entry renumbered to CR-5 and updated for the merged
reality. Merged tree: 189 passed (169 main + 21 lane E - 1 retired),
frontend build clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
generate_path() fills the foundation2 stub with the deterministic v1 rules:
top-N events by confidence (ties -> earlier, then shot chronologically);
per event, the registered camera whose pose track is nearest the event
time; keyframes at t_event - lead_s (clamped >= 0) and t_event + duration;
nearest-beat snapping when beats.json exists (missing/corrupt file -> no
snapping); FOV = 2*atan(h/(2*fy)) like scene3d.snapTo. Every pose goes
through the frozen geometry.colmap_to_threejs and the camPath quaternion
is emitted in Three.js [x,y,z,w] order (pitfall #1 — locked by test).
Degradation: no events / no registered poses -> valid empty camPath with a
note; the response always round-trips through camPath.fromJSON.
frontend/src/director.js wires the pre-wired hidden 🎬 button: POST
/api/director, camPath.fromJSON(response), auto-play via the existing
#btn-path flow; empty path or fetch failure surfaces on the button and
recovers — never a crash.
Acceptance (test_director.py, 14 tests): structure-locked against the
frozen camPath shape (exact key sets, unit quats, monotone times);
top-3 grid == [1,4,8,11,14] incl. shared-keyframe dedupe; conversion
matches colmap_to_threejs + mat_to_quat exactly; snapping puts every key
on the beat grid; unregistered poses excluded; API route + CLI dispatch
real output. Suite 148 passed; frontend build clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
run_features() fills the foundation2 stub: librosa beat/tempo tracking +
onset-strength detection at a 10 ms hop (librosa's default 32 ms grid at
16 kHz quantizes past the +/-50 ms acceptance bound), tempo from the median
inter-beat interval, times mapped to t_global via the frozen timebase
helpers. Degradation per house style: missing videos/WAV -> summary with
note, nothing written; silent/short/beatless audio -> valid-empty
beats.json, never a crash.
Acceptance (test_audio_features.py, 7 tests): synthetic 120 BPM grid
recovered — tempo 120.0 (+/-2 required), worst beat 30 ms off grid
(+/-50 required); the three ground-truth bangs appear as strong onsets;
silence/short/missing all degrade cleanly; CLI dispatch end-to-end.
CR-4: minimal stub-era assertion updates in test_phase5_api.py (CLI stub
loop shrinks to capsule) per the approved CR-1 precedent — see
plan/CHANGE_REQUESTS.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
build_capsule copies frontend/dist (clear error if missing), copies referenced media,
bakes every read-only API response extension-less at the live API's relative paths
(manifest with capsule:true + has_capture:false, events, anchors, poses, beats,
pointcloud, splat, audio via the same lazy extract_audio_hq cache, plus M16 saved
paths — listing baked as api/paths/index.html so per-id files can share the prefix),
injects window.__F4D_API_BASE__="" ahead of the app script, and ships a stdlib
serve.py (206 Range slicing incl. suffix ranges + 416, JSON content types for the
extension-less api/* files). Optional artifacts degrade by omission.
test_capsule.py: bundle structure, baked-JSON deep-equal vs the live route functions,
injection position, degrade/overwrite-guard/no-dist errors, CLI dispatch (CR-4), and
serve.py exercised over real HTTP in a subprocess (200/206/416/404, content types,
mp4+m4a magic bytes). Suite: 169 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Master clock: the selected soundtrack (new GET /api/audio/{id} —
listening-quality stereo AAC, lazily ffmpeg-extracted and cached in
data/work/audio_hq/) plays through WebAudio, and t_global derives from
AudioContext.currentTime. Every <video> is muted picture-only, so hard
seeks and playbackRate corrections are inaudible (the unmuted video
used to warble on every nudge). Graceful fallback to the old
performance.now() clock + <video> audio when a track can't load, with
an in-place upgrade if the decode lands mid-play. Drift is compensated
in the track's playbackRate; _resync() now recomputes from the live
clock since the soundtrack keeps playing while rAF is throttled.
Spatial audio (🎧 3D audio, shown when poses exist): every enabled
camera's track plays through an HRTF PannerNode at its live
reconstructed pose, gains normalized 1/sqrt(N); the listener follows
the viewer camera each frame (free roam, follow-cam, and M9 paths).
Tracks that finish decoding mid-play join the mix in place; toggling,
seeking, rate and enable changes rebuild the graph at the current
t_global with no timeline jump.
The window.__f4d debug hook is now also available in production builds
via localStorage f4dDebug=1 (headless panes throttle rAF; the pump is
the only way to drive the loop there).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pairwise_offsets now computes each signal's rFFT once at a shared padded
length (2*max_len) and reuses it for every pair: O(K) forward FFTs +
cheap per-pair spectrum products instead of O(K^2) full FFTs. Each
pair's lag search stays capped at +/-(la+lb)/2 — the same window the
per-pair transform used — and gcc_phat() itself is unchanged for
callers (drift windows, tests). Measured 1.8x on 6 cams x 10 min;
the advantage grows with camera count.
New test locks the cached path against direct gcc_phat, including
unequal-length signals where the padded sizes genuinely differ.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Records rather than streams: each device records a clip WITH AUDIO and uploads
it; the normal offline pipeline takes over. Fits the design almost for free —
MediaRecorder's webm/mp4 are already ingestible, and audio-based sync means
devices need no clock sync and needn't start together.
- capture.py: POST /api/capture/upload (streamed, sanitized name, size cap,
partial cleanup), heartbeat/devices registry, GET /capture page.
- static/capture.html: standalone page (no build step) — device picker, live
preview, record->upload, 1fps snapshot heartbeat, 'who's shooting what'
monitor, and secure-context detection that tells you to use
(getUserMedia needs HTTPS — the #1 gotcha for phones).
- OPT-IN via FESTIVAL4D_CAPTURE=1; manifest reports has_capture and the viewer
only then shows a Capture link. Public deploy leaves it off (open upload
endpoint would be unsafe) — noted in DEPLOY.md.
- Adds python-multipart dep. Suite 103 -> 119.
Also fixes test_manifest_shape, which was already RED on main: f12b6e0 added
has_splat to the manifest without updating the exact-set lock. Updated to the
true set (now also has_capture); the lock is what caught both drifts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- GET /api/splat serves data/work/splat.ply; manifest reports has_splat
- scene3d: lazy-load @mkkellogg/gaussian-splats-3d DropInViewer when a splat
exists (rigs/anchors/paths overlay as before); point-cloud fallback intact
- scripts/splat_via_modelbeast.sh: train the splat on the MODELBEAST fleet
(colmap_poses -> brush_train via the mb queue) and drop it in place
- docs/modelbeast-crossover.md: full crossover map (fleet compute, volumetric
capture w/ CorridorKey + audio sync, per-moment splats)
Verified in-browser: 32MB room splat rendered via /api/splat with overlays live.
Fixes the sole Round-3 wart: annotating one object from two views now yields
ONE anchor (the single-view fallback is superseded in place by the
triangulation, same id) instead of a sibling pair. Adds DELETE /api/anchors/{id}
(unlinks annotations, 404 on unknown) + a deletable anchor list in the
correction panel. db gains get/update/delete_anchor + get_annotations(event_id).
CR-3 filed (additive). Suite 101 -> 103; frontend build clean; delete UI
verified live (state + DOM + 3D scene stay consistent).
Also: docs/ideas.md (future extensions), plan/ISSUES.md (seeded with the
gemini-2.5-flash retirement, FIXED), plan/status/phase4.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Google retired gemini-2.5-flash for new API keys (404). Verified
gemini-3.1-flash-lite live: inline video + structured output, 3/3 synthetic
candidates classified. FESTIVAL4D_GEMINI_MODEL overrides. Round 3 addendum
documents the local .env (gitignored) with Gemini + OpenRouter creds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fills events_ai.py against the frozen M7 contract:
- detect_candidates: RMS x spectral-flux, local-max over a 10s neighborhood,
>=P90, with an AC-RMS (std) silence/DC floor so flat audio yields no events.
Matches the fixture ground-truth pulses [3,10,17] within ~0.02s (<0.5s).
- GeminiClassifier (native video), ClaudeClassifier (claude-opus-4-8,
messages.parse, adaptive thinking, no temperature), LocalClassifier
(OpenAI-compatible, 6 frames, 1 retry). SDKs imported lazily.
- get_classifier: FESTIVAL4D_CLASSIFIER selection; unconfigured -> None
(candidates-only, logged).
- prepare_inputs: ffmpeg 3s clip (<=720p, audio kept) + 6 JPEGs (<=768px).
- run_events: reference-audio detect -> classify (insert-once in final state,
since db.py has no update-event helper); per-candidate exception isolation;
whole-track replaces machine events (keeps user edits), windowed is additive.
test_events_ai.py: candidate-vs-ground-truth (no ffmpeg/API), stub-classifier
orchestration, exception isolation, degradation, provider selection, silence.
Also updates the now-obsolete api stub test (test_detect_events_endpoint) to the
real M3 {result, events} shape; frozen api.py unchanged (see CR-1). pytest: 90 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fills the foundation stubs in ingest.py and audio_sync.py.
- ingest: ffprobe probe + mono 16 kHz WAV extraction per video (ffmpeg);
idempotent run_ingest reuses existing rows so synthetic -> ingest -> sync
works without unique-constraint collisions.
- sync: PHAT-whitened GCC-PHAT pairwise offsets with sub-sample parabolic
peak refinement, scored by peak-to-second-peak ratio; confidence-weighted
global least-squares solve with cycle-consistency rejection (>50 ms);
windowed drift estimation (ppm) with a 5 ppm deadband; disconnected
sync-graph components -> offset_ms=None (spec pitfall #5).
- Persists offset_ms/drift_ppm/sync_confidence via db.update_video_sync and
exports data/work/sync.json.
Verified on the synthetic fixture: synthetic -> ingest -> sync recovers the
ground-truth offsets (0 / +1370 / -842 ms) to <0.001 ms and drift 0 ppm,
well inside the spec M1 tolerances (+/-10 ms, +/-3 ppm). pytest: 43 passed
(24 foundation + 19 new in test_ingest.py and test_audio_sync.py).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
