festifun/plan/20-phase5.md

Phase 5 — Friends, Director's Cut & Capsules (M10–M17)

Canonical spec for the phase-5 milestones. M0–M9 live in ../OPUS_BUILD_INSTRUCTIONS.md and every contract frozen there still applies (timebase, DB schema, pose convention, classifier contract, synthetic fixture). Standing protocol (status files, directives loop, evidence discipline, worktrees) is unchanged from README.md; the newest round of DIRECTIVES.md always wins.

Context — what shipped since v0.1.0 (read before planning anything)

• Splat rendering: GET /api/splat + viewer DropInViewer with point-cloud fallback (docs/modelbeast-crossover.md).
• Live capture: FESTIVAL4D_CAPTURE=1 mounts /capture (README).
• WebAudio master clock: t_global derives from AudioContext.currentTime; GET /api/audio/{id} serves a lazily-extracted stereo AAC per video; all <video> elements are muted picture-only, with graceful fallback to the old performance.now() clock (frontend/src/transport.js header comment is the reference).
• Spatial audio (🎧): per-camera HRTF PannerNodes at live poses; listener follows the viewer camera (transport.updateSpatial, fed by scene3d._updateAudio).
• Sync solver: pairwise GCC-PHAT now reuses one rFFT per signal (audio_sync.pairwise_offsets).
• Suite floor is 121 passed. It only goes up.

Phase map

Phase 5a (serial, ONE agent)      Phase 5b (parallel, one agent per lane)     Phase 5c (serial, ONE agent)
┌───────────────────────┐         ┌ lane/e-director ── M10 M11 M12 ─┐         ┌───────────────────────────┐
│ foundation2           │         ├ lane/f-social ──── M13 M14 M15 ─┤         │ integration2              │
│ contracts + stubs +   │──merge──┤                                 ├──merge──│ full-system pass +        │
│ UI wiring points      │         └ lane/g-capsule ─── M16 M17 ─────┘         │ real-footage field test   │
│ branch: foundation2   │         (independent — merge in any order)          │ branch: integration2      │
└───────────────────────┘         └───────────────────────────────────        └───────────────────────────┘

Lanes depend only on foundation2, never on each other. Everything is developed and verified against the synthetic fixture first; real footage is integration2's job.

Milestones

M10 — Audio features: beats & onsets (lane E, backend)

backend/festival4d/audio_features.py, entrypoint run_features(), CLI python -m festival4d features. Load the reference video's ingest WAV (16 kHz mono; the reference timeline is t_global). Beat/tempo tracking + onset-strength detection (librosa is already a dependency). Write data/work/beats.json (shape frozen below); GET /api/beats serves it, 404 when absent. Quiet/short/beatless audio must produce a valid, empty result — never a crash. Accept: synthetic master audio's known pulse grid recovered (beats within ±50 ms, tempo ±2 BPM); silence → empty-but-valid JSON; tests in backend/tests/test_audio_features.py.

M11 — Auto-director (lane E, backend + thin UI)

backend/festival4d/director.py, entrypoint generate_path(top_n=8, lead_s=2.0), CLI python -m festival4d direct, POST /api/director (stubbed by foundation2) returning a camPath-compatible JSON (contract below). Deterministic v1 rules — no ML: pick the top-N events by confidence (ties → earlier); for each, choose the registered camera with a pose nearest the event time (via db + geometry.colmap_to_threejs); keyframes at t_event − lead_s and t_event + duration; when beats.json exists, snap keyframe times to the nearest beat; FOV from that camera's intrinsics (same formula as scene3d.snapTo). Frontend: fill the stub frontend/src/director.js — the hidden 🎬 Auto-path button fetches /api/director and loads it via camPath.fromJSON (do not edit camPath.js). Accept: output round-trips through camPath.fromJSON (structure-locked test against the frozen shape); covers the top-N events; keyframe times land on beats when beats exist; live browser evidence of the path playing.

M12 — Cinematic export (lane E, frontend)

Fill the stub frontend/src/exportVideo.js. The hidden ⏺ Export button: seek to the first keyframe, play the path, capture scene3d canvas via captureStream(30) + the soundtrack via the transport.captureAudioStream() hook (foundation2 provides it — taps the WebAudio graph post-gain, spatial mix included), mux with MediaRecorder (vp9/opus webm), stop at the last keyframe, download festival4d-cut.webm. Restore all prior UI/transport state afterwards. Accept: exported file duration within ±5% of the path span, contains an audio track, plays in Chrome — browser evidence (file size, duration probe via ffprobe) in the status file.

M13 — Anchor manager & friend tags (lane F, frontend)

Fill the stub frontend/src/anchorPanel.js: an always-available collapsible panel listing every anchor (color dot, label, jump-to, delete, rename/recolor via the new PATCH /api/anchors/{id}). Jump-to uses the foundation2 hook scene3d.focusOn(x, y, z). Add the "friend tag" flow: a ＋ Tag action that runs the existing M8 bbox-annotation flow without an event and names the resulting anchor (friends become labeled, colored anchors that x-ray onto every video for free). Accept: create → rename → recolor → jump → delete round-trip live in the browser; labels visible in the 3D scene and video overlays; anchors survive reload.

M14 — Photo mode (lane F, frontend)

Fill the stub frontend/src/photoMode.js. 📷 / P: pause, hide helpers + gizmos + anchor labels (foundation2 hook scene3d.setHelpersVisible(false)), render one frame at 3840-wide (preserving aspect) to an offscreen target, download PNG, restore everything. Accept: PNG ≥ 3840 px wide with no grid/axes/frusta/gizmos; works in free-roam and follow-cam; with a splat present it renders the splat (verify on the synthetic point cloud + note the splat path).

M15 — Moment FX (lane F, frontend)

Fill the stub frontend/src/fx.js. When the playhead crosses a pyro/confetti event, fire a short particle burst at the event's resolved anchor (fall back to the stage centroid); on bass_drop, pulse the point-cloud/splat scale for ~a beat. Toggle via the hidden ✨ button; uses only scene3d.addObject/removeObject. Must fire when crossing markers in play and scrub. Accept: browser evidence of both effect types; frame rate stays ≥ 30 fps on the synthetic scene with FX enabled.

M16 — Server-side camera paths (lane G, backend + thin UI)

Implement the foundation2-stubbed routes over the new paths table: GET /api/paths (list), GET /api/paths/{id}, POST /api/paths {name, json}, DELETE /api/paths/{id} — json is the frozen camPath JSON as text, validated on POST. Fill the stub frontend/src/pathsStore.js: save-current-path (name prompt), load dropdown, delete — wired to the hidden path-save controls. Accept: endpoint tests (roundtrip deep-equal, 404s, invalid-JSON 422) in backend/tests/test_paths.py; live save → reload page → load reproduces keyframes exactly.

M17 — Memory capsule: static shareable export (lane G, backend)

backend/festival4d/capsule.py, CLI python -m festival4d capsule [--out DIR] (default data/capsule/). Build a zero-backend bundle: copy frontend/dist (error clearly if missing — tell the user to npm run build); copy media; bake every read-only API response to files at the same relative paths (api/manifest, api/events, api/anchors, api/beats, api/videos/{id}/poses, api/pointcloud, api/splat, api/audio/{id} — extracting audio if not yet cached); inject <script>window.__F4D_API_BASE__=""</script> into the copied index.html; set "capsule": true in the baked manifest (the frontend hides all write UI — foundation2 wires the flag); ship a stdlib serve.py in the bundle root that serves with HTTP Range support (video seeking needs 206s; plain http.server can't). Accept: python -m festival4d capsule on the synthetic project, then python serve.py in the bundle: app loads with zero requests to :8000, videos play and seek, 3D scene + timeline + spatial audio work, write UI absent; backend/tests/test_capsule.py covers bundle structure, baked-JSON validity, and serve.py Range (206) behavior.

Contracts (frozen at foundation2 merge)

1. beats.json (and the GET /api/beats body): {"tempo_bpm": float|null, "beats_s": [float], "onsets": [{"t_global_s": float, "strength": float 0..1}], "generated_by": "festival4d.audio_features"} — all times are t_global seconds.
2. camPath JSON (already shipped in camPath.js; now frozen): {"version": 1, "keyframes": [{"t_global": float, "pos": [x,y,z], "quat": [x,y,z,w], "fov": float}]} — Three.js scene space, quat order [x,y,z,w] (NOT COLMAP's [w,x,y,z]; convert via geometry.colmap_to_threejs on the backend, never inline).
3. paths table: paths(id INTEGER PK, name TEXT NOT NULL, created_at TEXT, path_json TEXT NOT NULL)
   • the four routes in M16.
4. PATCH /api/anchors/{id} {label?, color?} → updated anchor dict (404 on missing).
5. Runtime API base override: state.js resolves window.__F4D_API_BASE__ first, then VITE_API_BASE, then the localhost default. manifest.capsule: true ⇒ read-only UI.
6. transport.captureAudioStream() → MediaStream tapping the live WebAudio mix (and a matching releaseAudioStream()); scene3d.focusOn(x,y,z) and scene3d.setHelpersVisible(bool).
7. Frozen files during 5b: everything frozen in phase 1 plus frontend/src/{transport,scene3d,state,main,camPath,annotate}.js and frontend/index.html (foundation2 pre-wires every button — hidden — and every module import; lanes fill their own stub modules only). Believe a frozen file must change → plan/CHANGE_REQUESTS.md, work around locally, integration2 adjudicates.

foundation2 scope (Phase 5a, one agent, branch foundation2)

• DB: paths table + db.py helpers. API: /api/beats, /api/director, /api/paths*, PATCH /api/anchors/{id} — implemented against synthetic data where cheap, stubbed with the spec's graceful-degradation pattern where a lane owns the logic. CLI: features, direct, capsule subcommands dispatching into stubs.
• Backend stubs: audio_features.py, director.py, capsule.py (raise NotImplementedError; API/CLI catch and degrade, exactly like phase 1).
• Frontend: create empty stub modules director.js, exportVideo.js, anchorPanel.js, photoMode.js, fx.js, pathsStore.js; import them from main.js; add ALL new buttons to index.html (hidden until their module reports ready); add the transport/scene3d hooks and the __F4D_API_BASE__ + manifest.capsule wiring from the contracts above.
• Tests for every new route shape (mirroring test_api.py's exact-set style — note it locks the manifest key set; add capsule consciously).
• Update the ownership table below if anything moved. Suite ≥ 121 + your new tests. Merge fast.

integration2 scope (Phase 5c, one agent, branch integration2)

• Merge order irrelevant; resolve CHANGE_REQUESTS; full synthetic pass of every milestone (evidence per feature); raise the suite floor to the new count.
• Real-footage field test (the actual milestone): 2–4 real overlapping clips through ingest → sync → reconstruct → events → features → direct → serve, then capsule the result. Log every failure as a plan/ISSUES.md entry (ISSUE-1 is the format) — fix nothing mid-test.
• Tag v0.2.0 when accepted by the coordinator.

File ownership (Phase 5b)

Owner	Files
Lane E	backend/festival4d/audio_features.py, director.py, backend/tests/test_audio_features.py, test_director.py, frontend/src/director.js, exportVideo.js
Lane F	frontend/src/anchorPanel.js, photoMode.js, fx.js
Lane G	backend/festival4d/capsule.py, backend/tests/test_paths.py, test_capsule.py, frontend/src/pathsStore.js — the paths-route bodies were CRUD one-liners over the db helpers, so foundation2 implemented them (basic tests in test_phase5_api.py); lane G owns deepening test_paths.py + the UI
Every agent	its own plan/status/<lane>.md
Coordinator only	plan/DIRECTIVES.md
Frozen during 5b	contract item 7 above

Phase-5 pitfalls

1. Quaternion order. camPath is [x,y,z,w] (Three.js); COLMAP storage is [w,x,y,z]. Every past pose bug in this repo was a convention slip — go through geometry.colmap_to_threejs / lib/pose.js, never inline the math.
2. The pane you verify in throttles rAF (backgrounded tab). Drive the loop with the window.__f4d pump (localStorage.setItem("f4dDebug","1") in production builds); don't file "playback frozen" issues against your own hidden tab. MediaRecorder export (M12) needs a focused tab to capture real frames — say so in your evidence if you couldn't.
3. test_api.py locks response key-sets on purpose. Adding a manifest key or route means consciously updating the exact-set assertion in the same commit — a failing set-lock is a feature, not an obstacle.
4. Range serving. Any static path that serves video must answer 206 (spec M3 pitfall #3); python -m http.server does not — that's why M17 ships serve.py.
5. Never block on a missing optional input. No beats.json → director still works (no beat snapping). No splat → photo mode shoots the point cloud. No audio track → transport already falls back. Follow the degradation pattern everywhere; it's the house style.
6. Decoded audio is big (~23 MB/min stereo). M12/M17 must not decode all tracks just because they can — reuse what the transport already has.