# Phase 5 — Friends, Director's Cut & Capsules (M10–M17)

Canonical spec for the phase-5 milestones. **M0–M9 live in [`../OPUS_BUILD_INSTRUCTIONS.md`](../OPUS_BUILD_INSTRUCTIONS.md)
and every contract frozen there still applies** (timebase, DB schema, pose convention,
classifier contract, synthetic fixture). Standing protocol (status files, directives loop,
evidence discipline, worktrees) is unchanged from [`README.md`](README.md); the newest round of
[`DIRECTIVES.md`](DIRECTIVES.md) always wins.

## Context — what shipped since v0.1.0 (read before planning anything)

- **Splat rendering:** `GET /api/splat` + viewer `DropInViewer` with point-cloud fallback
  (`docs/modelbeast-crossover.md`).
- **Live capture:** `FESTIVAL4D_CAPTURE=1` mounts `/capture` (README).
- **WebAudio master clock:** `t_global` derives from `AudioContext.currentTime`;
  `GET /api/audio/{id}` serves a lazily-extracted stereo AAC per video; all `<video>` elements
  are muted picture-only, with graceful fallback to the old `performance.now()` clock
  (`frontend/src/transport.js` header comment is the reference).
- **Spatial audio (🎧):** per-camera HRTF `PannerNode`s at live poses; listener follows the
  viewer camera (`transport.updateSpatial`, fed by `scene3d._updateAudio`).
- **Sync solver:** pairwise GCC-PHAT now reuses one rFFT per signal (`audio_sync.pairwise_offsets`).
- **Suite floor is 121 passed.** It only goes up.

## Phase map

```
Phase 5a (serial, ONE agent)      Phase 5b (parallel, one agent per lane)     Phase 5c (serial, ONE agent)
┌───────────────────────┐         ┌ lane/e-director ── M10 M11 M12 ─┐         ┌───────────────────────────┐
│ foundation2           │         ├ lane/f-social ──── M13 M14 M15 ─┤         │ integration2              │
│ contracts + stubs +   │──merge──┤                                 ├──merge──│ full-system pass +        │
│ UI wiring points      │         └ lane/g-capsule ─── M16 M17 ─────┘         │ real-footage field test   │
│ branch: foundation2   │         (independent — merge in any order)          │ branch: integration2      │
└───────────────────────┘         └───────────────────────────────────        └───────────────────────────┘
```

Lanes depend **only** on foundation2, never on each other. Everything is developed and verified
against the synthetic fixture first; real footage is integration2's job.

## Milestones

### M10 — Audio features: beats & onsets (lane E, backend)
`backend/festival4d/audio_features.py`, entrypoint `run_features()`, CLI `python -m festival4d features`.
Load the **reference** video's ingest WAV (16 kHz mono; the reference timeline *is* `t_global`).
Beat/tempo tracking + onset-strength detection (librosa is already a dependency). Write
`data/work/beats.json` (shape frozen below); `GET /api/beats` serves it, 404 when absent.
Quiet/short/beatless audio must produce a *valid, empty* result — never a crash.
**Accept:** synthetic master audio's known pulse grid recovered (beats within ±50 ms, tempo ±2 BPM);
silence → empty-but-valid JSON; tests in `backend/tests/test_audio_features.py`.

### M11 — Auto-director (lane E, backend + thin UI)
`backend/festival4d/director.py`, entrypoint `generate_path(top_n=8, lead_s=2.0)`, CLI
`python -m festival4d direct`, `POST /api/director` (stubbed by foundation2) returning a
**camPath-compatible JSON** (contract below). Deterministic v1 rules — no ML:
pick the top-N events by confidence (ties → earlier); for each, choose the registered camera
with a pose nearest the event time (via `db` + `geometry.colmap_to_threejs`); keyframes at
`t_event − lead_s` and `t_event + duration`; when `beats.json` exists, snap keyframe times to
the nearest beat; FOV from that camera's intrinsics (same formula as `scene3d.snapTo`).
Frontend: fill the stub `frontend/src/director.js` — the hidden **🎬 Auto-path** button fetches
`/api/director` and loads it via `camPath.fromJSON` (do not edit camPath.js).
**Accept:** output round-trips through `camPath.fromJSON` (structure-locked test against the
frozen shape); covers the top-N events; keyframe times land on beats when beats exist; live
browser evidence of the path playing.

### M12 — Cinematic export (lane E, frontend)
Fill the stub `frontend/src/exportVideo.js`. The hidden **⏺ Export** button: seek to the first
keyframe, play the path, capture `scene3d` canvas via `captureStream(30)` + the soundtrack via
the `transport.captureAudioStream()` hook (foundation2 provides it — taps the WebAudio graph
post-gain, spatial mix included), mux with `MediaRecorder` (vp9/opus webm), stop at the last
keyframe, download `festival4d-cut.webm`. Restore all prior UI/transport state afterwards.
**Accept:** exported file duration within ±5% of the path span, contains an audio track, plays
in Chrome — browser evidence (file size, duration probe via ffprobe) in the status file.

### M13 — Anchor manager & friend tags (lane F, frontend)
Fill the stub `frontend/src/anchorPanel.js`: an always-available collapsible panel listing every
anchor (color dot, label, jump-to, delete, rename/recolor via the new `PATCH /api/anchors/{id}`).
Jump-to uses the foundation2 hook `scene3d.focusOn(x, y, z)`. Add the "friend tag" flow: a
**＋ Tag** action that runs the existing M8 bbox-annotation flow *without* an event and names the
resulting anchor (friends become labeled, colored anchors that x-ray onto every video for free).
**Accept:** create → rename → recolor → jump → delete round-trip live in the browser; labels
visible in the 3D scene and video overlays; anchors survive reload.

### M14 — Photo mode (lane F, frontend)
Fill the stub `frontend/src/photoMode.js`. **📷 / `P`**: pause, hide helpers + gizmos + anchor
labels (foundation2 hook `scene3d.setHelpersVisible(false)`), render one frame at 3840-wide
(preserving aspect) to an offscreen target, download PNG, restore everything.
**Accept:** PNG ≥ 3840 px wide with no grid/axes/frusta/gizmos; works in free-roam and
follow-cam; with a splat present it renders the splat (verify on the synthetic point cloud +
note the splat path).

### M15 — Moment FX (lane F, frontend)
Fill the stub `frontend/src/fx.js`. When the playhead crosses a `pyro`/`confetti` event, fire a
short particle burst at the event's resolved anchor (fall back to the stage centroid); on
`bass_drop`, pulse the point-cloud/splat scale for ~a beat. Toggle via the hidden **✨** button;
uses only `scene3d.addObject/removeObject`. Must fire when crossing markers in play *and* scrub.
**Accept:** browser evidence of both effect types; frame rate stays ≥ 30 fps on the synthetic
scene with FX enabled.

### M16 — Server-side camera paths (lane G, backend + thin UI)
Implement the foundation2-stubbed routes over the new `paths` table:
`GET /api/paths` (list), `GET /api/paths/{id}`, `POST /api/paths {name, json}`,
`DELETE /api/paths/{id}` — `json` is the frozen camPath JSON as text, validated on POST.
Fill the stub `frontend/src/pathsStore.js`: save-current-path (name prompt), load dropdown,
delete — wired to the hidden path-save controls.
**Accept:** endpoint tests (roundtrip deep-equal, 404s, invalid-JSON 422) in
`backend/tests/test_paths.py`; live save → reload page → load reproduces keyframes exactly.

### M17 — Memory capsule: static shareable export (lane G, backend)
`backend/festival4d/capsule.py`, CLI `python -m festival4d capsule [--out DIR]` (default
`data/capsule/`). Build a **zero-backend** bundle:
copy `frontend/dist` (error clearly if missing — tell the user to `npm run build`); copy media;
bake every read-only API response to files at the *same relative paths* (`api/manifest`,
`api/events`, `api/anchors`, `api/beats`, `api/videos/{id}/poses`, `api/pointcloud`, `api/splat`,
`api/audio/{id}` — extracting audio if not yet cached); inject
`<script>window.__F4D_API_BASE__=""</script>` into the copied `index.html`; set
`"capsule": true` in the baked manifest (the frontend hides all write UI — foundation2 wires
the flag); ship a stdlib **`serve.py`** in the bundle root that serves with HTTP Range support
(video seeking needs 206s; plain `http.server` can't).
**Accept:** `python -m festival4d capsule` on the synthetic project, then `python serve.py` in
the bundle: app loads with **zero** requests to :8000, videos play *and seek*, 3D scene +
timeline + spatial audio work, write UI absent; `backend/tests/test_capsule.py` covers bundle
structure, baked-JSON validity, and serve.py Range (206) behavior.

## Contracts (frozen at foundation2 merge)

1. **beats.json** (and the `GET /api/beats` body):
   `{"tempo_bpm": float|null, "beats_s": [float], "onsets": [{"t_global_s": float, "strength": float 0..1}], "generated_by": "festival4d.audio_features"}`
   — all times are `t_global` seconds.
2. **camPath JSON** (already shipped in `camPath.js`; now frozen):
   `{"version": 1, "keyframes": [{"t_global": float, "pos": [x,y,z], "quat": [x,y,z,w], "fov": float}]}`
   — **Three.js scene space**, quat order **[x,y,z,w]** (NOT COLMAP's [w,x,y,z]; convert via
   `geometry.colmap_to_threejs` on the backend, never inline).
3. **paths table:** `paths(id INTEGER PK, name TEXT NOT NULL, created_at TEXT, path_json TEXT NOT NULL)`
   + the four routes in M16.
4. **`PATCH /api/anchors/{id}`** `{label?, color?}` → updated anchor dict (404 on missing).
5. **Runtime API base override:** `state.js` resolves `window.__F4D_API_BASE__` first, then
   `VITE_API_BASE`, then the localhost default. **`manifest.capsule: true`** ⇒ read-only UI.
6. **`transport.captureAudioStream()`** → `MediaStream` tapping the live WebAudio mix (and a
   matching `releaseAudioStream()`); `scene3d.focusOn(x,y,z)` and `scene3d.setHelpersVisible(bool)`.
7. **Frozen files during 5b:** everything frozen in phase 1 **plus**
   `frontend/src/{transport,scene3d,state,main,camPath,annotate}.js` and `frontend/index.html`
   (foundation2 pre-wires every button — hidden — and every module import; lanes fill their own
   stub modules only). Believe a frozen file must change → `plan/CHANGE_REQUESTS.md`, work
   around locally, integration2 adjudicates.

## foundation2 scope (Phase 5a, one agent, branch `foundation2`)

- DB: `paths` table + `db.py` helpers. API: `/api/beats`, `/api/director`, `/api/paths*`,
  `PATCH /api/anchors/{id}` — implemented against synthetic data where cheap, stubbed with the
  spec's graceful-degradation pattern where a lane owns the logic. CLI: `features`, `direct`,
  `capsule` subcommands dispatching into stubs.
- Backend stubs: `audio_features.py`, `director.py`, `capsule.py` (raise `NotImplementedError`;
  API/CLI catch and degrade, exactly like phase 1).
- Frontend: create empty stub modules `director.js`, `exportVideo.js`, `anchorPanel.js`,
  `photoMode.js`, `fx.js`, `pathsStore.js`; import them from `main.js`; add ALL new buttons to
  `index.html` (hidden until their module reports ready); add the transport/scene3d hooks and
  the `__F4D_API_BASE__` + `manifest.capsule` wiring from the contracts above.
- Tests for every new route shape (mirroring `test_api.py`'s exact-set style — note it locks
  the manifest key set; add `capsule` consciously).
- Update the ownership table below if anything moved. Suite ≥ 121 + your new tests. Merge fast.

## integration2 scope (Phase 5c, one agent, branch `integration2`)

- Merge order irrelevant; resolve CHANGE_REQUESTS; full synthetic pass of every milestone
  (evidence per feature); raise the suite floor to the new count.
- **Real-footage field test** (the actual milestone): 2–4 real overlapping clips through
  `ingest → sync → reconstruct → events → features → direct → serve`, then capsule the result.
  Log every failure as a `plan/ISSUES.md` entry (ISSUE-1 is the format) — fix nothing mid-test.
- Tag `v0.2.0` when accepted by the coordinator.

## File ownership (Phase 5b)

| Owner | Files |
|---|---|
| Lane E | `backend/festival4d/audio_features.py`, `director.py`, `backend/tests/test_audio_features.py`, `test_director.py`, `frontend/src/director.js`, `exportVideo.js` |
| Lane F | `frontend/src/anchorPanel.js`, `photoMode.js`, `fx.js` |
| Lane G | `backend/festival4d/capsule.py`, `backend/tests/test_paths.py`, `test_capsule.py`, `frontend/src/pathsStore.js` — the paths-route bodies were CRUD one-liners over the db helpers, so foundation2 implemented them (basic tests in `test_phase5_api.py`); lane G owns deepening `test_paths.py` + the UI |
| Every agent | its own `plan/status/<lane>.md` |
| **Coordinator only** | `plan/DIRECTIVES.md` |
| **Frozen during 5b** | contract item 7 above |

## Phase-5 pitfalls

1. **Quaternion order.** camPath is `[x,y,z,w]` (Three.js); COLMAP storage is `[w,x,y,z]`.
   Every past pose bug in this repo was a convention slip — go through `geometry.colmap_to_threejs`
   / `lib/pose.js`, never inline the math.
2. **The pane you verify in throttles rAF** (backgrounded tab). Drive the loop with the
   `window.__f4d` pump (`localStorage.setItem("f4dDebug","1")` in production builds); don't
   file "playback frozen" issues against your own hidden tab. MediaRecorder export (M12) needs
   a *focused* tab to capture real frames — say so in your evidence if you couldn't.
3. **`test_api.py` locks response key-sets on purpose.** Adding a manifest key or route means
   consciously updating the exact-set assertion in the same commit — a failing set-lock is a
   feature, not an obstacle.
4. **Range serving.** Any static path that serves video must answer 206 (spec M3 pitfall #3);
   `python -m http.server` does not — that's why M17 ships `serve.py`.
5. **Never block on a missing optional input.** No beats.json → director still works (no beat
   snapping). No splat → photo mode shoots the point cloud. No audio track → transport already
   falls back. Follow the degradation pattern everywhere; it's the house style.
6. **Decoded audio is big** (~23 MB/min stereo). M12/M17 must not decode all tracks just
   because they can — reuse what the transport already has.