51650c728e
- DB: paths table (contract #3) + helpers; patch_anchor (label/color) - API: GET /api/beats, POST /api/director (valid-empty degradation), /api/paths CRUD (validated POST -> 422), PATCH /api/anchors/{id}; manifest gains capsule:false (test_api.py key-set lock updated consciously) - CLI: features | direct | capsule subcommands -> lane stubs (exit-2 degradation) - Backend stubs with frozen contracts in docstrings: audio_features, director, capsule - Frontend: six lane stub modules imported from main.js; all phase-5 buttons pre-wired hidden in index.html; write-ui/body.capsule read-only mode; __F4D_API_BASE__ runtime override in state.js - Hooks (contract #6): transport.captureAudioStream/releaseAudioStream, scene3d.focusOn, scene3d.setHelpersVisible (+registerHelper; camPath gizmos) - Suite 121 -> 127 passed; npm build clean; live-browser verified (status file) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
199 lines
14 KiB
Markdown
199 lines
14 KiB
Markdown
# Phase 5 — Friends, Director's Cut & Capsules (M10–M17)
|
||
|
||
Canonical spec for the phase-5 milestones. **M0–M9 live in [`../OPUS_BUILD_INSTRUCTIONS.md`](../OPUS_BUILD_INSTRUCTIONS.md)
|
||
and every contract frozen there still applies** (timebase, DB schema, pose convention,
|
||
classifier contract, synthetic fixture). Standing protocol (status files, directives loop,
|
||
evidence discipline, worktrees) is unchanged from [`README.md`](README.md); the newest round of
|
||
[`DIRECTIVES.md`](DIRECTIVES.md) always wins.
|
||
|
||
## Context — what shipped since v0.1.0 (read before planning anything)
|
||
|
||
- **Splat rendering:** `GET /api/splat` + viewer `DropInViewer` with point-cloud fallback
|
||
(`docs/modelbeast-crossover.md`).
|
||
- **Live capture:** `FESTIVAL4D_CAPTURE=1` mounts `/capture` (README).
|
||
- **WebAudio master clock:** `t_global` derives from `AudioContext.currentTime`;
|
||
`GET /api/audio/{id}` serves a lazily-extracted stereo AAC per video; all `<video>` elements
|
||
are muted picture-only, with graceful fallback to the old `performance.now()` clock
|
||
(`frontend/src/transport.js` header comment is the reference).
|
||
- **Spatial audio (🎧):** per-camera HRTF `PannerNode`s at live poses; listener follows the
|
||
viewer camera (`transport.updateSpatial`, fed by `scene3d._updateAudio`).
|
||
- **Sync solver:** pairwise GCC-PHAT now reuses one rFFT per signal (`audio_sync.pairwise_offsets`).
|
||
- **Suite floor is 121 passed.** It only goes up.
|
||
|
||
## Phase map
|
||
|
||
```
|
||
Phase 5a (serial, ONE agent) Phase 5b (parallel, one agent per lane) Phase 5c (serial, ONE agent)
|
||
┌───────────────────────┐ ┌ lane/e-director ── M10 M11 M12 ─┐ ┌───────────────────────────┐
|
||
│ foundation2 │ ├ lane/f-social ──── M13 M14 M15 ─┤ │ integration2 │
|
||
│ contracts + stubs + │──merge──┤ ├──merge──│ full-system pass + │
|
||
│ UI wiring points │ └ lane/g-capsule ─── M16 M17 ─────┘ │ real-footage field test │
|
||
│ branch: foundation2 │ (independent — merge in any order) │ branch: integration2 │
|
||
└───────────────────────┘ └─────────────────────────────────── └───────────────────────────┘
|
||
```
|
||
|
||
Lanes depend **only** on foundation2, never on each other. Everything is developed and verified
|
||
against the synthetic fixture first; real footage is integration2's job.
|
||
|
||
## Milestones
|
||
|
||
### M10 — Audio features: beats & onsets (lane E, backend)
|
||
`backend/festival4d/audio_features.py`, entrypoint `run_features()`, CLI `python -m festival4d features`.
|
||
Load the **reference** video's ingest WAV (16 kHz mono; the reference timeline *is* `t_global`).
|
||
Beat/tempo tracking + onset-strength detection (librosa is already a dependency). Write
|
||
`data/work/beats.json` (shape frozen below); `GET /api/beats` serves it, 404 when absent.
|
||
Quiet/short/beatless audio must produce a *valid, empty* result — never a crash.
|
||
**Accept:** synthetic master audio's known pulse grid recovered (beats within ±50 ms, tempo ±2 BPM);
|
||
silence → empty-but-valid JSON; tests in `backend/tests/test_audio_features.py`.
|
||
|
||
### M11 — Auto-director (lane E, backend + thin UI)
|
||
`backend/festival4d/director.py`, entrypoint `generate_path(top_n=8, lead_s=2.0)`, CLI
|
||
`python -m festival4d direct`, `POST /api/director` (stubbed by foundation2) returning a
|
||
**camPath-compatible JSON** (contract below). Deterministic v1 rules — no ML:
|
||
pick the top-N events by confidence (ties → earlier); for each, choose the registered camera
|
||
with a pose nearest the event time (via `db` + `geometry.colmap_to_threejs`); keyframes at
|
||
`t_event − lead_s` and `t_event + duration`; when `beats.json` exists, snap keyframe times to
|
||
the nearest beat; FOV from that camera's intrinsics (same formula as `scene3d.snapTo`).
|
||
Frontend: fill the stub `frontend/src/director.js` — the hidden **🎬 Auto-path** button fetches
|
||
`/api/director` and loads it via `camPath.fromJSON` (do not edit camPath.js).
|
||
**Accept:** output round-trips through `camPath.fromJSON` (structure-locked test against the
|
||
frozen shape); covers the top-N events; keyframe times land on beats when beats exist; live
|
||
browser evidence of the path playing.
|
||
|
||
### M12 — Cinematic export (lane E, frontend)
|
||
Fill the stub `frontend/src/exportVideo.js`. The hidden **⏺ Export** button: seek to the first
|
||
keyframe, play the path, capture `scene3d` canvas via `captureStream(30)` + the soundtrack via
|
||
the `transport.captureAudioStream()` hook (foundation2 provides it — taps the WebAudio graph
|
||
post-gain, spatial mix included), mux with `MediaRecorder` (vp9/opus webm), stop at the last
|
||
keyframe, download `festival4d-cut.webm`. Restore all prior UI/transport state afterwards.
|
||
**Accept:** exported file duration within ±5% of the path span, contains an audio track, plays
|
||
in Chrome — browser evidence (file size, duration probe via ffprobe) in the status file.
|
||
|
||
### M13 — Anchor manager & friend tags (lane F, frontend)
|
||
Fill the stub `frontend/src/anchorPanel.js`: an always-available collapsible panel listing every
|
||
anchor (color dot, label, jump-to, delete, rename/recolor via the new `PATCH /api/anchors/{id}`).
|
||
Jump-to uses the foundation2 hook `scene3d.focusOn(x, y, z)`. Add the "friend tag" flow: a
|
||
**＋ Tag** action that runs the existing M8 bbox-annotation flow *without* an event and names the
|
||
resulting anchor (friends become labeled, colored anchors that x-ray onto every video for free).
|
||
**Accept:** create → rename → recolor → jump → delete round-trip live in the browser; labels
|
||
visible in the 3D scene and video overlays; anchors survive reload.
|
||
|
||
### M14 — Photo mode (lane F, frontend)
|
||
Fill the stub `frontend/src/photoMode.js`. **📷 / `P`**: pause, hide helpers + gizmos + anchor
|
||
labels (foundation2 hook `scene3d.setHelpersVisible(false)`), render one frame at 3840-wide
|
||
(preserving aspect) to an offscreen target, download PNG, restore everything.
|
||
**Accept:** PNG ≥ 3840 px wide with no grid/axes/frusta/gizmos; works in free-roam and
|
||
follow-cam; with a splat present it renders the splat (verify on the synthetic point cloud +
|
||
note the splat path).
|
||
|
||
### M15 — Moment FX (lane F, frontend)
|
||
Fill the stub `frontend/src/fx.js`. When the playhead crosses a `pyro`/`confetti` event, fire a
|
||
short particle burst at the event's resolved anchor (fall back to the stage centroid); on
|
||
`bass_drop`, pulse the point-cloud/splat scale for ~a beat. Toggle via the hidden **✨** button;
|
||
uses only `scene3d.addObject/removeObject`. Must fire when crossing markers in play *and* scrub.
|
||
**Accept:** browser evidence of both effect types; frame rate stays ≥ 30 fps on the synthetic
|
||
scene with FX enabled.
|
||
|
||
### M16 — Server-side camera paths (lane G, backend + thin UI)
|
||
Implement the foundation2-stubbed routes over the new `paths` table:
|
||
`GET /api/paths` (list), `GET /api/paths/{id}`, `POST /api/paths {name, json}`,
|
||
`DELETE /api/paths/{id}` — `json` is the frozen camPath JSON as text, validated on POST.
|
||
Fill the stub `frontend/src/pathsStore.js`: save-current-path (name prompt), load dropdown,
|
||
delete — wired to the hidden path-save controls.
|
||
**Accept:** endpoint tests (roundtrip deep-equal, 404s, invalid-JSON 422) in
|
||
`backend/tests/test_paths.py`; live save → reload page → load reproduces keyframes exactly.
|
||
|
||
### M17 — Memory capsule: static shareable export (lane G, backend)
|
||
`backend/festival4d/capsule.py`, CLI `python -m festival4d capsule [--out DIR]` (default
|
||
`data/capsule/`). Build a **zero-backend** bundle:
|
||
copy `frontend/dist` (error clearly if missing — tell the user to `npm run build`); copy media;
|
||
bake every read-only API response to files at the *same relative paths* (`api/manifest`,
|
||
`api/events`, `api/anchors`, `api/beats`, `api/videos/{id}/poses`, `api/pointcloud`, `api/splat`,
|
||
`api/audio/{id}` — extracting audio if not yet cached); inject
|
||
`<script>window.__F4D_API_BASE__=""</script>` into the copied `index.html`; set
|
||
`"capsule": true` in the baked manifest (the frontend hides all write UI — foundation2 wires
|
||
the flag); ship a stdlib **`serve.py`** in the bundle root that serves with HTTP Range support
|
||
(video seeking needs 206s; plain `http.server` can't).
|
||
**Accept:** `python -m festival4d capsule` on the synthetic project, then `python serve.py` in
|
||
the bundle: app loads with **zero** requests to :8000, videos play *and seek*, 3D scene +
|
||
timeline + spatial audio work, write UI absent; `backend/tests/test_capsule.py` covers bundle
|
||
structure, baked-JSON validity, and serve.py Range (206) behavior.
|
||
|
||
## Contracts (frozen at foundation2 merge)
|
||
|
||
1. **beats.json** (and the `GET /api/beats` body):
|
||
`{"tempo_bpm": float|null, "beats_s": [float], "onsets": [{"t_global_s": float, "strength": float 0..1}], "generated_by": "festival4d.audio_features"}`
|
||
— all times are `t_global` seconds.
|
||
2. **camPath JSON** (already shipped in `camPath.js`; now frozen):
|
||
`{"version": 1, "keyframes": [{"t_global": float, "pos": [x,y,z], "quat": [x,y,z,w], "fov": float}]}`
|
||
— **Three.js scene space**, quat order **[x,y,z,w]** (NOT COLMAP's [w,x,y,z]; convert via
|
||
`geometry.colmap_to_threejs` on the backend, never inline).
|
||
3. **paths table:** `paths(id INTEGER PK, name TEXT NOT NULL, created_at TEXT, path_json TEXT NOT NULL)`
|
||
+ the four routes in M16.
|
||
4. **`PATCH /api/anchors/{id}`** `{label?, color?}` → updated anchor dict (404 on missing).
|
||
5. **Runtime API base override:** `state.js` resolves `window.__F4D_API_BASE__` first, then
|
||
`VITE_API_BASE`, then the localhost default. **`manifest.capsule: true`** ⇒ read-only UI.
|
||
6. **`transport.captureAudioStream()`** → `MediaStream` tapping the live WebAudio mix (and a
|
||
matching `releaseAudioStream()`); `scene3d.focusOn(x,y,z)` and `scene3d.setHelpersVisible(bool)`.
|
||
7. **Frozen files during 5b:** everything frozen in phase 1 **plus**
|
||
`frontend/src/{transport,scene3d,state,main,camPath,annotate}.js` and `frontend/index.html`
|
||
(foundation2 pre-wires every button — hidden — and every module import; lanes fill their own
|
||
stub modules only). Believe a frozen file must change → `plan/CHANGE_REQUESTS.md`, work
|
||
around locally, integration2 adjudicates.
|
||
|
||
## foundation2 scope (Phase 5a, one agent, branch `foundation2`)
|
||
|
||
- DB: `paths` table + `db.py` helpers. API: `/api/beats`, `/api/director`, `/api/paths*`,
|
||
`PATCH /api/anchors/{id}` — implemented against synthetic data where cheap, stubbed with the
|
||
spec's graceful-degradation pattern where a lane owns the logic. CLI: `features`, `direct`,
|
||
`capsule` subcommands dispatching into stubs.
|
||
- Backend stubs: `audio_features.py`, `director.py`, `capsule.py` (raise `NotImplementedError`;
|
||
API/CLI catch and degrade, exactly like phase 1).
|
||
- Frontend: create empty stub modules `director.js`, `exportVideo.js`, `anchorPanel.js`,
|
||
`photoMode.js`, `fx.js`, `pathsStore.js`; import them from `main.js`; add ALL new buttons to
|
||
`index.html` (hidden until their module reports ready); add the transport/scene3d hooks and
|
||
the `__F4D_API_BASE__` + `manifest.capsule` wiring from the contracts above.
|
||
- Tests for every new route shape (mirroring `test_api.py`'s exact-set style — note it locks
|
||
the manifest key set; add `capsule` consciously).
|
||
- Update the ownership table below if anything moved. Suite ≥ 121 + your new tests. Merge fast.
|
||
|
||
## integration2 scope (Phase 5c, one agent, branch `integration2`)
|
||
|
||
- Merge order irrelevant; resolve CHANGE_REQUESTS; full synthetic pass of every milestone
|
||
(evidence per feature); raise the suite floor to the new count.
|
||
- **Real-footage field test** (the actual milestone): 2–4 real overlapping clips through
|
||
`ingest → sync → reconstruct → events → features → direct → serve`, then capsule the result.
|
||
Log every failure as a `plan/ISSUES.md` entry (ISSUE-1 is the format) — fix nothing mid-test.
|
||
- Tag `v0.2.0` when accepted by the coordinator.
|
||
|
||
## File ownership (Phase 5b)
|
||
|
||
| Owner | Files |
|
||
|---|---|
|
||
| Lane E | `backend/festival4d/audio_features.py`, `director.py`, `backend/tests/test_audio_features.py`, `test_director.py`, `frontend/src/director.js`, `exportVideo.js` |
|
||
| Lane F | `frontend/src/anchorPanel.js`, `photoMode.js`, `fx.js` |
|
||
| Lane G | `backend/festival4d/capsule.py`, `backend/tests/test_paths.py`, `test_capsule.py`, `frontend/src/pathsStore.js` — the paths-route bodies were CRUD one-liners over the db helpers, so foundation2 implemented them (basic tests in `test_phase5_api.py`); lane G owns deepening `test_paths.py` + the UI |
|
||
| Every agent | its own `plan/status/<lane>.md` |
|
||
| **Coordinator only** | `plan/DIRECTIVES.md` |
|
||
| **Frozen during 5b** | contract item 7 above |
|
||
|
||
## Phase-5 pitfalls
|
||
|
||
1. **Quaternion order.** camPath is `[x,y,z,w]` (Three.js); COLMAP storage is `[w,x,y,z]`.
|
||
Every past pose bug in this repo was a convention slip — go through `geometry.colmap_to_threejs`
|
||
/ `lib/pose.js`, never inline the math.
|
||
2. **The pane you verify in throttles rAF** (backgrounded tab). Drive the loop with the
|
||
`window.__f4d` pump (`localStorage.setItem("f4dDebug","1")` in production builds); don't
|
||
file "playback frozen" issues against your own hidden tab. MediaRecorder export (M12) needs
|
||
a *focused* tab to capture real frames — say so in your evidence if you couldn't.
|
||
3. **`test_api.py` locks response key-sets on purpose.** Adding a manifest key or route means
|
||
consciously updating the exact-set assertion in the same commit — a failing set-lock is a
|
||
feature, not an obstacle.
|
||
4. **Range serving.** Any static path that serves video must answer 206 (spec M3 pitfall #3);
|
||
`python -m http.server` does not — that's why M17 ships `serve.py`.
|
||
5. **Never block on a missing optional input.** No beats.json → director still works (no beat
|
||
snapping). No splat → photo mode shoots the point cloud. No audio track → transport already
|
||
falls back. Follow the degradation pattern everywhere; it's the house style.
|
||
6. **Decoded audio is big** (~23 MB/min stereo). M12/M17 must not decode all tracks just
|
||
because they can — reuse what the transport already has.
|