545f7a347d
plan/20-phase5.md is the phase-5 canonical spec: M10 beats/onsets, M11 auto-director, M12 cinematic export (lane E); M13 anchor manager + friend tags, M14 photo mode, M15 moment FX (lane F); M16 server-side camera paths, M17 static memory-capsule export (lane G). foundation2 freezes the new contracts (beats.json, camPath JSON, paths table, PATCH anchors, __F4D_API_BASE__ / manifest.capsule, transport/scene3d hooks) and pre-wires all UI so parallel lanes never touch shared files. DIRECTIVES Round 4 records the fresh-eyes upgrades (FFT caching, WebAudio master clock, spatial audio; suite floor now 121), ratifies the additive contract amendments, and sets run order: foundation2 -> E, G, F (own worktrees, mandatory) -> integration2 (real-footage field test, v0.2.0 proposal). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
51 lines
3.2 KiB
Markdown
51 lines
3.2 KiB
Markdown
# Festival 4D — Future Ideas
|
|
|
|
Parking lot for post-v0.1.0 extensions (spec §5 future-extensions). Not committed work — a place
|
|
to capture direction so the prototype's scope stays honest. Roughly ordered by payoff-to-effort.
|
|
|
|
> **Status 2026-07-17:** splatting ✅ shipped (see `modelbeast-crossover.md`); spatial audio ✅
|
|
> shipped (🎧, WebAudio master clock). Cinematic export → **M12**, multi-modal audio analysis →
|
|
> **M10/M11**, anchor manager → **M13**, path persistence → **M16** — all now specced in
|
|
> `plan/20-phase5.md`. Still parked here: realtime ingest, object/artist tracking,
|
|
> per-project workspaces, sync-graph visualization, per-moment splats (crossover doc §3).
|
|
|
|
## Neural rendering — Gaussian Splatting
|
|
Replace the sparse COLMAP point cloud with a **3D Gaussian Splatting** model for photorealistic
|
|
free-roam instead of a dot cloud. COLMAP poses + images already feed splatting trainers
|
|
(e.g. `gsplat`, Inria 3DGS) directly, so the reconstruction step's output is reusable. The viewer
|
|
would swap `Points` for a splat renderer (a WebGL splat viewer, or bake to a mesh). Biggest visual
|
|
upgrade; heaviest compute (needs a GPU training pass per project).
|
|
|
|
## Cinematic export — render the camera path to MP4
|
|
M9 already produces keyframed god's-eye paths. Add a headless render: step `t_global` at a fixed
|
|
fps, drive the Three.js camera along the path, capture frames (offscreen canvas → `captureStream`
|
|
or server-side `puppeteer` + `ffmpeg`), and mux to MP4. Turns the app from a viewer into a
|
|
"director's cut" clip exporter. Path JSON export/import is already in place, so this is additive.
|
|
|
|
## Real-time / streaming ingest
|
|
Instead of pre-recorded files, ingest live phone feeds (WebRTC/RTMP). Requires: rolling audio
|
|
sync on a sliding window (GCC-PHAT already windowed for drift — reuse it), incremental SfM or
|
|
pose tracking against a pre-built map, and a streaming transport in the frontend. Large effort;
|
|
the offline pipeline is the right foundation.
|
|
|
|
## Multi-modal audio analysis for moment detection
|
|
Today's `events` step finds candidates from RMS + spectral flux, then a VLM classifies video.
|
|
Add **audio-native** analysis in parallel: beat/tempo tracking, drop detection from
|
|
spectrograms, song-boundary segmentation. Fuse audio-derived and vision-derived labels for
|
|
higher-confidence events and beat-aligned auto camera cuts. Cheap; complements the existing
|
|
classifier providers.
|
|
|
|
## Object / artist tracking
|
|
Extend M8 (manual bbox → 3D anchor) with automatic per-frame tracking (e.g. a detector +
|
|
tracker), so an anchor follows the lead performer across time instead of being static. Enables
|
|
a "follow artist" auto-camera mode in the 3D viewer.
|
|
|
|
## Quality-of-life
|
|
- **Standalone anchor manager** — the delete UI currently lives in the event correction panel;
|
|
a dedicated always-available anchor list would decouple anchor management from events.
|
|
- **Persist camera paths server-side** — paths are JSON export/import only; a `paths` table +
|
|
endpoints would let them live with the project.
|
|
- **Per-project workspaces** — the app assumes one project at a time (SQLite at `data/project.db`).
|
|
- **Sync-graph visualization** — show which videos aligned into which connected component when
|
|
some clips share no audio.
|