545f7a347d
plan/20-phase5.md is the phase-5 canonical spec: M10 beats/onsets, M11 auto-director, M12 cinematic export (lane E); M13 anchor manager + friend tags, M14 photo mode, M15 moment FX (lane F); M16 server-side camera paths, M17 static memory-capsule export (lane G). foundation2 freezes the new contracts (beats.json, camPath JSON, paths table, PATCH anchors, __F4D_API_BASE__ / manifest.capsule, transport/scene3d hooks) and pre-wires all UI so parallel lanes never touch shared files. DIRECTIVES Round 4 records the fresh-eyes upgrades (FFT caching, WebAudio master clock, spatial audio; suite floor now 121), ratifies the additive contract amendments, and sets run order: foundation2 -> E, G, F (own worktrees, mandatory) -> integration2 (real-footage field test, v0.2.0 proposal). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
3.2 KiB
Festival 4D — Future Ideas
Parking lot for post-v0.1.0 extensions (spec §5 future-extensions). Not committed work — a place to capture direction so the prototype's scope stays honest. Roughly ordered by payoff-to-effort.
Status 2026-07-17: splatting ✅ shipped (see
modelbeast-crossover.md); spatial audio ✅ shipped (🎧, WebAudio master clock). Cinematic export → M12, multi-modal audio analysis → M10/M11, anchor manager → M13, path persistence → M16 — all now specced inplan/20-phase5.md. Still parked here: realtime ingest, object/artist tracking, per-project workspaces, sync-graph visualization, per-moment splats (crossover doc §3).
Neural rendering — Gaussian Splatting
Replace the sparse COLMAP point cloud with a 3D Gaussian Splatting model for photorealistic
free-roam instead of a dot cloud. COLMAP poses + images already feed splatting trainers
(e.g. gsplat, Inria 3DGS) directly, so the reconstruction step's output is reusable. The viewer
would swap Points for a splat renderer (a WebGL splat viewer, or bake to a mesh). Biggest visual
upgrade; heaviest compute (needs a GPU training pass per project).
Cinematic export — render the camera path to MP4
M9 already produces keyframed god's-eye paths. Add a headless render: step t_global at a fixed
fps, drive the Three.js camera along the path, capture frames (offscreen canvas → captureStream
or server-side puppeteer + ffmpeg), and mux to MP4. Turns the app from a viewer into a
"director's cut" clip exporter. Path JSON export/import is already in place, so this is additive.
Real-time / streaming ingest
Instead of pre-recorded files, ingest live phone feeds (WebRTC/RTMP). Requires: rolling audio sync on a sliding window (GCC-PHAT already windowed for drift — reuse it), incremental SfM or pose tracking against a pre-built map, and a streaming transport in the frontend. Large effort; the offline pipeline is the right foundation.
Multi-modal audio analysis for moment detection
Today's events step finds candidates from RMS + spectral flux, then a VLM classifies video.
Add audio-native analysis in parallel: beat/tempo tracking, drop detection from
spectrograms, song-boundary segmentation. Fuse audio-derived and vision-derived labels for
higher-confidence events and beat-aligned auto camera cuts. Cheap; complements the existing
classifier providers.
Object / artist tracking
Extend M8 (manual bbox → 3D anchor) with automatic per-frame tracking (e.g. a detector + tracker), so an anchor follows the lead performer across time instead of being static. Enables a "follow artist" auto-camera mode in the 3D viewer.
Quality-of-life
- Standalone anchor manager — the delete UI currently lives in the event correction panel; a dedicated always-available anchor list would decouple anchor management from events.
- Persist camera paths server-side — paths are JSON export/import only; a
pathstable + endpoints would let them live with the project. - Per-project workspaces — the app assumes one project at a time (SQLite at
data/project.db). - Sync-graph visualization — show which videos aligned into which connected component when some clips share no audio.