m3ultra 545f7a347d Plan: Phase 5 spec + directives round 4 (M10-M17, lanes E/F/G)

plan/20-phase5.md is the phase-5 canonical spec: M10 beats/onsets,
M11 auto-director, M12 cinematic export (lane E); M13 anchor manager +
friend tags, M14 photo mode, M15 moment FX (lane F); M16 server-side
camera paths, M17 static memory-capsule export (lane G). foundation2
freezes the new contracts (beats.json, camPath JSON, paths table,
PATCH anchors, __F4D_API_BASE__ / manifest.capsule, transport/scene3d
hooks) and pre-wires all UI so parallel lanes never touch shared files.

DIRECTIVES Round 4 records the fresh-eyes upgrades (FFT caching,
WebAudio master clock, spatial audio; suite floor now 121), ratifies
the additive contract amendments, and sets run order:
foundation2 -> E, G, F (own worktrees, mandatory) -> integration2
(real-footage field test, v0.2.0 proposal).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

2026-07-17 17:08:44 +10:00

3.2 KiB

Raw Blame History

Festival 4D — Future Ideas

Parking lot for post-v0.1.0 extensions (spec §5 future-extensions). Not committed work — a place to capture direction so the prototype's scope stays honest. Roughly ordered by payoff-to-effort.

Status 2026-07-17: splatting ✅ shipped (see modelbeast-crossover.md); spatial audio ✅ shipped (🎧, WebAudio master clock). Cinematic export → M12, multi-modal audio analysis → M10/M11, anchor manager → M13, path persistence → M16 — all now specced in plan/20-phase5.md. Still parked here: realtime ingest, object/artist tracking, per-project workspaces, sync-graph visualization, per-moment splats (crossover doc §3).

Neural rendering — Gaussian Splatting

Replace the sparse COLMAP point cloud with a 3D Gaussian Splatting model for photorealistic free-roam instead of a dot cloud. COLMAP poses + images already feed splatting trainers (e.g. gsplat, Inria 3DGS) directly, so the reconstruction step's output is reusable. The viewer would swap Points for a splat renderer (a WebGL splat viewer, or bake to a mesh). Biggest visual upgrade; heaviest compute (needs a GPU training pass per project).

Cinematic export — render the camera path to MP4

M9 already produces keyframed god's-eye paths. Add a headless render: step t_global at a fixed fps, drive the Three.js camera along the path, capture frames (offscreen canvas → captureStream or server-side puppeteer + ffmpeg), and mux to MP4. Turns the app from a viewer into a "director's cut" clip exporter. Path JSON export/import is already in place, so this is additive.

Real-time / streaming ingest

Instead of pre-recorded files, ingest live phone feeds (WebRTC/RTMP). Requires: rolling audio sync on a sliding window (GCC-PHAT already windowed for drift — reuse it), incremental SfM or pose tracking against a pre-built map, and a streaming transport in the frontend. Large effort; the offline pipeline is the right foundation.

Today's events step finds candidates from RMS + spectral flux, then a VLM classifies video. Add audio-native analysis in parallel: beat/tempo tracking, drop detection from spectrograms, song-boundary segmentation. Fuse audio-derived and vision-derived labels for higher-confidence events and beat-aligned auto camera cuts. Cheap; complements the existing classifier providers.

Object / artist tracking

Extend M8 (manual bbox → 3D anchor) with automatic per-frame tracking (e.g. a detector + tracker), so an anchor follows the lead performer across time instead of being static. Enables a "follow artist" auto-camera mode in the 3D viewer.

Quality-of-life

Standalone anchor manager — the delete UI currently lives in the event correction panel; a dedicated always-available anchor list would decouple anchor management from events.
Persist camera paths server-side — paths are JSON export/import only; a paths table + endpoints would let them live with the project.
Per-project workspaces — the app assumes one project at a time (SQLite at data/project.db).
Sync-graph visualization — show which videos aligned into which connected component when some clips share no audio.

3.2 KiB Raw Blame History