# Festival 4D Turn multiple fan-shot smartphone videos of the same concert into a synchronized, explorable **4D experience**: time-aligned multi-video playback, 3D scene reconstruction with camera poses, a free-roam "god's eye" viewer, AR-style overlays projected onto each video, AI-tagged moments on a shared timeline, click-to-place 3D annotations, and keyframed cinematic fly-throughs. Everything works **from pixels and audio alone** — no depth sensors, no IMU logs. It runs fully offline and local; the only optional cloud piece is moment classification. --- ## What you can do - **Scrub a shared timeline** and watch every camera stay locked to the same instant, aligned by their audio (no manual frame-matching). - **Fly through the reconstructed scene** in 3D — orbit freely, snap to any real camera, or play a keyframed cinematic path. - **See an "x-ray" HUD**: 3D anchor points (stage corners, tagged objects) projected onto every video, tracking even when occluded by crowd or scaffolding. - **Jump between AI-tagged moments** (bass drops, pyro, crowd waves…) on the timeline, and correct any misclassification. - **Click to locate things in 3D**: draw a box around an object in one or two videos and the app triangulates its 3D position, dropping an anchor that appears everywhere at once. - **Hear the scene in 3D** 🎧: toggle spatial audio and every camera becomes a positional sound source at its reconstructed pose — fly toward a camera and you hear *that* spot in the crowd get closer. --- ## Prerequisites | Tool | Required? | Notes | |---|---|---| | **Python 3.11+** + [`uv`](https://docs.astral.sh/uv/) (or venv+pip) | yes | backend | | **ffmpeg** / **ffprobe** on `PATH` | yes | audio extraction, frame sampling, clip cutting | | **Node 18+** | yes | frontend (Vite + Three.js) | | **COLMAP** on `PATH` | optional | 3D reconstruction. Without it the app still runs — you get synced videos + timeline, just no 3D scene. `brew install colmap` on macOS. | | A classifier API key | optional | AI moment labels. Default **Gemini flash** (`GEMINI_API_KEY`; model `gemini-3.1-flash-lite`, override with `FESTIVAL4D_GEMINI_MODEL`); also Claude or any OpenAI-compatible/local/OpenRouter endpoint via `FESTIVAL4D_CLASSIFIER`. Without a key you still get audio-detected candidate moments, just unlabeled. | --- ## Quickstart (synthetic demo — no footage needed) The synthetic fixture generates three fake camera videos of the same fake stage, with known audio offsets and known 3D geometry — so you can see the whole app working end-to-end before you have any real footage. ```bash # 1. backend env uv venv --python 3.12 uv pip install -e ".[dev]" # 2. generate the synthetic project (fake videos + poses + point cloud + seeded events) uv run python -m festival4d synthetic # 3. serve the API (http://127.0.0.1:8000) uv run python -m festival4d serve # 4. in another terminal, run the frontend cd frontend npm install npm run dev # opens http://localhost:5173 (or the next free port) ``` Open the printed URL. You should see the 3D scene with three camera frusta around a stage box, the three synced video players, stage-corner anchors projected onto each video, and colored moment markers on the timeline. --- ## Capture: record straight from webcams & phones Don't have footage yet? Record it with whatever cameras you have — USB webcams, the laptop cam, and any phone with a browser. **No app install.** Cameras *record* clips that feed the normal pipeline (this isn't live streaming — see `docs/ideas.md` for that). This works because of two properties of the design: browser recordings (`.webm`, or `.mp4` on iOS) are already ingestible formats, and **sync is audio-based** — so the devices need **no clock sync and don't even have to start together**. They only have to record the same moment and hear the same sound. ```bash # capture is OPT-IN (it accepts uploads) — enable it explicitly: FESTIVAL4D_CAPTURE=1 uv run python -m festival4d serve ``` Then open **`/capture`** on each device (a 📹 Capture link also appears in the viewer): name the device, pick a camera + mic, hit record, hit stop — the clip uploads into `data/raw/`. The page shows a live "who's shooting what" monitor of every connected camera. When you're done: ```bash uv run python -m festival4d ingest && uv run python -m festival4d sync # then reconstruct / events ``` ### Getting a phone connected (the HTTPS bit) Browsers **only grant camera access on a secure origin**. A phone opening `http://:8000` gets *no camera*, silently. The easiest fix if you use Tailscale — it issues a real cert, so there's nothing to trust manually: ```bash tailscale serve --bg 8000 # prints https://..ts.net ``` Join the phone to the tailnet and open `https://..ts.net/capture`. (Without Tailscale: any HTTPS reverse proxy works; `localhost` is also treated as secure, which is why the laptop's own cameras work with no setup.) ### Capture tips - **Every clip needs audio**, and all cameras must hear the same sound — that's the only thing aligning them. Testing indoors? Play music. - **USB bandwidth is the real limit** on multiple webcams: two 1080p cams on one controller often fails. Drop to 720p or use separate ports/hubs. - Follow the *Shooting tips* below for reconstruction-friendly angles. - Capture routes stay **unmounted** unless `FESTIVAL4D_CAPTURE=1` — a public deployment must never expose an open upload endpoint. ## Real-footage workflow 1. **Shoot / gather** 2–4 videos of the same performance from different positions (see *Shooting tips* below) and drop the files into `data/raw/`. 2. **Run the pipeline:** ```bash uv run python -m festival4d ingest # probe videos, extract 16 kHz mono audio uv run python -m festival4d sync # GCC-PHAT audio alignment (offsets + drift) uv run python -m festival4d reconstruct # COLMAP structure-from-motion → camera poses + point cloud uv run python -m festival4d events # detect audio moments + AI-classify them uv run python -m festival4d serve # serve it ``` Each step is independent and re-runnable. `sync` alone already gives you locked multi-video playback; `reconstruct` adds the 3D scene and overlays; `events` adds the tagged timeline. 3. **Explore** in the browser (`cd frontend && npm run dev`). If COLMAP can't reconstruct your footage (common with dark, motion-blurred, or low-overlap clips), the pipeline says so and leaves you with the synced-video experience — it never corrupts existing data. To enable AI moment labels, set a key before `events`: ```bash export GEMINI_API_KEY=... # default provider (Gemini flash, native video) # or: export FESTIVAL4D_CLASSIFIER=claude ANTHROPIC_API_KEY=... # or: export FESTIVAL4D_CLASSIFIER=local FESTIVAL4D_OPENAI_BASE_URL=... FESTIVAL4D_OPENAI_MODEL=... ``` --- ## Using the viewer | Action | How | |---|---| | Play / pause | `Space` or the ⏵ button | | Nudge time ±1 s (±5 s) | `←` / `→` (hold `Shift`) | | Scrub | drag the timeline | | Pick the audio you hear | click a video (one soundtrack at a time) | | Spatial audio | 🎧 **3D audio** (needs poses): every camera plays from its position in the scene; the mix follows the viewer camera as you fly | | Enable/disable a video | ◉ on the video | | Orbit the 3D scene | drag in the 3D pane | | Snap the 3D view to a camera | click its frustum, its ⛶ button, or press `1`–`9` | | Free roam again | `Esc` / `0` / **Free roam** | | Correct a moment / annotate | click a timeline marker → panel | | Place a 3D anchor | in the panel, **Annotate location**, then drag a box on a video (repeat on a 2nd video to triangulate) | | Add a camera keyframe | **＋ Key** or `K` (captures the current free-roam view at the current time) | | Play the keyframed path | **▶ Path** · export/import with ⤓ / ⤒ | The top-right **sync error** panel shows each video's alignment error in ms while playing — a dev aid; it should stay green (<50 ms) once footage is well-synced. --- ## Architecture ``` data/raw/ ──ingest──▶ data/work/audio ──sync────▶ videos.offset_ms / drift_ppm ┐ │ ──frames──▶ data/work/frames ──reconstruct──▶ camera_poses + points.ply │ reference audio ──events──▶ events (audio candidates → AI labels) │ SQLite (data/project.db) ◀─────────┘ │ FastAPI (backend/festival4d/api.py) │ /api/manifest, /poses, /pointcloud, │ /events, /anchors, /annotations ▼ Vite + Three.js SPA (frontend/src) transport (master clock) · videoGrid + overlays · scene3d · timeline annotate (M8) · camPath (M9) ``` - **Backend** — Python: `ingest`/`audio_sync` (media + GCC-PHAT sync), `frames`/`sfm`/`geometry` (COLMAP orchestration + pose math + triangulation), `events_ai` (moment detection + pluggable classifiers), `resolve` (annotation→3D), `api` (FastAPI, Range-capable video serving), `db` (SQLAlchemy/SQLite), `synthetic` (the test fixture). - **Frontend** — vanilla JS + Three.js. A single master clock (`transport.js`) derives each video's local time from the shared timeline and continuously corrects playback. When possible the clock is **audio**: the selected soundtrack (`GET /api/audio/{id}`, extracted+cached server-side) plays through WebAudio and `t_global` derives from `AudioContext.currentTime` — so every video is muted picture-only and can be seeked/rate-trimmed with zero audible artifacts (falls back to a `performance.now()` clock + `