Festival 4D

Turn multiple fan-shot smartphone videos of the same concert into a synchronized, explorable 4D experience: time-aligned multi-video playback, 3D scene reconstruction with camera poses, a free-roam "god's eye" viewer, AR-style overlays projected onto each video, AI-tagged moments on a shared timeline, click-to-place 3D annotations, and keyframed cinematic fly-throughs.

Everything works from pixels and audio alone — no depth sensors, no IMU logs. It runs fully offline and local; the only optional cloud piece is moment classification.

---

What you can do

• Scrub a shared timeline and watch every camera stay locked to the same instant, aligned by their audio (no manual frame-matching).
• Fly through the reconstructed scene in 3D — orbit freely, snap to any real camera, or play a keyframed cinematic path.
• See an "x-ray" HUD: 3D anchor points (stage corners, tagged objects) projected onto every video, tracking even when occluded by crowd or scaffolding.
• Jump between AI-tagged moments (bass drops, pyro, crowd waves…) on the timeline, and correct any misclassification.
• Click to locate things in 3D: draw a box around an object in one or two videos and the app triangulates its 3D position, dropping an anchor that appears everywhere at once.
• Hear the scene in 3D 🎧: toggle spatial audio and every camera becomes a positional sound source at its reconstructed pose — fly toward a camera and you hear that spot in the crowd get closer.

---

Prerequisites

Tool	Required?	Notes
Python 3.11+ + uv (or venv+pip)	yes	backend
ffmpeg / ffprobe on PATH	yes	audio extraction, frame sampling, clip cutting
Node 18+	yes	frontend (Vite + Three.js)
COLMAP on PATH	optional	3D reconstruction. Without it the app still runs — you get synced videos + timeline, just no 3D scene. brew install colmap on macOS.
A classifier API key	optional	AI moment labels. Default Gemini flash (GEMINI_API_KEY; model gemini-3.1-flash-lite, override with FESTIVAL4D_GEMINI_MODEL); also Claude or any OpenAI-compatible/local/OpenRouter endpoint via FESTIVAL4D_CLASSIFIER. Without a key you still get audio-detected candidate moments, just unlabeled.

---

Quickstart (synthetic demo — no footage needed)

The synthetic fixture generates three fake camera videos of the same fake stage, with known audio offsets and known 3D geometry — so you can see the whole app working end-to-end before you have any real footage.

# 1. backend env
uv venv --python 3.12
uv pip install -e ".[dev]"

# 2. generate the synthetic project (fake videos + poses + point cloud + seeded events)
uv run python -m festival4d synthetic

# 3. serve the API (http://127.0.0.1:8000)
uv run python -m festival4d serve

# 4. in another terminal, run the frontend
cd frontend
npm install
npm run dev            # opens http://localhost:5173 (or the next free port)

Open the printed URL. You should see the 3D scene with three camera frusta around a stage box, the three synced video players, stage-corner anchors projected onto each video, and colored moment markers on the timeline.

---

Capture: record straight from webcams & phones

Don't have footage yet? Record it with whatever cameras you have — USB webcams, the laptop cam, and any phone with a browser. No app install. Cameras record clips that feed the normal pipeline (this isn't live streaming — see docs/ideas.md for that).

This works because of two properties of the design: browser recordings (.webm, or .mp4 on iOS) are already ingestible formats, and sync is audio-based — so the devices need no clock sync and don't even have to start together. They only have to record the same moment and hear the same sound.

# capture is OPT-IN (it accepts uploads) — enable it explicitly:
FESTIVAL4D_CAPTURE=1 uv run python -m festival4d serve

Then open /capture on each device (a 📹 Capture link also appears in the viewer): name the device, pick a camera + mic, hit record, hit stop — the clip uploads into data/raw/. The page shows a live "who's shooting what" monitor of every connected camera. When you're done:

uv run python -m festival4d ingest && uv run python -m festival4d sync   # then reconstruct / events

Getting a phone connected (the HTTPS bit)

Browsers only grant camera access on a secure origin. A phone opening http://<lan-ip>:8000 gets no camera, silently. The easiest fix if you use Tailscale — it issues a real cert, so there's nothing to trust manually:

tailscale serve --bg 8000        # prints https://<machine>.<tailnet>.ts.net

Join the phone to the tailnet and open https://<machine>.<tailnet>.ts.net/capture. (Without Tailscale: any HTTPS reverse proxy works; localhost is also treated as secure, which is why the laptop's own cameras work with no setup.)

Capture tips

• Every clip needs audio, and all cameras must hear the same sound — that's the only thing aligning them. Testing indoors? Play music.
• USB bandwidth is the real limit on multiple webcams: two 1080p cams on one controller often fails. Drop to 720p or use separate ports/hubs.
• Follow the Shooting tips below for reconstruction-friendly angles.
• Capture routes stay unmounted unless FESTIVAL4D_CAPTURE=1 — a public deployment must never expose an open upload endpoint.

Real-footage workflow

1. Shoot / gather 2–4 videos of the same performance from different positions (see Shooting tips below) and drop the files into data/raw/.

2. Run the pipeline:

   uv run python -m festival4d ingest        # probe videos, extract 16 kHz mono audio
   uv run python -m festival4d sync          # GCC-PHAT audio alignment (offsets + drift)
   uv run python -m festival4d reconstruct   # COLMAP structure-from-motion → camera poses + point cloud
   uv run python -m festival4d events        # detect audio moments + AI-classify them
   uv run python -m festival4d serve         # serve it
   

   Each step is independent and re-runnable. sync alone already gives you locked multi-video playback; reconstruct adds the 3D scene and overlays; events adds the tagged timeline.

3. Explore in the browser (cd frontend && npm run dev).

If COLMAP can't reconstruct your footage (common with dark, motion-blurred, or low-overlap clips), the pipeline says so and leaves you with the synced-video experience — it never corrupts existing data.

To enable AI moment labels, set a key before events:

export GEMINI_API_KEY=...             # default provider (Gemini flash, native video)
# or: export FESTIVAL4D_CLASSIFIER=claude  ANTHROPIC_API_KEY=...
# or: export FESTIVAL4D_CLASSIFIER=local   FESTIVAL4D_OPENAI_BASE_URL=... FESTIVAL4D_OPENAI_MODEL=...

---

Using the viewer

Action	How
Play / pause	Space or the ⏵ button
Nudge time ±1 s (±5 s)	← / → (hold Shift)
Scrub	drag the timeline
Pick the audio you hear	click a video (one soundtrack at a time)
Spatial audio	🎧 3D audio (needs poses): every camera plays from its position in the scene; the mix follows the viewer camera as you fly
Enable/disable a video	◉ on the video
Orbit the 3D scene	drag in the 3D pane
Snap the 3D view to a camera	click its frustum, its ⛶ button, or press 1–9
Free roam again	Esc / 0 / Free roam
Correct a moment / annotate	click a timeline marker → panel
Place a 3D anchor	in the panel, Annotate location, then drag a box on a video (repeat on a 2nd video to triangulate)
Add a camera keyframe	＋ Key or K (captures the current free-roam view at the current time)
Play the keyframed path	▶ Path · export/import with ⤓ / ⤒

The top-right sync error panel shows each video's alignment error in ms while playing — a dev aid; it should stay green (<50 ms) once footage is well-synced.

---

Architecture

data/raw/ ──ingest──▶ data/work/audio ──sync────▶ videos.offset_ms / drift_ppm  ┐
                                                                                 │
          ──frames──▶ data/work/frames ──reconstruct──▶ camera_poses + points.ply
                                                                                 │
                        reference audio ──events──▶ events (audio candidates → AI labels)
                                                                                 │
                                              SQLite (data/project.db) ◀─────────┘
                                                        │
                                              FastAPI  (backend/festival4d/api.py)
                                                        │  /api/manifest, /poses, /pointcloud,
                                                        │  /events, /anchors, /annotations
                                                        ▼
                                        Vite + Three.js SPA (frontend/src)
             transport (master clock) · videoGrid + overlays · scene3d · timeline
                                        annotate (M8) · camPath (M9)

• Backend — Python: ingest/audio_sync (media + GCC-PHAT sync), frames/sfm/geometry (COLMAP orchestration + pose math + triangulation), events_ai (moment detection + pluggable classifiers), resolve (annotation→3D), api (FastAPI, Range-capable video serving), db (SQLAlchemy/SQLite), synthetic (the test fixture).
• Frontend — vanilla JS + Three.js. A single master clock (transport.js) derives each video's local time from the shared timeline and continuously corrects playback. When possible the clock is audio: the selected soundtrack (GET /api/audio/{id}, extracted+cached server-side) plays through WebAudio and t_global derives from AudioContext.currentTime — so every video is muted picture-only and can be seeked/rate-trimmed with zero audible artifacts (falls back to a performance.now() clock + <video> audio if the track can't load). lib/pose.js is the one COLMAP→Three.js pose conversion, mirrored from geometry.py and locked by a shared test.
• Timebase contract: t_video = (t_global − offset_ms/1000) · (1 + drift_ppm·1e−6); the reference video has offset 0.

Full design + milestone history: OPUS_BUILD_INSTRUCTIONS.md; the parallel build plan and per-lane status live in plan/.

---

Shooting tips (for good reconstructions)

Concert footage is genuinely hard for structure-from-motion. To give COLMAP a chance:

• Spread the cameras out but keep overlapping views — each pair of cameras should see some of the same stage/structure. No overlap → they can't be related in 3D.
• Keep some static structure in frame (stage edges, truss, speaker stacks). A frame that's all moving crowd and lights has nothing stable to triangulate.
• Avoid pure zoom — physically moving parallax reconstructs far better than zooming.
• Brighter, sharper is better — motion blur and near-dark frames are the main failure cause.
• 2–4 phones is the sweet spot for a first reconstruction; more is fine but slower.

Audio sync is far more forgiving: any clips that share audible sound (the same music/claps) will align, even across otherwise-unrelated angles.

---

Tests

uv run pytest            # backend
cd frontend && npm run build   # frontend typecheck/build

---

Non-goals (this prototype)

No accounts/auth, no cloud storage, no realtime ingest, no NeRF/Gaussian splatting, no mobile UI, no Docker. Single local user, one project at a time.