8dca2f5154
Google retired gemini-2.5-flash for new API keys (404). Verified gemini-3.1-flash-lite live: inline video + structured output, 3/3 synthetic candidates classified. FESTIVAL4D_GEMINI_MODEL overrides. Round 3 addendum documents the local .env (gitignored) with Gemini + OpenRouter creds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
190 lines
8.8 KiB
Markdown
190 lines
8.8 KiB
Markdown
# Festival 4D
|
||
|
||
Turn multiple fan-shot smartphone videos of the same concert into a synchronized,
|
||
explorable **4D experience**: time-aligned multi-video playback, 3D scene reconstruction
|
||
with camera poses, a free-roam "god's eye" viewer, AR-style overlays projected onto each
|
||
video, AI-tagged moments on a shared timeline, click-to-place 3D annotations, and keyframed
|
||
cinematic fly-throughs.
|
||
|
||
Everything works **from pixels and audio alone** — no depth sensors, no IMU logs. It runs
|
||
fully offline and local; the only optional cloud piece is moment classification.
|
||
|
||
---
|
||
|
||
## What you can do
|
||
|
||
- **Scrub a shared timeline** and watch every camera stay locked to the same instant, aligned
|
||
by their audio (no manual frame-matching).
|
||
- **Fly through the reconstructed scene** in 3D — orbit freely, snap to any real camera, or
|
||
play a keyframed cinematic path.
|
||
- **See an "x-ray" HUD**: 3D anchor points (stage corners, tagged objects) projected onto every
|
||
video, tracking even when occluded by crowd or scaffolding.
|
||
- **Jump between AI-tagged moments** (bass drops, pyro, crowd waves…) on the timeline, and
|
||
correct any misclassification.
|
||
- **Click to locate things in 3D**: draw a box around an object in one or two videos and the
|
||
app triangulates its 3D position, dropping an anchor that appears everywhere at once.
|
||
|
||
---
|
||
|
||
## Prerequisites
|
||
|
||
| Tool | Required? | Notes |
|
||
|---|---|---|
|
||
| **Python 3.11+** + [`uv`](https://docs.astral.sh/uv/) (or venv+pip) | yes | backend |
|
||
| **ffmpeg** / **ffprobe** on `PATH` | yes | audio extraction, frame sampling, clip cutting |
|
||
| **Node 18+** | yes | frontend (Vite + Three.js) |
|
||
| **COLMAP** on `PATH` | optional | 3D reconstruction. Without it the app still runs — you get synced videos + timeline, just no 3D scene. `brew install colmap` on macOS. |
|
||
| A classifier API key | optional | AI moment labels. Default **Gemini flash** (`GEMINI_API_KEY`; model `gemini-3.1-flash-lite`, override with `FESTIVAL4D_GEMINI_MODEL`); also Claude or any OpenAI-compatible/local/OpenRouter endpoint via `FESTIVAL4D_CLASSIFIER`. Without a key you still get audio-detected candidate moments, just unlabeled. |
|
||
|
||
---
|
||
|
||
## Quickstart (synthetic demo — no footage needed)
|
||
|
||
The synthetic fixture generates three fake camera videos of the same fake stage, with known
|
||
audio offsets and known 3D geometry — so you can see the whole app working end-to-end before
|
||
you have any real footage.
|
||
|
||
```bash
|
||
# 1. backend env
|
||
uv venv --python 3.12
|
||
uv pip install -e ".[dev]"
|
||
|
||
# 2. generate the synthetic project (fake videos + poses + point cloud + seeded events)
|
||
uv run python -m festival4d synthetic
|
||
|
||
# 3. serve the API (http://127.0.0.1:8000)
|
||
uv run python -m festival4d serve
|
||
|
||
# 4. in another terminal, run the frontend
|
||
cd frontend
|
||
npm install
|
||
npm run dev # opens http://localhost:5173 (or the next free port)
|
||
```
|
||
|
||
Open the printed URL. You should see the 3D scene with three camera frusta around a stage box,
|
||
the three synced video players, stage-corner anchors projected onto each video, and colored
|
||
moment markers on the timeline.
|
||
|
||
---
|
||
|
||
## Real-footage workflow
|
||
|
||
1. **Shoot / gather** 2–4 videos of the same performance from different positions (see
|
||
*Shooting tips* below) and drop the files into `data/raw/`.
|
||
2. **Run the pipeline:**
|
||
|
||
```bash
|
||
uv run python -m festival4d ingest # probe videos, extract 16 kHz mono audio
|
||
uv run python -m festival4d sync # GCC-PHAT audio alignment (offsets + drift)
|
||
uv run python -m festival4d reconstruct # COLMAP structure-from-motion → camera poses + point cloud
|
||
uv run python -m festival4d events # detect audio moments + AI-classify them
|
||
uv run python -m festival4d serve # serve it
|
||
```
|
||
|
||
Each step is independent and re-runnable. `sync` alone already gives you locked multi-video
|
||
playback; `reconstruct` adds the 3D scene and overlays; `events` adds the tagged timeline.
|
||
3. **Explore** in the browser (`cd frontend && npm run dev`).
|
||
|
||
If COLMAP can't reconstruct your footage (common with dark, motion-blurred, or low-overlap
|
||
clips), the pipeline says so and leaves you with the synced-video experience — it never
|
||
corrupts existing data.
|
||
|
||
To enable AI moment labels, set a key before `events`:
|
||
|
||
```bash
|
||
export GEMINI_API_KEY=... # default provider (Gemini flash, native video)
|
||
# or: export FESTIVAL4D_CLASSIFIER=claude ANTHROPIC_API_KEY=...
|
||
# or: export FESTIVAL4D_CLASSIFIER=local FESTIVAL4D_OPENAI_BASE_URL=... FESTIVAL4D_OPENAI_MODEL=...
|
||
```
|
||
|
||
---
|
||
|
||
## Using the viewer
|
||
|
||
| Action | How |
|
||
|---|---|
|
||
| Play / pause | `Space` or the ⏵ button |
|
||
| Nudge time ±1 s (±5 s) | `←` / `→` (hold `Shift`) |
|
||
| Scrub | drag the timeline |
|
||
| Pick the audio you hear | click a video (only one is unmuted at a time) |
|
||
| Enable/disable a video | ◉ on the video |
|
||
| Orbit the 3D scene | drag in the 3D pane |
|
||
| Snap the 3D view to a camera | click its frustum, its ⛶ button, or press `1`–`9` |
|
||
| Free roam again | `Esc` / `0` / **Free roam** |
|
||
| Correct a moment / annotate | click a timeline marker → panel |
|
||
| Place a 3D anchor | in the panel, **Annotate location**, then drag a box on a video (repeat on a 2nd video to triangulate) |
|
||
| Add a camera keyframe | **＋ Key** or `K` (captures the current free-roam view at the current time) |
|
||
| Play the keyframed path | **▶ Path** · export/import with ⤓ / ⤒ |
|
||
|
||
The top-right **sync error** panel shows each video's alignment error in ms while playing —
|
||
a dev aid; it should stay green (<50 ms) once footage is well-synced.
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
```
|
||
data/raw/ ──ingest──▶ data/work/audio ──sync────▶ videos.offset_ms / drift_ppm ┐
|
||
│
|
||
──frames──▶ data/work/frames ──reconstruct──▶ camera_poses + points.ply
|
||
│
|
||
reference audio ──events──▶ events (audio candidates → AI labels)
|
||
│
|
||
SQLite (data/project.db) ◀─────────┘
|
||
│
|
||
FastAPI (backend/festival4d/api.py)
|
||
│ /api/manifest, /poses, /pointcloud,
|
||
│ /events, /anchors, /annotations
|
||
▼
|
||
Vite + Three.js SPA (frontend/src)
|
||
transport (master clock) · videoGrid + overlays · scene3d · timeline
|
||
annotate (M8) · camPath (M9)
|
||
```
|
||
|
||
- **Backend** — Python: `ingest`/`audio_sync` (media + GCC-PHAT sync), `frames`/`sfm`/`geometry`
|
||
(COLMAP orchestration + pose math + triangulation), `events_ai` (moment detection + pluggable
|
||
classifiers), `resolve` (annotation→3D), `api` (FastAPI, Range-capable video serving), `db`
|
||
(SQLAlchemy/SQLite), `synthetic` (the test fixture).
|
||
- **Frontend** — vanilla JS + Three.js. A single master clock (`transport.js`) derives each
|
||
video's local time from the shared timeline and continuously corrects playback; `lib/pose.js`
|
||
is the one COLMAP→Three.js pose conversion, mirrored from `geometry.py` and locked by a shared
|
||
test.
|
||
- **Timebase contract:** `t_video = (t_global − offset_ms/1000) · (1 + drift_ppm·1e−6)`; the
|
||
reference video has offset 0.
|
||
|
||
Full design + milestone history: [`OPUS_BUILD_INSTRUCTIONS.md`](OPUS_BUILD_INSTRUCTIONS.md);
|
||
the parallel build plan and per-lane status live in [`plan/`](plan/).
|
||
|
||
---
|
||
|
||
## Shooting tips (for good reconstructions)
|
||
|
||
Concert footage is genuinely hard for structure-from-motion. To give COLMAP a chance:
|
||
|
||
- **Spread the cameras out** but keep **overlapping views** — each pair of cameras should see
|
||
some of the same stage/structure. No overlap → they can't be related in 3D.
|
||
- **Keep some static structure in frame** (stage edges, truss, speaker stacks). A frame that's
|
||
all moving crowd and lights has nothing stable to triangulate.
|
||
- **Avoid pure zoom** — physically moving parallax reconstructs far better than zooming.
|
||
- **Brighter, sharper is better** — motion blur and near-dark frames are the main failure cause.
|
||
- **2–4 phones is the sweet spot** for a first reconstruction; more is fine but slower.
|
||
|
||
Audio sync is far more forgiving: any clips that share audible sound (the same music/claps)
|
||
will align, even across otherwise-unrelated angles.
|
||
|
||
---
|
||
|
||
## Tests
|
||
|
||
```bash
|
||
uv run pytest # backend
|
||
cd frontend && npm run build # frontend typecheck/build
|
||
```
|
||
|
||
---
|
||
|
||
## Non-goals (this prototype)
|
||
|
||
No accounts/auth, no cloud storage, no realtime ingest, no NeRF/Gaussian splatting, no mobile
|
||
UI, no Docker. Single local user, one project at a time.
|