230 lines
22 KiB
Markdown
230 lines
22 KiB
Markdown
# Festival 4D — Build Instructions
|
||
|
||
**Audience:** Claude Opus 4.8 executing autonomously in this repo (`/Users/m3ultra/Documents/festifun`).
|
||
**Goal:** Turn multiple fan-shot smartphone videos of the same concert into a synchronized, explorable 4D experience: time-aligned multi-video playback, 3D scene reconstruction with camera poses, a free-roam "god's eye" viewer, AR-style overlays projected onto each video, and AI-tagged moments on a shared timeline.
|
||
|
||
This is an entertainment/replay project (same family as sports "bullet time" replay). Keep all naming, prompts, and UI copy in the concert domain.
|
||
|
||
---
|
||
|
||
> **Parallel execution:** this spec is the canonical reference; the work is organized into lanes for multiple concurrent agents — see [`plan/README.md`](plan/README.md) for the phase map, branch protocol, file ownership, and your lane brief. If you were given a lane brief, it overrides the serial ordering note below (your lane's milestones still apply verbatim).
|
||
|
||
## 0. Ground rules
|
||
|
||
- **Build milestones in dependency order.** Each milestone has acceptance criteria; do not start the next until the current one passes. When executing serially (single agent), go M0→M9; when executing in lanes, follow `plan/`.
|
||
- **Work without real footage.** The user may not have multi-camera concert videos yet. Every milestone must be verifiable with the synthetic fixtures described in M0. When real footage arrives, the same pipeline runs on it unchanged.
|
||
- **Prefer boring tech.** SQLite (not PostgreSQL — this is a local, single-user, offline-first tool; do not add a DB server dependency). Vanilla JS + Three.js via Vite (no React). FastAPI + uvicorn. Python 3.11+, managed with `uv` if available, else venv + pip.
|
||
- **COLMAP is an external binary.** Detect it at runtime (`shutil.which("colmap")`); if missing, print install instructions (`brew install colmap` on macOS) and — critically — the whole app must still run using the synthetic-poses fixture. Never make COLMAP a hard import-time dependency.
|
||
- **Commit per milestone** with a message describing what now works. Initialize git first (`git init`).
|
||
|
||
## 1. Repository layout
|
||
|
||
```
|
||
festifun/
|
||
├── README.md # setup + usage, written last (M9)
|
||
├── pyproject.toml # backend deps: fastapi, uvicorn, numpy, scipy, librosa,
|
||
│ # soundfile, sqlalchemy, pydantic, opencv-python-headless,
|
||
│ # google-genai, anthropic, openai (last three = classifier providers)
|
||
├── backend/
|
||
│ ├── festival4d/
|
||
│ │ ├── __init__.py
|
||
│ │ ├── cli.py # `python -m festival4d <cmd>`: ingest | sync | reconstruct | events | serve
|
||
│ │ ├── config.py # paths, constants
|
||
│ │ ├── db.py # SQLAlchemy engine/session, schema
|
||
│ │ ├── ingest.py # probe videos (ffprobe), extract audio, register in DB
|
||
│ │ ├── audio_sync.py # GCC-PHAT pairwise offsets + global solve
|
||
│ │ ├── frames.py # sharpness-aware frame sampling for SfM
|
||
│ │ ├── sfm.py # COLMAP orchestration, model export, normalization, interpolation
|
||
│ │ ├── geometry.py # quaternion slerp, ray casting, triangulation, coord conversions
|
||
│ │ ├── events_ai.py # audio candidate detection + Claude classification
|
||
│ │ ├── api.py # FastAPI app
|
||
│ │ └── synthetic.py # test-fixture generator (see M0)
|
||
│ └── tests/ # pytest; every numeric module gets a test
|
||
├── frontend/
|
||
│ ├── index.html
|
||
│ ├── package.json # vite + three
|
||
│ └── src/
|
||
│ ├── main.js
|
||
│ ├── state.js # global store: manifest, playhead, playing, selected camera
|
||
│ ├── transport.js # master clock + per-video sync (M4)
|
||
│ ├── videoGrid.js # <video> grid + overlay canvases
|
||
│ ├── scene3d.js # Three.js scene: point cloud, frusta, orbit controls
|
||
│ ├── overlays.js # 3D→2D anchor projection (M6)
|
||
│ ├── timeline.js # scrubber + event markers
|
||
│ ├── annotate.js # bbox drawing + submission (M8)
|
||
│ └── camPath.js # keyframed camera paths (M9)
|
||
└── data/
|
||
├── raw/ # user drops source videos here (gitignored)
|
||
├── work/ # extracted wavs, frames, colmap workspace (gitignored)
|
||
└── project.db # SQLite (gitignored)
|
||
```
|
||
|
||
## 2. Data model (SQLite via SQLAlchemy)
|
||
|
||
```
|
||
videos(id, filename, duration_s, fps, width, height, offset_ms REAL, drift_ppm REAL,
|
||
sync_confidence REAL, created_at)
|
||
camera_poses(id, video_id FK, frame_idx INT, t_video_s REAL,
|
||
qw,qx,qy,qz, tx,ty,tz, -- COLMAP convention: world→camera
|
||
fx,fy,cx,cy, -- pinhole intrinsics after undistortion
|
||
registered BOOL) -- False = interpolated, not solved
|
||
anchors(id, label, x,y,z, color) -- 3D points to overlay (stage corners etc.)
|
||
events(id, t_global_s REAL, duration_s REAL, event_type TEXT, confidence REAL,
|
||
description TEXT, source TEXT) -- source: 'audio_auto' | 'ai' | 'user'
|
||
annotations(id, video_id FK, t_video_s REAL, x0,y0,x1,y1, -- normalized [0,1] bbox
|
||
resolved_anchor_id FK NULL)
|
||
```
|
||
|
||
Timebase convention (write this in `config.py` docstring and stick to it everywhere):
|
||
**`t_global` is the master timeline in seconds.** For video *v*: `t_video = (t_global - offset_ms/1000) * (1 + drift_ppm * 1e-6)`. The reference video has offset 0.
|
||
|
||
## 3. Milestones
|
||
|
||
### M0 — Scaffold + synthetic fixtures
|
||
- Repo layout above, deps installed, `python -m festival4d --help` works, `pytest` runs, Vite dev server serves a hello page.
|
||
- `synthetic.py` generates a full fake project so every later layer is testable without footage:
|
||
- **Fake videos:** use ffmpeg to render 3 short (20 s, 640×360, 30 fps) clips of the *same* procedurally animated scene from different angles — simplest: a `testsrc2`/`drawtext` composite is fine, but each clip must share a common audio track (a music-like WAV you synthesize with numpy: beat pulses + tones) *shifted by known offsets* (e.g. 0 ms, +1370 ms, −842 ms) and with distinct visuals. Store ground-truth offsets in a JSON next to them.
|
||
- **Fake poses:** 3 camera trajectories on arcs around origin looking at a fake "stage" (a box of 3D points at the origin) + a sparse point cloud (random points on the box + ground plane). Write them into `camera_poses` and a `points.ply` in the exact same format the real COLMAP path will produce.
|
||
- **Accept:** `python -m festival4d synthetic` populates `data/raw` + DB; tests pass.
|
||
|
||
### M1 — Ingest + audio sync
|
||
- `ingest`: ffprobe each file in `data/raw`, insert into `videos`, extract mono 16 kHz WAV per video (ffmpeg) into `data/work/audio/`.
|
||
- `sync` (in `audio_sync.py`):
|
||
- **Use GCC-PHAT, not raw cross-correlation.** Concert audio is loud, clipped, and reverberant; PHAT weighting (whiten the cross-spectrum: `R = X·conj(Y); R /= |R|+eps; r = irfft(R)`) is far more robust. Implement with numpy/scipy FFTs; librosa only for loading.
|
||
- Compute **all pairwise offsets**, not just first-vs-rest. Score each pair by peak-to-second-peak ratio. Then solve globally: offsets are edges in a graph, per-video offsets are node values; least-squares solve the overdetermined system, weight by confidence, and reject pairs that violate cycle consistency (`off_ab + off_bc − off_ac` > 50 ms).
|
||
- **Drift check:** re-run GCC-PHAT on 10 s windows every 30 s along the overlap; fit a line through window-offsets. Store slope as `drift_ppm`. Phone clocks drift tens of ppm — over a 10-minute set that's audible/visible. If |drift| < 5 ppm store 0.
|
||
- Persist `offset_ms`, `drift_ppm`, `sync_confidence` to DB and export `data/work/sync.json`.
|
||
- **Tests:** synthesize one signal, shift copies by known amounts (+ add noise + small resample to simulate drift), assert recovered offsets within ±10 ms and drift within ±3 ppm. Also run against the M0 fixture and assert the known ground-truth offsets.
|
||
- **Accept:** sync on synthetic clips recovers ground truth.
|
||
|
||
### M2 — Reconstruction (COLMAP) + pose export
|
||
- `frames.py`: sample candidate frames ~2/s per video; within each 0.5 s window keep the sharpest frame (variance of Laplacian via OpenCV). Write JPEGs named `{video_id}_{frame_idx}.jpg`.
|
||
- `sfm.py`: drive the COLMAP CLI via subprocess:
|
||
1. `feature_extractor` — one camera per video (`--ImageReader.single_camera_per_folder 1`, put each video's frames in its own subfolder), `OPENCV` camera model.
|
||
2. `sequential_matcher` within each video + `exhaustive_matcher` across (frame counts here are small; exhaustive is fine at this scale).
|
||
3. `mapper` (incremental reconstruction), then `image_undistorter`/`model_converter` to get a TXT model.
|
||
4. Parse `images.txt` / `cameras.txt` / `points3D.txt` yourself (formats are documented in COLMAP docs; simple line parsing — no pycolmap dependency needed).
|
||
- **Normalize the scene** before storing: translate so the point-cloud centroid is at origin; scale so the camera-position bounding sphere has radius 10; rotate so world-up = the average of camera up-vectors (phones are held roughly upright — this heuristic works well; write it as such in a comment). Apply the same similarity transform to poses and points.
|
||
- **Interpolate poses** for unregistered frames between registered ones: slerp rotation, lerp translation, mark `registered=False`; do not extrapolate beyond first/last registered frame of each video.
|
||
- Export `data/work/points.ply` (binary little-endian PLY) and fill `camera_poses`.
|
||
- **Failure handling:** if COLMAP registers < 60 % of frames or < 2 videos into one model, print a clear diagnostic (likely causes: too-dark footage, motion blur, no view overlap) and leave the synthetic/previous poses untouched. Concert footage is genuinely hard for SfM (moving lights, crowds, blur) — the app must degrade gracefully to "synced videos, no 3D".
|
||
- **Tests:** parser unit tests on hand-written COLMAP TXT snippets; normalization test (centroid≈0, radius≈10, up≈+Y); slerp test.
|
||
- **Accept:** on the synthetic fixture (which skips COLMAP and injects known poses) the whole export path runs; if COLMAP is installed, the pipeline runs end-to-end on any real folder of images without crashing.
|
||
|
||
### M3 — API
|
||
FastAPI app (`api.py`), CORS enabled for the Vite origin, mounted static serving of `data/raw` **with HTTP Range support** (Starlette's `StaticFiles` supports Range — required or `<video>` seeking breaks).
|
||
|
||
```
|
||
GET /api/manifest → {videos:[{id,filename,url,duration_s,fps,width,height,
|
||
offset_ms,drift_ppm}], t_global_max, has_poses}
|
||
GET /api/videos/{id}/poses → [{frame_idx,t_video_s,q:[w,x,y,z],t:[x,y,z],
|
||
intrinsics:{fx,fy,cx,cy},registered}]
|
||
GET /api/pointcloud → points.ply file
|
||
GET /api/anchors → [...] POST /api/anchors
|
||
GET /api/events → [...]
|
||
POST /api/events/detect {t_global_s?, window_s?} → runs M7 detection (or on one window)
|
||
POST /api/annotations {video_id, t_video_s, bbox:[x0,y0,x1,y1]} → {anchor_id?, point?} (M8)
|
||
```
|
||
`cli.py serve` runs uvicorn. **Accept:** with the synthetic project, `curl` of each endpoint returns sane data; a browser can seek the served videos.
|
||
|
||
### M4 — Synchronized multi-video playback (the heart of the app)
|
||
- Layout: left = 3D canvas placeholder, right = grid of `<video>` elements (one per video), bottom = timeline scrubber + play/pause + speed control.
|
||
- **Master clock** in `transport.js`: `t_global = t0 + (performance.now() - wall0)/1000 * rate` while playing. Never trust any single video element as the clock.
|
||
- Per video, each animation frame (use `requestVideoFrameCallback` where available, else rAF): compute target `t_video` from the timebase formula; compare to `video.currentTime`:
|
||
- error > 150 ms → hard seek (`fastSeek` if available);
|
||
- 20–150 ms → nudge `playbackRate` (clamp to master rate ×[0.95, 1.05]) until converged;
|
||
- < 20 ms → `playbackRate = rate`.
|
||
- Videos whose `t_video` falls outside `[0, duration]` pause and dim (they weren't recording at that global time).
|
||
- Audio: only one video unmuted at a time (click to select "audio source"); all others muted — otherwise phasing chaos.
|
||
- **Accept:** with the 3 synthetic clips (known offsets), scrubbing and playback keep all clips visibly locked to the shared audio events; drift error indicator (dev overlay showing per-video sync error in ms) stays < 50 ms during continuous playback.
|
||
|
||
### M5 — 3D viewer
|
||
- `scene3d.js`: load `/api/pointcloud` (PLY via three/examples PLYLoader) as a `Points` cloud; draw each video's camera path as a line and current pose as a frustum wireframe (colored per video); OrbitControls for free roam.
|
||
- **Coordinate conversion — do this exactly, it's the classic pitfall.** COLMAP stores world→camera: `x_cam = R x_world + t`, camera axes = (+x right, +y down, +z forward). Three.js cameras look down −z with +y up. Per pose:
|
||
```
|
||
R = quat_to_mat(q) // world→cam
|
||
C = -Rᵀ t // camera center in world coords
|
||
R_c2w = Rᵀ
|
||
R_three = R_c2w * diag(1, -1, -1) // flip y,z of camera-local axes
|
||
camera.position = C; camera.setRotationFromMatrix(R_three)
|
||
```
|
||
Implement once in a shared helper with a unit test in the backend mirror (`geometry.py`) so both sides agree.
|
||
- On each playhead tick, update frusta from the pose at the nearest `t_video` (poses are keyed by frame; binary-search + use interpolated entries).
|
||
- **Snap-to-camera:** clicking a frustum (or a per-video button) tweens the viewer camera to that video's pose and keeps following it while playing; intrinsics → `PerspectiveCamera`: `fov = 2·atan(H/(2·fy))·180/π`, `aspect = W/H` (assume centered principal point for the prototype). "Free roam" button detaches.
|
||
- **Accept:** on synthetic data, orbiting shows 3 camera arcs around the stage box; snap-to-camera view of camera A roughly matches what video A shows (verifiable exactly on synthetic data since it's generated from those poses).
|
||
|
||
### M6 — Anchor overlays ("x-ray" HUD)
|
||
- Each `<video>` gets a transparent `<canvas>` overlay, resolution-matched to the displayed video size (handle devicePixelRatio + object-fit letterboxing math carefully — compute the video's content rect inside the element).
|
||
- For each anchor and each visible video: build the Three.js camera for that video's *current* pose (M5 helper), `anchor.project(camera)`, convert NDC → canvas px, skip if behind camera (`vec.z > 1` after project, or check the w-divide sign properly by transforming to camera space first and requiring `z_cam < 0`).
|
||
- Draw a dot + label; anchors render regardless of real-world occlusion — that *is* the x-ray feature. Style: small circle, 1 px white halo, label text.
|
||
- Seed default anchors on the synthetic stage (4 box corners named "Stage FL/FR/BL/BR").
|
||
- **Accept:** on synthetic data, the projected corners track the drawn stage box across all 3 videos while playing (exact on synthetic since geometry is ground truth).
|
||
|
||
### M7 — Moment detection + AI classification
|
||
Two stages — cheap audio candidates first, then vision LLM on candidates only:
|
||
- **Candidates (`events_ai.py`):** on the reference video's audio compute RMS energy and spectral-flux onset strength (librosa); find peaks that are local maxima over a 10 s neighborhood and above the 90th percentile → candidate `t_global` moments (bass drops, pyro bangs, crowd roars). Insert as `events(source='audio_auto', event_type='candidate')`.
|
||
- **Classification — pluggable provider (`events_ai.py`).** The shared contract is a Pydantic model and a tiny protocol; providers are interchangeable and selected by env var:
|
||
```python
|
||
class MomentClassification(BaseModel):
|
||
event_type: Literal["bass_drop","pyro","confetti","crowd_wave",
|
||
"artist_moment","light_show","quiet_moment","other"]
|
||
confidence: float # 0..1
|
||
description: str # one sentence, human-readable
|
||
|
||
class MomentClassifier(Protocol):
|
||
def classify(self, clip_path: Path, frames: list[Path]) -> MomentClassification: ...
|
||
```
|
||
Selection: `FESTIVAL4D_CLASSIFIER=gemini|claude|local` (default `gemini`). Inputs prepared once per candidate: a 3 s MP4 clip around `t_global` from the best-covering video (ffmpeg, ≤720p, keep it under ~15 MB) **and** 6 sampled JPEGs (≤768 px long edge) for providers without video input.
|
||
|
||
1. **`GeminiClassifier` (default — cheapest, native video).** `google-genai` SDK, model `gemini-2.5-flash`, needs `GEMINI_API_KEY`. Send the clip inline (video bytes are supported directly for small files) and enforce structured output:
|
||
```python
|
||
from google import genai
|
||
client = genai.Client() # reads GEMINI_API_KEY
|
||
resp = client.models.generate_content(
|
||
model="gemini-2.5-flash",
|
||
contents=[
|
||
genai.types.Part.from_bytes(data=clip_bytes, mime_type="video/mp4"),
|
||
"This is a 3-second concert crowd clip around a detected audio spike. Classify the moment.",
|
||
],
|
||
config={"response_mime_type": "application/json",
|
||
"response_schema": MomentClassification},
|
||
)
|
||
result = MomentClassification.model_validate_json(resp.text)
|
||
```
|
||
2. **`ClaudeClassifier`** (optional, `ANTHROPIC_API_KEY`): Anthropic Python SDK, model `claude-opus-4-8`, the 6 frames as base64 image blocks, structured output via `client.messages.parse(..., thinking={"type":"adaptive"}, output_format=MomentClassification)` → `resp.parsed_output`. Do **not** pass `temperature` (rejected on Opus 4.8).
|
||
3. **`LocalClassifier`** (offline/free): any OpenAI-compatible endpoint — Ollama/LM Studio with a vision model (e.g. `qwen2.5-vl`), or **OpenRouter** by just pointing the same class at `https://openrouter.ai/api/v1`. Config: `FESTIVAL4D_OPENAI_BASE_URL`, `FESTIVAL4D_OPENAI_KEY`, `FESTIVAL4D_OPENAI_MODEL`. Send the 6 frames as image_url/base64 parts, request JSON, validate with `MomentClassification.model_validate_json` and one retry on validation failure (local models are sloppier about JSON).
|
||
|
||
Common behavior for all providers: catch exceptions per candidate — one failure must not abort the batch; log and continue. If the selected provider's key/endpoint is unconfigured, run candidate detection only, store `event_type='candidate'`, and log that classification was skipped. Add one pytest with a stub classifier to verify the orchestration loop independent of any API.
|
||
- Frontend: colored markers on the timeline (color by event_type, legend in a corner); hover → description tooltip; click → jump playhead there.
|
||
- **Accept:** on synthetic audio (which has synthesized "bangs"), candidates land within ±0.5 s of the ground-truth pulse times; with an API key present, classification runs and markers render.
|
||
|
||
### M8 — Correction / annotation → 3D
|
||
- Clicking a timeline event opens a small panel: shown classification, a dropdown to correct `event_type` (writes `source='user'`), and an "annotate location" mode.
|
||
- Annotate mode: user drags a rectangle on any video's overlay canvas → normalized bbox → `POST /api/annotations`.
|
||
- Backend resolution (`geometry.py`):
|
||
- Cast a ray from that camera through the bbox center: unproject with the pose+intrinsics at `t_video`.
|
||
- If **another annotation exists for the same event from a different video**: triangulate (closest point between the two rays; reject if rays are near-parallel or midpoint gap > 0.5 scene units).
|
||
- Else: fall back to the **nearest point-cloud point to the ray** (within a 0.3-unit cylinder); if none, return the ray at depth = distance to point-cloud centroid.
|
||
- Store the resulting 3D point as a new anchor linked to the event → it immediately appears in all overlays and the 3D scene (label = event description).
|
||
- **Accept:** on synthetic data, annotating the same stage corner from two videos produces an anchor within 0.2 units of the true corner.
|
||
|
||
### M9 — God's-eye extras + README
|
||
- **Keyframed camera paths (`camPath.js`):** "add keyframe" captures current free-roam pose at current `t_global`; keyframes render as gizmos; playback interpolates position with Catmull-Rom and orientation with slerp while the transport plays; export/import path as JSON.
|
||
- Polish: loading states, per-video enable/disable, keyboard shortcuts (space = play/pause, ←/→ = ±1 s, 1..9 = snap to camera N).
|
||
- **README.md:** prerequisites (ffmpeg required; COLMAP optional; a classifier key optional — `GEMINI_API_KEY` default, or Claude/local/OpenRouter via `FESTIVAL4D_CLASSIFIER`), quickstart (`synthetic` → `serve` → `npm run dev`), then the real-footage workflow (`drop files in data/raw` → `ingest` → `sync` → `reconstruct` → `events`), architecture sketch, and a short "shooting tips for good reconstructions" section (overlap between viewpoints, avoid pure zoom, keep some static structure in frame, brighter footage works better).
|
||
- **Accept:** a fresh clone following README reaches the working synthetic demo with no undocumented steps.
|
||
|
||
## 4. Known pitfalls (read before coding)
|
||
|
||
1. **COLMAP↔Three.js axes** — the diag(1,−1,−1) flip in M5. Get the shared helper + test done before any rendering.
|
||
2. **HTML5 video is not frame-accurate.** Design the transport around continuous correction (M4), never around "seek then trust currentTime".
|
||
3. **Range requests** — without them, seeking silently breaks. Verify with `curl -H "Range: bytes=0-100"` → expect 206.
|
||
4. **Letterboxing math** in overlays — `object-fit: contain` means canvas px ≠ video px; compute the content rect.
|
||
5. **Only sync what overlaps.** Two clips with no common audio can't be aligned; the pairwise-graph design (M1) handles this — videos in disconnected components get `offset_ms=NULL` and are excluded from the timeline with a UI notice.
|
||
6. **`.gitignore`** `data/raw data/work *.db node_modules` from M0 — media files must never be committed.
|
||
7. **Classifier providers**: default is Gemini 2.5 Flash (`GEMINI_API_KEY`) with native video input; Claude option uses `claude-opus-4-8` with adaptive thinking, no sampling params, `messages.parse` (model IDs never get date suffixes); local/OpenRouter option is any OpenAI-compatible vision endpoint. All must degrade to candidates-only when unconfigured.
|
||
|
||
## 5. Explicit non-goals for this prototype
|
||
|
||
No user accounts/auth, no cloud storage, no realtime ingest, no NeRF/Gaussian splatting (leave a `docs/ideas.md` note), no mobile UI, no Docker. Single local user, single project at a time.
|