Go to file

m3ultra 157c783e25 Lane A (M1): ingest + GCC-PHAT audio sync Fills the foundation stubs in ingest.py and audio_sync.py. - ingest: ffprobe probe + mono 16 kHz WAV extraction per video (ffmpeg); idempotent run_ingest reuses existing rows so synthetic -> ingest -> sync works without unique-constraint collisions. - sync: PHAT-whitened GCC-PHAT pairwise offsets with sub-sample parabolic peak refinement, scored by peak-to-second-peak ratio; confidence-weighted global least-squares solve with cycle-consistency rejection (>50 ms); windowed drift estimation (ppm) with a 5 ppm deadband; disconnected sync-graph components -> offset_ms=None (spec pitfall #5). - Persists offset_ms/drift_ppm/sync_confidence via db.update_video_sync and exports data/work/sync.json. Verified on the synthetic fixture: synthetic -> ingest -> sync recovers the ground-truth offsets (0 / +1370 / -842 ms) to <0.001 ms and drift 0 ppm, well inside the spec M1 tolerances (+/-10 ms, +/-3 ppm). pytest: 43 passed (24 foundation + 19 new in test_ingest.py and test_audio_sync.py). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>		2026-07-16 01:13:56 +10:00
backend	Lane A (M1): ingest + GCC-PHAT audio sync	2026-07-16 01:13:56 +10:00
frontend	Foundation (M0 + M3 + frozen contracts): scaffold, synthetic fixtures, full API	2026-07-16 00:45:55 +10:00
plan	Lane A (M1): ingest + GCC-PHAT audio sync	2026-07-16 01:13:56 +10:00
.gitignore	Festival 4D build spec + parallel lane plan for Opus agents	2026-07-16 00:06:54 +10:00
OPUS_BUILD_INSTRUCTIONS.md	Festival 4D build spec + parallel lane plan for Opus agents	2026-07-16 00:06:54 +10:00
pyproject.toml	Foundation (M0 + M3 + frozen contracts): scaffold, synthetic fixtures, full API	2026-07-16 00:45:55 +10:00
README.md	Foundation (M0 + M3 + frozen contracts): scaffold, synthetic fixtures, full API	2026-07-16 00:45:55 +10:00
uv.lock	Foundation (M0 + M3 + frozen contracts): scaffold, synthetic fixtures, full API	2026-07-16 00:45:55 +10:00

README.md

Festival 4D

Turn multiple fan-shot smartphone videos of the same concert into a synchronized, explorable 4D experience: time-aligned multi-video playback, 3D scene reconstruction with camera poses, a free-roam "god's eye" viewer, AR-style overlays projected onto each video, and AI-tagged moments on a shared timeline.

Status: foundation phase. The scaffold, database, frozen contracts (API, pose math, DB schema), and a synthetic fixture generator are in place. Feature lanes (media sync, reconstruction, viewer, AI events) build on top. The full README with the real-footage workflow is written in the integration phase (M9). See OPUS_BUILD_INSTRUCTIONS.md for the canonical spec and plan/ for the execution plan.

Prerequisites

Python 3.11+ and uv (or venv + pip)
ffmpeg / ffprobe on your PATH (required)
COLMAP (optional — reconstruction degrades gracefully without it)
A classifier API key (optional — GEMINI_API_KEY by default; see events_ai.py)
Node 18+ for the frontend

Quickstart (synthetic demo — no footage needed)

# 1. backend env
uv venv --python 3.12
uv pip install -e ".[dev]"

# 2. generate the synthetic fixture project (fake videos + poses + point cloud + events)
uv run python -m festival4d synthetic

# 3. serve the API (http://127.0.0.1:8000)
uv run python -m festival4d serve

# 4. in another terminal, the frontend
cd frontend
npm install
npm run dev            # http://localhost:5173

Backend CLI

python -m festival4d synthetic     # generate the synthetic fixture (M0)
python -m festival4d ingest        # probe videos + extract audio      (lane A / M1)
python -m festival4d sync          # GCC-PHAT audio alignment           (lane A / M1)
python -m festival4d reconstruct   # COLMAP SfM + pose export           (lane B / M2)
python -m festival4d events        # audio candidates + AI classify     (lane D / M7)
python -m festival4d serve         # FastAPI app                        (M3)

Tests

uv run pytest