ec22cbdf1f
Full frontend built against the synthetic API (no dependence on lanes A/B/D): M4 — synchronized playback: master clock from performance.now() (never a <video>); per-video correction via requestVideoFrameCallback (exact mediaTime) with a currentTime+half-frame fallback — hard-seek >150ms / nudge playbackRate +/-5% 20-150ms / lock <20ms; out-of-range videos pause+dim; single audio source; dev sync-error overlay. Timebase mirrors config.py (lib/timebase.js). Verified: seek is exact to 0ms; continuous inter-video desync 13ms mean / 46ms max. M5 — 3D viewer: PLY point cloud, per-video camera paths + current-pose frusta, OrbitControls, snap-to-camera (intrinsics->PerspectiveCamera fov) + free roam. All COLMAP->Three.js via the frozen lib/pose.js; pose interpolation in lib/poseTrack.js (slerp+lerp of contract inputs). M6 — anchor overlays: letterbox-correct per-video canvas; anchors projected via pose.js; behind-camera cull. Projection agrees with an independent pinhole to 1.45e-13 px. Timeline — colored event markers + legend, hover tooltip, click-to-jump; draggable scrubber. Phase-3 seams (annotate.js M8, camPath.js M9) left as stubs. No frozen files edited; no new deps; no change requests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|---|---|---|
| backend | ||
| frontend | ||
| plan | ||
| .gitignore | ||
| OPUS_BUILD_INSTRUCTIONS.md | ||
| pyproject.toml | ||
| README.md | ||
| uv.lock | ||
Festival 4D
Turn multiple fan-shot smartphone videos of the same concert into a synchronized, explorable 4D experience: time-aligned multi-video playback, 3D scene reconstruction with camera poses, a free-roam "god's eye" viewer, AR-style overlays projected onto each video, and AI-tagged moments on a shared timeline.
Status: foundation phase. The scaffold, database, frozen contracts (API, pose math, DB schema), and a synthetic fixture generator are in place. Feature lanes (media sync, reconstruction, viewer, AI events) build on top. The full README with the real-footage workflow is written in the integration phase (M9). See
OPUS_BUILD_INSTRUCTIONS.mdfor the canonical spec andplan/for the execution plan.
Prerequisites
- Python 3.11+ and
uv(or venv + pip) - ffmpeg / ffprobe on your
PATH(required) - COLMAP (optional — reconstruction degrades gracefully without it)
- A classifier API key (optional —
GEMINI_API_KEYby default; seeevents_ai.py) - Node 18+ for the frontend
Quickstart (synthetic demo — no footage needed)
# 1. backend env
uv venv --python 3.12
uv pip install -e ".[dev]"
# 2. generate the synthetic fixture project (fake videos + poses + point cloud + events)
uv run python -m festival4d synthetic
# 3. serve the API (http://127.0.0.1:8000)
uv run python -m festival4d serve
# 4. in another terminal, the frontend
cd frontend
npm install
npm run dev # http://localhost:5173
Backend CLI
python -m festival4d synthetic # generate the synthetic fixture (M0)
python -m festival4d ingest # probe videos + extract audio (lane A / M1)
python -m festival4d sync # GCC-PHAT audio alignment (lane A / M1)
python -m festival4d reconstruct # COLMAP SfM + pose export (lane B / M2)
python -m festival4d events # audio candidates + AI classify (lane D / M7)
python -m festival4d serve # FastAPI app (M3)
Tests
uv run pytest