festifun/OPUS_BUILD_INSTRUCTIONS.md

Festival 4D — Build Instructions

Audience: Claude Opus 4.8 executing autonomously in this repo (/Users/m3ultra/Documents/festifun). Goal: Turn multiple fan-shot smartphone videos of the same concert into a synchronized, explorable 4D experience: time-aligned multi-video playback, 3D scene reconstruction with camera poses, a free-roam "god's eye" viewer, AR-style overlays projected onto each video, and AI-tagged moments on a shared timeline.

This is an entertainment/replay project (same family as sports "bullet time" replay). Keep all naming, prompts, and UI copy in the concert domain.

---

Parallel execution: this spec is the canonical reference; the work is organized into lanes for multiple concurrent agents — see plan/README.md for the phase map, branch protocol, file ownership, and your lane brief. If you were given a lane brief, it overrides the serial ordering note below (your lane's milestones still apply verbatim).

0. Ground rules

• Build milestones in dependency order. Each milestone has acceptance criteria; do not start the next until the current one passes. When executing serially (single agent), go M0→M9; when executing in lanes, follow plan/.
• Work without real footage. The user may not have multi-camera concert videos yet. Every milestone must be verifiable with the synthetic fixtures described in M0. When real footage arrives, the same pipeline runs on it unchanged.
• Prefer boring tech. SQLite (not PostgreSQL — this is a local, single-user, offline-first tool; do not add a DB server dependency). Vanilla JS + Three.js via Vite (no React). FastAPI + uvicorn. Python 3.11+, managed with uv if available, else venv + pip.
• COLMAP is an external binary. Detect it at runtime (shutil.which("colmap")); if missing, print install instructions (brew install colmap on macOS) and — critically — the whole app must still run using the synthetic-poses fixture. Never make COLMAP a hard import-time dependency.
• Commit per milestone with a message describing what now works. Initialize git first (git init).

1. Repository layout

festifun/
├── README.md                  # setup + usage, written last (M9)
├── pyproject.toml             # backend deps: fastapi, uvicorn, numpy, scipy, librosa,
│                              #   soundfile, sqlalchemy, pydantic, opencv-python-headless,
│                              #   google-genai, anthropic, openai   (last three = classifier providers)
├── backend/
│   ├── festival4d/
│   │   ├── __init__.py
│   │   ├── cli.py             # `python -m festival4d <cmd>`: ingest | sync | reconstruct | events | serve
│   │   ├── config.py          # paths, constants
│   │   ├── db.py              # SQLAlchemy engine/session, schema
│   │   ├── ingest.py          # probe videos (ffprobe), extract audio, register in DB
│   │   ├── audio_sync.py      # GCC-PHAT pairwise offsets + global solve
│   │   ├── frames.py          # sharpness-aware frame sampling for SfM
│   │   ├── sfm.py             # COLMAP orchestration, model export, normalization, interpolation
│   │   ├── geometry.py        # quaternion slerp, ray casting, triangulation, coord conversions
│   │   ├── events_ai.py       # audio candidate detection + Claude classification
│   │   ├── api.py             # FastAPI app
│   │   └── synthetic.py       # test-fixture generator (see M0)
│   └── tests/                 # pytest; every numeric module gets a test
├── frontend/
│   ├── index.html
│   ├── package.json           # vite + three
│   └── src/
│       ├── main.js
│       ├── state.js           # global store: manifest, playhead, playing, selected camera
│       ├── transport.js       # master clock + per-video sync (M4)
│       ├── videoGrid.js       # <video> grid + overlay canvases
│       ├── scene3d.js         # Three.js scene: point cloud, frusta, orbit controls
│       ├── overlays.js        # 3D→2D anchor projection (M6)
│       ├── timeline.js        # scrubber + event markers
│       ├── annotate.js        # bbox drawing + submission (M8)
│       └── camPath.js         # keyframed camera paths (M9)
└── data/
    ├── raw/                   # user drops source videos here (gitignored)
    ├── work/                  # extracted wavs, frames, colmap workspace (gitignored)
    └── project.db             # SQLite (gitignored)

2. Data model (SQLite via SQLAlchemy)

videos(id, filename, duration_s, fps, width, height, offset_ms REAL, drift_ppm REAL,
       sync_confidence REAL, created_at)
camera_poses(id, video_id FK, frame_idx INT, t_video_s REAL,
             qw,qx,qy,qz, tx,ty,tz,            -- COLMAP convention: world→camera
             fx,fy,cx,cy,                       -- pinhole intrinsics after undistortion
             registered BOOL)                   -- False = interpolated, not solved
anchors(id, label, x,y,z, color)                -- 3D points to overlay (stage corners etc.)
events(id, t_global_s REAL, duration_s REAL, event_type TEXT, confidence REAL,
       description TEXT, source TEXT)           -- source: 'audio_auto' | 'ai' | 'user'
annotations(id, video_id FK, t_video_s REAL, x0,y0,x1,y1,  -- normalized [0,1] bbox
            resolved_anchor_id FK NULL)

Timebase convention (write this in config.py docstring and stick to it everywhere): t_global is the master timeline in seconds. For video v: t_video = (t_global - offset_ms/1000) * (1 + drift_ppm * 1e-6). The reference video has offset 0.

3. Milestones

M0 — Scaffold + synthetic fixtures

• Repo layout above, deps installed, python -m festival4d --help works, pytest runs, Vite dev server serves a hello page.
• synthetic.py generates a full fake project so every later layer is testable without footage:
  • Fake videos: use ffmpeg to render 3 short (20 s, 640×360, 30 fps) clips of the same procedurally animated scene from different angles — simplest: a testsrc2/drawtext composite is fine, but each clip must share a common audio track (a music-like WAV you synthesize with numpy: beat pulses + tones) shifted by known offsets (e.g. 0 ms, +1370 ms, −842 ms) and with distinct visuals. Store ground-truth offsets in a JSON next to them.
  • Fake poses: 3 camera trajectories on arcs around origin looking at a fake "stage" (a box of 3D points at the origin) + a sparse point cloud (random points on the box + ground plane). Write them into camera_poses and a points.ply in the exact same format the real COLMAP path will produce.
• Accept: python -m festival4d synthetic populates data/raw + DB; tests pass.

M1 — Ingest + audio sync

• ingest: ffprobe each file in data/raw, insert into videos, extract mono 16 kHz WAV per video (ffmpeg) into data/work/audio/.
• sync (in audio_sync.py):
  • Use GCC-PHAT, not raw cross-correlation. Concert audio is loud, clipped, and reverberant; PHAT weighting (whiten the cross-spectrum: R = X·conj(Y); R /= |R|+eps; r = irfft(R)) is far more robust. Implement with numpy/scipy FFTs; librosa only for loading.
  • Compute all pairwise offsets, not just first-vs-rest. Score each pair by peak-to-second-peak ratio. Then solve globally: offsets are edges in a graph, per-video offsets are node values; least-squares solve the overdetermined system, weight by confidence, and reject pairs that violate cycle consistency (off_ab + off_bc − off_ac > 50 ms).
  • Drift check: re-run GCC-PHAT on 10 s windows every 30 s along the overlap; fit a line through window-offsets. Store slope as drift_ppm. Phone clocks drift tens of ppm — over a 10-minute set that's audible/visible. If |drift| < 5 ppm store 0.
  • Persist offset_ms, drift_ppm, sync_confidence to DB and export data/work/sync.json.
• Tests: synthesize one signal, shift copies by known amounts (+ add noise + small resample to simulate drift), assert recovered offsets within ±10 ms and drift within ±3 ppm. Also run against the M0 fixture and assert the known ground-truth offsets.
• Accept: sync on synthetic clips recovers ground truth.

M2 — Reconstruction (COLMAP) + pose export

• frames.py: sample candidate frames ~2/s per video; within each 0.5 s window keep the sharpest frame (variance of Laplacian via OpenCV). Write JPEGs named {video_id}_{frame_idx}.jpg.
• sfm.py: drive the COLMAP CLI via subprocess:
  1. feature_extractor — one camera per video (--ImageReader.single_camera_per_folder 1, put each video's frames in its own subfolder), OPENCV camera model.
  2. sequential_matcher within each video + exhaustive_matcher across (frame counts here are small; exhaustive is fine at this scale).
  3. mapper (incremental reconstruction), then image_undistorter/model_converter to get a TXT model.
  4. Parse images.txt / cameras.txt / points3D.txt yourself (formats are documented in COLMAP docs; simple line parsing — no pycolmap dependency needed).
• Normalize the scene before storing: translate so the point-cloud centroid is at origin; scale so the camera-position bounding sphere has radius 10; rotate so world-up = the average of camera up-vectors (phones are held roughly upright — this heuristic works well; write it as such in a comment). Apply the same similarity transform to poses and points.
• Interpolate poses for unregistered frames between registered ones: slerp rotation, lerp translation, mark registered=False; do not extrapolate beyond first/last registered frame of each video.
• Export data/work/points.ply (binary little-endian PLY) and fill camera_poses.
• Failure handling: if COLMAP registers < 60 % of frames or < 2 videos into one model, print a clear diagnostic (likely causes: too-dark footage, motion blur, no view overlap) and leave the synthetic/previous poses untouched. Concert footage is genuinely hard for SfM (moving lights, crowds, blur) — the app must degrade gracefully to "synced videos, no 3D".
• Tests: parser unit tests on hand-written COLMAP TXT snippets; normalization test (centroid≈0, radius≈10, up≈+Y); slerp test.
• Accept: on the synthetic fixture (which skips COLMAP and injects known poses) the whole export path runs; if COLMAP is installed, the pipeline runs end-to-end on any real folder of images without crashing.

M3 — API

FastAPI app (api.py), CORS enabled for the Vite origin, mounted static serving of data/raw with HTTP Range support (Starlette's StaticFiles supports Range — required or <video> seeking breaks).

GET  /api/manifest            → {videos:[{id,filename,url,duration_s,fps,width,height,
                                          offset_ms,drift_ppm}], t_global_max, has_poses}
GET  /api/videos/{id}/poses   → [{frame_idx,t_video_s,q:[w,x,y,z],t:[x,y,z],
                                  intrinsics:{fx,fy,cx,cy},registered}]
GET  /api/pointcloud          → points.ply file
GET  /api/anchors             → [...]        POST /api/anchors
GET  /api/events              → [...]
POST /api/events/detect      {t_global_s?, window_s?}  → runs M7 detection (or on one window)
POST /api/annotations        {video_id, t_video_s, bbox:[x0,y0,x1,y1]} → {anchor_id?, point?}  (M8)

cli.py serve runs uvicorn. Accept: with the synthetic project, curl of each endpoint returns sane data; a browser can seek the served videos.

M4 — Synchronized multi-video playback (the heart of the app)

• Layout: left = 3D canvas placeholder, right = grid of <video> elements (one per video), bottom = timeline scrubber + play/pause + speed control.
• Master clock in transport.js: t_global = t0 + (performance.now() - wall0)/1000 * rate while playing. Never trust any single video element as the clock.
• Per video, each animation frame (use requestVideoFrameCallback where available, else rAF): compute target t_video from the timebase formula; compare to video.currentTime:
  • error > 150 ms → hard seek (fastSeek if available);
  • 20–150 ms → nudge playbackRate (clamp to master rate ×[0.95, 1.05]) until converged;
  • < 20 ms → playbackRate = rate.
• Videos whose t_video falls outside [0, duration] pause and dim (they weren't recording at that global time).
• Audio: only one video unmuted at a time (click to select "audio source"); all others muted — otherwise phasing chaos.
• Accept: with the 3 synthetic clips (known offsets), scrubbing and playback keep all clips visibly locked to the shared audio events; drift error indicator (dev overlay showing per-video sync error in ms) stays < 50 ms during continuous playback.

M5 — 3D viewer

• scene3d.js: load /api/pointcloud (PLY via three/examples PLYLoader) as a Points cloud; draw each video's camera path as a line and current pose as a frustum wireframe (colored per video); OrbitControls for free roam.
• Coordinate conversion — do this exactly, it's the classic pitfall. COLMAP stores world→camera: x_cam = R x_world + t, camera axes = (+x right, +y down, +z forward). Three.js cameras look down −z with +y up. Per pose:

  R = quat_to_mat(q)            // world→cam
  C = -Rᵀ t                     // camera center in world coords
  R_c2w = Rᵀ
  R_three = R_c2w * diag(1, -1, -1)   // flip y,z of camera-local axes
  camera.position = C; camera.setRotationFromMatrix(R_three)
  

  Implement once in a shared helper with a unit test in the backend mirror (geometry.py) so both sides agree.
• On each playhead tick, update frusta from the pose at the nearest t_video (poses are keyed by frame; binary-search + use interpolated entries).
• Snap-to-camera: clicking a frustum (or a per-video button) tweens the viewer camera to that video's pose and keeps following it while playing; intrinsics → PerspectiveCamera: fov = 2·atan(H/(2·fy))·180/π, aspect = W/H (assume centered principal point for the prototype). "Free roam" button detaches.
• Accept: on synthetic data, orbiting shows 3 camera arcs around the stage box; snap-to-camera view of camera A roughly matches what video A shows (verifiable exactly on synthetic data since it's generated from those poses).

M6 — Anchor overlays ("x-ray" HUD)

• Each <video> gets a transparent <canvas> overlay, resolution-matched to the displayed video size (handle devicePixelRatio + object-fit letterboxing math carefully — compute the video's content rect inside the element).
• For each anchor and each visible video: build the Three.js camera for that video's current pose (M5 helper), anchor.project(camera), convert NDC → canvas px, skip if behind camera (vec.z > 1 after project, or check the w-divide sign properly by transforming to camera space first and requiring z_cam < 0).
• Draw a dot + label; anchors render regardless of real-world occlusion — that is the x-ray feature. Style: small circle, 1 px white halo, label text.
• Seed default anchors on the synthetic stage (4 box corners named "Stage FL/FR/BL/BR").
• Accept: on synthetic data, the projected corners track the drawn stage box across all 3 videos while playing (exact on synthetic since geometry is ground truth).

M7 — Moment detection + AI classification

Two stages — cheap audio candidates first, then vision LLM on candidates only:

• Candidates (events_ai.py): on the reference video's audio compute RMS energy and spectral-flux onset strength (librosa); find peaks that are local maxima over a 10 s neighborhood and above the 90th percentile → candidate t_global moments (bass drops, pyro bangs, crowd roars). Insert as events(source='audio_auto', event_type='candidate').

• Classification — pluggable provider (events_ai.py). The shared contract is a Pydantic model and a tiny protocol; providers are interchangeable and selected by env var:

  class MomentClassification(BaseModel):
      event_type: Literal["bass_drop","pyro","confetti","crowd_wave",
                          "artist_moment","light_show","quiet_moment","other"]
      confidence: float          # 0..1
      description: str           # one sentence, human-readable
  
  class MomentClassifier(Protocol):
      def classify(self, clip_path: Path, frames: list[Path]) -> MomentClassification: ...
  

  Selection: FESTIVAL4D_CLASSIFIER=gemini|claude|local (default gemini). Inputs prepared once per candidate: a 3 s MP4 clip around t_global from the best-covering video (ffmpeg, ≤720p, keep it under ~15 MB) and 6 sampled JPEGs (≤768 px long edge) for providers without video input.

  1. GeminiClassifier (default — cheapest, native video). google-genai SDK, model gemini-2.5-flash, needs GEMINI_API_KEY. Send the clip inline (video bytes are supported directly for small files) and enforce structured output:

     from google import genai
     client = genai.Client()   # reads GEMINI_API_KEY
     resp = client.models.generate_content(
         model="gemini-2.5-flash",
         contents=[
             genai.types.Part.from_bytes(data=clip_bytes, mime_type="video/mp4"),
             "This is a 3-second concert crowd clip around a detected audio spike. Classify the moment.",
         ],
         config={"response_mime_type": "application/json",
                 "response_schema": MomentClassification},
     )
     result = MomentClassification.model_validate_json(resp.text)
     

  2. ClaudeClassifier (optional, ANTHROPIC_API_KEY): Anthropic Python SDK, model claude-opus-4-8, the 6 frames as base64 image blocks, structured output via client.messages.parse(..., thinking={"type":"adaptive"}, output_format=MomentClassification) → resp.parsed_output. Do not pass temperature (rejected on Opus 4.8).
  3. LocalClassifier (offline/free): any OpenAI-compatible endpoint — Ollama/LM Studio with a vision model (e.g. qwen2.5-vl), or OpenRouter by just pointing the same class at https://openrouter.ai/api/v1. Config: FESTIVAL4D_OPENAI_BASE_URL, FESTIVAL4D_OPENAI_KEY, FESTIVAL4D_OPENAI_MODEL. Send the 6 frames as image_url/base64 parts, request JSON, validate with MomentClassification.model_validate_json and one retry on validation failure (local models are sloppier about JSON).

  Common behavior for all providers: catch exceptions per candidate — one failure must not abort the batch; log and continue. If the selected provider's key/endpoint is unconfigured, run candidate detection only, store event_type='candidate', and log that classification was skipped. Add one pytest with a stub classifier to verify the orchestration loop independent of any API.

• Frontend: colored markers on the timeline (color by event_type, legend in a corner); hover → description tooltip; click → jump playhead there.

• Accept: on synthetic audio (which has synthesized "bangs"), candidates land within ±0.5 s of the ground-truth pulse times; with an API key present, classification runs and markers render.

M8 — Correction / annotation → 3D

• Clicking a timeline event opens a small panel: shown classification, a dropdown to correct event_type (writes source='user'), and an "annotate location" mode.
• Annotate mode: user drags a rectangle on any video's overlay canvas → normalized bbox → POST /api/annotations.
• Backend resolution (geometry.py):
  • Cast a ray from that camera through the bbox center: unproject with the pose+intrinsics at t_video.
  • If another annotation exists for the same event from a different video: triangulate (closest point between the two rays; reject if rays are near-parallel or midpoint gap > 0.5 scene units).
  • Else: fall back to the nearest point-cloud point to the ray (within a 0.3-unit cylinder); if none, return the ray at depth = distance to point-cloud centroid.
  • Store the resulting 3D point as a new anchor linked to the event → it immediately appears in all overlays and the 3D scene (label = event description).
• Accept: on synthetic data, annotating the same stage corner from two videos produces an anchor within 0.2 units of the true corner.

M9 — God's-eye extras + README

• Keyframed camera paths (camPath.js): "add keyframe" captures current free-roam pose at current t_global; keyframes render as gizmos; playback interpolates position with Catmull-Rom and orientation with slerp while the transport plays; export/import path as JSON.
• Polish: loading states, per-video enable/disable, keyboard shortcuts (space = play/pause, ←/→ = ±1 s, 1..9 = snap to camera N).
• README.md: prerequisites (ffmpeg required; COLMAP optional; a classifier key optional — GEMINI_API_KEY default, or Claude/local/OpenRouter via FESTIVAL4D_CLASSIFIER), quickstart (synthetic → serve → npm run dev), then the real-footage workflow (drop files in data/raw → ingest → sync → reconstruct → events), architecture sketch, and a short "shooting tips for good reconstructions" section (overlap between viewpoints, avoid pure zoom, keep some static structure in frame, brighter footage works better).
• Accept: a fresh clone following README reaches the working synthetic demo with no undocumented steps.

4. Known pitfalls (read before coding)

1. COLMAP↔Three.js axes — the diag(1,−1,−1) flip in M5. Get the shared helper + test done before any rendering.
2. HTML5 video is not frame-accurate. Design the transport around continuous correction (M4), never around "seek then trust currentTime".
3. Range requests — without them, seeking silently breaks. Verify with curl -H "Range: bytes=0-100" → expect 206.
4. Letterboxing math in overlays — object-fit: contain means canvas px ≠ video px; compute the content rect.
5. Only sync what overlaps. Two clips with no common audio can't be aligned; the pairwise-graph design (M1) handles this — videos in disconnected components get offset_ms=NULL and are excluded from the timeline with a UI notice.
6. .gitignore data/raw data/work *.db node_modules from M0 — media files must never be committed.
7. Classifier providers: default is Gemini 2.5 Flash (GEMINI_API_KEY) with native video input; Claude option uses claude-opus-4-8 with adaptive thinking, no sampling params, messages.parse (model IDs never get date suffixes); local/OpenRouter option is any OpenAI-compatible vision endpoint. All must degrade to candidates-only when unconfigured.

5. Explicit non-goals for this prototype

No user accounts/auth, no cloud storage, no realtime ingest, no NeRF/Gaussian splatting (leave a docs/ideas.md note), no mobile UI, no Docker. Single local user, single project at a time.