DIRECTIVES round 1: all lanes merged & verified (96 tests), integration begins

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

README.md

@@ -0,0 +1,57 @@
+# Festival 4D
+
+Turn multiple fan-shot smartphone videos of the same concert into a synchronized,
+explorable **4D experience**: time-aligned multi-video playback, 3D scene reconstruction
+with camera poses, a free-roam "god's eye" viewer, AR-style overlays projected onto each
+video, and AI-tagged moments on a shared timeline.
+
+> **Status: foundation phase.** The scaffold, database, frozen contracts (API, pose math,
+> DB schema), and a synthetic fixture generator are in place. Feature lanes (media sync,
+> reconstruction, viewer, AI events) build on top. The full README with the real-footage
+> workflow is written in the integration phase (M9). See
+> [`OPUS_BUILD_INSTRUCTIONS.md`](OPUS_BUILD_INSTRUCTIONS.md) for the canonical spec and
+> [`plan/`](plan/) for the execution plan.
+
+## Prerequisites
+
+- **Python 3.11+** and [`uv`](https://docs.astral.sh/uv/) (or venv + pip)
+- **ffmpeg** / **ffprobe** on your `PATH` (required)
+- **COLMAP** (optional — reconstruction degrades gracefully without it)
+- A classifier API key (optional — `GEMINI_API_KEY` by default; see `events_ai.py`)
+- **Node 18+** for the frontend
+
+## Quickstart (synthetic demo — no footage needed)
+
+```bash
+# 1. backend env
+uv venv --python 3.12
+uv pip install -e ".[dev]"
+
+# 2. generate the synthetic fixture project (fake videos + poses + point cloud + events)
+uv run python -m festival4d synthetic
+
+# 3. serve the API (http://127.0.0.1:8000)
+uv run python -m festival4d serve
+
+# 4. in another terminal, the frontend
+cd frontend
+npm install
+npm run dev            # http://localhost:5173
+```
+
+## Backend CLI
+
+```
+python -m festival4d synthetic     # generate the synthetic fixture (M0)
+python -m festival4d ingest        # probe videos + extract audio      (lane A / M1)
+python -m festival4d sync          # GCC-PHAT audio alignment           (lane A / M1)
+python -m festival4d reconstruct   # COLMAP SfM + pose export           (lane B / M2)
+python -m festival4d events        # audio candidates + AI classify     (lane D / M7)
+python -m festival4d serve         # FastAPI app                        (M3)
+```
+
+## Tests
+
+```bash
+uv run pytest
+```

backend/festival4d/__init__.py

@@ -0,0 +1,8 @@
+"""Festival 4D — synchronized, explorable 4D concert replay.
+
+See ``OPUS_BUILD_INSTRUCTIONS.md`` for the canonical spec and ``plan/`` for the
+parallel execution plan. This package is the backend: CLI, database, synthetic
+fixture generator, media/reconstruction/AI pipelines, and the FastAPI app.
+"""
+
+__version__ = "0.1.0"

backend/festival4d/__main__.py

@@ -0,0 +1,6 @@
+"""Enable ``python -m festival4d <cmd>``."""
+
+from festival4d.cli import main
+
+if __name__ == "__main__":
+    raise SystemExit(main())

backend/festival4d/api.py

@@ -0,0 +1,209 @@
+"""FastAPI app (spec M3). FROZEN after foundation — lanes never edit routes or shapes.
+
+Serves the synthetic project fully so lane C can treat this API as finished:
+
+    GET  /api/manifest            -> {videos:[{id,filename,url,duration_s,fps,width,height,
+                                              offset_ms,drift_ppm}], t_global_max, has_poses}
+    GET  /api/videos/{id}/poses   -> [{frame_idx,t_video_s,q:[w,x,y,z],t:[x,y,z],
+                                      intrinsics:{fx,fy,cx,cy},registered}]
+    GET  /api/pointcloud          -> points.ply (binary; Range-capable)
+    GET  /api/anchors             -> [...]        POST /api/anchors
+    GET  /api/events              -> [...]
+    POST /api/events/detect       {t_global_s?, window_s?}  -> runs M7 detection
+    POST /api/annotations         {video_id, t_video_s, bbox:[x0,y0,x1,y1]} -> {anchor_id?, point?}
+
+Source videos in ``data/raw`` are mounted at ``/media`` via Starlette ``StaticFiles``,
+which supports HTTP Range (required for ``<video>`` seeking; verify ``curl -H "Range:
+bytes=0-100"`` -> 206). Stubbed lane entrypoints (events detection, M8 annotation
+resolution) raise ``NotImplementedError``; we catch it here and degrade to a 200 with a
+note so the frozen routes never 500 while a lane is still stubbed.
+"""
+
+from __future__ import annotations
+
+import logging
+
+from fastapi import FastAPI, HTTPException
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.responses import FileResponse
+from fastapi.staticfiles import StaticFiles
+from pydantic import BaseModel, Field
+
+from festival4d import config, db
+
+log = logging.getLogger("festival4d.api")
+
+app = FastAPI(title="Festival 4D", version="0.1.0")
+
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=config.FRONTEND_ORIGINS,
+    allow_origin_regex=config.FRONTEND_ORIGIN_REGEX,
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+
+# Bind the DB and ensure the media directory exists so the static mount is valid even
+# before the synthetic fixture has been generated.
+db.init_engine()
+db.init_db()
+config.RAW_DIR.mkdir(parents=True, exist_ok=True)
+
+# HTTP Range-capable static serving of source videos (spec M3, pitfall #3).
+app.mount("/media", StaticFiles(directory=config.RAW_DIR), name="media")
+
+
+# ---------------------------------------------------------------------------
+# Request models
+# ---------------------------------------------------------------------------
+class AnchorIn(BaseModel):
+    label: str
+    x: float
+    y: float
+    z: float
+    color: str | None = None
+
+
+class DetectIn(BaseModel):
+    t_global_s: float | None = None
+    window_s: float | None = None
+
+
+class AnnotationIn(BaseModel):
+    video_id: int
+    t_video_s: float
+    bbox: list[float] = Field(min_length=4, max_length=4)  # [x0, y0, x1, y1] normalized
+
+
+# ---------------------------------------------------------------------------
+# Serialization helpers
+# ---------------------------------------------------------------------------
+def _video_dict(v) -> dict:
+    return {
+        "id": v.id,
+        "filename": v.filename,
+        "url": f"/media/{v.filename}",
+        "duration_s": v.duration_s,
+        "fps": v.fps,
+        "width": v.width,
+        "height": v.height,
+        "offset_ms": v.offset_ms,
+        "drift_ppm": v.drift_ppm,
+    }
+
+
+def _pose_dict(p) -> dict:
+    return {
+        "frame_idx": p.frame_idx,
+        "t_video_s": p.t_video_s,
+        "q": [p.qw, p.qx, p.qy, p.qz],
+        "t": [p.tx, p.ty, p.tz],
+        "intrinsics": {"fx": p.fx, "fy": p.fy, "cx": p.cx, "cy": p.cy},
+        "registered": p.registered,
+    }
+
+
+def _anchor_dict(a) -> dict:
+    return {"id": a.id, "label": a.label, "x": a.x, "y": a.y, "z": a.z, "color": a.color}
+
+
+def _event_dict(e) -> dict:
+    return {
+        "id": e.id,
+        "t_global_s": e.t_global_s,
+        "duration_s": e.duration_s,
+        "event_type": e.event_type,
+        "confidence": e.confidence,
+        "description": e.description,
+        "source": e.source,
+    }
+
+
+# ---------------------------------------------------------------------------
+# Routes
+# ---------------------------------------------------------------------------
+@app.get("/api/health")
+def health() -> dict:
+    return {"status": "ok"}
+
+
+@app.get("/api/manifest")
+def manifest() -> dict:
+    videos = db.get_videos()
+    t_global_max = 0.0
+    for v in videos:
+        offset_s = (v.offset_ms or 0.0) / 1000.0
+        t_global_max = max(t_global_max, offset_s + v.duration_s)
+    return {
+        "videos": [_video_dict(v) for v in videos],
+        "t_global_max": t_global_max,
+        "has_poses": db.has_poses(),
+    }
+
+
+@app.get("/api/videos/{video_id}/poses")
+def video_poses(video_id: int) -> list[dict]:
+    if db.get_video(video_id) is None:
+        raise HTTPException(status_code=404, detail=f"no video with id={video_id}")
+    return [_pose_dict(p) for p in db.get_poses(video_id)]
+
+
+@app.get("/api/pointcloud")
+def pointcloud() -> FileResponse:
+    if not config.POINTS_PLY.exists():
+        raise HTTPException(status_code=404, detail="no point cloud (run `synthetic` or `reconstruct`)")
+    return FileResponse(
+        config.POINTS_PLY,
+        media_type="application/octet-stream",
+        filename="points.ply",
+    )
+
+
+@app.get("/api/anchors")
+def get_anchors() -> list[dict]:
+    return [_anchor_dict(a) for a in db.get_anchors()]
+
+
+@app.post("/api/anchors")
+def create_anchor(payload: AnchorIn) -> dict:
+    anchor = db.add_anchor(payload.label, payload.x, payload.y, payload.z, payload.color)
+    return _anchor_dict(anchor)
+
+
+@app.get("/api/events")
+def get_events() -> list[dict]:
+    return [_event_dict(e) for e in db.get_events()]
+
+
+@app.post("/api/events/detect")
+def detect_events(payload: DetectIn) -> dict:
+    from festival4d import events_ai
+
+    try:
+        result = events_ai.run_events(t_global_s=payload.t_global_s, window_s=payload.window_s)
+    except NotImplementedError as exc:
+        log.info("events detection not implemented yet: %s", exc)
+        return {
+            "result": None,
+            "note": "moment detection not implemented yet (lane D / M7)",
+            "events": [_event_dict(e) for e in db.get_events()],
+        }
+    return {"result": result, "events": [_event_dict(e) for e in db.get_events()]}
+
+
+@app.post("/api/annotations")
+def create_annotation(payload: AnnotationIn) -> dict:
+    x0, y0, x1, y1 = payload.bbox
+    annotation = db.add_annotation(payload.video_id, payload.t_video_s, x0, y0, x1, y1)
+
+    # M8 resolution (triangulate / nearest-point) is integration work; geometry stubs raise
+    # until lane B lands. Store the annotation now; resolve to an anchor later.
+    anchor_id = None
+    point = None
+    note = "annotation stored; 3D resolution is M8/integration work"
+    return {
+        "annotation_id": annotation.id,
+        "anchor_id": anchor_id,
+        "point": point,
+        "note": note,
+    }

backend/festival4d/audio_sync.py

@@ -0,0 +1,414 @@
+"""GCC-PHAT pairwise audio offsets + global graph solve + drift (spec M1, lane A).
+
+Algorithm (spec M1):
+
+- **GCC-PHAT**, not raw cross-correlation: whiten the cross-spectrum
+  ``R = X·conj(Y); R /= |R| + eps; r = irfft(R)`` — robust to loud/clipped/reverberant
+  concert audio. numpy FFTs; librosa only for loading (in :func:`run_sync`).
+- All **pairwise** offsets, each scored by peak-to-second-peak ratio; then a global
+  least-squares solve over the offset graph, weighted by confidence, rejecting pairs that
+  violate cycle consistency (``|off_ab + off_bc − off_ac| > 50 ms``, generalized here as a
+  residual test against the fitted global solution).
+- **Drift**: re-run GCC-PHAT on 10 s windows every 30 s along the overlap; fit a line to
+  ``delay(t)``; store the slope (dimensionless rate) as ``drift_ppm`` (store 0 if < 5 ppm).
+- Disconnected sync-graph components → ``offset_ms = None`` (never a guess; spec pitfall #5).
+- Persist via ``db.update_video_sync`` and export ``config.SYNC_JSON``.
+
+Sign conventions (kept consistent with ``config``'s timebase):
+
+    A video *v*'s audio sample at local time ``τ`` equals the master signal at global time
+    ``offset_s + τ`` (drift 0). So for two videos ``a``/``b``:
+    ``offset_a − offset_b = −gcc_phat(a, b)`` seconds, where :func:`gcc_phat` returns the
+    delay of its first argument relative to its second. Under drift ``d`` (= ``drift_ppm·1e−6``),
+    the windowed delay of ``v`` vs the reference is ``d·t − offset_s`` — a line whose slope is
+    the drift, recovered by :func:`estimate_drift`.
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+
+import numpy as np
+
+from festival4d import config, db
+from festival4d.ingest import audio_wav_path
+
+log = logging.getLogger("festival4d.audio_sync")
+
+_EPS = 1e-10
+# Edges weaker than this peak-to-second-peak ratio carry no reliable alignment
+# (flat/ambiguous correlation) and are dropped before the global solve.
+_MIN_EDGE_CONFIDENCE = 1.15
+# A pairwise offset whose residual against the fitted global solution exceeds this is a
+# cycle-consistency violation (spec M1); it is rejected and the component is re-solved.
+_CYCLE_TOLERANCE_MS = 50.0
+# Drift smaller than this magnitude is stored as exactly 0 (spec M1).
+_DRIFT_DEADBAND_PPM = 5.0
+
+
+# ---------------------------------------------------------------------------
+# Core correlation
+# ---------------------------------------------------------------------------
+def _peak_ratio(env: np.ndarray, peak: int, sr: int, guard_s: float = 0.004) -> float:
+    """Peak-to-second-peak ratio of a correlation envelope (the spec's edge score).
+
+    The second peak is the largest value outside a small guard band around the main peak,
+    so the main lobe's own shoulders don't count as competition.
+    """
+    main = float(env[peak])
+    if main <= _EPS:
+        return 0.0
+    guard = max(1, int(round(guard_s * sr)))
+    masked = env.copy()
+    masked[max(0, peak - guard):min(len(env), peak + guard + 1)] = 0.0
+    second = float(masked.max()) if masked.size else 0.0
+    if second <= _EPS:
+        return 1.0e6
+    return main / second
+
+
+def gcc_phat(sig: np.ndarray, ref: np.ndarray, sr: int,
+             max_lag_s: float | None = None) -> tuple[float, float]:
+    """GCC-PHAT time delay of ``sig`` relative to ``ref`` (spec M1, lane A).
+
+    Returns ``(offset_s, confidence)`` where ``offset_s`` is positive when ``sig`` *lags*
+    ``ref`` (i.e. ``sig[n] ≈ ref[n − offset_s·sr]``), and confidence is the
+    peak-to-second-peak ratio.
+    """
+    sig = np.asarray(sig, dtype=np.float64)
+    ref = np.asarray(ref, dtype=np.float64)
+    # Remove DC so the whitened correlation keys on structure, not a bias term.
+    sig = sig - sig.mean()
+    ref = ref - ref.mean()
+
+    n = len(sig) + len(ref)              # linear (zero-padded) correlation → no circular wrap
+    SIG = np.fft.rfft(sig, n)
+    REF = np.fft.rfft(ref, n)
+    R = SIG * np.conj(REF)
+    R /= np.abs(R) + _EPS                # PHAT weighting: flatten magnitude, keep phase
+    cc = np.fft.irfft(R, n)              # lag 0 at index 0; negative lags wrap to the tail
+
+    max_lag = n // 2 if max_lag_s is None else int(round(max_lag_s * sr))
+    max_lag = max(1, min(max_lag, n // 2))
+    # Re-center to contiguous lags [−max_lag … +max_lag].
+    cc = np.concatenate((cc[-max_lag:], cc[:max_lag + 1]))
+    env = np.abs(cc)
+    peak = int(np.argmax(env))
+    # Parabolic sub-sample refinement of the peak location (3-point fit around the max),
+    # so precision isn't capped at one sample (62.5 µs at 16 kHz) and drift slopes stay clean.
+    delta = 0.0
+    if 0 < peak < len(env) - 1:
+        a, b, c = env[peak - 1], env[peak], env[peak + 1]
+        denom = a - 2.0 * b + c
+        if abs(denom) > _EPS:
+            delta = float(np.clip(0.5 * (a - c) / denom, -0.5, 0.5))
+    offset_s = ((peak - max_lag) + delta) / sr
+    return offset_s, _peak_ratio(env, peak, sr)
+
+
+# ---------------------------------------------------------------------------
+# Pairwise graph
+# ---------------------------------------------------------------------------
+def pairwise_offsets(signals: dict[int, np.ndarray], sr: int) -> list[dict]:
+    """All pairwise GCC-PHAT offsets with confidences (spec M1, lane A).
+
+    ``signals`` maps ``video_id -> mono samples``. Returns a list of
+    ``{a, b, offset_s, confidence}`` edges, where ``offset_s`` is the *relative start
+    offset* ``offset_a − offset_b`` in seconds (see the module's sign conventions).
+    """
+    ids = sorted(signals)
+    edges: list[dict] = []
+    for i in range(len(ids)):
+        for j in range(i + 1, len(ids)):
+            a, b = ids[i], ids[j]
+            delay_s, conf = gcc_phat(signals[a], signals[b], sr)
+            edges.append({"a": a, "b": b, "offset_s": -delay_s, "confidence": conf})
+    return edges
+
+
+def _components(ids: list[int], edges: list[dict]) -> dict[int, list[int]]:
+    """Connected components of the offset graph (union-find over usable edges)."""
+    parent = {i: i for i in ids}
+
+    def find(x: int) -> int:
+        while parent[x] != x:
+            parent[x] = parent[parent[x]]
+            x = parent[x]
+        return x
+
+    for e in edges:
+        parent[find(e["a"])] = find(e["b"])
+    comps: dict[int, list[int]] = {}
+    for i in ids:
+        comps.setdefault(find(i), []).append(i)
+    return comps
+
+
+def _connected(nodes: list[int], root: int, edges: list[dict]) -> bool:
+    """Whether every node is reachable from ``root`` over ``edges``."""
+    adj: dict[int, list[int]] = {n: [] for n in nodes}
+    for e in edges:
+        adj[e["a"]].append(e["b"])
+        adj[e["b"]].append(e["a"])
+    seen = {root}
+    stack = [root]
+    while stack:
+        for y in adj[stack.pop()]:
+            if y not in seen:
+                seen.add(y)
+                stack.append(y)
+    return len(seen) == len(nodes)
+
+
+def _weighted_lstsq(nodes: list[int], ref: int, edges: list[dict]) -> dict[int, float]:
+    """Confidence-weighted least-squares offsets (seconds) with ``ref`` pinned to 0.
+
+    Each edge contributes the constraint ``offset_a − offset_b = offset_s``, weighted by
+    ``sqrt(confidence)``. Solved for all nodes except the pinned reference.
+    """
+    unknown = [n for n in nodes if n != ref]
+    col = {n: k for k, n in enumerate(unknown)}
+    A = np.zeros((len(edges), len(unknown)))
+    b = np.zeros(len(edges))
+    w = np.zeros(len(edges))
+    for r, e in enumerate(edges):
+        if e["a"] in col:
+            A[r, col[e["a"]]] += 1.0
+        if e["b"] in col:
+            A[r, col[e["b"]]] -= 1.0
+        b[r] = e["offset_s"]             # ref terms are 0, so they drop out of A
+        w[r] = np.sqrt(max(e["confidence"], _EPS))
+    x, *_ = np.linalg.lstsq(A * w[:, None], b * w, rcond=None)
+    sol = {ref: 0.0}
+    for n in unknown:
+        sol[n] = float(x[col[n]])
+    return sol
+
+
+def _solve_component(nodes: list[int], edges: list[dict]) -> dict[int, float]:
+    """Solve one connected component, rejecting cycle-inconsistent edges iteratively.
+
+    The reference (smallest ``video_id``) is pinned to 0. While the graph is over-determined
+    (more edges than a spanning tree needs) and the worst edge residual exceeds
+    ``_CYCLE_TOLERANCE_MS``, drop that edge (unless doing so would disconnect a node) and
+    re-solve. Returns offsets in seconds.
+    """
+    nodes = sorted(nodes)
+    ref = min(nodes)
+    edges = list(edges)
+    sol = _weighted_lstsq(nodes, ref, edges)
+    while len(edges) > len(nodes) - 1:
+        worst, worst_res = None, 0.0
+        for e in edges:
+            res = abs((sol[e["a"]] - sol[e["b"]]) - e["offset_s"])
+            if res > worst_res:
+                worst, worst_res = e, res
+        if worst is None or worst_res * 1000.0 <= _CYCLE_TOLERANCE_MS:
+            break
+        trial = [e for e in edges if e is not worst]
+        if not _connected(nodes, ref, trial):
+            break                        # can't drop it without orphaning a node
+        log.info("audio_sync: rejecting inconsistent edge %d-%d (residual %.1f ms)",
+                 worst["a"], worst["b"], worst_res * 1000.0)
+        edges = trial
+        sol = _weighted_lstsq(nodes, ref, edges)
+    return sol
+
+
+def solve_global_offsets(edges: list[dict], video_ids: list[int]) -> dict[int, float | None]:
+    """Least-squares global solve over the pairwise-offset graph (spec M1, lane A).
+
+    The reference component (largest; ties broken by smallest ``video_id``) is solved with
+    its smallest-id node pinned to 0. Every video outside it — including any with no usable
+    edges — gets ``None`` (spec pitfall #5). Returns ``video_id -> offset_ms`` (or ``None``).
+    """
+    ids = sorted(set(video_ids))
+    usable = [
+        e for e in edges
+        if e["a"] in ids and e["b"] in ids
+        and e["confidence"] >= _MIN_EDGE_CONFIDENCE
+        and np.isfinite(e["offset_s"])
+    ]
+    result: dict[int, float | None] = {i: None for i in ids}
+    if not ids:
+        return result
+
+    comps = _components(ids, usable)
+    # Reference component: the one that aligns the most cameras. A lone reference (no
+    # overlap with anyone) is still "aligned" at 0; everyone else stays None.
+    ref_comp = max(comps.values(), key=lambda c: (len(c), -min(c)))
+    if len(ref_comp) == 1:
+        result[ref_comp[0]] = 0.0
+        return result
+
+    ref_set = set(ref_comp)
+    comp_edges = [e for e in usable if e["a"] in ref_set and e["b"] in ref_set]
+    for i, off_s in _solve_component(ref_comp, comp_edges).items():
+        result[i] = off_s * 1000.0
+    return result
+
+
+# ---------------------------------------------------------------------------
+# Drift
+# ---------------------------------------------------------------------------
+def estimate_drift(sig: np.ndarray, ref: np.ndarray, sr: int,
+                   window_s: float = 10.0, hop_s: float = 30.0) -> float:
+    """Estimate clock drift in ppm by fitting window-offsets over the overlap (spec M1).
+
+    Slides a ``window_s`` window every ``hop_s`` over the common span, measures each
+    window's GCC-PHAT delay, and fits a line ``delay ≈ slope·t + b``. ``slope`` is the
+    dimensionless drift rate; ``drift_ppm = slope·1e6``. Returns 0 when there are too few
+    windows to fit or the estimate is within the ±5 ppm deadband.
+    """
+    sig = np.asarray(sig, dtype=np.float64)
+    ref = np.asarray(ref, dtype=np.float64)
+    n = min(len(sig), len(ref))
+    win = int(round(window_s * sr))
+    hop = int(round(hop_s * sr))
+    if win < sr or n < win or hop < 1:   # need a window of at least ~1 s that fits
+        return 0.0
+
+    centers: list[float] = []
+    delays: list[float] = []
+    start = 0
+    while start + win <= n:
+        delay_s, conf = gcc_phat(sig[start:start + win], ref[start:start + win], sr,
+                                 max_lag_s=window_s / 2.0)
+        if conf >= _MIN_EDGE_CONFIDENCE:
+            centers.append((start + win / 2.0) / sr)
+            delays.append(delay_s)
+        start += hop
+
+    if len(centers) < 2:
+        return 0.0
+    slope = float(np.polyfit(np.array(centers), np.array(delays), 1)[0])
+    drift_ppm = slope * 1.0e6
+    return 0.0 if abs(drift_ppm) < _DRIFT_DEADBAND_PPM else drift_ppm
+
+
+# ---------------------------------------------------------------------------
+# Orchestration
+# ---------------------------------------------------------------------------
+def _reference_id(offsets_ms: dict[int, float | None]) -> int | None:
+    """The reference video: smallest id among those with a solved (non-None) offset."""
+    solved = [i for i, o in offsets_ms.items() if o is not None]
+    return min(solved) if solved else None
+
+
+def _confidence_by_video(edges: list[dict], offsets_ms: dict[int, float | None],
+                         ref_id: int | None) -> dict[int, float | None]:
+    """Per-video sync confidence in [0, 1): best incident edge, mapped ``1 − 1/ratio``.
+
+    The reference is 1.0; unsolved videos are None. A raw peak ratio of 1 (ambiguous) maps
+    to 0, 2 to 0.5, large ratios approach 1 — a bounded, honest confidence.
+    """
+    best: dict[int, float] = {}
+    for e in edges:
+        if offsets_ms.get(e["a"]) is None or offsets_ms.get(e["b"]) is None:
+            continue
+        r = e["confidence"]
+        best[e["a"]] = max(best.get(e["a"], 0.0), r)
+        best[e["b"]] = max(best.get(e["b"], 0.0), r)
+    out: dict[int, float | None] = {}
+    for vid, off in offsets_ms.items():
+        if off is None:
+            out[vid] = None
+        elif vid == ref_id:
+            out[vid] = 1.0
+        else:
+            ratio = best.get(vid)
+            out[vid] = None if ratio is None else float(max(0.0, 1.0 - 1.0 / ratio))
+    return out
+
+
+def run_sync() -> dict:
+    """Sync every ingested video: pairwise GCC-PHAT, global solve, drift, persist.
+
+    Entrypoint for ``python -m festival4d sync``. Writes ``offset_ms``/``drift_ppm``/
+    ``sync_confidence`` via ``db`` and exports ``config.SYNC_JSON``. Returns the solution.
+    """
+    import librosa  # spec: librosa only for loading; import lazily (heavy module)
+
+    db.init_engine()
+    db.init_db()
+    videos = db.get_videos()
+    if not videos:
+        raise RuntimeError("no videos in DB — run `python -m festival4d ingest` first")
+
+    sr = config.AUDIO_SAMPLE_RATE
+    signals: dict[int, np.ndarray] = {}
+    missing: list[str] = []
+    for v in videos:
+        wav = audio_wav_path(v.id)
+        if not wav.exists():
+            missing.append(v.filename)
+            continue
+        y, _ = librosa.load(str(wav), sr=sr, mono=True)
+        signals[v.id] = y.astype(np.float64)
+    if missing:
+        log.warning("audio_sync: no extracted audio for %s — run ingest; "
+                    "they will be left unsynced (offset=None)", ", ".join(missing))
+
+    edges = pairwise_offsets(signals, sr) if len(signals) >= 2 else []
+    offsets_ms = solve_global_offsets(edges, list(signals.keys()))
+    # Videos with no audio at all are unsynced too.
+    for v in videos:
+        offsets_ms.setdefault(v.id, None)
+
+    ref_id = _reference_id(offsets_ms)
+    ref_sig = signals.get(ref_id) if ref_id is not None else None
+    confidence = _confidence_by_video(edges, offsets_ms, ref_id)
+
+    solution = {
+        "reference_video_id": ref_id,
+        "reference_filename": next((v.filename for v in videos if v.id == ref_id), None),
+        "sample_rate": sr,
+        "videos": [],
+        "edges": [
+            {"a": e["a"], "b": e["b"],
+             "offset_ms": round(e["offset_s"] * 1000.0, 3),
+             "confidence": round(float(e["confidence"]), 4)}
+            for e in edges
+        ],
+        "components": sorted(
+            (sorted(c) for c in _components(sorted(signals), [
+                e for e in edges if e["confidence"] >= _MIN_EDGE_CONFIDENCE
+            ]).values()),
+            key=lambda c: (-len(c), c),
+        ),
+        "generated_by": "festival4d.audio_sync",
+    }
+
+    for v in videos:
+        off = offsets_ms.get(v.id)
+        if off is None:
+            drift = None
+        elif v.id == ref_id:
+            drift = 0.0
+        elif ref_sig is not None and v.id in signals:
+            drift = estimate_drift(signals[v.id], ref_sig, sr)
+        else:
+            drift = 0.0
+        conf = confidence.get(v.id)
+        db.update_video_sync(v.id, off, drift, conf)
+        solution["videos"].append({
+            "video_id": v.id, "filename": v.filename,
+            "offset_ms": None if off is None else round(off, 3),
+            "drift_ppm": None if drift is None else round(drift, 4),
+            "sync_confidence": None if conf is None else round(conf, 4),
+            "connected": off is not None,
+        })
+        log.info("audio_sync: %s offset=%s drift=%s conf=%s",
+                 v.filename,
+                 "None" if off is None else f"{off:+.1f}ms",
+                 "None" if drift is None else f"{drift:+.2f}ppm",
+                 "None" if conf is None else f"{conf:.2f}")
+
+    config.SYNC_JSON.parent.mkdir(parents=True, exist_ok=True)
+    config.SYNC_JSON.write_text(json.dumps(solution, indent=2))
+    log.info("audio_sync: wrote %s (%d synced, %d unsynced)",
+             config.SYNC_JSON,
+             sum(1 for vv in solution["videos"] if vv["connected"]),
+             sum(1 for vv in solution["videos"] if not vv["connected"]))
+    return solution

backend/festival4d/cli.py

@@ -0,0 +1,130 @@
+"""Command-line entrypoint: ``python -m festival4d <cmd>``.
+
+FROZEN after foundation. Lanes fill in the bodies of the functions this dispatches
+to (in ``ingest.py``, ``audio_sync.py``, ``sfm.py``, ``events_ai.py``); they never
+edit this file or ``api.py``.
+
+Subcommands: ``synthetic | ingest | sync | reconstruct | events | serve``.
+
+Every subcommand dispatches to a single lane entrypoint. While a lane is still a stub,
+its entrypoint raises :class:`NotImplementedError`; we catch that here and print a clear
+"not implemented yet" message with a non-zero exit code, rather than a traceback. Once the
+lane fills the body, the same dispatch runs it for real — no change to this file needed.
+"""
+
+from __future__ import annotations
+
+import argparse
+import logging
+import sys
+
+log = logging.getLogger("festival4d.cli")
+
+
+def _cmd_synthetic(args: argparse.Namespace) -> int:
+    from festival4d import synthetic
+
+    synthetic.build()
+    return 0
+
+
+def _cmd_ingest(args: argparse.Namespace) -> int:
+    from festival4d import ingest
+
+    ingest.run_ingest()
+    return 0
+
+
+def _cmd_sync(args: argparse.Namespace) -> int:
+    from festival4d import audio_sync
+
+    audio_sync.run_sync()
+    return 0
+
+
+def _cmd_reconstruct(args: argparse.Namespace) -> int:
+    from festival4d import sfm
+
+    sfm.run_reconstruct()
+    return 0
+
+
+def _cmd_events(args: argparse.Namespace) -> int:
+    from festival4d import events_ai
+
+    result = events_ai.run_events(t_global_s=args.t_global_s, window_s=args.window_s)
+    log.info("events: %s", result)
+    return 0
+
+
+def _cmd_serve(args: argparse.Namespace) -> int:
+    import uvicorn
+
+    uvicorn.run(
+        "festival4d.api:app",
+        host=args.host,
+        port=args.port,
+        reload=args.reload,
+    )
+    return 0
+
+
+def build_parser() -> argparse.ArgumentParser:
+    from festival4d import config
+
+    parser = argparse.ArgumentParser(
+        prog="festival4d",
+        description="Festival 4D — synchronized, explorable 4D concert replay.",
+    )
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    p_syn = sub.add_parser("synthetic", help="generate the synthetic fixture project (M0)")
+    p_syn.set_defaults(func=_cmd_synthetic)
+
+    p_ing = sub.add_parser("ingest", help="probe videos + extract audio (lane A / M1)")
+    p_ing.set_defaults(func=_cmd_ingest)
+
+    p_sync = sub.add_parser("sync", help="GCC-PHAT audio alignment (lane A / M1)")
+    p_sync.set_defaults(func=_cmd_sync)
+
+    p_rec = sub.add_parser("reconstruct", help="COLMAP SfM + pose export (lane B / M2)")
+    p_rec.set_defaults(func=_cmd_reconstruct)
+
+    p_evt = sub.add_parser("events", help="audio candidates + AI classification (lane D / M7)")
+    p_evt.add_argument("--t-global-s", dest="t_global_s", type=float, default=None,
+                       help="detect within a single window centered here (default: whole track)")
+    p_evt.add_argument("--window-s", dest="window_s", type=float, default=None,
+                       help="window width in seconds when --t-global-s is given")
+    p_evt.set_defaults(func=_cmd_events)
+
+    p_srv = sub.add_parser("serve", help="run the FastAPI app (M3)")
+    p_srv.add_argument("--host", default=config.API_HOST)
+    p_srv.add_argument("--port", type=int, default=config.API_PORT)
+    p_srv.add_argument("--reload", action="store_true", help="uvicorn autoreload (dev)")
+    p_srv.set_defaults(func=_cmd_serve)
+
+    return parser
+
+
+def main(argv: list[str] | None = None) -> int:
+    logging.basicConfig(
+        level=logging.INFO,
+        format="%(asctime)s %(levelname)-7s %(name)s: %(message)s",
+    )
+    parser = build_parser()
+    args = parser.parse_args(argv)
+    try:
+        return args.func(args)
+    except NotImplementedError as exc:
+        detail = f" ({exc})" if str(exc) else ""
+        print(
+            f"festival4d {args.command}: not implemented yet{detail}.\n"
+            f"This subcommand is owned by a feature lane that hasn't landed. "
+            f"Run `python -m festival4d synthetic` for the working demo path.",
+            file=sys.stderr,
+        )
+        return 2
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

backend/festival4d/config.py

@@ -0,0 +1,78 @@
+"""Paths, constants, and the timebase convention.
+
+Timebase convention (FROZEN CONTRACT — do not change without a change request):
+
+    ``t_global`` is the master timeline in seconds.
+    For a video *v* with ``offset_ms`` and ``drift_ppm``::
+
+        t_video = (t_global - offset_ms / 1000) * (1 + drift_ppm * 1e-6)
+
+    The reference video has ``offset_ms == 0`` (and ``drift_ppm == 0``). A positive
+    ``offset_ms`` means the video started recording *later* than the master zero, so at a
+    given ``t_global`` its local playhead is earlier. Use :func:`t_video_from_global` and
+    :func:`t_global_from_video` everywhere — never re-derive the algebra inline. The
+    frontend mirrors :func:`t_video_from_global` in ``transport.js``.
+"""
+
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+# ---------------------------------------------------------------------------
+# Paths. The data root can be overridden with FESTIVAL4D_DATA_DIR (used by tests
+# to run against a temporary project without touching the repo's data/).
+# ---------------------------------------------------------------------------
+REPO_ROOT = Path(__file__).resolve().parents[2]
+
+
+def _data_dir() -> Path:
+    env = os.environ.get("FESTIVAL4D_DATA_DIR")
+    return Path(env).resolve() if env else REPO_ROOT / "data"
+
+
+DATA_DIR = _data_dir()
+RAW_DIR = DATA_DIR / "raw"          # user drops source videos here (gitignored)
+WORK_DIR = DATA_DIR / "work"        # extracted wavs, frames, colmap workspace (gitignored)
+AUDIO_DIR = WORK_DIR / "audio"      # per-video mono 16 kHz wavs (ingest)
+FRAMES_DIR = WORK_DIR / "frames"    # sampled JPEGs for SfM
+COLMAP_DIR = WORK_DIR / "colmap"    # COLMAP workspace
+DB_PATH = DATA_DIR / "project.db"   # SQLite (gitignored)
+
+POINTS_PLY = WORK_DIR / "points.ply"        # reconstructed / synthetic point cloud
+SYNC_JSON = WORK_DIR / "sync.json"          # exported sync solution (lane A)
+GROUND_TRUTH_JSON = WORK_DIR / "ground_truth.json"  # synthetic fixture ground truth
+
+# ---------------------------------------------------------------------------
+# Constants
+# ---------------------------------------------------------------------------
+FRONTEND_ORIGINS = [
+    "http://localhost:5173",
+    "http://127.0.0.1:5173",
+]
+# Local-only, single-user, offline tool: allow any localhost origin so a Vite dev server on
+# a fallback port (5174, …) still works. Kept alongside the explicit 5173 default.
+FRONTEND_ORIGIN_REGEX = r"http://(localhost|127\.0\.0\.1)(:\d+)?"
+API_HOST = "127.0.0.1"
+API_PORT = 8000
+
+AUDIO_SAMPLE_RATE = 16_000          # ingest extracts mono 16 kHz for sync (M1)
+
+# Synthetic fixture parameters (M0). Ground-truth offsets are deliberately odd,
+# non-round numbers so a naive integer-second guesser can't accidentally "pass".
+SYNTH_VIDEO_W = 640
+SYNTH_VIDEO_H = 360
+SYNTH_FPS = 30
+SYNTH_DURATION_S = 20.0
+SYNTH_AUDIO_SR = 48_000
+SYNTH_OFFSETS_MS = (0.0, 1370.0, -842.0)   # cam0 is the reference (offset 0)
+
+
+def t_video_from_global(t_global: float, offset_ms: float, drift_ppm: float = 0.0) -> float:
+    """Map master-timeline seconds to a video's local playhead seconds (FROZEN)."""
+    return (t_global - offset_ms / 1000.0) * (1.0 + drift_ppm * 1e-6)
+
+
+def t_global_from_video(t_video: float, offset_ms: float, drift_ppm: float = 0.0) -> float:
+    """Inverse of :func:`t_video_from_global` (FROZEN)."""
+    return t_video / (1.0 + drift_ppm * 1e-6) + offset_ms / 1000.0

backend/festival4d/db.py

@@ -0,0 +1,397 @@
+"""SQLAlchemy engine/session and schema (spec §2) plus small CRUD helpers.
+
+FROZEN CONTRACT after foundation: the schema (table/column names and types) and the
+helper signatures are what every lane depends on. Lanes write through these helpers only.
+
+Schema (spec §2)::
+
+    videos(id, filename, duration_s, fps, width, height, offset_ms, drift_ppm,
+           sync_confidence, created_at)
+    camera_poses(id, video_id FK, frame_idx, t_video_s, qw,qx,qy,qz, tx,ty,tz,
+                 fx,fy,cx,cy, registered)      # COLMAP world->camera; registered=False => interpolated
+    anchors(id, label, x,y,z, color)
+    events(id, t_global_s, duration_s, event_type, confidence, description, source)
+    annotations(id, video_id FK, t_video_s, x0,y0,x1,y1, resolved_anchor_id FK NULL)
+
+Engine management: :func:`init_engine` (re)binds the module to a SQLite file. It defaults
+to ``config.DB_PATH`` but tests point it at a temp file. Helpers open and commit their own
+short-lived sessions and return detached ORM instances (``expire_on_commit=False``), so the
+returned objects are safe to read after the session closes.
+"""
+
+from __future__ import annotations
+
+from contextlib import contextmanager
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Iterable, Iterator, Sequence
+
+from sqlalchemy import (
+    Boolean,
+    Float,
+    ForeignKey,
+    Integer,
+    String,
+    create_engine,
+    delete,
+    select,
+)
+from sqlalchemy.engine import Engine
+from sqlalchemy.orm import DeclarativeBase, Mapped, Session, mapped_column, sessionmaker
+
+from festival4d import config
+
+
+# ---------------------------------------------------------------------------
+# ORM models
+# ---------------------------------------------------------------------------
+class Base(DeclarativeBase):
+    pass
+
+
+class Video(Base):
+    __tablename__ = "videos"
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True)
+    filename: Mapped[str] = mapped_column(String, nullable=False, unique=True)
+    duration_s: Mapped[float] = mapped_column(Float, nullable=False)
+    fps: Mapped[float] = mapped_column(Float, nullable=False)
+    width: Mapped[int] = mapped_column(Integer, nullable=False)
+    height: Mapped[int] = mapped_column(Integer, nullable=False)
+    offset_ms: Mapped[float | None] = mapped_column(Float, nullable=True)
+    drift_ppm: Mapped[float | None] = mapped_column(Float, nullable=True)
+    sync_confidence: Mapped[float | None] = mapped_column(Float, nullable=True)
+    created_at: Mapped[str] = mapped_column(String, nullable=False)
+
+
+class CameraPose(Base):
+    __tablename__ = "camera_poses"
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True)
+    video_id: Mapped[int] = mapped_column(ForeignKey("videos.id"), nullable=False, index=True)
+    frame_idx: Mapped[int] = mapped_column(Integer, nullable=False)
+    t_video_s: Mapped[float] = mapped_column(Float, nullable=False)
+    # COLMAP world->camera quaternion [w,x,y,z] + translation
+    qw: Mapped[float] = mapped_column(Float, nullable=False)
+    qx: Mapped[float] = mapped_column(Float, nullable=False)
+    qy: Mapped[float] = mapped_column(Float, nullable=False)
+    qz: Mapped[float] = mapped_column(Float, nullable=False)
+    tx: Mapped[float] = mapped_column(Float, nullable=False)
+    ty: Mapped[float] = mapped_column(Float, nullable=False)
+    tz: Mapped[float] = mapped_column(Float, nullable=False)
+    # pinhole intrinsics after undistortion
+    fx: Mapped[float] = mapped_column(Float, nullable=False)
+    fy: Mapped[float] = mapped_column(Float, nullable=False)
+    cx: Mapped[float] = mapped_column(Float, nullable=False)
+    cy: Mapped[float] = mapped_column(Float, nullable=False)
+    registered: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
+
+
+class Anchor(Base):
+    __tablename__ = "anchors"
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True)
+    label: Mapped[str] = mapped_column(String, nullable=False)
+    x: Mapped[float] = mapped_column(Float, nullable=False)
+    y: Mapped[float] = mapped_column(Float, nullable=False)
+    z: Mapped[float] = mapped_column(Float, nullable=False)
+    color: Mapped[str | None] = mapped_column(String, nullable=True)
+
+
+class Event(Base):
+    __tablename__ = "events"
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True)
+    t_global_s: Mapped[float] = mapped_column(Float, nullable=False)
+    duration_s: Mapped[float | None] = mapped_column(Float, nullable=True)
+    event_type: Mapped[str] = mapped_column(String, nullable=False)
+    confidence: Mapped[float | None] = mapped_column(Float, nullable=True)
+    description: Mapped[str | None] = mapped_column(String, nullable=True)
+    source: Mapped[str] = mapped_column(String, nullable=False)  # 'audio_auto' | 'ai' | 'user'
+
+
+class Annotation(Base):
+    __tablename__ = "annotations"
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True)
+    video_id: Mapped[int] = mapped_column(ForeignKey("videos.id"), nullable=False, index=True)
+    t_video_s: Mapped[float] = mapped_column(Float, nullable=False)
+    x0: Mapped[float] = mapped_column(Float, nullable=False)
+    y0: Mapped[float] = mapped_column(Float, nullable=False)
+    x1: Mapped[float] = mapped_column(Float, nullable=False)
+    y1: Mapped[float] = mapped_column(Float, nullable=False)
+    resolved_anchor_id: Mapped[int | None] = mapped_column(
+        ForeignKey("anchors.id"), nullable=True
+    )
+
+
+# ---------------------------------------------------------------------------
+# Engine / session management
+# ---------------------------------------------------------------------------
+_engine: Engine | None = None
+_SessionLocal: sessionmaker[Session] | None = None
+_bound_path: Path | None = None
+
+
+def init_engine(db_path: str | Path | None = None) -> Engine:
+    """(Re)bind the module to a SQLite database file and return the engine.
+
+    Called with no argument uses ``config.DB_PATH``. Idempotent for the same path.
+    """
+    global _engine, _SessionLocal, _bound_path
+    path = Path(db_path) if db_path is not None else config.DB_PATH
+    if _engine is not None and _bound_path == path:
+        return _engine
+    path.parent.mkdir(parents=True, exist_ok=True)
+    if _engine is not None:
+        _engine.dispose()
+    _engine = create_engine(
+        f"sqlite:///{path}",
+        connect_args={"check_same_thread": False},
+        future=True,
+    )
+    _SessionLocal = sessionmaker(bind=_engine, expire_on_commit=False, future=True)
+    _bound_path = path
+    return _engine
+
+
+def engine() -> Engine:
+    if _engine is None:
+        init_engine()
+    assert _engine is not None
+    return _engine
+
+
+@contextmanager
+def session_scope() -> Iterator[Session]:
+    """Transactional session context; commits on success, rolls back on error."""
+    if _SessionLocal is None:
+        init_engine()
+    assert _SessionLocal is not None
+    session = _SessionLocal()
+    try:
+        yield session
+        session.commit()
+    except Exception:
+        session.rollback()
+        raise
+    finally:
+        session.close()
+
+
+def init_db() -> None:
+    """Create all tables if they do not exist."""
+    Base.metadata.create_all(engine())
+
+
+def reset_db() -> None:
+    """Drop and recreate all tables (used by the synthetic fixture generator)."""
+    Base.metadata.drop_all(engine())
+    Base.metadata.create_all(engine())
+
+
+def _now_iso() -> str:
+    return datetime.now(timezone.utc).isoformat()
+
+
+# ---------------------------------------------------------------------------
+# Videos
+# ---------------------------------------------------------------------------
+def add_video(
+    filename: str,
+    duration_s: float,
+    fps: float,
+    width: int,
+    height: int,
+    offset_ms: float | None = None,
+    drift_ppm: float | None = None,
+    sync_confidence: float | None = None,
+) -> Video:
+    """Insert a video row and return it (used by ingest + synthetic)."""
+    with session_scope() as s:
+        video = Video(
+            filename=filename,
+            duration_s=duration_s,
+            fps=fps,
+            width=width,
+            height=height,
+            offset_ms=offset_ms,
+            drift_ppm=drift_ppm,
+            sync_confidence=sync_confidence,
+            created_at=_now_iso(),
+        )
+        s.add(video)
+        s.flush()
+        s.refresh(video)
+        return video
+
+
+def get_videos() -> list[Video]:
+    with session_scope() as s:
+        return list(s.scalars(select(Video).order_by(Video.id)))
+
+
+def get_video(video_id: int) -> Video | None:
+    with session_scope() as s:
+        return s.get(Video, video_id)
+
+
+def update_video_sync(
+    video_id: int,
+    offset_ms: float | None,
+    drift_ppm: float | None,
+    sync_confidence: float | None,
+) -> None:
+    """Persist the audio-sync solution for a video (lane A / M1)."""
+    with session_scope() as s:
+        video = s.get(Video, video_id)
+        if video is None:
+            raise KeyError(f"no video with id={video_id}")
+        video.offset_ms = offset_ms
+        video.drift_ppm = drift_ppm
+        video.sync_confidence = sync_confidence
+
+
+# ---------------------------------------------------------------------------
+# Camera poses
+# ---------------------------------------------------------------------------
+def set_poses(video_id: int, poses: Iterable[dict]) -> int:
+    """Replace all poses for a video with ``poses`` (lane B / M2, and synthetic).
+
+    Each dict must have keys: frame_idx, t_video_s, qw, qx, qy, qz, tx, ty, tz,
+    fx, fy, cx, cy, and optionally ``registered`` (default True). Returns the count.
+
+    This is atomic: existing poses are only deleted inside the same transaction that
+    inserts the new ones, so a caller that raises before returning never corrupts the DB.
+    """
+    rows = list(poses)
+    with session_scope() as s:
+        s.execute(delete(CameraPose).where(CameraPose.video_id == video_id))
+        for p in rows:
+            s.add(
+                CameraPose(
+                    video_id=video_id,
+                    frame_idx=int(p["frame_idx"]),
+                    t_video_s=float(p["t_video_s"]),
+                    qw=float(p["qw"]), qx=float(p["qx"]), qy=float(p["qy"]), qz=float(p["qz"]),
+                    tx=float(p["tx"]), ty=float(p["ty"]), tz=float(p["tz"]),
+                    fx=float(p["fx"]), fy=float(p["fy"]), cx=float(p["cx"]), cy=float(p["cy"]),
+                    registered=bool(p.get("registered", True)),
+                )
+            )
+    return len(rows)
+
+
+def get_poses(video_id: int) -> list[CameraPose]:
+    with session_scope() as s:
+        return list(
+            s.scalars(
+                select(CameraPose)
+                .where(CameraPose.video_id == video_id)
+                .order_by(CameraPose.t_video_s)
+            )
+        )
+
+
+def has_poses() -> bool:
+    with session_scope() as s:
+        return s.scalar(select(CameraPose.id).limit(1)) is not None
+
+
+# ---------------------------------------------------------------------------
+# Anchors
+# ---------------------------------------------------------------------------
+def add_anchor(label: str, x: float, y: float, z: float, color: str | None = None) -> Anchor:
+    with session_scope() as s:
+        anchor = Anchor(label=label, x=x, y=y, z=z, color=color)
+        s.add(anchor)
+        s.flush()
+        s.refresh(anchor)
+        return anchor
+
+
+def get_anchors() -> list[Anchor]:
+    with session_scope() as s:
+        return list(s.scalars(select(Anchor).order_by(Anchor.id)))
+
+
+# ---------------------------------------------------------------------------
+# Events
+# ---------------------------------------------------------------------------
+def add_event(
+    t_global_s: float,
+    event_type: str,
+    source: str,
+    duration_s: float | None = None,
+    confidence: float | None = None,
+    description: str | None = None,
+) -> Event:
+    with session_scope() as s:
+        event = Event(
+            t_global_s=t_global_s,
+            duration_s=duration_s,
+            event_type=event_type,
+            confidence=confidence,
+            description=description,
+            source=source,
+        )
+        s.add(event)
+        s.flush()
+        s.refresh(event)
+        return event
+
+
+def get_events() -> list[Event]:
+    with session_scope() as s:
+        return list(s.scalars(select(Event).order_by(Event.t_global_s)))
+
+
+def clear_events(source: str | None = None) -> int:
+    """Delete events, optionally only those from a given ``source``. Returns count deleted."""
+    with session_scope() as s:
+        stmt = delete(Event)
+        if source is not None:
+            stmt = stmt.where(Event.source == source)
+        result = s.execute(stmt)
+        return int(result.rowcount or 0)
+
+
+# ---------------------------------------------------------------------------
+# Annotations
+# ---------------------------------------------------------------------------
+def add_annotation(
+    video_id: int,
+    t_video_s: float,
+    x0: float,
+    y0: float,
+    x1: float,
+    y1: float,
+    resolved_anchor_id: int | None = None,
+) -> Annotation:
+    with session_scope() as s:
+        annotation = Annotation(
+            video_id=video_id,
+            t_video_s=t_video_s,
+            x0=x0, y0=y0, x1=x1, y1=y1,
+            resolved_anchor_id=resolved_anchor_id,
+        )
+        s.add(annotation)
+        s.flush()
+        s.refresh(annotation)
+        return annotation
+
+
+def get_annotations(video_id: int | None = None) -> list[Annotation]:
+    with session_scope() as s:
+        stmt = select(Annotation).order_by(Annotation.id)
+        if video_id is not None:
+            stmt = stmt.where(Annotation.video_id == video_id)
+        return list(s.scalars(stmt))
+
+
+def set_annotation_anchor(annotation_id: int, anchor_id: int) -> None:
+    with session_scope() as s:
+        annotation = s.get(Annotation, annotation_id)
+        if annotation is None:
+            raise KeyError(f"no annotation with id={annotation_id}")
+        annotation.resolved_anchor_id = anchor_id

backend/festival4d/events_ai.py

@@ -0,0 +1,545 @@
+"""Moment detection + pluggable AI classification (spec M7, lane D).
+
+Two stages: cheap audio candidates first, then a vision LLM on candidates only.
+
+This module defines the **FROZEN classifier contract** — :class:`MomentClassification`
+(Pydantic model) and :class:`MomentClassifier` (Protocol) — exactly as spec M7. Lanes and
+integration depend on these shapes. Lane D fills the rest: candidate detection, the provider
+classes, provider selection, and the ``run_events`` entrypoint (called by both ``cli.py`` and
+``api.py``'s ``POST /api/events/detect``).
+
+Provider selection (lane D): ``FESTIVAL4D_CLASSIFIER=gemini|claude|local`` (default
+``gemini``). Inputs prepared once per candidate: a ~3 s MP4 clip around ``t_global`` from
+the best-covering video and 6 sampled JPEGs for providers without video input. Common
+behavior: catch exceptions per candidate (one failure never aborts the batch); if the
+selected provider is unconfigured, run candidate detection only (``event_type='candidate'``)
+and log that classification was skipped.
+
+The module imports with only ``pydantic`` present — heavy numeric libs (``numpy``,
+``librosa``, ``scipy``, ``soundfile``) and provider SDKs (``google-genai``, ``anthropic``,
+``openai``) are imported lazily inside the functions that need them, never at load.
+
+DB note (lane D): ``db.py`` (frozen) exposes ``add_event`` and ``clear_events`` but *no*
+update-event helper, so instead of "insert audio_auto candidate, later update to ai" this
+module inserts each event **once in its final state** — ``source='ai'`` with the classified
+type when a provider succeeds, else ``source='audio_auto'`` with ``event_type='candidate'``.
+The end state in the DB is identical to the spec's two-step description.
+"""
+
+from __future__ import annotations
+
+import base64
+import logging
+import os
+import shutil
+import subprocess
+from pathlib import Path
+from typing import Literal, Protocol, runtime_checkable
+
+from pydantic import BaseModel, Field
+
+from festival4d import config, db
+
+log = logging.getLogger("festival4d.events_ai")
+
+# The closed set of moment classes (spec M7). Lane C colors timeline markers by these.
+EventType = Literal[
+    "bass_drop", "pyro", "confetti", "crowd_wave",
+    "artist_moment", "light_show", "quiet_moment", "other",
+]
+EVENT_TYPES: tuple[str, ...] = (
+    "bass_drop", "pyro", "confetti", "crowd_wave",
+    "artist_moment", "light_show", "quiet_moment", "other",
+)
+
+# ---------------------------------------------------------------------------
+# Tuning constants (lane D). Detection is verified against the synthetic fixture's
+# ground-truth pulse times (spec M0/M7); see backend/tests/test_events_ai.py.
+# ---------------------------------------------------------------------------
+STFT_HOP = 512                 # librosa frame hop (samples)
+STFT_FRAME = 2048              # RMS window length (samples)
+NEIGHBORHOOD_S = 10.0          # a candidate must be the max over this centered window
+PEAK_PERCENTILE = 90.0         # ...and above this percentile of the detection function
+MERGE_S = 1.0                  # merge peaks closer than this, keeping the strongest
+MIN_SIGNAL_STD = 1e-3         # AC-RMS floor (~−60 dBFS): silence AND constant/DC have ~0 std
+CANDIDATE_DURATION_S = 1.0     # stored event duration (a moment, not an interval)
+DEFAULT_WINDOW_S = 6.0         # window width when run_events is given only t_global_s
+
+CLIP_SECONDS = 3.0             # length of the classifier clip (spec M7: ~3 s)
+NUM_FRAMES = 6                 # sampled JPEGs for providers without video input
+FRAME_LONG_EDGE = 768          # max long-edge px for sampled frames (spec M7: ≤768)
+CLIP_MAX_HEIGHT = 720          # clip capped at ≤720p (spec M7)
+AUDIO_EXTRACT_SR = 48_000      # sample rate when extracting reference audio via ffmpeg
+
+
+class MomentClassification(BaseModel):
+    """Structured classifier output (FROZEN CONTRACT, spec M7)."""
+
+    event_type: EventType
+    confidence: float = Field(ge=0.0, le=1.0)   # 0..1
+    description: str                            # one sentence, human-readable
+
+
+@runtime_checkable
+class MomentClassifier(Protocol):
+    """Interchangeable classifier provider (FROZEN CONTRACT, spec M7)."""
+
+    def classify(self, clip_path: Path, frames: list[Path]) -> MomentClassification:
+        """Classify one candidate moment from a short clip + sampled frames."""
+        ...
+
+
+_CLASSIFY_PROMPT = (
+    "This is a ~3-second clip from a concert, captured around a detected audio spike. "
+    "Classify the single most salient moment in it. Choose event_type from: "
+    "bass_drop, pyro, confetti, crowd_wave, artist_moment, light_show, quiet_moment, other. "
+    "Give a confidence in [0, 1] and one short human-readable sentence describing what happens."
+)
+
+
+# ---------------------------------------------------------------------------
+# Provider classes (lane D). Each imports its SDK lazily so the module loads with
+# only pydantic present; construction reads the key/endpoint from the environment.
+# ---------------------------------------------------------------------------
+class GeminiClassifier:
+    """Default provider — Gemini 2.5 Flash, native video input (``GEMINI_API_KEY``)."""
+
+    model = "gemini-2.5-flash"
+
+    def __init__(self) -> None:
+        from google import genai
+
+        self._genai = genai
+        self._client = genai.Client()  # reads GEMINI_API_KEY from the environment
+
+    def classify(self, clip_path: Path, frames: list[Path]) -> MomentClassification:
+        genai = self._genai
+        clip_bytes = Path(clip_path).read_bytes()
+        resp = self._client.models.generate_content(
+            model=self.model,
+            contents=[
+                genai.types.Part.from_bytes(data=clip_bytes, mime_type="video/mp4"),
+                _CLASSIFY_PROMPT,
+            ],
+            config={
+                "response_mime_type": "application/json",
+                "response_schema": MomentClassification,
+            },
+        )
+        return MomentClassification.model_validate_json(resp.text)
+
+
+class ClaudeClassifier:
+    """Optional provider — Claude Opus 4.8, 6 frames as images (``ANTHROPIC_API_KEY``).
+
+    Uses ``client.messages.parse(..., thinking={'type': 'adaptive'},
+    output_format=MomentClassification)`` -> ``resp.parsed_output``; does **not** pass
+    ``temperature`` (rejected on Opus 4.8). Model id ``claude-opus-4-8`` (never date-suffixed).
+    """
+
+    model = "claude-opus-4-8"
+
+    def __init__(self) -> None:
+        import anthropic
+
+        self._client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment
+
+    def classify(self, clip_path: Path, frames: list[Path]) -> MomentClassification:
+        content: list[dict] = []
+        for fp in frames[:NUM_FRAMES]:
+            data = base64.standard_b64encode(Path(fp).read_bytes()).decode("ascii")
+            content.append({
+                "type": "image",
+                "source": {"type": "base64", "media_type": "image/jpeg", "data": data},
+            })
+        content.append({"type": "text", "text": _CLASSIFY_PROMPT})
+        resp = self._client.messages.parse(
+            model=self.model,
+            max_tokens=1024,
+            thinking={"type": "adaptive"},
+            output_format=MomentClassification,
+            messages=[{"role": "user", "content": content}],
+        )
+        return resp.parsed_output
+
+
+class LocalClassifier:
+    """Offline/free provider — any OpenAI-compatible vision endpoint (Ollama/LM Studio/
+    OpenRouter). Config: ``FESTIVAL4D_OPENAI_BASE_URL``, ``FESTIVAL4D_OPENAI_KEY``,
+    ``FESTIVAL4D_OPENAI_MODEL``. Sends the 6 frames as image parts; validates JSON with one
+    retry on failure (local models are sloppier about JSON).
+    """
+
+    def __init__(self) -> None:
+        from openai import OpenAI
+
+        base_url = os.environ["FESTIVAL4D_OPENAI_BASE_URL"]
+        # Local servers (Ollama/LM Studio) accept any non-empty key; OpenRouter needs a real one.
+        api_key = os.environ.get("FESTIVAL4D_OPENAI_KEY") or "not-needed"
+        self._model = os.environ["FESTIVAL4D_OPENAI_MODEL"]
+        self._client = OpenAI(base_url=base_url, api_key=api_key)
+
+    def classify(self, clip_path: Path, frames: list[Path]) -> MomentClassification:
+        from pydantic import ValidationError
+
+        schema = MomentClassification.model_json_schema()
+        content: list[dict] = [{
+            "type": "text",
+            "text": (
+                _CLASSIFY_PROMPT
+                + " Respond with ONLY a JSON object matching this schema: "
+                + repr(schema)
+            ),
+        }]
+        for fp in frames[:NUM_FRAMES]:
+            b64 = base64.standard_b64encode(Path(fp).read_bytes()).decode("ascii")
+            content.append({
+                "type": "image_url",
+                "image_url": {"url": f"data:image/jpeg;base64,{b64}"},
+            })
+        messages = [{"role": "user", "content": content}]
+
+        last_exc: Exception | None = None
+        for attempt in range(2):  # one retry on validation failure
+            resp = self._client.chat.completions.create(
+                model=self._model,
+                messages=messages,
+                response_format={"type": "json_object"},
+            )
+            text = resp.choices[0].message.content or ""
+            try:
+                return MomentClassification.model_validate_json(text)
+            except ValidationError as exc:
+                last_exc = exc
+                log.warning("events: local classifier returned invalid JSON (attempt %d)",
+                            attempt + 1)
+        assert last_exc is not None
+        raise last_exc
+
+
+_PROVIDERS: dict[str, tuple[type, tuple[str, ...]]] = {
+    # name -> (class, required environment variables)
+    "gemini": (GeminiClassifier, ("GEMINI_API_KEY",)),
+    "claude": (ClaudeClassifier, ("ANTHROPIC_API_KEY",)),
+    "local": (LocalClassifier, ("FESTIVAL4D_OPENAI_BASE_URL", "FESTIVAL4D_OPENAI_MODEL")),
+}
+
+
+def get_classifier(name: str | None = None) -> MomentClassifier | None:
+    """Select a provider by name or ``FESTIVAL4D_CLASSIFIER`` (default ``gemini``).
+
+    Returns a ready classifier, or ``None`` if the selected provider is unconfigured (no
+    key/endpoint) or fails to initialize — callers then run candidates-only.
+    """
+    name = (name or os.environ.get("FESTIVAL4D_CLASSIFIER") or "gemini").lower()
+    entry = _PROVIDERS.get(name)
+    if entry is None:
+        log.warning("events: unknown classifier %r; running candidates-only", name)
+        return None
+    cls, required = entry
+    missing = [var for var in required if not os.environ.get(var)]
+    if missing:
+        log.info("events: provider %r unconfigured (missing %s); classification skipped, "
+                 "storing candidates only", name, ", ".join(missing))
+        return None
+    try:
+        return cls()
+    except Exception:  # SDK import/construction failure -> degrade, don't crash the batch
+        log.exception("events: failed to initialize %r classifier; running candidates-only", name)
+        return None
+
+
+# ---------------------------------------------------------------------------
+# Candidate detection (spec M7). Verified against the fixture ground-truth pulses.
+# ---------------------------------------------------------------------------
+def detect_candidates(audio: "object", sr: int) -> list[float]:
+    """Audio candidate moments on the reference track (spec M7, lane D).
+
+    Compute RMS energy and spectral-flux onset strength (librosa), combine into a single
+    detection function (product — bangs dominate in both, so the product isolates them
+    cleanly from the regular beat floor), and return the times (reference-local seconds) of
+    peaks that are local maxima over a ~10 s neighborhood and above the 90th percentile.
+    Verified against the fixture's ground-truth pulse times (±0.5 s).
+    """
+    import librosa
+    import numpy as np
+    from scipy.ndimage import maximum_filter1d
+
+    y = np.asarray(audio, dtype=np.float32).reshape(-1)
+    if y.size < STFT_FRAME:
+        return []
+    # A reference track with no real moments (silence, constant/DC, or a bare noise floor)
+    # has ~0 AC energy. The percentile threshold below is purely *relative*, so on such a
+    # flat signal it would flag ~1 spurious peak per second; gate on AC-RMS (std) first —
+    # this catches silence and DC alike (peak amplitude does not: a DC offset has a large
+    # peak but no transients, only STFT-boundary artifacts).
+    if float(np.std(y)) < MIN_SIGNAL_STD:
+        return []
+
+    rms = librosa.feature.rms(y=y, hop_length=STFT_HOP, frame_length=STFT_FRAME)[0]
+    flux = librosa.onset.onset_strength(y=y, sr=sr, hop_length=STFT_HOP)
+    n = min(len(rms), len(flux))
+    if n == 0:
+        return []
+    rms, flux = rms[:n], flux[:n]
+    eps = 1e-12
+    strength = (rms / (rms.max() + eps)) * (flux / (flux.max() + eps))
+    # No transient structure at all (e.g. constant/DC audio -> flat detection function).
+    if float(strength.max()) <= eps:
+        return []
+
+    times = librosa.frames_to_time(np.arange(n), sr=sr, hop_length=STFT_HOP)
+    win = max(1, int(round(NEIGHBORHOOD_S * sr / STFT_HOP)))
+    if win % 2 == 0:
+        win += 1
+    # A frame is a local maximum if it equals the max over its centered ~10 s window. Because
+    # the fixture's bangs are >5 s apart, each is the unique max in its own window — a
+    # min-distance peak picker would instead merge the 7 s-spaced bangs and drop one.
+    is_local_max = strength >= (maximum_filter1d(strength, size=win, mode="nearest") - eps)
+    threshold = float(np.percentile(strength, PEAK_PERCENTILE))
+    idx = np.where(is_local_max & (strength >= threshold))[0]
+
+    # Greedy merge of near-duplicate peaks (plateaus), keeping the strongest.
+    peaks: list[float] = []
+    for i in sorted(idx.tolist(), key=lambda j: -strength[j]):
+        t = float(times[i])
+        if all(abs(t - p) >= MERGE_S for p in peaks):
+            peaks.append(t)
+    return sorted(peaks)
+
+
+# ---------------------------------------------------------------------------
+# Reference-audio loading + classifier-input preparation (ffmpeg).
+# ---------------------------------------------------------------------------
+def _select_reference(videos: list) -> "object | None":
+    """The reference video (spec §2: offset 0). Fall back to the smallest |offset|."""
+    if not videos:
+        return None
+    exact = [v for v in videos if v.offset_ms == 0]
+    if exact:
+        return exact[0]
+    with_offset = [v for v in videos if v.offset_ms is not None]
+    if with_offset:
+        return min(with_offset, key=lambda v: abs(v.offset_ms))
+    return videos[0]
+
+
+def _load_reference_audio(video) -> "tuple[object, int] | None":
+    """Load the reference video's audio as mono float samples.
+
+    Independent of lane A: extract the audio straight from the raw video with ffmpeg (a
+    reused ingest WAV would couple us to lane A's naming). Returns ``(samples, sr)`` or
+    ``None`` if the raw video or ffmpeg is unavailable.
+    """
+    import tempfile
+
+    import numpy as np
+    import soundfile as sf
+
+    raw = config.RAW_DIR / video.filename
+    if not raw.exists():
+        log.warning("events: raw video %s not found under %s", video.filename, config.RAW_DIR)
+        return None
+    if shutil.which("ffmpeg") is None:
+        log.warning("events: ffmpeg not on PATH; cannot extract reference audio")
+        return None
+
+    with tempfile.TemporaryDirectory() as tmp:
+        wav = Path(tmp) / "ref.wav"
+        cmd = [
+            "ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
+            "-i", str(raw), "-vn", "-ac", "1", "-ar", str(AUDIO_EXTRACT_SR),
+            "-f", "wav", str(wav),
+        ]
+        try:
+            subprocess.run(cmd, check=True, capture_output=True)
+        except subprocess.CalledProcessError as exc:
+            log.warning("events: ffmpeg audio extraction failed: %s", exc.stderr.decode("utf-8", "ignore"))
+            return None
+        samples, sr = sf.read(str(wav), dtype="float32", always_2d=False)
+    if getattr(samples, "ndim", 1) > 1:
+        samples = samples.mean(axis=1)
+    return np.asarray(samples, dtype=np.float32), int(sr)
+
+
+def _select_best_video(videos: list, t_global_s: float) -> "tuple[object, float] | None":
+    """The video whose 3 s window around ``t_global`` is best-covered (most centered)."""
+    best = None
+    best_score = None
+    best_t = 0.0
+    for v in videos:
+        t_video = config.t_video_from_global(t_global_s, v.offset_ms or 0.0, v.drift_ppm or 0.0)
+        margin = min(t_video, (v.duration_s or 0.0) - t_video)  # ≥ CLIP_SECONDS/2 => fully covers
+        if best_score is None or margin > best_score:
+            best_score, best, best_t = margin, v, t_video
+    if best is None:
+        return None
+    return best, best_t
+
+
+def _events_workdir() -> Path:
+    d = config.WORK_DIR / "events"
+    d.mkdir(parents=True, exist_ok=True)
+    return d
+
+
+def prepare_inputs(t_global_s: float) -> tuple[Path, list[Path]]:
+    """Prepare classifier inputs for a candidate: a ~3 s MP4 clip + up to 6 sampled JPEGs (M7).
+
+    Picks the best-covering video, then uses ffmpeg to cut a ≤720p clip (audio kept, for
+    Gemini's native-video path) and sample 6 frames (≤768 px long edge). Raises if ffmpeg
+    or a usable source video is missing — ``run_events`` isolates that per candidate.
+    """
+    if shutil.which("ffmpeg") is None:
+        raise RuntimeError("ffmpeg not found on PATH — required to prepare classifier inputs")
+
+    videos = db.get_videos()
+    picked = _select_best_video(videos, t_global_s)
+    if picked is None:
+        raise RuntimeError("no videos available to prepare classifier inputs")
+    video, t_video = picked
+    raw = config.RAW_DIR / video.filename
+    if not raw.exists():
+        raise RuntimeError(f"source video {raw} not found")
+
+    duration = video.duration_s or (t_video + CLIP_SECONDS)
+    start = max(0.0, min(t_video - CLIP_SECONDS / 2.0, max(0.0, duration - CLIP_SECONDS)))
+
+    workdir = _events_workdir()
+    tag = f"cand_{t_global_s:0.3f}".replace("-", "m")
+    clip_path = workdir / f"{tag}.mp4"
+    frames_dir = workdir / tag
+    if frames_dir.exists():
+        for old in frames_dir.glob("frame_*.jpg"):
+            old.unlink()
+    frames_dir.mkdir(parents=True, exist_ok=True)
+
+    # 3 s clip, downscaled only if taller than 720p, audio preserved, faststart for inline bytes.
+    subprocess.run(
+        ["ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
+         "-ss", f"{start:.3f}", "-i", str(raw), "-t", f"{CLIP_SECONDS:.3f}",
+         "-vf", f"scale=-2:'min({CLIP_MAX_HEIGHT},ih)'",
+         "-c:v", "libx264", "-preset", "veryfast", "-crf", "28", "-pix_fmt", "yuv420p",
+         "-c:a", "aac", "-b:a", "96k", "-movflags", "+faststart", str(clip_path)],
+        check=True, capture_output=True,
+    )
+    # 6 frames evenly across the 3 s window, long edge ≤ 768 px.
+    scale = (f"scale='if(gt(iw,ih),min({FRAME_LONG_EDGE},iw),-2)':"
+             f"'if(gt(iw,ih),-2,min({FRAME_LONG_EDGE},ih))'")
+    fps = NUM_FRAMES / CLIP_SECONDS
+    subprocess.run(
+        ["ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
+         "-ss", f"{start:.3f}", "-i", str(raw), "-t", f"{CLIP_SECONDS:.3f}",
+         "-vf", f"fps={fps},{scale}", "-frames:v", str(NUM_FRAMES),
+         str(frames_dir / "frame_%02d.jpg")],
+        check=True, capture_output=True,
+    )
+    frames = sorted(frames_dir.glob("frame_*.jpg"))
+    return clip_path, frames
+
+
+# ---------------------------------------------------------------------------
+# Orchestration (spec M7). Entrypoint for the CLI + POST /api/events/detect.
+# ---------------------------------------------------------------------------
+def _classify_one(t_global_s: float, classifier: "MomentClassifier | None") -> tuple[str, str, float | None, str | None]:
+    """Resolve one candidate to (event_type, source, confidence, description).
+
+    No classifier -> a bare ``audio_auto`` candidate. Otherwise classify, isolating any
+    failure (SDK error, bad clip) by falling back to a candidate — one failure never aborts
+    the batch (spec M7).
+    """
+    if classifier is None:
+        return "candidate", "audio_auto", None, None
+    try:
+        clip_path, frames = prepare_inputs(t_global_s)
+        result = classifier.classify(clip_path, frames)
+        return result.event_type, "ai", result.confidence, result.description
+    except Exception:
+        log.exception("events: classification failed at t_global=%.3f s; storing as candidate",
+                      t_global_s)
+        return "candidate", "audio_auto", None, None
+
+
+def run_events(t_global_s: float | None = None, window_s: float | None = None) -> dict:
+    """Detect audio candidates and (if a provider is configured) classify them.
+
+    Entrypoint for ``python -m festival4d events`` and ``POST /api/events/detect``. Detects
+    on the reference track, maps candidate times to ``t_global``, then inserts events via
+    ``db`` (``source='audio_auto'`` for bare candidates, ``source='ai'`` for classified ones;
+    see the module docstring on why this is a single insert per event). Per-candidate
+    exceptions are isolated. Returns a summary dict.
+
+    Whole-track detection (``t_global_s is None``) is authoritative: it first clears prior
+    machine-generated events (``audio_auto`` + ``ai``) while preserving user edits. A windowed
+    call (``t_global_s`` given, optional ``window_s``) only *adds* candidates inside the
+    window and clears nothing, so it never disturbs the rest of the timeline.
+    """
+    db.init_engine()
+    db.init_db()
+
+    selected = (os.environ.get("FESTIVAL4D_CLASSIFIER") or "gemini").lower()
+    videos = db.get_videos()
+    reference = _select_reference(videos)
+    if reference is None:
+        log.warning("events: no videos registered; run `synthetic` or `ingest` first")
+        return _summary(selected, False, 0, 0, 0, None, note="no videos registered")
+
+    loaded = _load_reference_audio(reference)
+    if loaded is None:
+        return _summary(selected, False, 0, 0, 0, None,
+                        note="no reference audio (missing raw video or ffmpeg)")
+    audio, sr = loaded
+
+    local_times = detect_candidates(audio, sr)
+    off = reference.offset_ms or 0.0
+    drift = reference.drift_ppm or 0.0
+    cand_globals = [config.t_global_from_video(t, off, drift) for t in local_times]
+
+    window = None
+    if t_global_s is not None:
+        w = window_s if window_s is not None else DEFAULT_WINDOW_S
+        lo, hi = t_global_s - w / 2.0, t_global_s + w / 2.0
+        cand_globals = [t for t in cand_globals if lo <= t <= hi]
+        window = {"t_global_s": t_global_s, "window_s": w}
+    else:
+        # Whole-track detection replaces the machine timeline; user events are kept.
+        db.clear_events("audio_auto")
+        db.clear_events("ai")
+
+    classifier = get_classifier(selected)
+    n_ai = n_auto = 0
+    for t_global in cand_globals:
+        event_type, source, confidence, description = _classify_one(t_global, classifier)
+        db.add_event(
+            t_global_s=t_global,
+            event_type=event_type,
+            source=source,
+            duration_s=CANDIDATE_DURATION_S,
+            confidence=confidence,
+            description=description,
+        )
+        if source == "ai":
+            n_ai += 1
+        else:
+            n_auto += 1
+
+    summary = _summary(selected, classifier is not None, len(cand_globals), n_ai, n_auto, window)
+    log.info("events: reference=%s candidates=%d classified=%d candidates_only=%d provider=%s",
+             reference.filename, len(cand_globals), n_ai, n_auto,
+             selected if classifier is not None else f"{selected} (unconfigured)")
+    return summary
+
+
+def _summary(provider: str, configured: bool, candidates: int, classified: int,
+             candidates_only: int, window: dict | None, note: str | None = None) -> dict:
+    out = {
+        "provider": provider,
+        "classifier_configured": configured,
+        "candidates": candidates,
+        "classified": classified,
+        "candidates_only": candidates_only,
+        "window": window,
+    }
+    if note is not None:
+        out["note"] = note
+    return out

backend/festival4d/frames.py

@@ -0,0 +1,88 @@
+"""Sharpness-aware frame sampling for SfM (spec M2, lane B).
+
+STUB — lane B fills the bodies; the public signatures below are frozen (``sfm.py`` calls
+these). Sample candidate frames ~2/s per video; within each 0.5 s window keep the sharpest
+frame (variance of Laplacian via OpenCV). Write JPEGs named ``{video_id}_{frame_idx}.jpg``
+into ``config.FRAMES_DIR`` (one subfolder per video for COLMAP's single-camera-per-folder).
+"""
+
+from __future__ import annotations
+
+import logging
+from pathlib import Path
+
+import numpy as np
+
+log = logging.getLogger("festival4d.frames")
+
+
+def sharpness(image) -> float:
+    """Variance of the Laplacian of an image (OpenCV) — higher is sharper.
+
+    Accepts a BGR or grayscale ``ndarray`` (as returned by ``cv2.VideoCapture.read``).
+    The variance of the Laplacian is the standard focus/blur measure: a sharp frame has
+    strong high-frequency edges (high variance), a blurred one is smooth (low variance).
+    """
+    import cv2
+
+    img = np.asarray(image)
+    if img.ndim == 3:
+        img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
+    return float(cv2.Laplacian(img, cv2.CV_64F).var())
+
+
+def sample_frames(video_path: Path, video_id: int, out_dir: Path,
+                  target_fps: float = 2.0, window_s: float = 0.5) -> list[Path]:
+    """Sample the sharpest frame per ``window_s`` window from ``video_path`` (spec M2).
+
+    Decodes every frame, buckets them into non-overlapping ``window_s``-second windows, and
+    keeps the single sharpest frame (variance of Laplacian) in each window. This yields
+    ~``1 / window_s`` frames per second (≈ ``target_fps`` at the defaults), biased toward the
+    in-focus frames COLMAP needs. Writes JPEGs named ``{video_id}_{frame_idx}.jpg`` (the
+    original decoded ``frame_idx``, so ``t_video = frame_idx / fps``) into ``out_dir``.
+
+    Returns the written JPEG paths, ordered by frame index.
+    """
+    import cv2
+
+    out_dir = Path(out_dir)
+    out_dir.mkdir(parents=True, exist_ok=True)
+
+    cap = cv2.VideoCapture(str(video_path))
+    if not cap.isOpened():
+        raise RuntimeError(f"could not open video for frame sampling: {video_path}")
+    try:
+        fps = float(cap.get(cv2.CAP_PROP_FPS))
+        if not np.isfinite(fps) or fps <= 0.0:
+            log.warning("frames: %s reported fps=%s; falling back to 30.0", video_path, fps)
+            fps = 30.0
+        bucket_s = float(window_s) if window_s and window_s > 0 else 0.5
+
+        # window_idx -> (sharpness, frame_idx, frame_bgr). Only the current best per window
+        # is retained, so memory stays ~O(number of windows), not O(frames).
+        best: dict[int, tuple[float, int, np.ndarray]] = {}
+        frame_idx = 0
+        while True:
+            ok, frame = cap.read()
+            if not ok:
+                break
+            t = frame_idx / fps
+            win = int(t / bucket_s)
+            s = sharpness(frame)
+            cur = best.get(win)
+            if cur is None or s > cur[0]:
+                best[win] = (s, frame_idx, frame)
+            frame_idx += 1
+    finally:
+        cap.release()
+
+    written: list[Path] = []
+    for win in sorted(best):
+        _, fidx, frame = best[win]
+        path = out_dir / f"{video_id}_{fidx}.jpg"
+        if not cv2.imwrite(str(path), frame):
+            raise RuntimeError(f"failed to write frame JPEG: {path}")
+        written.append(path)
+    log.info("frames: %s -> %d frames sampled into %s", Path(video_path).name,
+             len(written), out_dir)
+    return written

backend/festival4d/geometry.py

@@ -0,0 +1,315 @@
+"""Coordinate conversions, quaternion math, ray casting, triangulation.
+
+The single most important thing in this file is :func:`colmap_to_threejs` — the
+COLMAP world->camera pose to Three.js camera conversion (spec M5). It is a **FROZEN
+CONTRACT**: it ships in foundation with a passing unit test (``test_geometry.py``), and
+the JS mirror lives in ``frontend/src/lib/pose.js`` with the *same* embedded test
+vectors. Lanes B and C **consume** it; they never reimplement or modify it.
+
+The remaining functions (:func:`slerp_pose`, :func:`ray_from_pixel`,
+:func:`triangulate_rays`, :func:`nearest_point_on_ray`) are stubs with frozen
+signatures for lane B (spec M2 + M8). Lane B fills the bodies and adds their tests.
+
+Conventions
+-----------
+- Quaternions are COLMAP order **[w, x, y, z]** (scalar first), unit norm.
+- A pose ``(q, t)`` is world->camera: ``x_cam = R(q) @ x_world + t``.
+- COLMAP camera axes: +x right, +y down, +z forward (into the scene).
+- Three.js cameras look down -z with +y up; hence the ``diag(1, -1, -1)`` flip.
+"""
+
+from __future__ import annotations
+
+import numpy as np
+from numpy.typing import ArrayLike, NDArray
+
+# Camera-axis flip that takes COLMAP camera-local axes (x right, y down, z forward)
+# to Three.js camera-local axes (x right, y up, z backward). Part of the frozen contract.
+_FLIP_YZ = np.diag([1.0, -1.0, -1.0])
+
+
+def quat_to_mat(q: ArrayLike) -> NDArray[np.float64]:
+    """Convert a unit quaternion ``[w, x, y, z]`` to a 3x3 rotation matrix.
+
+    Hamilton convention, right-handed, active rotation: the returned ``R`` is COLMAP's
+    world->camera matrix when ``q`` is a COLMAP pose quaternion. FROZEN (used by the
+    conversion contract).
+    """
+    w, x, y, z = (float(v) for v in np.asarray(q, dtype=np.float64).reshape(4))
+    n = w * w + x * x + y * y + z * z
+    if n < 1e-12:
+        raise ValueError("quaternion has near-zero norm")
+    s = 2.0 / n
+    wx, wy, wz = s * w * x, s * w * y, s * w * z
+    xx, xy, xz = s * x * x, s * x * y, s * x * z
+    yy, yz, zz = s * y * y, s * y * z, s * z * z
+    return np.array(
+        [
+            [1.0 - (yy + zz), xy - wz, xz + wy],
+            [xy + wz, 1.0 - (xx + zz), yz - wx],
+            [xz - wy, yz + wx, 1.0 - (xx + yy)],
+        ],
+        dtype=np.float64,
+    )
+
+
+def mat_to_quat(R: ArrayLike) -> NDArray[np.float64]:
+    """Convert a 3x3 rotation matrix to a unit quaternion ``[w, x, y, z]`` (w >= 0).
+
+    Inverse of :func:`quat_to_mat`. Used by tests and the pose export path.
+    """
+    m = np.asarray(R, dtype=np.float64).reshape(3, 3)
+    trace = m[0, 0] + m[1, 1] + m[2, 2]
+    if trace > 0.0:
+        s = np.sqrt(trace + 1.0) * 2.0
+        w = 0.25 * s
+        x = (m[2, 1] - m[1, 2]) / s
+        y = (m[0, 2] - m[2, 0]) / s
+        z = (m[1, 0] - m[0, 1]) / s
+    elif m[0, 0] > m[1, 1] and m[0, 0] > m[2, 2]:
+        s = np.sqrt(1.0 + m[0, 0] - m[1, 1] - m[2, 2]) * 2.0
+        w = (m[2, 1] - m[1, 2]) / s
+        x = 0.25 * s
+        y = (m[0, 1] + m[1, 0]) / s
+        z = (m[0, 2] + m[2, 0]) / s
+    elif m[1, 1] > m[2, 2]:
+        s = np.sqrt(1.0 + m[1, 1] - m[0, 0] - m[2, 2]) * 2.0
+        w = (m[0, 2] - m[2, 0]) / s
+        x = (m[0, 1] + m[1, 0]) / s
+        y = 0.25 * s
+        z = (m[1, 2] + m[2, 1]) / s
+    else:
+        s = np.sqrt(1.0 + m[2, 2] - m[0, 0] - m[1, 1]) * 2.0
+        w = (m[1, 0] - m[0, 1]) / s
+        x = (m[0, 2] + m[2, 0]) / s
+        y = (m[1, 2] + m[2, 1]) / s
+        z = 0.25 * s
+    q = np.array([w, x, y, z], dtype=np.float64)
+    q /= np.linalg.norm(q)
+    if q[0] < 0:  # canonical sign: non-negative scalar part
+        q = -q
+    return q
+
+
+def colmap_to_threejs(q: ArrayLike, t: ArrayLike) -> tuple[NDArray[np.float64], NDArray[np.float64]]:
+    """Convert a COLMAP world->camera pose to a Three.js camera pose (spec M5).
+
+    **FROZEN CONTRACT.** Mirrored in ``frontend/src/lib/pose.js``; do not change the math
+    without a change request and a synchronized update to both sides + their test vectors.
+
+    Parameters
+    ----------
+    q : array_like, shape (4,)
+        COLMAP world->camera quaternion ``[w, x, y, z]``.
+    t : array_like, shape (3,)
+        COLMAP world->camera translation ``[tx, ty, tz]``.
+
+    Returns
+    -------
+    position : ndarray, shape (3,)
+        Camera center in world coordinates, ``C = -R^T t``. Assign to ``camera.position``.
+    rotation_matrix : ndarray, shape (3, 3)
+        Three.js camera world rotation ``R_three = R^T @ diag(1, -1, -1)``. Assign via
+        ``camera.setRotationFromMatrix(...)`` (a proper rotation, det = +1).
+    """
+    R = quat_to_mat(q)                      # world -> cam
+    t_vec = np.asarray(t, dtype=np.float64).reshape(3)
+    R_c2w = R.T                             # cam -> world
+    position = -R_c2w @ t_vec              # camera center in world coords
+    rotation_matrix = R_c2w @ _FLIP_YZ     # flip camera-local y,z for Three.js
+    return position, rotation_matrix
+
+
+# ---------------------------------------------------------------------------
+# Lane B (spec M2 + M8): pose interpolation, pixel ray casting, two-view triangulation,
+# and the single-view nearest-point fallback. Signatures FROZEN; bodies implemented below.
+# ---------------------------------------------------------------------------
+def slerp_pose(
+    q0: ArrayLike,
+    t0: ArrayLike,
+    q1: ArrayLike,
+    t1: ArrayLike,
+    alpha: float,
+) -> tuple[NDArray[np.float64], NDArray[np.float64]]:
+    """Interpolate between two world->camera poses (spec M2 pose interpolation).
+
+    Spherically interpolate the rotation (slerp on the shorter arc, handling the
+    double-cover sign) and linearly interpolate the translation, at fraction
+    ``alpha in [0, 1]`` from pose 0 to pose 1.
+
+    Parameters
+    ----------
+    q0, q1 : array_like, shape (4,)
+        COLMAP quaternions ``[w, x, y, z]`` at the endpoints.
+    t0, t1 : array_like, shape (3,)
+        COLMAP translations at the endpoints.
+    alpha : float
+        Interpolation fraction; 0 returns pose 0, 1 returns pose 1.
+
+    Returns
+    -------
+    (q, t) : the interpolated quaternion ``[w, x, y, z]`` and translation ``[x, y, z]``.
+    """
+    a = float(alpha)
+    q0 = np.asarray(q0, dtype=np.float64).reshape(4)
+    q1 = np.asarray(q1, dtype=np.float64).reshape(4)
+    n0 = np.linalg.norm(q0)
+    n1 = np.linalg.norm(q1)
+    if n0 < 1e-12 or n1 < 1e-12:
+        raise ValueError("slerp endpoint quaternion has near-zero norm")
+    q0 = q0 / n0
+    q1 = q1 / n1
+
+    # Double cover: pick the sign of q1 that lies on the same hemisphere as q0, so slerp
+    # takes the shorter arc (a rotation and its negation are the same orientation).
+    dot = float(np.dot(q0, q1))
+    if dot < 0.0:
+        q1 = -q1
+        dot = -dot
+    dot = min(1.0, max(-1.0, dot))
+
+    if dot > 0.9995:
+        # Endpoints almost coincide: nlerp is numerically safe and visually identical.
+        q_interp = q0 + a * (q1 - q0)
+    else:
+        theta_0 = np.arccos(dot)
+        sin_0 = np.sin(theta_0)
+        s0 = np.sin((1.0 - a) * theta_0) / sin_0
+        s1 = np.sin(a * theta_0) / sin_0
+        q_interp = s0 * q0 + s1 * q1
+    q_interp = q_interp / np.linalg.norm(q_interp)
+
+    t0 = np.asarray(t0, dtype=np.float64).reshape(3)
+    t1 = np.asarray(t1, dtype=np.float64).reshape(3)
+    t_interp = (1.0 - a) * t0 + a * t1
+    return q_interp, t_interp
+
+
+def ray_from_pixel(
+    q: ArrayLike,
+    t: ArrayLike,
+    fx: float,
+    fy: float,
+    cx: float,
+    cy: float,
+    px: float,
+    py: float,
+) -> tuple[NDArray[np.float64], NDArray[np.float64]]:
+    """Unproject a pixel into a world-space ray (spec M8 annotation resolution).
+
+    Given the world->camera pose ``(q, t)`` and pinhole intrinsics, build the ray that
+    passes through image pixel ``(px, py)``.
+
+    Returns
+    -------
+    origin : ndarray, shape (3,)
+        Ray origin = camera center in world coords.
+    direction : ndarray, shape (3,)
+        Unit ray direction in world coords, pointing into the scene.
+    """
+    R = quat_to_mat(q)                          # world -> cam
+    t_vec = np.asarray(t, dtype=np.float64).reshape(3)
+    origin = -R.T @ t_vec                        # camera center in world coords
+
+    # Pinhole back-projection. A world point X projects with x_cam = R X + t and
+    # px = fx * x_cam.x / x_cam.z + cx, py = fy * x_cam.y / x_cam.z + cy
+    # (COLMAP camera axes: +x right, +y down, +z forward). So the camera-space direction
+    # through pixel (px, py) is [(px-cx)/fx, (py-cy)/fy, 1], pointing forward into the scene.
+    d_cam = np.array([(px - cx) / fx, (py - cy) / fy, 1.0], dtype=np.float64)
+    d_world = R.T @ d_cam                         # rotate direction cam -> world
+    norm = np.linalg.norm(d_world)
+    if norm < 1e-12:
+        raise ValueError("degenerate ray direction")
+    return origin, d_world / norm
+
+
+def triangulate_rays(
+    origin_a: ArrayLike,
+    dir_a: ArrayLike,
+    origin_b: ArrayLike,
+    dir_b: ArrayLike,
+) -> tuple[NDArray[np.float64], float]:
+    """Closest point between two world-space rays (spec M8 two-view triangulation).
+
+    Returns
+    -------
+    point : ndarray, shape (3,)
+        Midpoint of the shortest segment connecting the two rays.
+    gap : float
+        Length of that shortest segment (the mutual-nearest-approach distance). Callers
+        reject the triangulation when the rays are near-parallel or ``gap`` exceeds the
+        spec threshold (0.5 scene units).
+    """
+    oa = np.asarray(origin_a, dtype=np.float64).reshape(3)
+    ob = np.asarray(origin_b, dtype=np.float64).reshape(3)
+    da = np.asarray(dir_a, dtype=np.float64).reshape(3)
+    db = np.asarray(dir_b, dtype=np.float64).reshape(3)
+    na, nb = np.linalg.norm(da), np.linalg.norm(db)
+    if na < 1e-12 or nb < 1e-12:
+        raise ValueError("triangulate_rays: zero-length direction")
+    da = da / na
+    db = db / nb
+
+    # Shortest segment between two lines P(s)=oa+s*da, Q(u)=ob+u*db. Minimize |P-Q|^2.
+    # With unit directions: b = da.db, denom = 1 - b^2 (0 when parallel).
+    w0 = oa - ob
+    b = float(np.dot(da, db))
+    d = float(np.dot(da, w0))
+    e = float(np.dot(db, w0))
+    denom = 1.0 - b * b
+    if denom < 1e-9:
+        # Near-parallel: no unique closest pair. Anchor on oa, take the closest point on
+        # line b to it; gap is the line-to-line perpendicular distance. Callers reject
+        # near-parallel rays up front, so this branch just stays numerically safe.
+        s = 0.0
+        u = e
+    else:
+        s = (b * e - d) / denom
+        u = (e - b * d) / denom
+    pa = oa + s * da
+    pb = ob + u * db
+    point = 0.5 * (pa + pb)
+    gap = float(np.linalg.norm(pa - pb))
+    return point, gap
+
+
+def nearest_point_on_ray(
+    origin: ArrayLike,
+    direction: ArrayLike,
+    points: ArrayLike,
+    radius: float = 0.3,
+) -> NDArray[np.float64] | None:
+    """Nearest point-cloud point to a ray, within a cylinder (spec M8 single-view fallback).
+
+    Among ``points`` (shape ``(N, 3)``) find the one whose perpendicular distance to the
+    ray is smallest, considering only points within ``radius`` of the ray and in front of
+    the origin.
+
+    Returns
+    -------
+    point : ndarray shape (3,) or None
+        The selected point-cloud point, or ``None`` if none lie within ``radius``.
+    """
+    o = np.asarray(origin, dtype=np.float64).reshape(3)
+    d = np.asarray(direction, dtype=np.float64).reshape(3)
+    nd = np.linalg.norm(d)
+    if nd < 1e-12:
+        raise ValueError("nearest_point_on_ray: zero-length direction")
+    d = d / nd
+
+    pts = np.asarray(points, dtype=np.float64).reshape(-1, 3)
+    if len(pts) == 0:
+        return None
+
+    v = pts - o                          # origin -> each point
+    proj = v @ d                         # signed distance along the ray
+    perp = v - np.outer(proj, d)         # component perpendicular to the ray
+    perp_dist = np.linalg.norm(perp, axis=1)
+
+    # In front of the origin and inside the cylinder of the given radius.
+    mask = (proj > 0.0) & (perp_dist <= radius)
+    if not np.any(mask):
+        return None
+    idx_in = np.where(mask)[0]
+    best = idx_in[np.argmin(perp_dist[idx_in])]
+    return pts[best].copy()

backend/festival4d/ingest.py

@@ -0,0 +1,138 @@
+"""Video ingest: probe files in ``data/raw`` and extract audio (spec M1, lane A).
+
+For each file in ``config.RAW_DIR``: ffprobe it, insert/update the ``videos`` row via
+``db`` helpers, and extract a mono 16 kHz WAV into ``config.AUDIO_DIR`` (one per video,
+keyed by ``video_id``) — that WAV is exactly what ``audio_sync.run_sync`` loads.
+
+``run_ingest`` is idempotent: re-running after the synthetic fixture (which already
+registered the videos) reuses the existing rows and just re-extracts audio, so the
+demo path ``synthetic -> ingest -> sync`` works without unique-constraint collisions.
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+import shutil
+import subprocess
+from pathlib import Path
+
+from festival4d import config, db
+
+log = logging.getLogger("festival4d.ingest")
+
+# Container extensions we treat as ingestible video (audio is pulled from any of them).
+_VIDEO_EXTS = {".mp4", ".mov", ".m4v", ".mkv", ".avi", ".webm"}
+
+
+def _require(tool: str) -> None:
+    if shutil.which(tool) is None:
+        raise RuntimeError(
+            f"{tool} not found on PATH — required for ingest (macOS: `brew install ffmpeg`)"
+        )
+
+
+def audio_wav_path(video_id: int) -> Path:
+    """Canonical path of a video's extracted mono WAV.
+
+    Shared convention between ingest (writer) and :mod:`festival4d.audio_sync` (reader),
+    so neither side has to guess filenames.
+    """
+    return config.AUDIO_DIR / f"{video_id}.wav"
+
+
+def _parse_fps(rate: str | None) -> float | None:
+    """Parse an ffprobe frame-rate string (``"30/1"`` or ``"29.97"``) to fps."""
+    if not rate or rate in ("0/0", "N/A"):
+        return None
+    if "/" in rate:
+        num, den = rate.split("/", 1)
+        den = float(den)
+        return float(num) / den if den else None
+    return float(rate)
+
+
+def probe_video(path: Path) -> dict:
+    """ffprobe a video; return ``{duration_s, fps, width, height}`` (lane A / M1)."""
+    _require("ffprobe")
+    out = subprocess.run(
+        ["ffprobe", "-v", "error", "-select_streams", "v:0",
+         "-show_entries",
+         "stream=width,height,avg_frame_rate,r_frame_rate:format=duration",
+         "-of", "json", str(path)],
+        check=True, capture_output=True, text=True,
+    ).stdout
+    data = json.loads(out)
+    streams = data.get("streams") or []
+    if not streams:
+        raise ValueError(f"{path.name}: no video stream found")
+    stream = streams[0]
+    # avg_frame_rate is the true average; r_frame_rate is a fallback for odd containers.
+    fps = _parse_fps(stream.get("avg_frame_rate")) or _parse_fps(stream.get("r_frame_rate")) or 0.0
+    duration = float(data.get("format", {}).get("duration") or 0.0)
+    return {
+        "duration_s": duration,
+        "fps": float(fps),
+        "width": int(stream["width"]),
+        "height": int(stream["height"]),
+    }
+
+
+def extract_audio(path: Path, out_wav: Path, sample_rate: int = 16_000) -> Path:
+    """Extract a mono ``sample_rate`` 16-bit PCM WAV from ``path`` via ffmpeg (lane A / M1)."""
+    _require("ffmpeg")
+    out_wav.parent.mkdir(parents=True, exist_ok=True)
+    subprocess.run(
+        ["ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
+         "-i", str(path),
+         "-vn",                       # drop video
+         "-ac", "1",                  # mono
+         "-ar", str(sample_rate),     # resample
+         "-c:a", "pcm_s16le",         # uncompressed PCM (lossless for sync)
+         str(out_wav)],
+        check=True,
+    )
+    return out_wav
+
+
+def run_ingest() -> list[dict]:
+    """Ingest every video in ``config.RAW_DIR``: probe, register in DB, extract audio.
+
+    Entrypoint for ``python -m festival4d ingest``. Returns a per-video summary list.
+    """
+    db.init_engine()
+    db.init_db()
+
+    files = sorted(
+        p for p in config.RAW_DIR.glob("*") if p.suffix.lower() in _VIDEO_EXTS
+    )
+    if not files:
+        log.warning("ingest: no video files in %s (drop clips there first)", config.RAW_DIR)
+        return []
+
+    existing = {v.filename: v for v in db.get_videos()}
+    summary: list[dict] = []
+    for path in files:
+        info = probe_video(path)
+        video = existing.get(path.name)
+        reused = video is not None
+        if video is None:
+            video = db.add_video(
+                filename=path.name,
+                duration_s=info["duration_s"], fps=info["fps"],
+                width=info["width"], height=info["height"],
+            )
+        wav = audio_wav_path(video.id)
+        extract_audio(path, wav)
+        summary.append({
+            "video_id": video.id, "filename": path.name,
+            "duration_s": info["duration_s"], "fps": info["fps"],
+            "width": info["width"], "height": info["height"],
+            "audio_wav": str(wav), "reused_db_row": reused,
+        })
+        log.info("ingest: %s -> video_id=%d (%.1fs, %.2f fps, %dx%d)%s",
+                 path.name, video.id, info["duration_s"], info["fps"],
+                 info["width"], info["height"], "  [reused row]" if reused else "")
+
+    log.info("ingest: %d video(s) ready; audio in %s", len(summary), config.AUDIO_DIR)
+    return summary

backend/festival4d/sfm.py

@@ -0,0 +1,608 @@
+"""COLMAP orchestration, model parsing, scene normalization, pose export (spec M2, lane B).
+
+Pipeline (spec M2):
+- Sample frames (``frames.sample_frames``), then drive the COLMAP CLI via subprocess:
+  ``feature_extractor`` (OPENCV model, single camera per folder) -> ``exhaustive_matcher``
+  (small frame counts; exhaustive covers within- and cross-video pairs) -> ``mapper`` ->
+  ``image_undistorter`` + ``model_converter`` to a PINHOLE TXT model. Parse ``images.txt`` /
+  ``cameras.txt`` / ``points3D.txt`` by hand (no pycolmap).
+- **Normalize** the scene: centroid -> origin, camera bounding sphere radius -> 10, world-up
+  -> average camera up. The same similarity transform is applied to poses and points.
+- **Interpolate** poses for unregistered frames (slerp rotation, lerp translation, mark
+  ``registered=False``); never extrapolate past the first/last registered frame of a video.
+- Export ``config.POINTS_PLY`` (binary little-endian, matching ``synthetic.write_ply``) and
+  fill ``camera_poses`` via ``db.set_poses``.
+- **Failure handling** (spec M2): COLMAP optional at runtime (``shutil.which("colmap")``); if
+  it registers < 60% of sampled frames or < 2 videos into one model, print a clear diagnostic
+  and leave existing poses untouched — never corrupt the DB. The app degrades to "synced
+  videos, no 3D".
+"""
+
+from __future__ import annotations
+
+import logging
+import functools
+import struct
+import subprocess
+from pathlib import Path
+
+import numpy as np
+
+from festival4d import config, db, frames
+from festival4d.geometry import mat_to_quat, quat_to_mat, slerp_pose
+
+log = logging.getLogger("festival4d.sfm")
+
+# Minimum share of sampled frames COLMAP must register, and minimum videos in one model,
+# for the reconstruction to be trusted (spec M2 failure handling).
+MIN_REGISTERED_FRACTION = 0.60
+MIN_VIDEOS_IN_MODEL = 2
+NORMALIZED_SPHERE_RADIUS = 10.0
+
+# How COLMAP camera models lay out PARAMS -> (fx, fy, cx, cy). Distortion terms are ignored:
+# the stored intrinsics are the pinhole part (exact after image_undistorter's PINHOLE output,
+# approximate if a distorted model is parsed directly). Covers the models COLMAP emits here.
+_INTRINSICS_FROM_PARAMS = {
+    "SIMPLE_PINHOLE": lambda p: (p[0], p[0], p[1], p[2]),
+    "PINHOLE": lambda p: (p[0], p[1], p[2], p[3]),
+    "SIMPLE_RADIAL": lambda p: (p[0], p[0], p[1], p[2]),
+    "RADIAL": lambda p: (p[0], p[0], p[1], p[2]),
+    "SIMPLE_RADIAL_FISHEYE": lambda p: (p[0], p[0], p[1], p[2]),
+    "RADIAL_FISHEYE": lambda p: (p[0], p[0], p[1], p[2]),
+    "OPENCV": lambda p: (p[0], p[1], p[2], p[3]),
+    "OPENCV_FISHEYE": lambda p: (p[0], p[1], p[2], p[3]),
+    "FULL_OPENCV": lambda p: (p[0], p[1], p[2], p[3]),
+    "FOV": lambda p: (p[0], p[1], p[2], p[3]),
+    "THIN_PRISM_FISHEYE": lambda p: (p[0], p[1], p[2], p[3]),
+}
+
+
+def colmap_available() -> bool:
+    """Whether the COLMAP binary is on PATH (spec M2 optional-at-runtime rule)."""
+    import shutil
+
+    return shutil.which("colmap") is not None
+
+
+# ---------------------------------------------------------------------------
+# COLMAP TXT-model parsers (hand-written; no pycolmap dependency)
+# ---------------------------------------------------------------------------
+def _iter_data_lines(path: Path):
+    """Yield non-empty, non-comment lines from a COLMAP TXT file."""
+    with open(path, "r", encoding="utf-8") as f:
+        for line in f:
+            line = line.strip()
+            if line and not line.startswith("#"):
+                yield line
+
+
+def parse_images_txt(path: Path) -> list[dict]:
+    """Parse a COLMAP ``images.txt`` into per-image pose records (spec M2, lane B).
+
+    Format: two lines per image. The first is
+    ``IMAGE_ID QW QX QY QZ TX TY TZ CAMERA_ID NAME``; the second is the 2D keypoint list
+    (ignored here). Returns one dict per image with the world->camera quaternion
+    ``[w,x,y,z]``, translation, camera id, and image name.
+    """
+    records: list[dict] = []
+    take_pose_line = True  # image data alternates: pose line, then points2D line
+    for line in _iter_data_lines(path):
+        if take_pose_line:
+            parts = line.split()
+            if len(parts) < 10:
+                raise ValueError(f"malformed images.txt pose line: {line!r}")
+            records.append({
+                "image_id": int(parts[0]),
+                "qw": float(parts[1]), "qx": float(parts[2]),
+                "qy": float(parts[3]), "qz": float(parts[4]),
+                "tx": float(parts[5]), "ty": float(parts[6]), "tz": float(parts[7]),
+                "camera_id": int(parts[8]),
+                "name": " ".join(parts[9:]),  # NAME may itself contain spaces
+            })
+        take_pose_line = not take_pose_line
+    return records
+
+
+def parse_cameras_txt(path: Path) -> dict[int, dict]:
+    """Parse a COLMAP ``cameras.txt`` into ``camera_id -> intrinsics`` (spec M2, lane B).
+
+    Format per line: ``CAMERA_ID MODEL WIDTH HEIGHT PARAMS[]``. Returns, per camera id, a
+    dict with ``model``, ``width``, ``height``, pinhole ``fx, fy, cx, cy`` and the raw
+    ``params`` list.
+    """
+    cameras: dict[int, dict] = {}
+    for line in _iter_data_lines(path):
+        parts = line.split()
+        if len(parts) < 4:
+            raise ValueError(f"malformed cameras.txt line: {line!r}")
+        cam_id = int(parts[0])
+        model = parts[1]
+        width, height = int(parts[2]), int(parts[3])
+        params = [float(x) for x in parts[4:]]
+        if model not in _INTRINSICS_FROM_PARAMS:
+            raise ValueError(f"unsupported COLMAP camera model {model!r}")
+        try:
+            fx, fy, cx, cy = _INTRINSICS_FROM_PARAMS[model](params)
+        except IndexError:
+            raise ValueError(
+                f"cameras.txt: model {model} has too few PARAMS: {line!r}"
+            ) from None
+        cameras[cam_id] = {
+            "model": model, "width": width, "height": height,
+            "fx": float(fx), "fy": float(fy), "cx": float(cx), "cy": float(cy),
+            "params": params,
+        }
+    return cameras
+
+
+def parse_points3d_txt(path: Path) -> tuple[np.ndarray, np.ndarray]:
+    """Parse a COLMAP ``points3D.txt`` into ``(points Nx3 float32, colors Nx3 uint8)``.
+
+    Format per line: ``POINT3D_ID X Y Z R G B ERROR TRACK[]`` (track ignored).
+    """
+    xyz: list[tuple[float, float, float]] = []
+    rgb: list[tuple[int, int, int]] = []
+    for line in _iter_data_lines(path):
+        parts = line.split()
+        if len(parts) < 7:
+            raise ValueError(f"malformed points3D.txt line: {line!r}")
+        xyz.append((float(parts[1]), float(parts[2]), float(parts[3])))
+        rgb.append((int(parts[4]), int(parts[5]), int(parts[6])))
+    if not xyz:
+        return (np.zeros((0, 3), dtype=np.float32), np.zeros((0, 3), dtype=np.uint8))
+    return (np.asarray(xyz, dtype=np.float32), np.asarray(rgb, dtype=np.uint8))
+
+
+# ---------------------------------------------------------------------------
+# Scene normalization
+# ---------------------------------------------------------------------------
+def _rotation_aligning(a: np.ndarray, b: np.ndarray) -> np.ndarray:
+    """Shortest-arc rotation matrix that maps unit vector ``a`` onto unit vector ``b``."""
+    a = a / np.linalg.norm(a)
+    b = b / np.linalg.norm(b)
+    v = np.cross(a, b)
+    c = float(np.dot(a, b))
+    s = float(np.linalg.norm(v))
+    if s < 1e-12:
+        if c > 0:                       # already aligned
+            return np.eye(3)
+        # antiparallel: rotate 180 deg about any axis perpendicular to a
+        perp = np.array([1.0, 0.0, 0.0])
+        if abs(a[0]) > 0.9:
+            perp = np.array([0.0, 1.0, 0.0])
+        axis = np.cross(a, perp)
+        axis /= np.linalg.norm(axis)
+        return 2.0 * np.outer(axis, axis) - np.eye(3)
+    vx = np.array([[0.0, -v[2], v[1]], [v[2], 0.0, -v[0]], [-v[1], v[0], 0.0]])
+    return np.eye(3) + vx + vx @ vx * ((1.0 - c) / (s * s))
+
+
+def _camera_center(pose: dict) -> np.ndarray:
+    """World-space camera center ``C = -R^T t`` for a world->camera pose dict."""
+    R = quat_to_mat([pose["qw"], pose["qx"], pose["qy"], pose["qz"]])
+    t = np.array([pose["tx"], pose["ty"], pose["tz"]], dtype=np.float64)
+    return -R.T @ t
+
+
+def _camera_up(pose: dict) -> np.ndarray:
+    """World-space camera up vector. COLMAP camera +y is *down*, so up = R^T @ (0,-1,0)."""
+    R = quat_to_mat([pose["qw"], pose["qx"], pose["qy"], pose["qz"]])
+    return R.T @ np.array([0.0, -1.0, 0.0])
+
+
+def normalize_scene(points, poses):
+    """Similarity transform: centroid->origin, camera sphere radius->10, up->+Y (spec M2).
+
+    ``points`` is an ``(N,3)`` array; ``poses`` a list of world->camera pose dicts (keys
+    ``qw,qx,qy,qz,tx,ty,tz`` plus any extras, which are preserved). The single similarity
+    transform ``X' = s * Rn @ (X - c)`` (translate to the point centroid, rotate the average
+    camera-up onto +Y, scale so the camera bounding sphere has radius 10) is applied to both
+    the points and the poses.
+
+    Returns ``(points_new float32 (N,3), poses_new list[dict])``.
+    """
+    pts = np.asarray(points, dtype=np.float64).reshape(-1, 3)
+    poses = list(poses)
+
+    # 1. Translate to the point-cloud centroid.
+    centroid = pts.mean(axis=0) if len(pts) else np.zeros(3)
+
+    # 2. Rotate so the average camera-up aligns with +Y (phones held roughly upright).
+    if poses:
+        ups = np.array([_camera_up(p) for p in poses])
+        up_avg = ups.mean(axis=0)
+        if np.linalg.norm(up_avg) > 1e-9:
+            Rn = _rotation_aligning(up_avg, np.array([0.0, 1.0, 0.0]))
+        else:
+            Rn = np.eye(3)
+    else:
+        Rn = np.eye(3)
+
+    # 3. Scale so the farthest camera center from the centroid sits at radius 10.
+    if poses:
+        centers = np.array([_camera_center(p) for p in poses])
+        max_dist = float(np.max(np.linalg.norm(centers - centroid, axis=1)))
+        scale = NORMALIZED_SPHERE_RADIUS / max_dist if max_dist > 1e-9 else 1.0
+    else:
+        scale = 1.0
+
+    # Apply X' = s * Rn @ (X - c) to points.
+    points_new = (scale * (pts - centroid) @ Rn.T).astype(np.float32)
+
+    # Apply the matching transform to each world->camera pose:
+    #   R' = R @ Rn^T ,  t' = s * (t + R @ c)   (derived so projections are unchanged).
+    poses_new: list[dict] = []
+    for p in poses:
+        R = quat_to_mat([p["qw"], p["qx"], p["qy"], p["qz"]])
+        t = np.array([p["tx"], p["ty"], p["tz"]], dtype=np.float64)
+        R_new = R @ Rn.T
+        t_new = scale * (t + R @ centroid)
+        q_new = mat_to_quat(R_new)
+        out = dict(p)
+        out["qw"], out["qx"], out["qy"], out["qz"] = (
+            float(q_new[0]), float(q_new[1]), float(q_new[2]), float(q_new[3]))
+        out["tx"], out["ty"], out["tz"] = float(t_new[0]), float(t_new[1]), float(t_new[2])
+        poses_new.append(out)
+
+    return points_new, poses_new
+
+
+# ---------------------------------------------------------------------------
+# Pose interpolation
+# ---------------------------------------------------------------------------
+def interpolate_poses(registered: list[dict], sampled: list[dict]) -> list[dict]:
+    """Fill unregistered sampled frames of one video between registered ones (spec M2).
+
+    ``registered`` are solved poses (with ``registered=True``); ``sampled`` are the frames
+    that were fed to COLMAP as ``{frame_idx, t_video_s, ...}``. For each sampled frame that
+    lies strictly between the first and last registered frame but wasn't solved, slerp the
+    rotation and lerp the translation of the two bracketing registered poses (intrinsics
+    inherited from the earlier neighbor), marking it ``registered=False``. Frames outside the
+    registered span are dropped (no extrapolation). Returns all poses sorted by ``t_video_s``.
+    """
+    reg = sorted(registered, key=lambda p: p["t_video_s"])
+    if not reg:
+        return []
+    reg_frames = {int(p["frame_idx"]) for p in reg}
+    t_first, t_last = reg[0]["t_video_s"], reg[-1]["t_video_s"]
+
+    out: list[dict] = list(reg)
+    for sf in sampled:
+        fidx = int(sf["frame_idx"])
+        tv = float(sf["t_video_s"])
+        if fidx in reg_frames:
+            continue
+        if tv <= t_first or tv >= t_last:
+            continue  # no extrapolation beyond the registered span
+        # bracketing registered neighbors
+        lo = max((p for p in reg if p["t_video_s"] <= tv), key=lambda p: p["t_video_s"])
+        hi = min((p for p in reg if p["t_video_s"] >= tv), key=lambda p: p["t_video_s"])
+        span = hi["t_video_s"] - lo["t_video_s"]
+        alpha = 0.0 if span <= 1e-12 else (tv - lo["t_video_s"]) / span
+        q, t = slerp_pose(
+            [lo["qw"], lo["qx"], lo["qy"], lo["qz"]], [lo["tx"], lo["ty"], lo["tz"]],
+            [hi["qw"], hi["qx"], hi["qy"], hi["qz"]], [hi["tx"], hi["ty"], hi["tz"]],
+            alpha,
+        )
+        out.append({
+            "frame_idx": fidx, "t_video_s": tv,
+            "qw": float(q[0]), "qx": float(q[1]), "qy": float(q[2]), "qz": float(q[3]),
+            "tx": float(t[0]), "ty": float(t[1]), "tz": float(t[2]),
+            "fx": lo["fx"], "fy": lo["fy"], "cx": lo["cx"], "cy": lo["cy"],
+            "registered": False,
+        })
+    out.sort(key=lambda p: p["t_video_s"])
+    return out
+
+
+# ---------------------------------------------------------------------------
+# COLMAP CLI orchestration
+# ---------------------------------------------------------------------------
+@functools.lru_cache(maxsize=None)
+def _colmap_help(command: str) -> str:
+    """Cached ``colmap <command> --help`` text, for build-aware option detection.
+
+    COLMAP prints its option list to stderr, so both streams are captured.
+    """
+    try:
+        proc = subprocess.run(
+            ["colmap", command, "--help"], capture_output=True, text=True
+        )
+        return (proc.stdout or "") + "\n" + (proc.stderr or "")
+    except Exception:
+        return ""
+
+
+def _cpu_flag(command: str) -> list[str]:
+    """Force CPU SIFT for ``command`` if this COLMAP build exposes a ``use_gpu`` option.
+
+    The option prefix differs by version: COLMAP 3.x uses ``SiftExtraction`` /
+    ``SiftMatching``, 4.x uses ``FeatureExtraction`` / ``FeatureMatching``. We read the option
+    name from ``--help`` so the pipeline runs headless on either (and on CPU-only builds that
+    omit the option entirely, we pass nothing). Returns e.g. ``["--FeatureExtraction.use_gpu",
+    "0"]`` or ``[]``.
+    """
+    for line in _colmap_help(command).splitlines():
+        line = line.strip()
+        if line.startswith("--") and ".use_gpu" in line:
+            return [line.split()[0], "0"]
+    return []
+
+
+def _run_colmap_step(args: list[str]) -> bool:
+    """Run one ``colmap <args>`` step; return True on success, False (logged) on failure."""
+    cmd = ["colmap", *args]
+    log.info("colmap: %s", " ".join(args[:2]))
+    try:
+        proc = subprocess.run(cmd, capture_output=True, text=True)
+    except FileNotFoundError:
+        log.error("colmap binary vanished from PATH")
+        return False
+    if proc.returncode != 0:
+        log.error("colmap %s failed (rc=%d): %s", args[0], proc.returncode,
+                  (proc.stderr or proc.stdout or "").strip()[-500:])
+        return False
+    return True
+
+
+def _registered_image_count(model_dir: Path) -> int | None:
+    """Number of registered images in a COLMAP model dir, or None if it has no image list.
+
+    The mapper writes binary models by default; ``images.bin`` begins with a little-endian
+    ``uint64`` giving the registered-image count, which is the exact metric we want (file
+    size would instead track total keypoint observations and mis-rank multi-component runs).
+    Falls back to counting the TXT model's two-lines-per-image list.
+    """
+    images_txt = model_dir / "images.txt"
+    images_bin = model_dir / "images.bin"
+    if images_txt.exists():
+        return sum(1 for _ in _iter_data_lines(images_txt)) // 2
+    if images_bin.exists():
+        with open(images_bin, "rb") as f:
+            header = f.read(8)
+        return struct.unpack("<Q", header)[0] if len(header) == 8 else 0
+    return None
+
+
+def _largest_model_dir(sparse_dir: Path) -> Path | None:
+    """Return the reconstruction subdir (``0``, ``1``, ...) with the most registered images."""
+    candidates = [d for d in sparse_dir.iterdir() if d.is_dir()] if sparse_dir.exists() else []
+    best, best_n = None, -1
+    for d in candidates:
+        n = _registered_image_count(d)
+        if n is None:
+            continue
+        if n > best_n:
+            best, best_n = d, n
+    return best
+
+
+def run_colmap(frames_dir: Path, workspace: Path) -> Path | None:
+    """Drive the COLMAP CLI and return the exported TXT-model directory (spec M2, lane B).
+
+    ``frames_dir`` holds one subfolder of JPEGs per video (single camera per folder). Runs
+    feature extraction (CPU SIFT, OPENCV model), exhaustive matching, incremental mapping,
+    then undistortion + TXT conversion. Returns the directory containing ``images.txt`` /
+    ``cameras.txt`` / ``points3D.txt``, or ``None`` if COLMAP produced no usable model
+    (caller degrades gracefully).
+    """
+    workspace = Path(workspace)
+    # Start each run from a clean workspace: a stale database.db makes feature_extractor
+    # fail ("images already exist"), and a leftover sparse/ model would poison the pick.
+    if workspace.exists():
+        import shutil
+        shutil.rmtree(workspace)
+    workspace.mkdir(parents=True, exist_ok=True)
+    database = workspace / "database.db"
+    sparse = workspace / "sparse"
+    sparse.mkdir(exist_ok=True)
+
+    ok = _run_colmap_step([
+        "feature_extractor",
+        "--database_path", str(database),
+        "--image_path", str(frames_dir),
+        "--ImageReader.single_camera_per_folder", "1",
+        "--ImageReader.camera_model", "OPENCV",
+        *_cpu_flag("feature_extractor"),
+    ])
+    if not ok:
+        return None
+
+    # Exhaustive matching covers within- and cross-video pairs; frame counts are small.
+    if not _run_colmap_step([
+        "exhaustive_matcher",
+        "--database_path", str(database),
+        *_cpu_flag("exhaustive_matcher"),
+    ]):
+        return None
+
+    if not _run_colmap_step([
+        "mapper",
+        "--database_path", str(database),
+        "--image_path", str(frames_dir),
+        "--output_path", str(sparse),
+    ]):
+        return None
+
+    model = _largest_model_dir(sparse)
+    if model is None:
+        log.error("colmap mapper produced no reconstruction")
+        return None
+
+    # Undistort -> PINHOLE model, then convert to TXT. Fall back to converting the raw
+    # (possibly distorted) sparse model directly if undistortion fails.
+    txt_dir = workspace / "model_txt"
+    txt_dir.mkdir(exist_ok=True)
+    dense = workspace / "dense"
+    undistorted = _run_colmap_step([
+        "image_undistorter",
+        "--image_path", str(frames_dir),
+        "--input_path", str(model),
+        "--output_path", str(dense),
+        "--output_type", "COLMAP",
+    ])
+    convert_input = (dense / "sparse") if undistorted and (dense / "sparse").exists() else model
+    if not _run_colmap_step([
+        "model_converter",
+        "--input_path", str(convert_input),
+        "--output_path", str(txt_dir),
+        "--output_type", "TXT",
+    ]):
+        return None
+    if not (txt_dir / "images.txt").exists():
+        log.error("colmap model_converter produced no images.txt")
+        return None
+    return txt_dir
+
+
+# ---------------------------------------------------------------------------
+# Full reconstruction entrypoint
+# ---------------------------------------------------------------------------
+def _video_id_and_frame(name: str) -> tuple[int, int] | None:
+    """Recover ``(video_id, frame_idx)`` from a ``{video_id}_{frame_idx}.jpg`` image name."""
+    stem = Path(name).stem
+    if "_" not in stem:
+        return None
+    vid_s, frame_s = stem.rsplit("_", 1)
+    try:
+        return int(vid_s), int(frame_s)
+    except ValueError:
+        return None
+
+
+def _diagnostic(msg: str) -> None:
+    """Emit a clear operator-facing diagnostic (spec M2 graceful degradation)."""
+    log.warning("reconstruct: %s", msg)
+    print(f"[reconstruct] {msg}")
+
+
+def run_reconstruct() -> dict:
+    """Full reconstruction: sample frames, run COLMAP, normalize, interpolate, export.
+
+    Entrypoint for ``python -m festival4d reconstruct``. Degrades gracefully when COLMAP is
+    absent or the reconstruction is too weak (spec M2 failure handling): existing poses are
+    left untouched and the DB is never corrupted. Returns a summary dict with a ``status``.
+    """
+    db.init_engine()
+    db.init_db()
+    videos = db.get_videos()
+    if not videos:
+        _diagnostic("no videos in the project — run `synthetic` or `ingest` first.")
+        return {"status": "no_videos"}
+
+    if not colmap_available():
+        _diagnostic(
+            "COLMAP not found on PATH — skipping 3D reconstruction. Install it "
+            "(`brew install colmap` on macOS) to enable pose/point-cloud solving. "
+            "Existing (synthetic/previous) poses are left untouched; the app still runs as "
+            "synced videos without 3D."
+        )
+        return {"status": "skipped_no_colmap", "videos": len(videos)}
+
+    # 1. Sample sharp frames per video into one subfolder each (single camera per folder).
+    frames_root = config.FRAMES_DIR
+    per_video_sampled: dict[int, list[dict]] = {}
+    for v in videos:
+        video_path = config.RAW_DIR / v.filename
+        if not video_path.exists():
+            _diagnostic(f"video file missing, skipping: {video_path}")
+            continue
+        out_dir = frames_root / str(v.id)
+        paths = frames.sample_frames(video_path, v.id, out_dir)
+        fps = float(v.fps) if v.fps and v.fps > 0 else 30.0
+        sampled = []
+        for p in paths:
+            parsed = _video_id_and_frame(p.name)
+            if parsed is None:
+                continue
+            _, fidx = parsed
+            sampled.append({"frame_idx": fidx, "t_video_s": fidx / fps})
+        per_video_sampled[v.id] = sorted(sampled, key=lambda s: s["frame_idx"])
+
+    total_sampled = sum(len(s) for s in per_video_sampled.values())
+    if total_sampled == 0:
+        _diagnostic("no frames could be sampled from any video — nothing to reconstruct.")
+        return {"status": "no_frames"}
+
+    # 2. Run COLMAP.
+    model_dir = run_colmap(frames_root, config.COLMAP_DIR)
+    if model_dir is None:
+        _diagnostic(
+            "COLMAP produced no usable reconstruction. Likely causes: too-dark footage, "
+            "motion blur, or insufficient view overlap between cameras. Existing poses left "
+            "untouched — the app degrades to synced videos, no 3D."
+        )
+        return {"status": "failed_no_model", "sampled": total_sampled}
+
+    # 3. Parse the model.
+    images = parse_images_txt(model_dir / "images.txt")
+    cameras = parse_cameras_txt(model_dir / "cameras.txt")
+    points, colors = parse_points3d_txt(model_dir / "points3D.txt")
+
+    # 4. Attach each registered image to its video + intrinsics.
+    registered_by_video: dict[int, list[dict]] = {}
+    for img in images:
+        parsed = _video_id_and_frame(img["name"])
+        if parsed is None:
+            continue
+        video_id, frame_idx = parsed
+        cam = cameras.get(img["camera_id"])
+        if cam is None:
+            continue
+        fps = next((float(v.fps) for v in videos if v.id == video_id), 30.0) or 30.0
+        registered_by_video.setdefault(video_id, []).append({
+            "frame_idx": frame_idx, "t_video_s": frame_idx / fps,
+            "qw": img["qw"], "qx": img["qx"], "qy": img["qy"], "qz": img["qz"],
+            "tx": img["tx"], "ty": img["ty"], "tz": img["tz"],
+            "fx": cam["fx"], "fy": cam["fy"], "cx": cam["cx"], "cy": cam["cy"],
+            "registered": True, "video_id": video_id,
+        })
+
+    registered_count = sum(len(v) for v in registered_by_video.values())
+    videos_in_model = len(registered_by_video)
+    frac = registered_count / total_sampled if total_sampled else 0.0
+    if frac < MIN_REGISTERED_FRACTION or videos_in_model < MIN_VIDEOS_IN_MODEL:
+        _diagnostic(
+            f"weak reconstruction: {registered_count}/{total_sampled} frames "
+            f"({frac:.0%}) registered across {videos_in_model} video(s); need "
+            f">= {MIN_REGISTERED_FRACTION:.0%} of frames and >= {MIN_VIDEOS_IN_MODEL} "
+            "videos. Likely causes: too-dark footage, motion blur, or too little view "
+            "overlap. Existing poses left untouched — degrading to synced videos, no 3D."
+        )
+        return {
+            "status": "failed_weak", "sampled": total_sampled,
+            "registered": registered_count, "videos_in_model": videos_in_model,
+        }
+
+    # 5. Normalize the whole scene (points + all registered poses) with one transform.
+    all_registered = [p for v in registered_by_video.values() for p in v]
+    norm_points, norm_registered = normalize_scene(points, all_registered)
+    norm_by_video: dict[int, list[dict]] = {}
+    for p in norm_registered:
+        norm_by_video.setdefault(p["video_id"], []).append(p)
+
+    # 6. Interpolate unregistered sampled frames, then write poses per video (atomic replace).
+    interpolated_total = 0
+    for video_id, reg in norm_by_video.items():
+        sampled = per_video_sampled.get(video_id, [])
+        final = interpolate_poses(reg, sampled)
+        interpolated_total += sum(1 for p in final if not p.get("registered", True))
+        db.set_poses(video_id, final)
+
+    # 7. Export the normalized point cloud in the frozen PLY format.
+    from festival4d.synthetic import write_ply
+    write_ply(config.POINTS_PLY, norm_points, colors)
+
+    summary = {
+        "status": "ok",
+        "videos_in_model": videos_in_model,
+        "registered": registered_count,
+        "interpolated": interpolated_total,
+        "points": int(len(norm_points)),
+        "points_ply": str(config.POINTS_PLY),
+    }
+    log.info("reconstruct: done — %s", summary)
+    print(f"[reconstruct] reconstructed {videos_in_model} videos, "
+          f"{registered_count} registered + {interpolated_total} interpolated poses, "
+          f"{len(norm_points)} points -> {config.POINTS_PLY}")
+    return summary

backend/festival4d/synthetic.py

@@ -0,0 +1,556 @@
+"""Synthetic fixture generator (spec M0) — the universal test bed.
+
+``python -m festival4d synthetic`` builds a complete fake project so every later layer is
+testable without real footage:
+
+- **3 fake videos** (``cam0/1/2.mp4``, 640x360, 30 fps, 20 s), each a distinct ``testsrc2``
+  visual but sharing the *same* synthesized music-like audio, shifted by known offsets
+  (0, +1370, -842 ms). ``cam0`` is the reference (offset 0).
+- **Fake poses**: 3 camera trajectories on arcs around a "stage" box at the origin,
+  looking at the stage, written into ``camera_poses`` (COLMAP world->camera convention).
+- **Point cloud**: random points on the stage box + ground plane, written to
+  ``points.ply`` in the exact binary-little-endian format the real COLMAP path produces.
+- **Seeded events + anchors** so the timeline and overlays have data immediately.
+- **``ground_truth.json``**: offsets, drift, audio pulse (bang) times, events, and stage
+  corners — part of the frozen contract (lanes A and D assert against it).
+
+FROZEN CONTRACT after foundation. The module is factored so the numeric pieces
+(audio shifts, poses, PLY round-trip) are unit-testable without invoking ffmpeg.
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+import shutil
+import struct
+import subprocess
+import tempfile
+import wave
+from pathlib import Path
+
+import numpy as np
+
+from festival4d import config, db
+from festival4d.geometry import mat_to_quat
+
+log = logging.getLogger("festival4d.synthetic")
+
+# Audio timeline: the master signal covers global time [G_START, G_END] so that even the
+# negatively-offset video (started before global 0) samples a valid window.
+G_START = -2.0
+G_END = 24.0
+BEAT_INTERVAL_S = 0.5                       # regular beat clicks
+BANG_TIMES_S = (3.0, 10.0, 17.0)            # loud transients (M7 candidates); well separated
+TONE_FREQS_HZ = (220.0, 330.0)
+
+# Camera intrinsics (pinhole, after undistortion) for the synthetic cameras.
+SYNTH_HFOV_DEG = 60.0
+
+# Stage geometry (world coords, +y up, +z toward the cameras/audience).
+STAGE_HALF_W = 3.0      # x extent
+STAGE_HALF_D = 2.0      # z extent
+STAGE_HEIGHT = 1.0      # y top
+STAGE_TARGET = (0.0, 0.5, 0.0)   # cameras aim here
+
+STAGE_CORNERS = {
+    "Stage FL": (-STAGE_HALF_W, STAGE_HEIGHT, STAGE_HALF_D),
+    "Stage FR": (STAGE_HALF_W, STAGE_HEIGHT, STAGE_HALF_D),
+    "Stage BL": (-STAGE_HALF_W, STAGE_HEIGHT, -STAGE_HALF_D),
+    "Stage BR": (STAGE_HALF_W, STAGE_HEIGHT, -STAGE_HALF_D),
+}
+
+# Seeded timeline events (pre-classified) so lane C can render markers immediately.
+SEED_EVENTS = [
+    (3.0, "bass_drop", 0.94, "Bass drops and the crowd erupts as the beat hits."),
+    (6.0, "crowd_wave", 0.72, "A wave ripples through the audience."),
+    (8.0, "quiet_moment", 0.61, "A brief hush before the next build."),
+    (10.0, "pyro", 0.88, "Pyro columns fire across the front of the stage."),
+    (13.0, "light_show", 0.90, "Sweeping beams sync to the breakdown."),
+    (17.0, "confetti", 0.81, "Confetti cannons burst over the crowd."),
+    (19.0, "artist_moment", 0.77, "The artist steps to the edge for a solo."),
+]
+
+# Fonts to try for the "CAM N" label (optional nicety; skipped if none exist).
+_FONT_CANDIDATES = (
+    "/System/Library/Fonts/Supplemental/Arial.ttf",
+    "/System/Library/Fonts/Supplemental/Helvetica.ttc",
+    "/Library/Fonts/Arial.ttf",
+    "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf",
+)
+
+_PLY_DTYPE = np.dtype(
+    [("x", "<f4"), ("y", "<f4"), ("z", "<f4"),
+     ("red", "u1"), ("green", "u1"), ("blue", "u1")]
+)
+
+
+# ---------------------------------------------------------------------------
+# Audio
+# ---------------------------------------------------------------------------
+def synth_master_audio(sr: int = config.SYNTH_AUDIO_SR) -> tuple[np.ndarray, dict]:
+    """Synthesize the master music-like signal over global time [G_START, G_END].
+
+    Returns the float32 signal (peak-normalized to ~0.95) and a ground-truth dict with the
+    sample rate, the window start, and the beat/bang times (global seconds).
+    """
+    n = int(round((G_END - G_START) * sr))
+    t = G_START + np.arange(n) / sr
+    sig = np.zeros(n, dtype=np.float64)
+
+    # Low tone bed (kept quiet so it never dominates the transients GCC-PHAT keys on).
+    for f in TONE_FREQS_HZ:
+        sig += 0.05 * np.sin(2.0 * np.pi * f * t)
+
+    # Regular beat clicks: short 2 kHz bursts with a fast exponential decay -> sharp,
+    # broadband-ish transients that give a clean cross-correlation peak.
+    beat_times = np.arange(np.ceil(G_START / BEAT_INTERVAL_S) * BEAT_INTERVAL_S,
+                           G_END, BEAT_INTERVAL_S)
+    for bt in beat_times:
+        _add_transient(sig, t, sr, center=bt, amp=0.25, freq=2000.0, decay_s=0.03)
+
+    # Loud "bangs" (bass drops / pyro): louder, longer, low-frequency thump + noise burst.
+    rng = np.random.default_rng(7)
+    for bang in BANG_TIMES_S:
+        _add_transient(sig, t, sr, center=bang, amp=0.9, freq=60.0, decay_s=0.18)
+        _add_noise_burst(sig, t, sr, center=bang, amp=0.5, decay_s=0.12, rng=rng)
+
+    # Faint noise floor for realism (SNR stays high so sync remains exact).
+    sig += 0.004 * rng.standard_normal(n)
+
+    peak = float(np.max(np.abs(sig)))
+    if peak > 0:
+        sig *= 0.95 / peak
+
+    ground_truth = {
+        "sample_rate": sr,
+        "g_start_s": G_START,
+        "g_end_s": G_END,
+        "beat_interval_s": BEAT_INTERVAL_S,
+        "bang_times_global_s": list(BANG_TIMES_S),
+        "pulse_times_global_s": list(BANG_TIMES_S),  # alias lane D matches candidates against
+    }
+    return sig.astype(np.float32), ground_truth
+
+
+def _add_transient(sig, t, sr, center, amp, freq, decay_s):
+    """Add an exponentially decaying sinusoid burst centered at global time ``center``."""
+    length = int(decay_s * 5 * sr)
+    start = int(round((center - G_START) * sr))
+    i0 = max(0, start)
+    i1 = min(len(sig), start + length)
+    if i1 <= i0:
+        return
+    local = (np.arange(i0, i1) - start) / sr
+    env = np.exp(-local / decay_s)
+    sig[i0:i1] += amp * env * np.sin(2.0 * np.pi * freq * local)
+
+
+def _add_noise_burst(sig, t, sr, center, amp, decay_s, rng):
+    length = int(decay_s * 5 * sr)
+    start = int(round((center - G_START) * sr))
+    i0 = max(0, start)
+    i1 = min(len(sig), start + length)
+    if i1 <= i0:
+        return
+    local = (np.arange(i0, i1) - start) / sr
+    env = np.exp(-local / decay_s)
+    sig[i0:i1] += amp * env * rng.standard_normal(i1 - i0)
+
+
+def slice_for_offset(master: np.ndarray, sr: int, offset_ms: float,
+                     duration_s: float) -> np.ndarray:
+    """Extract a video's audio window from the master (spec M0 known-offset shift).
+
+    A video with ``offset_ms`` has local time ``tau`` mapping to global ``offset_ms/1000 +
+    tau`` (drift 0), so its audio is the master window starting at that global time.
+    """
+    start_global = offset_ms / 1000.0
+    start = int(round((start_global - G_START) * sr))
+    length = int(round(duration_s * sr))
+    if start < 0 or start + length > len(master):
+        raise ValueError(f"offset {offset_ms} ms window falls outside the master signal")
+    return master[start:start + length]
+
+
+def write_wav(path: Path, signal: np.ndarray, sr: int) -> None:
+    """Write a mono 16-bit PCM WAV (stdlib only)."""
+    path.parent.mkdir(parents=True, exist_ok=True)
+    clipped = np.clip(signal, -1.0, 1.0)
+    pcm = (clipped * 32767.0).astype("<i2")
+    with wave.open(str(path), "wb") as w:
+        w.setnchannels(1)
+        w.setsampwidth(2)
+        w.setframerate(sr)
+        w.writeframes(pcm.tobytes())
+
+
+def read_wav(path: Path) -> tuple[np.ndarray, int]:
+    """Read a mono 16-bit PCM WAV back to float [-1, 1] (used by tests)."""
+    with wave.open(str(path), "rb") as w:
+        sr = w.getframerate()
+        frames = w.readframes(w.getnframes())
+    pcm = np.frombuffer(frames, dtype="<i2").astype(np.float64) / 32767.0
+    return pcm, sr
+
+
+# ---------------------------------------------------------------------------
+# Geometry: poses + point cloud
+# ---------------------------------------------------------------------------
+def intrinsics(width: int, height: int) -> dict:
+    fx = (width / 2.0) / np.tan(np.radians(SYNTH_HFOV_DEG) / 2.0)
+    return {"fx": float(fx), "fy": float(fx), "cx": width / 2.0, "cy": height / 2.0}
+
+
+def look_at_colmap(center, target, world_up=(0.0, 1.0, 0.0)) -> tuple[np.ndarray, np.ndarray]:
+    """Build a COLMAP world->camera pose (q=[w,x,y,z], t) for a camera at ``center``
+    looking at ``target`` with the given world up. Camera axes: +x right, +y down, +z fwd.
+    """
+    C = np.asarray(center, dtype=np.float64)
+    T = np.asarray(target, dtype=np.float64)
+    up = np.asarray(world_up, dtype=np.float64)
+
+    z_c = T - C
+    z_c /= np.linalg.norm(z_c)
+    if abs(np.dot(up, z_c)) > 0.999:          # looking near-vertical: pick a safe up
+        up = np.array([0.0, 0.0, 1.0])
+    up_c = up - np.dot(up, z_c) * z_c          # image up (perp to forward)
+    up_c /= np.linalg.norm(up_c)
+    x_c = np.cross(z_c, up_c)                   # right
+    y_c = -up_c                                 # down
+
+    R_c2w = np.column_stack([x_c, y_c, z_c])   # camera->world
+    R_w2c = R_c2w.T                             # world->camera (COLMAP R)
+    t = -R_w2c @ C
+    q = mat_to_quat(R_w2c)
+    return q, t
+
+
+def camera_track(cam_index: int, duration_s: float, fps: float,
+                 width: int, height: int) -> list[dict]:
+    """One camera's trajectory: a gentle arc in front of the stage, ~2 poses/second."""
+    base_az = np.radians([-35.0, 0.0, 35.0][cam_index])
+    base_radius = 9.0
+    base_height = [1.6, 2.4, 2.0][cam_index]
+    sweep = np.radians(12.0)
+    intr = intrinsics(width, height)
+
+    poses = []
+    step = 0.5  # seconds -> ~2 poses/second
+    n = int(round(duration_s / step)) + 1
+    for k in range(n):
+        t_video = min(k * step, duration_s)
+        phase = 2.0 * np.pi * (t_video / duration_s)
+        az = base_az + sweep * np.sin(phase)
+        radius = base_radius + 0.5 * np.sin(phase + cam_index)
+        height_y = base_height + 0.3 * np.sin(phase * 2.0)
+        center = (radius * np.sin(az), height_y, radius * np.cos(az))
+        q, t = look_at_colmap(center, STAGE_TARGET)
+        poses.append({
+            "frame_idx": int(round(t_video * fps)),
+            "t_video_s": float(t_video),
+            "qw": float(q[0]), "qx": float(q[1]), "qy": float(q[2]), "qz": float(q[3]),
+            "tx": float(t[0]), "ty": float(t[1]), "tz": float(t[2]),
+            **intr,
+            "registered": True,
+        })
+    return poses
+
+
+def generate_point_cloud(seed: int = 42) -> tuple[np.ndarray, np.ndarray]:
+    """Random points on the stage box + ground plane (with the named corners included).
+
+    Returns ``(points Nx3 float32, colors Nx3 uint8)``.
+    """
+    rng = np.random.default_rng(seed)
+    pts: list[np.ndarray] = []
+    cols: list[np.ndarray] = []
+
+    hw, hd, h = STAGE_HALF_W, STAGE_HALF_D, STAGE_HEIGHT
+    stage_color = np.array([210, 90, 60])       # warm stage
+    ground_color = np.array([70, 80, 70])       # dim ground
+
+    # Stage box: sample points on all 6 faces.
+    n_face = 260
+    faces = [
+        # (fixed axis, value, u range, v range) -> build points
+        ("x", -hw), ("x", hw), ("z", -hd), ("z", hd), ("y", 0.0), ("y", h),
+    ]
+    for axis, val in faces:
+        u = rng.uniform(-hw, hw, n_face)
+        v = rng.uniform(0.0, h, n_face) if axis in ("x", "z") else rng.uniform(-hd, hd, n_face)
+        w = rng.uniform(-hd, hd, n_face)
+        if axis == "x":
+            p = np.column_stack([np.full(n_face, val), v, w])
+        elif axis == "z":
+            p = np.column_stack([u, v, np.full(n_face, val)])
+        else:  # y
+            p = np.column_stack([u, np.full(n_face, val), w])
+        pts.append(p)
+        jitter = rng.integers(-15, 16, size=(n_face, 3))
+        cols.append(np.clip(stage_color + jitter, 0, 255))
+
+    # Ground plane: scattered points on y=0 out to a radius, excluding the stage footprint.
+    n_ground = 1600
+    gx = rng.uniform(-12.0, 12.0, n_ground)
+    gz = rng.uniform(-12.0, 12.0, n_ground)
+    on_stage = (np.abs(gx) < hw) & (np.abs(gz) < hd)
+    gx, gz = gx[~on_stage], gz[~on_stage]
+    gp = np.column_stack([gx, np.zeros_like(gx), gz])
+    pts.append(gp)
+    jitter = rng.integers(-12, 13, size=(len(gx), 3))
+    cols.append(np.clip(ground_color + jitter, 0, 255))
+
+    # Include the exact named stage corners so the M8 nearest-point fallback can hit them.
+    corners = np.array(list(STAGE_CORNERS.values()))
+    pts.append(corners)
+    cols.append(np.tile(np.array([255, 230, 120]), (len(corners), 1)))
+
+    points = np.vstack(pts).astype(np.float32)
+    colors = np.vstack(cols).astype(np.uint8)
+    return points, colors
+
+
+def write_ply(path: Path, points: np.ndarray, colors: np.ndarray) -> None:
+    """Write a binary-little-endian PLY (x,y,z float + red,green,blue uchar). FROZEN format."""
+    points = np.asarray(points, dtype=np.float32).reshape(-1, 3)
+    colors = np.asarray(colors, dtype=np.uint8).reshape(-1, 3)
+    if len(points) != len(colors):
+        raise ValueError("points and colors length mismatch")
+    n = len(points)
+    arr = np.empty(n, dtype=_PLY_DTYPE)
+    arr["x"], arr["y"], arr["z"] = points[:, 0], points[:, 1], points[:, 2]
+    arr["red"], arr["green"], arr["blue"] = colors[:, 0], colors[:, 1], colors[:, 2]
+    assert arr.dtype.itemsize == 15  # no padding — 3*4 + 3*1
+
+    header = (
+        "ply\n"
+        "format binary_little_endian 1.0\n"
+        "comment Festival 4D point cloud\n"
+        f"element vertex {n}\n"
+        "property float x\n"
+        "property float y\n"
+        "property float z\n"
+        "property uchar red\n"
+        "property uchar green\n"
+        "property uchar blue\n"
+        "end_header\n"
+    )
+    path.parent.mkdir(parents=True, exist_ok=True)
+    with open(path, "wb") as f:
+        f.write(header.encode("ascii"))
+        f.write(arr.tobytes())
+
+
+def read_ply(path: Path) -> tuple[np.ndarray, np.ndarray]:
+    """Read a PLY written by :func:`write_ply` (used by tests). Returns (points, colors)."""
+    with open(path, "rb") as f:
+        line = f.readline()
+        if line.strip() != b"ply":
+            raise ValueError("not a PLY file")
+        n = 0
+        while True:
+            line = f.readline()
+            if not line:
+                raise ValueError("unexpected EOF in PLY header")
+            if line.startswith(b"element vertex"):
+                n = int(line.split()[-1])
+            if line.strip() == b"end_header":
+                break
+        arr = np.frombuffer(f.read(n * _PLY_DTYPE.itemsize), dtype=_PLY_DTYPE, count=n)
+    points = np.column_stack([arr["x"], arr["y"], arr["z"]]).astype(np.float32)
+    colors = np.column_stack([arr["red"], arr["green"], arr["blue"]]).astype(np.uint8)
+    return points, colors
+
+
+# ---------------------------------------------------------------------------
+# Video rendering (ffmpeg)
+# ---------------------------------------------------------------------------
+def _find_font() -> str | None:
+    for candidate in _FONT_CANDIDATES:
+        if Path(candidate).exists():
+            return candidate
+    return None
+
+
+_DRAWTEXT_AVAILABLE: bool | None = None
+
+
+def _has_drawtext() -> bool:
+    """Whether this ffmpeg build includes the ``drawtext`` filter (needs libfreetype)."""
+    global _DRAWTEXT_AVAILABLE
+    if _DRAWTEXT_AVAILABLE is None:
+        try:
+            out = subprocess.run(
+                ["ffmpeg", "-hide_banner", "-filters"],
+                check=True, capture_output=True, text=True,
+            ).stdout
+            _DRAWTEXT_AVAILABLE = any(
+                line.split()[1] == "drawtext"
+                for line in out.splitlines()
+                if len(line.split()) > 1
+            )
+        except Exception:
+            _DRAWTEXT_AVAILABLE = False
+    return _DRAWTEXT_AVAILABLE
+
+
+def render_video(out_path: Path, wav_path: Path, cam_index: int,
+                 width: int, height: int, fps: float, duration_s: float) -> None:
+    """Render one synthetic clip: a distinct ``testsrc2`` visual muxed with ``wav_path``."""
+    if shutil.which("ffmpeg") is None:
+        raise RuntimeError("ffmpeg not found on PATH — required for the synthetic fixture")
+    out_path.parent.mkdir(parents=True, exist_ok=True)
+
+    hue = cam_index * 47
+    vf = f"hue=h={hue}"
+    font = _find_font()
+    if font and _has_drawtext():
+        vf += (f",drawtext=fontfile='{font}':text='CAM {cam_index}':"
+               "x=24:y=24:fontsize=44:fontcolor=white:box=1:boxcolor=black@0.5")
+
+    src = f"testsrc2=size={width}x{height}:rate={fps}:duration={duration_s}"
+    cmd = [
+        "ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
+        "-f", "lavfi", "-i", src,
+        "-i", str(wav_path),
+        "-vf", vf,
+        "-c:v", "libx264", "-pix_fmt", "yuv420p", "-preset", "veryfast", "-crf", "28",
+        "-profile:v", "baseline", "-level", "3.1",
+        "-movflags", "+faststart",
+        "-c:a", "aac", "-b:a", "128k",
+        "-shortest",
+        str(out_path),
+    ]
+    subprocess.run(cmd, check=True)
+
+
+def ffprobe_video(path: Path) -> dict:
+    """Probe width/height/fps/duration of a video with ffprobe."""
+    out = subprocess.run(
+        ["ffprobe", "-v", "error", "-select_streams", "v:0",
+         "-show_entries", "stream=width,height,avg_frame_rate:format=duration",
+         "-of", "json", str(path)],
+        check=True, capture_output=True, text=True,
+    ).stdout
+    data = json.loads(out)
+    stream = data["streams"][0]
+    num, den = stream["avg_frame_rate"].split("/")
+    fps = float(num) / float(den) if float(den) else 0.0
+    return {
+        "width": int(stream["width"]),
+        "height": int(stream["height"]),
+        "fps": fps,
+        "duration_s": float(data["format"]["duration"]),
+    }
+
+
+# ---------------------------------------------------------------------------
+# Orchestration
+# ---------------------------------------------------------------------------
+def build(base_dir: Path | str | None = None, duration_s: float | None = None,
+          run_ffmpeg: bool = True) -> dict:
+    """Generate the full synthetic project. Returns a summary dict.
+
+    ``base_dir`` defaults to ``config.DATA_DIR``. ``duration_s`` defaults to the fixture's
+    20 s (tests may pass a shorter value). ``run_ffmpeg=False`` skips video rendering (for
+    fast, hermetic tests of the DB/pose/PLY path).
+    """
+    base = Path(base_dir) if base_dir is not None else config.DATA_DIR
+    duration = float(duration_s) if duration_s is not None else config.SYNTH_DURATION_S
+    raw = base / "raw"
+    work = base / "work"
+    raw.mkdir(parents=True, exist_ok=True)
+    work.mkdir(parents=True, exist_ok=True)
+
+    sr = config.SYNTH_AUDIO_SR
+    w, h, fps = config.SYNTH_VIDEO_W, config.SYNTH_VIDEO_H, config.SYNTH_FPS
+
+    log.info("synthetic: building project at %s (duration=%.1fs, ffmpeg=%s)",
+             base, duration, run_ffmpeg)
+
+    db.init_engine(base / "project.db")
+    db.reset_db()
+
+    master, audio_gt = synth_master_audio(sr)
+
+    videos_gt = []
+    with tempfile.TemporaryDirectory() as tmp:
+        tmp = Path(tmp)
+        for i, offset_ms in enumerate(config.SYNTH_OFFSETS_MS):
+            filename = f"cam{i}.mp4"
+            out_path = raw / filename
+            clip = slice_for_offset(master, sr, offset_ms, duration)
+
+            if run_ffmpeg:
+                wav_path = tmp / f"cam{i}.wav"
+                write_wav(wav_path, clip, sr)
+                render_video(out_path, wav_path, i, w, h, fps, duration)
+                probed = ffprobe_video(out_path)
+                vw, vh, vfps, vdur = (probed["width"], probed["height"],
+                                      probed["fps"], probed["duration_s"])
+            else:
+                vw, vh, vfps, vdur = w, h, float(fps), duration
+
+            video = db.add_video(
+                filename=filename, duration_s=vdur, fps=vfps, width=vw, height=vh,
+                offset_ms=float(offset_ms), drift_ppm=0.0, sync_confidence=1.0,
+            )
+            db.set_poses(video.id, camera_track(i, duration, fps, w, h))
+            videos_gt.append({
+                "video_id": video.id, "filename": filename,
+                "offset_ms": float(offset_ms), "drift_ppm": 0.0,
+                "is_reference": i == 0,
+            })
+            log.info("synthetic: cam%d offset=%+.0fms -> %s", i, offset_ms, out_path.name)
+
+    # Point cloud
+    points, colors = generate_point_cloud()
+    write_ply(work / "points.ply", points, colors)
+
+    # Anchors (stage corners)
+    corner_colors = {"Stage FL": "#ff5533", "Stage FR": "#ffaa33",
+                     "Stage BL": "#33aaff", "Stage BR": "#aa55ff"}
+    for label, (x, y, z) in STAGE_CORNERS.items():
+        db.add_anchor(label, x, y, z, corner_colors.get(label))
+
+    # Seeded events
+    for t_global, event_type, conf, desc in SEED_EVENTS:
+        db.add_event(t_global_s=t_global, event_type=event_type, source="ai",
+                     duration_s=1.0, confidence=conf, description=desc)
+
+    # Ground truth JSON
+    ground_truth = {
+        "videos": videos_gt,
+        "reference_filename": "cam0.mp4",
+        "audio": audio_gt,
+        "events": [
+            {"t_global_s": t, "event_type": et, "confidence": c, "description": d}
+            for (t, et, c, d) in SEED_EVENTS
+        ],
+        "scene": {
+            "stage_corners": {k: list(v) for k, v in STAGE_CORNERS.items()},
+            "stage_target": list(STAGE_TARGET),
+            "point_count": int(len(points)),
+        },
+    }
+    gt_path = work / "ground_truth.json"
+    gt_path.write_text(json.dumps(ground_truth, indent=2))
+
+    summary = {
+        "base_dir": str(base),
+        "videos": [v["filename"] for v in videos_gt],
+        "points_ply": str(work / "points.ply"),
+        "point_count": int(len(points)),
+        "ground_truth": str(gt_path),
+        "events": len(SEED_EVENTS),
+        "anchors": len(STAGE_CORNERS),
+    }
+    log.info("synthetic: done — %d videos, %d points, %d events, %d anchors",
+             len(videos_gt), len(points), len(SEED_EVENTS), len(STAGE_CORNERS))
+    return summary
+
+
+if __name__ == "__main__":  # pragma: no cover
+    logging.basicConfig(level=logging.INFO)
+    print(json.dumps(build(), indent=2))

backend/tests/conftest.py

@@ -0,0 +1,13 @@
+"""Pytest session setup.
+
+Redirect the Festival 4D data directory to a throwaway temp dir for the whole test
+session, so tests never touch the repo's real ``data/`` (which may hold the demo fixture).
+Set before any ``festival4d.config`` import so the paths resolve to the temp dir.
+"""
+
+import os
+import tempfile
+
+os.environ.setdefault(
+    "FESTIVAL4D_DATA_DIR", tempfile.mkdtemp(prefix="festival4d-test-")
+)

backend/tests/test_api.py

@@ -0,0 +1,112 @@
+"""API contract tests (spec M3), against a synthetic fixture built into the test data dir.
+
+Locks the frozen response shapes lane C depends on, and verifies HTTP Range serving (206),
+CORS, and graceful degradation of the still-stubbed endpoints.
+"""
+
+from __future__ import annotations
+
+import shutil
+
+import pytest
+
+pytestmark = pytest.mark.skipif(
+    shutil.which("ffmpeg") is None or shutil.which("ffprobe") is None,
+    reason="ffmpeg/ffprobe required to build the API test fixture",
+)
+
+
+@pytest.fixture(scope="module")
+def client():
+    from fastapi.testclient import TestClient
+
+    from festival4d import synthetic
+
+    synthetic.build(duration_s=1.0, run_ffmpeg=True)  # into the temp data dir (conftest)
+
+    from festival4d import api  # imported after build so the /media mount + DB are ready
+
+    with TestClient(api.app) as c:
+        yield c
+
+
+def test_manifest_shape(client):
+    data = client.get("/api/manifest").json()
+    assert set(data) == {"videos", "t_global_max", "has_poses"}
+    assert data["has_poses"] is True
+    assert len(data["videos"]) == 3
+    v = data["videos"][0]
+    assert set(v) == {
+        "id", "filename", "url", "duration_s", "fps", "width", "height",
+        "offset_ms", "drift_ppm",
+    }
+    offsets = {vv["filename"]: vv["offset_ms"] for vv in data["videos"]}
+    assert offsets == {"cam0.mp4": 0.0, "cam1.mp4": 1370.0, "cam2.mp4": -842.0}
+
+
+def test_poses_shape(client):
+    poses = client.get("/api/videos/1/poses").json()
+    assert len(poses) > 0
+    p = poses[0]
+    assert set(p) == {"frame_idx", "t_video_s", "q", "t", "intrinsics", "registered"}
+    assert len(p["q"]) == 4 and len(p["t"]) == 3
+    assert set(p["intrinsics"]) == {"fx", "fy", "cx", "cy"}
+
+
+def test_missing_video_404(client):
+    assert client.get("/api/videos/999/poses").status_code == 404
+
+
+def test_anchors_get_and_post(client):
+    anchors = client.get("/api/anchors").json()
+    assert len(anchors) == 4
+    created = client.post(
+        "/api/anchors", json={"label": "T", "x": 1, "y": 2, "z": 3, "color": "#fff"}
+    ).json()
+    assert created["label"] == "T" and created["id"] > 0
+
+
+def test_events(client):
+    events = client.get("/api/events").json()
+    assert len(events) == len(__import__("festival4d.synthetic", fromlist=["x"]).SEED_EVENTS)
+    assert {"id", "t_global_s", "event_type", "source", "description"} <= set(events[0])
+
+
+def test_pointcloud_is_ply(client):
+    resp = client.get("/api/pointcloud")
+    assert resp.status_code == 200
+    assert resp.content[:3] == b"ply"
+
+
+def test_video_range_returns_206(client):
+    """Spec pitfall #3: <video> seeking needs HTTP Range -> 206."""
+    resp = client.get("/media/cam0.mp4", headers={"Range": "bytes=0-100"})
+    assert resp.status_code == 206
+    assert resp.headers["content-range"].startswith("bytes 0-100/")
+    assert len(resp.content) == 101
+
+
+def test_cors_allows_vite_origin(client):
+    resp = client.get("/api/manifest", headers={"Origin": "http://localhost:5173"})
+    assert resp.headers.get("access-control-allow-origin") == "http://localhost:5173"
+
+
+def test_detect_events_endpoint(client):
+    # Lane D (M7) has landed: POST /api/events/detect runs real detection and returns
+    # {result, events} (no more stub "note"). result is the run_events summary.
+    resp = client.post("/api/events/detect", json={})
+    assert resp.status_code == 200
+    data = resp.json()
+    assert "result" in data and "events" in data
+    assert {"candidates", "classified", "candidates_only", "classifier_configured"} <= set(data["result"])
+    assert isinstance(data["events"], list)
+    assert client.post("/api/events/detect", json={}).status_code == 200  # idempotent
+
+
+def test_annotation_stored(client):
+    data = client.post(
+        "/api/annotations",
+        json={"video_id": 1, "t_video_s": 0.5, "bbox": [0.4, 0.4, 0.6, 0.6]},
+    ).json()
+    assert data["annotation_id"] > 0
+    assert data["anchor_id"] is None                    # M8 resolution is integration work

backend/tests/test_audio_sync.py

@@ -0,0 +1,240 @@
+"""Audio-sync tests (spec M1, lane A).
+
+Pure-numpy where possible (no ffmpeg): GCC-PHAT sign/precision, pairwise+global solve
+recovering the M0 known offsets (±10 ms), disconnected components → None (pitfall #5),
+cycle-consistency rejection, drift recovery (±3 ppm), and the full ``run_sync`` persist +
+export path against synthesized WAVs. One ffmpeg-gated test runs the literal spec M1
+acceptance: ``synthetic -> ingest -> sync`` recovers ground truth.
+"""
+
+from __future__ import annotations
+
+import json
+import shutil
+
+import numpy as np
+import pytest
+
+from festival4d import audio_sync, config, db, ingest, synthetic
+
+
+# ---------------------------------------------------------------------------
+# helpers
+# ---------------------------------------------------------------------------
+def _broadband(n: int, seed: int = 0) -> np.ndarray:
+    """Seeded, mildly band-limited noise — a sharp, unambiguous correlation target."""
+    rng = np.random.default_rng(seed)
+    x = rng.standard_normal(n)
+    return np.convolve(x, np.ones(4) / 4.0, mode="same")
+
+
+def _delay_samples(x: np.ndarray, d: int) -> np.ndarray:
+    """Return ``y`` with ``y[n] = x[n - d]`` (x delayed by d samples, zero-filled)."""
+    y = np.zeros_like(x)
+    if d >= 0:
+        y[d:] = x[:len(x) - d] if d else x
+    else:
+        y[:d] = x[-d:]
+    return y
+
+
+def _clean_audio_dir() -> None:
+    """Remove any WAVs left in the shared session audio dir (ids are reused after reset_db)."""
+    config.AUDIO_DIR.mkdir(parents=True, exist_ok=True)
+    for wav in config.AUDIO_DIR.glob("*.wav"):
+        wav.unlink()
+
+
+# ---------------------------------------------------------------------------
+# GCC-PHAT
+# ---------------------------------------------------------------------------
+def test_gcc_phat_recovers_known_delay_positive_and_negative():
+    sr = 16_000
+    ref = _broadband(sr * 4)
+    for d in (0, 137, -211, 800):
+        sig = _delay_samples(ref, d)
+        offset_s, conf = audio_sync.gcc_phat(sig, ref, sr)
+        # positive offset_s means sig lags ref, i.e. matches a positive sample delay
+        assert abs(offset_s - d / sr) < 0.5 / sr, (d, offset_s * sr)
+        assert conf > 5.0
+
+
+def test_gcc_phat_confidence_drops_with_noise():
+    sr = 16_000
+    ref = _broadband(sr * 4, seed=1)
+    clean = _delay_samples(ref, 300)
+    rng = np.random.default_rng(2)
+    noisy = clean + 3.0 * rng.standard_normal(len(clean))
+    _, c_clean = audio_sync.gcc_phat(clean, ref, sr)
+    _, c_noisy = audio_sync.gcc_phat(noisy, ref, sr)
+    assert c_clean > c_noisy
+
+
+# ---------------------------------------------------------------------------
+# pairwise + global solve
+# ---------------------------------------------------------------------------
+def test_pairwise_and_solve_recover_synthetic_offsets():
+    """The M0 known offsets (0/+1370/−842 ms) recovered within the spec's ±10 ms."""
+    sr = config.AUDIO_SAMPLE_RATE
+    master, _ = synthetic.synth_master_audio(sr)
+    signals = {
+        i + 1: synthetic.slice_for_offset(master, sr, off, config.SYNTH_DURATION_S)
+        for i, off in enumerate(config.SYNTH_OFFSETS_MS)
+    }
+    edges = audio_sync.pairwise_offsets(signals, sr)
+    solved = audio_sync.solve_global_offsets(edges, list(signals))
+    for i, off in enumerate(config.SYNTH_OFFSETS_MS):
+        assert solved[i + 1] is not None
+        assert abs(solved[i + 1] - off) <= 10.0, (off, solved[i + 1])
+
+
+def test_pairwise_edge_sign_is_relative_start_offset():
+    """Edge ``offset_s`` = offset_a − offset_b (seconds)."""
+    sr = config.AUDIO_SAMPLE_RATE
+    master, _ = synthetic.synth_master_audio(sr)
+    a = synthetic.slice_for_offset(master, sr, 500.0, 6.0)   # id 1, offset +500 ms
+    b = synthetic.slice_for_offset(master, sr, -300.0, 6.0)  # id 2, offset −300 ms
+    edges = audio_sync.pairwise_offsets({1: a, 2: b}, sr)
+    assert len(edges) == 1
+    assert abs(edges[0]["offset_s"] * 1000.0 - (500.0 - -300.0)) <= 10.0
+
+
+def test_solve_disconnected_component_gets_none():
+    """A video with no usable edges is unsynced (offset None), not guessed (pitfall #5)."""
+    edges = [
+        {"a": 1, "b": 2, "offset_s": -0.5, "confidence": 30.0},
+        {"a": 2, "b": 3, "offset_s": -0.4, "confidence": 30.0},
+    ]
+    solved = audio_sync.solve_global_offsets(edges, [1, 2, 3, 4])
+    assert solved[1] == 0.0
+    assert abs(solved[2] - 500.0) < 1.0
+    assert abs(solved[3] - 900.0) < 1.0
+    assert solved[4] is None                       # isolated → None
+
+
+def test_solve_rejects_cycle_inconsistent_edge():
+    """A grossly inconsistent edge (>50 ms residual) is rejected; the rest still solve."""
+    # True offsets: 1=0, 2=+500 ms, 3=+900 ms. Edge 1-3 is corrupted (should be −0.9 s).
+    edges = [
+        {"a": 1, "b": 2, "offset_s": -0.5, "confidence": 20.0},
+        {"a": 2, "b": 3, "offset_s": -0.4, "confidence": 20.0},
+        {"a": 1, "b": 3, "offset_s": +5.0, "confidence": 20.0},   # bad
+    ]
+    solved = audio_sync.solve_global_offsets(edges, [1, 2, 3])
+    assert solved[1] == 0.0
+    assert abs(solved[2] - 500.0) < 1.0
+    assert abs(solved[3] - 900.0) < 1.0            # recovered despite the bad edge
+
+
+def test_low_confidence_edges_are_dropped():
+    """Edges below the confidence floor don't connect a component."""
+    edges = [{"a": 1, "b": 2, "offset_s": -0.5, "confidence": 1.01}]
+    solved = audio_sync.solve_global_offsets(edges, [1, 2])
+    # 1 is the lone reference at 0; 2 has only a rejected edge → None
+    assert solved[1] == 0.0 and solved[2] is None
+
+
+# ---------------------------------------------------------------------------
+# drift
+# ---------------------------------------------------------------------------
+def _make_drifted(ref: np.ndarray, sr: int, drift_ppm: float, offset_s: float = 0.0) -> np.ndarray:
+    """Build v from the config timebase: v(τ) = ref((1 − d)·τ + offset_s), d = ppm·1e−6."""
+    d = drift_ppm * 1e-6
+    local = np.arange(len(ref))
+    src = (1.0 - d) * local + offset_s * sr
+    return np.interp(src, np.arange(len(ref)), ref, left=0.0, right=0.0)
+
+
+def test_estimate_drift_recovers_known_ppm():
+    sr = 8_000
+    ref = _broadband(sr * 150, seed=5)             # 150 s so several 30 s-spaced windows fit
+    for ppm in (30.0, -22.0):
+        v = _make_drifted(ref, sr, ppm)
+        est = audio_sync.estimate_drift(v, ref, sr)
+        assert abs(est - ppm) <= 3.0, (ppm, est)
+
+
+def test_estimate_drift_zero_within_deadband():
+    sr = 8_000
+    ref = _broadband(sr * 150, seed=6)
+    assert audio_sync.estimate_drift(ref, ref, sr) == 0.0            # identical
+    assert audio_sync.estimate_drift(_make_drifted(ref, sr, 1.5), ref, sr) == 0.0  # < 5 ppm
+
+
+def test_estimate_drift_returns_zero_when_too_short():
+    sr = 16_000
+    ref = _broadband(sr * 15)                       # < 2 windows at 10 s / 30 s hop
+    assert audio_sync.estimate_drift(ref, ref, sr) == 0.0
+
+
+# ---------------------------------------------------------------------------
+# run_sync orchestration
+# ---------------------------------------------------------------------------
+def test_run_sync_persists_and_exports():
+    """Full run_sync against synthesized WAVs (no ffmpeg): DB + sync.json, ±10 ms."""
+    sr = config.AUDIO_SAMPLE_RATE
+    master, _ = synthetic.synth_master_audio(sr)
+    db.init_engine()
+    db.reset_db()
+    _clean_audio_dir()
+    gt = {}
+    for i, off in enumerate(config.SYNTH_OFFSETS_MS):
+        v = db.add_video(filename=f"cam{i}.mp4", duration_s=config.SYNTH_DURATION_S,
+                         fps=30.0, width=640, height=360)
+        clip = synthetic.slice_for_offset(master, sr, off, config.SYNTH_DURATION_S)
+        synthetic.write_wav(ingest.audio_wav_path(v.id), clip, sr)
+        gt[v.id] = off
+
+    solution = audio_sync.run_sync()
+
+    for v in db.get_videos():
+        assert v.offset_ms is not None
+        assert abs(v.offset_ms - gt[v.id]) <= 10.0, (gt[v.id], v.offset_ms)
+        assert v.drift_ppm == 0.0
+        assert 0.0 <= v.sync_confidence <= 1.0
+    ref = min(gt)
+    assert solution["reference_video_id"] == ref
+    assert db.get_video(ref).sync_confidence == 1.0
+    exported = json.loads(config.SYNC_JSON.read_text())
+    assert exported["reference_video_id"] == ref
+    assert len(exported["edges"]) == 3
+
+
+def test_run_sync_marks_video_without_audio_unsynced():
+    """A registered video with no extracted WAV is left offset=None, not guessed."""
+    sr = config.AUDIO_SAMPLE_RATE
+    master, _ = synthetic.synth_master_audio(sr)
+    db.init_engine()
+    db.reset_db()
+    _clean_audio_dir()
+    v0 = db.add_video(filename="cam0.mp4", duration_s=20.0, fps=30.0, width=640, height=360)
+    v1 = db.add_video(filename="cam1.mp4", duration_s=20.0, fps=30.0, width=640, height=360)
+    no_audio = db.add_video(filename="cam2.mp4", duration_s=20.0, fps=30.0, width=640, height=360)
+    synthetic.write_wav(ingest.audio_wav_path(v0.id),
+                        synthetic.slice_for_offset(master, sr, 0.0, 20.0), sr)
+    synthetic.write_wav(ingest.audio_wav_path(v1.id),
+                        synthetic.slice_for_offset(master, sr, 1370.0, 20.0), sr)
+    # no WAV for `no_audio`
+
+    audio_sync.run_sync()
+    assert db.get_video(no_audio.id).offset_ms is None
+    assert db.get_video(no_audio.id).sync_confidence is None
+    assert db.get_video(v0.id).offset_ms == 0.0
+
+
+# ---------------------------------------------------------------------------
+# ffmpeg-gated: the literal spec M1 acceptance (synthetic -> ingest -> sync)
+# ---------------------------------------------------------------------------
+@pytest.mark.skipif(shutil.which("ffmpeg") is None or shutil.which("ffprobe") is None,
+                    reason="ffmpeg/ffprobe required for the end-to-end sync acceptance")
+def test_synthetic_ingest_sync_recovers_ground_truth():
+    synthetic.build(duration_s=8.0, run_ffmpeg=True)   # into the temp data dir (conftest)
+    ingest.run_ingest()
+    audio_sync.run_sync()
+
+    gt = {v["filename"]: v["offset_ms"]
+          for v in json.loads(config.GROUND_TRUTH_JSON.read_text())["videos"]}
+    for v in db.get_videos():
+        assert v.offset_ms is not None, v.filename
+        assert abs(v.offset_ms - gt[v.filename]) <= 10.0, (v.filename, v.offset_ms, gt[v.filename])
+        assert abs(v.drift_ppm) <= 3.0

backend/tests/test_events_ai.py

@@ -0,0 +1,288 @@
+"""Tests for events_ai.py (lane D / spec M7).
+
+No test hits a real API or ffmpeg: candidate detection runs on the synthetic fixture's
+in-memory audio, and the classifier orchestration is exercised with stub classifiers and a
+monkeypatched input-preparer. This mirrors spec M7's acceptance:
+  - candidates land within ±0.5 s of the ground-truth pulse times, and
+  - the two-stage loop (detect -> classify) works independent of any provider.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+from pydantic import ValidationError
+
+from festival4d import config, db, events_ai, synthetic
+from festival4d.events_ai import (
+    ClaudeClassifier,
+    GeminiClassifier,
+    LocalClassifier,
+    MomentClassification,
+    MomentClassifier,
+    detect_candidates,
+    get_classifier,
+    run_events,
+)
+
+SR = config.SYNTH_AUDIO_SR
+
+
+# ---------------------------------------------------------------------------
+# Helpers / stub classifiers (no network)
+# ---------------------------------------------------------------------------
+def _reference_audio():
+    """cam0's audio: the master signal sliced at offset 0 (local == global time)."""
+    master, gt = synthetic.synth_master_audio(SR)
+    audio = synthetic.slice_for_offset(master, SR, 0.0, config.SYNTH_DURATION_S)
+    return audio, gt["pulse_times_global_s"]
+
+
+class StubClassifier:
+    """Always returns the same valid classification; counts calls."""
+
+    def __init__(self, event_type: str = "bass_drop") -> None:
+        self.calls = 0
+        self.event_type = event_type
+
+    def classify(self, clip_path: Path, frames: list[Path]) -> MomentClassification:
+        self.calls += 1
+        return MomentClassification(
+            event_type=self.event_type, confidence=0.9, description="stub classification"
+        )
+
+
+class FlakyClassifier:
+    """Raises on a chosen call to exercise per-candidate exception isolation."""
+
+    def __init__(self, fail_on_call: int = 2) -> None:
+        self.calls = 0
+        self.fail_on_call = fail_on_call
+
+    def classify(self, clip_path: Path, frames: list[Path]) -> MomentClassification:
+        self.calls += 1
+        if self.calls == self.fail_on_call:
+            raise RuntimeError("simulated provider failure")
+        return MomentClassification(event_type="pyro", confidence=0.6, description="ok")
+
+
+def _fresh_reference_db(offset_ms: float = 0.0) -> None:
+    """A clean DB with a single reference video (matching cam0)."""
+    db.init_engine()
+    db.reset_db()
+    db.add_video(
+        filename="cam0.mp4", duration_s=config.SYNTH_DURATION_S, fps=config.SYNTH_FPS,
+        width=config.SYNTH_VIDEO_W, height=config.SYNTH_VIDEO_H,
+        offset_ms=offset_ms, drift_ppm=0.0, sync_confidence=1.0,
+    )
+
+
+def _stub_run(monkeypatch, classifier):
+    """Wire run_events to run real detection on fixture audio + a stub classifier."""
+    audio, _ = _reference_audio()
+    monkeypatch.setattr(events_ai, "_load_reference_audio", lambda video: (audio, SR))
+    monkeypatch.setattr(events_ai, "get_classifier", lambda name=None: classifier)
+    monkeypatch.setattr(
+        events_ai, "prepare_inputs",
+        lambda t: (Path("/nonexistent/clip.mp4"), [Path("/nonexistent/frame_00.jpg")]),
+    )
+
+
+# ---------------------------------------------------------------------------
+# Candidate detection (the acceptance-critical numeric test)
+# ---------------------------------------------------------------------------
+def test_detect_candidates_matches_ground_truth():
+    audio, pulses = _reference_audio()
+    cands = detect_candidates(audio, SR)
+
+    # Recall: every ground-truth pulse has a candidate within ±0.5 s.
+    for pulse in pulses:
+        nearest = min(abs(c - pulse) for c in cands)
+        assert nearest < 0.5, f"pulse {pulse}s unmatched; nearest candidate {nearest:.3f}s away"
+
+    # Precision: no spurious candidates — each maps to a distinct ground-truth pulse.
+    for c in cands:
+        nearest = min(abs(c - pulse) for pulse in pulses)
+        assert nearest < 0.5, f"spurious candidate at {c:.3f}s ({nearest:.3f}s from any pulse)"
+    assert len(cands) == len(pulses), (cands, pulses)
+
+
+def test_detect_candidates_short_audio_is_empty():
+    import numpy as np
+
+    assert detect_candidates(np.zeros(16, dtype=np.float32), SR) == []
+
+
+def test_detect_candidates_silent_or_flat_audio_is_empty():
+    """Silence / DC / a bare noise floor must yield NO candidates, never fabricate events."""
+    import numpy as np
+
+    n = SR * 6
+    assert detect_candidates(np.zeros(n, dtype=np.float32), SR) == []          # digital silence
+    assert detect_candidates(np.full(n, 0.5, dtype=np.float32), SR) == []      # constant / DC
+    rng = np.random.default_rng(0)
+    assert detect_candidates((1e-4 * rng.standard_normal(n)).astype(np.float32), SR) == []  # noise floor
+
+
+# ---------------------------------------------------------------------------
+# Frozen contract sanity (MomentClassification / MomentClassifier)
+# ---------------------------------------------------------------------------
+def test_moment_classification_validates_bounds_and_enum():
+    ok = MomentClassification(event_type="pyro", confidence=0.5, description="x")
+    assert ok.event_type == "pyro"
+    with pytest.raises(ValidationError):
+        MomentClassification(event_type="pyro", confidence=1.5, description="x")  # >1
+    with pytest.raises(ValidationError):
+        MomentClassification(event_type="not_a_type", confidence=0.5, description="x")
+
+
+def test_stub_satisfies_classifier_protocol():
+    assert isinstance(StubClassifier(), MomentClassifier)
+
+
+# ---------------------------------------------------------------------------
+# Provider selection + degradation (no network — constructors don't call out)
+# ---------------------------------------------------------------------------
+def test_get_classifier_unconfigured_returns_none(monkeypatch):
+    for var in ("GEMINI_API_KEY", "ANTHROPIC_API_KEY",
+                "FESTIVAL4D_OPENAI_BASE_URL", "FESTIVAL4D_OPENAI_MODEL", "FESTIVAL4D_OPENAI_KEY"):
+        monkeypatch.delenv(var, raising=False)
+    assert get_classifier("gemini") is None
+    assert get_classifier("claude") is None
+    assert get_classifier("local") is None
+
+
+def test_get_classifier_unknown_provider_returns_none():
+    assert get_classifier("does-not-exist") is None
+
+
+def test_get_classifier_defaults_to_gemini(monkeypatch):
+    monkeypatch.delenv("FESTIVAL4D_CLASSIFIER", raising=False)
+    monkeypatch.setenv("GEMINI_API_KEY", "test-key")
+    assert isinstance(get_classifier(), GeminiClassifier)
+
+
+def test_get_classifier_local_when_configured(monkeypatch):
+    monkeypatch.setenv("FESTIVAL4D_OPENAI_BASE_URL", "http://localhost:11434/v1")
+    monkeypatch.setenv("FESTIVAL4D_OPENAI_MODEL", "qwen2.5-vl")
+    monkeypatch.delenv("FESTIVAL4D_OPENAI_KEY", raising=False)
+    clf = get_classifier("local")
+    assert isinstance(clf, LocalClassifier)
+    assert isinstance(clf, MomentClassifier)
+
+
+def test_get_classifier_claude_when_configured(monkeypatch):
+    monkeypatch.setenv("ANTHROPIC_API_KEY", "test-key")
+    assert isinstance(get_classifier("claude"), ClaudeClassifier)
+
+
+# ---------------------------------------------------------------------------
+# Orchestration loop (stub classifier — no API, no ffmpeg)
+# ---------------------------------------------------------------------------
+def test_run_events_classifies_all_candidates(monkeypatch):
+    _fresh_reference_db()
+    stub = StubClassifier(event_type="bass_drop")
+    _stub_run(monkeypatch, stub)
+
+    result = run_events()
+
+    assert result["candidates"] == 3
+    assert result["classified"] == 3
+    assert result["candidates_only"] == 0
+    assert result["classifier_configured"] is True
+    assert stub.calls == 3
+
+    events = db.get_events()
+    assert len(events) == 3
+    assert all(e.source == "ai" for e in events)
+    assert all(e.event_type == "bass_drop" for e in events)
+    # Candidate times map to t_global (reference offset 0) near the fixture pulses.
+    times = sorted(e.t_global_s for e in events)
+    for got, pulse in zip(times, (3.0, 10.0, 17.0)):
+        assert abs(got - pulse) < 0.5
+
+
+def test_run_events_unconfigured_stores_candidates_only(monkeypatch):
+    _fresh_reference_db()
+    audio, _ = _reference_audio()
+    monkeypatch.setattr(events_ai, "_load_reference_audio", lambda video: (audio, SR))
+    monkeypatch.setattr(events_ai, "get_classifier", lambda name=None: None)
+    # Degraded path must never prepare inputs. A *raised* tripwire would be swallowed by
+    # _classify_one's `except Exception`; record calls with a side effect instead.
+    prepared: list = []
+    monkeypatch.setattr(events_ai, "prepare_inputs", lambda t: prepared.append(t))
+
+    result = run_events()
+
+    assert prepared == []  # prepare_inputs was never called when unconfigured
+
+    assert result["candidates"] == 3
+    assert result["classified"] == 0
+    assert result["candidates_only"] == 3
+    assert result["classifier_configured"] is False
+
+    events = db.get_events()
+    assert len(events) == 3
+    assert all(e.source == "audio_auto" for e in events)
+    assert all(e.event_type == "candidate" for e in events)
+
+
+def test_run_events_isolates_per_candidate_failure(monkeypatch):
+    _fresh_reference_db()
+    flaky = FlakyClassifier(fail_on_call=2)  # 2nd candidate fails
+    _stub_run(monkeypatch, flaky)
+
+    result = run_events()
+
+    assert result["candidates"] == 3
+    assert result["classified"] == 2       # two succeeded
+    assert result["candidates_only"] == 1  # the failed one fell back to a candidate
+    assert flaky.calls == 3                # the batch was not aborted
+
+    events = db.get_events()
+    assert sum(e.source == "ai" for e in events) == 2
+    assert sum(e.source == "audio_auto" for e in events) == 1
+
+
+def test_run_events_whole_track_replaces_machine_events_keeps_user(monkeypatch):
+    _fresh_reference_db()
+    # Pre-existing events: a stale AI one and a user correction.
+    db.add_event(t_global_s=99.0, event_type="other", source="ai", description="stale")
+    db.add_event(t_global_s=5.0, event_type="crowd_wave", source="user", description="mine")
+    _stub_run(monkeypatch, StubClassifier())
+
+    run_events()
+
+    events = db.get_events()
+    user = [e for e in events if e.source == "user"]
+    ai = [e for e in events if e.source == "ai"]
+    assert len(user) == 1 and user[0].description == "mine"   # user edit preserved
+    assert len(ai) == 3                                       # fresh detection, stale AI gone
+    assert not any(e.description == "stale" for e in events)
+
+
+def test_run_events_windowed_adds_in_window_and_clears_nothing(monkeypatch):
+    _fresh_reference_db()
+    # A machine event OUTSIDE the window must survive — windowed detection is purely additive.
+    db.add_event(t_global_s=3.0, event_type="bass_drop", source="ai", description="pre-existing")
+    _stub_run(monkeypatch, StubClassifier())
+
+    result = run_events(t_global_s=10.0, window_s=4.0)  # window [8, 12] -> only the ~10 s pulse
+
+    assert result["candidates"] == 1
+    assert result["window"] == {"t_global_s": 10.0, "window_s": 4.0}
+    events = db.get_events()
+    assert any(e.description == "pre-existing" for e in events)  # out-of-window event preserved
+    assert len(events) == 2
+    new = [e for e in events if e.description != "pre-existing"]
+    assert len(new) == 1 and abs(new[0].t_global_s - 10.0) < 0.5
+
+
+def test_run_events_no_videos_degrades(monkeypatch):
+    db.init_engine()
+    db.reset_db()
+    result = run_events()
+    assert result["candidates"] == 0
+    assert "note" in result
+    assert db.get_events() == []

backend/tests/test_geometry.py

@@ -0,0 +1,333 @@
+"""Tests for geometry.py.
+
+The ``colmap_to_threejs`` tests below are a **FROZEN CONTRACT** (foundation). The embedded
+``POSE_TEST_VECTORS`` are duplicated verbatim in ``frontend/src/lib/pose.js`` so both sides
+provably agree — if you touch the conversion math, update both files and both vector sets,
+via a change request. Lane B owns this file but must not modify these three cases; it adds
+tests for slerp_pose / ray_from_pixel / triangulate_rays / nearest_point_on_ray below them.
+"""
+
+from __future__ import annotations
+
+import numpy as np
+import pytest
+from scipy.spatial.transform import Rotation
+
+from festival4d.geometry import (
+    colmap_to_threejs,
+    mat_to_quat,
+    quat_to_mat,
+)
+
+# Sqrt(2)/2, spelled out so it matches the JS constant character-for-character.
+SQRT1_2 = 0.7071067811865476
+
+# --- FROZEN test vectors (must equal POSE_TEST_VECTORS in frontend/src/lib/pose.js) ----
+# Each: COLMAP world->camera (q=[w,x,y,z], t) -> Three.js (position, rotation 3x3 row-major).
+POSE_TEST_VECTORS = [
+    {
+        "name": "identity",
+        "q": [1.0, 0.0, 0.0, 0.0],
+        "t": [0.0, 0.0, -10.0],
+        "position": [0.0, 0.0, 10.0],
+        "rotation": [[1.0, 0.0, 0.0], [0.0, -1.0, 0.0], [0.0, 0.0, -1.0]],
+    },
+    {
+        "name": "yaw90",  # +90 deg about world Y
+        "q": [SQRT1_2, 0.0, SQRT1_2, 0.0],
+        "t": [0.0, 0.0, 10.0],
+        "position": [10.0, 0.0, 0.0],
+        "rotation": [[0.0, 0.0, 1.0], [0.0, -1.0, 0.0], [1.0, 0.0, 0.0]],
+    },
+    {
+        "name": "lookat",  # camera at +Z looking at origin -> Three.js rotation is identity
+        "q": [0.0, 1.0, 0.0, 0.0],
+        "t": [0.0, 0.0, 8.0],
+        "position": [0.0, 0.0, 8.0],
+        "rotation": [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]],
+    },
+]
+
+
+@pytest.mark.parametrize("vec", POSE_TEST_VECTORS, ids=lambda v: v["name"])
+def test_colmap_to_threejs_known_vectors(vec):
+    position, rotation = colmap_to_threejs(vec["q"], vec["t"])
+    np.testing.assert_allclose(position, vec["position"], atol=1e-9)
+    np.testing.assert_allclose(rotation, vec["rotation"], atol=1e-9)
+
+
+@pytest.mark.parametrize("vec", POSE_TEST_VECTORS, ids=lambda v: v["name"])
+def test_threejs_rotation_is_proper(vec):
+    """R_three must be a proper rotation (orthonormal, det = +1)."""
+    _, R = colmap_to_threejs(vec["q"], vec["t"])
+    np.testing.assert_allclose(R @ R.T, np.eye(3), atol=1e-9)
+    assert abs(np.linalg.det(R) - 1.0) < 1e-9
+
+
+def _colmap_pose(R_w2c: np.ndarray, center: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
+    """Build a COLMAP (q, t) from a world->camera rotation and a world camera center."""
+    t = -R_w2c @ center
+    q = mat_to_quat(R_w2c)
+    return q, t
+
+
+def test_colmap_to_threejs_roundtrip_random_poses():
+    """Forward COLMAP->Three.js then invert it; recover the original (q, t)."""
+    rng = np.random.default_rng(20260716)
+    flip = np.diag([1.0, -1.0, -1.0])
+    for _ in range(50):
+        R_w2c = Rotation.random(random_state=rng).as_matrix()
+        center = rng.uniform(-10, 10, size=3)
+        q, t = _colmap_pose(R_w2c, center)
+
+        position, R_three = colmap_to_threejs(q, t)
+
+        # position is the camera center
+        np.testing.assert_allclose(position, center, atol=1e-9)
+
+        # invert: R_c2w = R_three @ flip  ->  R_w2c = R_c2w^T  ->  t = -R_w2c @ C
+        R_c2w = R_three @ flip
+        R_back = R_c2w.T
+        t_back = -R_back @ position
+        q_back = mat_to_quat(R_back)
+
+        np.testing.assert_allclose(t_back, t, atol=1e-9)
+        # quaternions are double-cover; compare canonical (w>=0) forms
+        q_canon = q if q[0] >= 0 else -q
+        np.testing.assert_allclose(q_back, q_canon, atol=1e-9)
+
+
+def test_quat_to_mat_matches_scipy_oracle():
+    """quat_to_mat agrees with an independent implementation (scipy)."""
+    rng = np.random.default_rng(11)
+    for _ in range(50):
+        # scipy quaternions are scalar-LAST [x,y,z,w]; ours are scalar-FIRST [w,x,y,z]
+        q_xyzw = rng.normal(size=4)
+        q_xyzw /= np.linalg.norm(q_xyzw)
+        q_wxyz = np.array([q_xyzw[3], q_xyzw[0], q_xyzw[1], q_xyzw[2]])
+        np.testing.assert_allclose(
+            quat_to_mat(q_wxyz), Rotation.from_quat(q_xyzw).as_matrix(), atol=1e-9
+        )
+
+
+# ===========================================================================
+# Lane B (spec M2 + M8) tests — below the frozen block. These exercise the
+# stubs foundation left: slerp_pose, ray_from_pixel, triangulate_rays,
+# nearest_point_on_ray. They do NOT touch the frozen colmap_to_threejs cases above.
+# ===========================================================================
+from scipy.spatial.transform import Slerp  # noqa: E402
+
+from festival4d.geometry import (  # noqa: E402
+    colmap_to_threejs,
+    nearest_point_on_ray,
+    ray_from_pixel,
+    slerp_pose,
+    triangulate_rays,
+)
+
+
+def _wxyz_to_xyzw(q):
+    q = np.asarray(q, dtype=float)
+    return np.array([q[1], q[2], q[3], q[0]])
+
+
+def _rand_unit_quat(rng):
+    q = rng.normal(size=4)
+    return q / np.linalg.norm(q)
+
+
+# --- slerp_pose ------------------------------------------------------------
+def test_slerp_pose_endpoints():
+    q0 = _rand_unit_quat(np.random.default_rng(1))
+    q1 = _rand_unit_quat(np.random.default_rng(2))
+    t0 = np.array([1.0, -2.0, 3.0])
+    t1 = np.array([-4.0, 5.0, 6.0])
+
+    q_a, t_a = slerp_pose(q0, t0, q1, t1, 0.0)
+    q_b, t_b = slerp_pose(q0, t0, q1, t1, 1.0)
+    # endpoints recover the endpoint *rotations* (quaternion up to sign) and translations
+    np.testing.assert_allclose(quat_to_mat(q_a), quat_to_mat(q0), atol=1e-12)
+    np.testing.assert_allclose(quat_to_mat(q_b), quat_to_mat(q1), atol=1e-12)
+    np.testing.assert_allclose(t_a, t0, atol=1e-12)
+    np.testing.assert_allclose(t_b, t1, atol=1e-12)
+
+
+def test_slerp_pose_matches_scipy_oracle():
+    rng = np.random.default_rng(20260716)
+    for _ in range(30):
+        q0 = _rand_unit_quat(rng)
+        q1 = _rand_unit_quat(rng)
+        oracle = Slerp([0.0, 1.0], Rotation.from_quat(
+            [_wxyz_to_xyzw(q0), _wxyz_to_xyzw(q1)]))
+        for alpha in (0.1, 0.25, 0.5, 0.73, 0.9):
+            q, _ = slerp_pose(q0, [0, 0, 0], q1, [0, 0, 0], alpha)
+            np.testing.assert_allclose(
+                quat_to_mat(q), oracle(alpha).as_matrix(), atol=1e-9)
+
+
+def test_slerp_pose_translation_is_linear():
+    q = np.array([1.0, 0.0, 0.0, 0.0])
+    t0 = np.array([0.0, 0.0, 0.0])
+    t1 = np.array([10.0, -4.0, 2.0])
+    for alpha in (0.0, 0.3, 0.5, 1.0):
+        _, t = slerp_pose(q, t0, q, t1, alpha)
+        np.testing.assert_allclose(t, (1 - alpha) * t0 + alpha * t1, atol=1e-12)
+
+
+def test_slerp_pose_double_cover_takes_short_arc():
+    """q1 and -q1 are the same rotation; slerp must yield the same (short-arc) result."""
+    rng = np.random.default_rng(99)
+    q0 = _rand_unit_quat(rng)
+    q1 = _rand_unit_quat(rng)
+    for alpha in (0.2, 0.5, 0.8):
+        qa, _ = slerp_pose(q0, [0, 0, 0], q1, [0, 0, 0], alpha)
+        qb, _ = slerp_pose(q0, [0, 0, 0], -q1, [0, 0, 0], alpha)
+        np.testing.assert_allclose(quat_to_mat(qa), quat_to_mat(qb), atol=1e-12)
+
+
+def test_slerp_pose_midpoint_is_half_angle():
+    """A 180-degree-ish pair: the midpoint rotation angle is half the endpoint angle."""
+    q0 = np.array([1.0, 0.0, 0.0, 0.0])                 # identity
+    ang = np.radians(100.0)
+    q1 = np.array([np.cos(ang / 2), 0.0, np.sin(ang / 2), 0.0])  # yaw 100 deg about Y
+    qm, _ = slerp_pose(q0, [0, 0, 0], q1, [0, 0, 0], 0.5)
+    # rotation angle of qm relative to identity should be ~50 deg
+    angle = 2.0 * np.arccos(min(1.0, abs(qm[0])))
+    assert abs(np.degrees(angle) - 50.0) < 1e-6
+
+
+# --- ray_from_pixel --------------------------------------------------------
+def test_ray_from_pixel_origin_is_camera_center():
+    """The ray origin equals the Three.js camera center from the frozen contract."""
+    rng = np.random.default_rng(5)
+    for _ in range(20):
+        q = _rand_unit_quat(rng)
+        t = rng.uniform(-5, 5, size=3)
+        position, _ = colmap_to_threejs(q, t)
+        origin, _ = ray_from_pixel(q, t, 600, 600, 320, 180, 320, 180)
+        np.testing.assert_allclose(origin, position, atol=1e-9)
+
+
+def test_ray_from_pixel_center_is_forward_axis():
+    """The center pixel unprojects along the camera forward (+z) axis, in world coords."""
+    q = np.array([1.0, 0.0, 0.0, 0.0])   # identity world->cam
+    t = np.array([0.0, 0.0, -10.0])      # camera center at (0,0,10)
+    origin, direction = ray_from_pixel(q, t, 500, 500, 320, 180, 320, 180)
+    np.testing.assert_allclose(origin, [0.0, 0.0, 10.0], atol=1e-9)
+    np.testing.assert_allclose(direction, [0.0, 0.0, 1.0], atol=1e-9)
+
+
+def _project_colmap(q, t, fx, fy, cx, cy, X):
+    """Forward COLMAP pinhole projection of a world point X -> pixel (px, py)."""
+    R = quat_to_mat(q)
+    xc = R @ np.asarray(X, float) + np.asarray(t, float)
+    return fx * xc[0] / xc[2] + cx, fy * xc[1] / xc[2] + cy, xc[2]
+
+
+def test_ray_from_pixel_is_projection_inverse():
+    """A world point in front of the camera lies exactly on the ray through its pixel."""
+    rng = np.random.default_rng(7)
+    for _ in range(50):
+        q = _rand_unit_quat(rng)
+        t = rng.uniform(-3, 3, size=3)
+        fx = fy = rng.uniform(400, 800)
+        cx, cy = 320.0, 180.0
+        C = -quat_to_mat(q).T @ t                    # camera center
+        forward = quat_to_mat(q).T @ np.array([0, 0, 1.0])
+        X = C + rng.uniform(2, 8) * forward + rng.uniform(-1, 1, size=3)  # in front
+        px, py, zc = _project_colmap(q, t, fx, fy, cx, cy, X)
+        if zc <= 0.1:
+            continue
+        origin, direction = ray_from_pixel(q, t, fx, fy, cx, cy, px, py)
+        to_X = X - origin
+        # X - origin must be parallel to direction and in front (positive projection)
+        cross = np.cross(to_X, direction)
+        assert np.linalg.norm(cross) < 1e-6 * (1 + np.linalg.norm(to_X))
+        assert np.dot(to_X, direction) > 0
+
+
+# --- triangulate_rays ------------------------------------------------------
+def test_triangulate_rays_intersecting():
+    P = np.array([1.0, 2.0, 3.0])
+    oa = np.array([0.0, 0.0, 0.0])
+    ob = np.array([4.0, 0.0, 0.0])
+    point, gap = triangulate_rays(oa, P - oa, ob, P - ob)
+    np.testing.assert_allclose(point, P, atol=1e-9)
+    assert gap < 1e-9
+
+
+def test_triangulate_rays_skew_known_geometry():
+    """Ray A along +x at z=0; ray B along +y at z=1. Closest points (0,0,0),(0,0,1)."""
+    oa = np.array([0.0, 0.0, 0.0]); da = np.array([1.0, 0.0, 0.0])
+    ob = np.array([0.0, 0.0, 1.0]); db = np.array([0.0, 1.0, 0.0])
+    point, gap = triangulate_rays(oa, da, ob, db)
+    np.testing.assert_allclose(point, [0.0, 0.0, 0.5], atol=1e-9)
+    assert abs(gap - 1.0) < 1e-9
+
+
+def test_triangulate_rays_parallel_is_safe():
+    """Parallel rays must not divide-by-zero; gap ~ their separation."""
+    oa = np.array([0.0, 0.0, 0.0]); da = np.array([1.0, 0.0, 0.0])
+    ob = np.array([0.0, 2.0, 0.0]); db = np.array([1.0, 0.0, 0.0])
+    point, gap = triangulate_rays(oa, da, ob, db)
+    assert np.all(np.isfinite(point))
+    assert abs(gap - 2.0) < 1e-6
+
+
+def test_triangulate_rays_from_two_cameras_recovers_point():
+    """Two cameras looking at a world point; triangulating their pixel rays recovers it."""
+    rng = np.random.default_rng(123)
+    P = np.array([0.5, 1.0, -0.5])
+    fx = fy = 600.0; cx, cy = 320.0, 180.0
+    rays = []
+    for center in ([6.0, 2.0, 6.0], [-6.0, 2.5, 6.0]):
+        C = np.array(center, float)
+        z = P - C; z /= np.linalg.norm(z)
+        up = np.array([0.0, 1.0, 0.0])
+        up = up - np.dot(up, z) * z; up /= np.linalg.norm(up)
+        x = np.cross(z, up); y = -up
+        R_c2w = np.column_stack([x, y, z]); R = R_c2w.T
+        t = -R @ C
+        q = mat_to_quat(R)
+        px, py, _ = _project_colmap(q, t, fx, fy, cx, cy, P)
+        rays.append(ray_from_pixel(q, t, fx, fy, cx, cy, px, py))
+    point, gap = triangulate_rays(rays[0][0], rays[0][1], rays[1][0], rays[1][1])
+    assert np.linalg.norm(point - P) < 0.2   # spec M8 acceptance tolerance
+    assert gap < 1e-6
+
+
+# --- nearest_point_on_ray --------------------------------------------------
+def test_nearest_point_on_ray_hits_within_radius():
+    o = np.array([0.0, 0.0, 0.0]); d = np.array([0.0, 0.0, 1.0])
+    points = np.array([
+        [0.1, 0.0, 5.0],    # perp 0.1, in front -> candidate
+        [0.05, 0.0, 3.0],   # perp 0.05, in front -> closest
+        [2.0, 0.0, 5.0],    # perp 2.0 -> outside radius
+    ])
+    got = nearest_point_on_ray(o, d, points, radius=0.3)
+    np.testing.assert_allclose(got, [0.05, 0.0, 3.0], atol=1e-12)
+
+
+def test_nearest_point_on_ray_excludes_outside_and_behind():
+    o = np.array([0.0, 0.0, 0.0]); d = np.array([0.0, 0.0, 1.0])
+    behind = np.array([[0.05, 0.0, -3.0]])     # in cylinder but behind origin
+    outside = np.array([[1.0, 0.0, 3.0]])      # in front but outside radius
+    assert nearest_point_on_ray(o, d, behind, radius=0.3) is None
+    assert nearest_point_on_ray(o, d, outside, radius=0.3) is None
+    assert nearest_point_on_ray(o, d, np.zeros((0, 3)), radius=0.3) is None
+
+
+def test_nearest_point_on_ray_synthetic_stage_corner():
+    """M8 single-view fallback: a ray toward a stage corner selects that corner point."""
+    from festival4d import synthetic
+
+    corner = np.array(synthetic.STAGE_CORNERS["Stage FL"], float)
+    points, _ = synthetic.generate_point_cloud()
+    # a camera looking straight at the corner; ray through the image center hits it
+    q, t = synthetic.look_at_colmap((5.0, 3.0, 9.0), corner)
+    intr = synthetic.intrinsics(640, 360)
+    origin, direction = ray_from_pixel(
+        q, t, intr["fx"], intr["fy"], intr["cx"], intr["cy"], intr["cx"], intr["cy"])
+    got = nearest_point_on_ray(origin, direction, points.astype(float), radius=0.3)
+    assert got is not None
+    np.testing.assert_allclose(got, corner, atol=1e-4)

backend/tests/test_ingest.py

@@ -0,0 +1,93 @@
+"""Ingest tests (spec M1, lane A).
+
+Pure unit tests for the path/fps helpers, plus ffmpeg-gated tests that probe a real
+rendered clip, extract audio, and run the full idempotent ``run_ingest``.
+"""
+
+from __future__ import annotations
+
+import shutil
+
+import numpy as np
+import pytest
+
+from festival4d import config, db, ingest, synthetic
+
+
+# ---------------------------------------------------------------------------
+# pure helpers
+# ---------------------------------------------------------------------------
+def test_audio_wav_path_convention():
+    assert ingest.audio_wav_path(7) == config.AUDIO_DIR / "7.wav"
+
+
+def test_parse_fps():
+    assert ingest._parse_fps("30/1") == 30.0
+    assert abs(ingest._parse_fps("30000/1001") - 29.97) < 0.01
+    assert ingest._parse_fps("25") == 25.0
+    assert ingest._parse_fps(None) is None
+    assert ingest._parse_fps("0/0") is None
+    assert ingest._parse_fps("N/A") is None
+
+
+def test_run_ingest_no_files_is_empty(tmp_path, monkeypatch):
+    """An empty raw dir yields an empty summary, not a crash."""
+    monkeypatch.setattr(config, "RAW_DIR", tmp_path / "empty_raw")
+    (tmp_path / "empty_raw").mkdir()
+    db.init_engine()
+    db.reset_db()
+    assert ingest.run_ingest() == []
+
+
+# ---------------------------------------------------------------------------
+# ffmpeg-gated
+# ---------------------------------------------------------------------------
+def _render_clip(tmp_path, duration_s=2.0, cam_index=0):
+    sr = config.AUDIO_SAMPLE_RATE
+    master, _ = synthetic.synth_master_audio(sr)
+    clip = synthetic.slice_for_offset(master, sr, 0.0, duration_s)
+    wav = tmp_path / "src.wav"
+    synthetic.write_wav(wav, clip, sr)
+    mp4 = tmp_path / f"cam{cam_index}.mp4"
+    synthetic.render_video(mp4, wav, cam_index, 640, 360, 30, duration_s)
+    return mp4
+
+
+@pytest.mark.skipif(shutil.which("ffmpeg") is None or shutil.which("ffprobe") is None,
+                    reason="ffmpeg/ffprobe required")
+def test_probe_video(tmp_path):
+    mp4 = _render_clip(tmp_path, duration_s=2.0)
+    info = ingest.probe_video(mp4)
+    assert info["width"] == 640 and info["height"] == 360
+    assert abs(info["fps"] - 30.0) < 1.0
+    assert 1.5 < info["duration_s"] < 2.6
+
+
+@pytest.mark.skipif(shutil.which("ffmpeg") is None or shutil.which("ffprobe") is None,
+                    reason="ffmpeg/ffprobe required")
+def test_extract_audio(tmp_path):
+    mp4 = _render_clip(tmp_path, duration_s=2.0)
+    out = ingest.extract_audio(mp4, tmp_path / "out.wav", sample_rate=16_000)
+    assert out.exists()
+    samples, sr = synthetic.read_wav(out)
+    assert sr == 16_000
+    assert samples.ndim == 1                          # mono
+    assert abs(len(samples) / sr - 2.0) < 0.3         # ~2 s
+
+
+@pytest.mark.skipif(shutil.which("ffmpeg") is None or shutil.which("ffprobe") is None,
+                    reason="ffmpeg/ffprobe required")
+def test_run_ingest_registers_and_is_idempotent():
+    synthetic.build(duration_s=2.0, run_ffmpeg=True)   # into the temp data dir (conftest)
+    summary = ingest.run_ingest()
+    assert len(summary) == 3
+    for s in summary:
+        assert ingest.audio_wav_path(s["video_id"]).exists()
+        samples, sr = synthetic.read_wav(ingest.audio_wav_path(s["video_id"]))
+        assert sr == 16_000 and samples.ndim == 1
+
+    # Second run reuses the existing rows (no unique-constraint crash) and re-extracts.
+    again = ingest.run_ingest()
+    assert len(again) == 3
+    assert all(s["reused_db_row"] for s in again)
+    assert len(db.get_videos()) == 3                  # no duplicate rows

backend/tests/test_sfm.py

@@ -0,0 +1,494 @@
+"""Tests for lane B: frames.py (sampling) + sfm.py (parsers, normalization,
+interpolation, and the graceful-degradation reconstruct pipeline).
+
+Everything here is verifiable without COLMAP: parsers run on hand-written TXT snippets, the
+export path runs on a hand-built (synthetic-pose) COLMAP model injected in place of a real
+COLMAP run, and the failure paths are exercised by monkeypatching. Spec M2 acceptance:
+"on the synthetic fixture (which skips COLMAP and injects known poses) the whole export path
+runs; if COLMAP is installed, the pipeline runs end-to-end without crashing."
+"""
+
+from __future__ import annotations
+
+import struct
+from pathlib import Path
+
+import numpy as np
+import pytest
+
+from festival4d import config, db, frames, sfm, synthetic
+from festival4d.geometry import quat_to_mat
+
+
+# ===========================================================================
+# COLMAP TXT parsers (hand-written snippets)
+# ===========================================================================
+IMAGES_TXT = """\
+# Image list with two lines of data per image:
+#   IMAGE_ID, QW, QX, QY, QZ, TX, TY, TZ, CAMERA_ID, NAME
+#   POINTS2D[] as (X, Y, POINT3D_ID)
+# Number of images: 2, mean observations per image: 2
+1 0.9998 0.0 0.02 0.0 -1.5 0.3 8.0 1 1/1_0.jpg
+480.0 270.0 5 500.1 260.2 -1
+2 0.7071 0.0 0.7071 0.0 2.0 0.1 7.5 2 2/2_30.jpg
+100.0 120.0 -1 300.0 200.0 5
+"""
+
+CAMERAS_TXT = """\
+# Camera list with one line of data per camera:
+#   CAMERA_ID, MODEL, WIDTH, HEIGHT, PARAMS[]
+# Number of cameras: 2
+1 PINHOLE 640 360 500.0 500.0 320.0 180.0
+2 OPENCV 640 360 510.0 511.0 321.0 181.0 0.01 -0.02 0.0 0.0
+"""
+
+POINTS3D_TXT = """\
+# 3D point list with one line of data per point:
+#   POINT3D_ID, X, Y, Z, R, G, B, ERROR, TRACK[] as (IMAGE_ID, POINT2D_IDX)
+# Number of points: 2, mean track length: 2
+5 1.0 2.0 3.0 200 100 50 0.5 1 0 2 1
+9 -1.0 0.0 4.0 10 20 30 0.8 1 1
+"""
+
+
+def test_parse_images_txt(tmp_path):
+    p = tmp_path / "images.txt"
+    p.write_text(IMAGES_TXT)
+    recs = sfm.parse_images_txt(p)
+    assert len(recs) == 2
+    assert recs[0]["image_id"] == 1
+    assert recs[0]["camera_id"] == 1
+    assert recs[0]["name"] == "1/1_0.jpg"
+    np.testing.assert_allclose(
+        [recs[0]["qw"], recs[0]["qx"], recs[0]["qy"], recs[0]["qz"]],
+        [0.9998, 0.0, 0.02, 0.0])
+    np.testing.assert_allclose(
+        [recs[1]["tx"], recs[1]["ty"], recs[1]["tz"]], [2.0, 0.1, 7.5])
+
+
+def test_parse_cameras_txt(tmp_path):
+    p = tmp_path / "cameras.txt"
+    p.write_text(CAMERAS_TXT)
+    cams = sfm.parse_cameras_txt(p)
+    assert set(cams) == {1, 2}
+    assert cams[1]["model"] == "PINHOLE"
+    assert (cams[1]["fx"], cams[1]["fy"], cams[1]["cx"], cams[1]["cy"]) == \
+        (500.0, 500.0, 320.0, 180.0)
+    # OPENCV: first 4 params are fx, fy, cx, cy; distortion ignored
+    assert (cams[2]["fx"], cams[2]["fy"], cams[2]["cx"], cams[2]["cy"]) == \
+        (510.0, 511.0, 321.0, 181.0)
+
+
+def test_parse_points3d_txt(tmp_path):
+    p = tmp_path / "points3D.txt"
+    p.write_text(POINTS3D_TXT)
+    points, colors = sfm.parse_points3d_txt(p)
+    assert points.shape == (2, 3) and colors.shape == (2, 3)
+    np.testing.assert_allclose(points[0], [1.0, 2.0, 3.0])
+    assert tuple(colors[0]) == (200, 100, 50)
+    assert points.dtype == np.float32 and colors.dtype == np.uint8
+
+
+def test_parse_points3d_empty(tmp_path):
+    p = tmp_path / "points3D.txt"
+    p.write_text("# header only\n")
+    points, colors = sfm.parse_points3d_txt(p)
+    assert points.shape == (0, 3) and colors.shape == (0, 3)
+
+
+# ===========================================================================
+# Scene normalization
+# ===========================================================================
+def _all_synth_poses():
+    poses = []
+    for i in range(3):
+        poses.extend(synthetic.camera_track(i, 20.0, 30, 640, 360))
+    return poses
+
+
+def test_normalize_scene_invariants():
+    points, _ = synthetic.generate_point_cloud()
+    poses = _all_synth_poses()
+    npoints, nposes = sfm.normalize_scene(points.astype(float), poses)
+
+    # centroid at origin
+    assert np.linalg.norm(npoints.mean(axis=0)) < 1e-3
+    # camera bounding sphere (from origin = point centroid) has radius 10
+    centers = np.array([sfm._camera_center(p) for p in nposes])
+    assert abs(float(np.max(np.linalg.norm(centers, axis=1))) - 10.0) < 1e-6
+    # average camera-up aligns with +Y
+    ups = np.array([sfm._camera_up(p) for p in nposes])
+    up_avg = ups.mean(axis=0)
+    up_avg /= np.linalg.norm(up_avg)
+    np.testing.assert_allclose(up_avg, [0.0, 1.0, 0.0], atol=1e-6)
+
+
+def test_normalize_scene_preserves_projection():
+    """A world point projects to the same pixel before and after normalization
+    (a similarity transform of the world must not change any image)."""
+    points, _ = synthetic.generate_point_cloud()
+    poses = _all_synth_poses()
+    intr = synthetic.intrinsics(640, 360)
+    for p in poses:
+        p.update(intr)
+    X = np.array([0.7, 0.9, 0.3])  # arbitrary world point
+
+    def project(pose, Xw):
+        R = quat_to_mat([pose["qw"], pose["qx"], pose["qy"], pose["qz"]])
+        t = np.array([pose["tx"], pose["ty"], pose["tz"]])
+        xc = R @ Xw + t
+        return np.array([pose["fx"] * xc[0] / xc[2] + pose["cx"],
+                         pose["fy"] * xc[1] / xc[2] + pose["cy"]])
+
+    npoints, nposes = sfm.normalize_scene(points.astype(float), poses)
+    # transform the same world point by X' = s*Rn@(X-c): recover s,Rn,c from the point map.
+    # Easier: the invariant is projection equality, so map X through the same transform used
+    # on the cloud by fitting it — instead, just check pixel equality using each pose pair.
+    # Reconstruct transform from three non-collinear cloud points is overkill; use the fact
+    # that projection is invariant, so compare original point's pixel to the transformed
+    # point's pixel where the transform is inferred from the point cloud centroid+scale.
+    c = points.astype(float).mean(axis=0)
+    centers = np.array([sfm._camera_center(p) for p in poses])
+    scale = 10.0 / float(np.max(np.linalg.norm(centers - c, axis=1)))
+    ups = np.array([sfm._camera_up(p) for p in poses])
+    up_avg = ups.mean(axis=0)
+    Rn = sfm._rotation_aligning(up_avg, np.array([0.0, 1.0, 0.0]))
+    Xn = scale * Rn @ (X - c)
+
+    for p_old, p_new in zip(poses, nposes):
+        np.testing.assert_allclose(project(p_old, X), project(p_new, Xn), atol=1e-6)
+
+
+# ===========================================================================
+# Pose interpolation
+# ===========================================================================
+def _pose(frame_idx, t, q, t3, registered=True):
+    return {"frame_idx": frame_idx, "t_video_s": t,
+            "qw": q[0], "qx": q[1], "qy": q[2], "qz": q[3],
+            "tx": t3[0], "ty": t3[1], "tz": t3[2],
+            "fx": 500.0, "fy": 500.0, "cx": 320.0, "cy": 180.0,
+            "registered": registered, "video_id": 1}
+
+
+def test_interpolate_poses_fills_gaps_no_extrapolation():
+    q0 = [1.0, 0.0, 0.0, 0.0]
+    ang = np.radians(60.0)
+    q1 = [np.cos(ang / 2), 0.0, np.sin(ang / 2), 0.0]  # 60 deg yaw about Y
+    registered = [_pose(0, 0.0, q0, [0, 0, 0]), _pose(60, 2.0, q1, [6, 0, 0])]
+    sampled = [{"frame_idx": f, "t_video_s": f / 30.0} for f in (0, 30, 60, 90)]
+
+    out = sfm.interpolate_poses(registered, sampled)
+    by_frame = {p["frame_idx"]: p for p in out}
+    assert set(by_frame) == {0, 30, 60}          # frame 90 dropped (beyond last registered)
+    assert by_frame[30]["registered"] is False    # interpolated
+    assert by_frame[0]["registered"] is True and by_frame[60]["registered"] is True
+    # midpoint translation is the lerp; rotation is ~30 deg
+    np.testing.assert_allclose(
+        [by_frame[30]["tx"], by_frame[30]["ty"], by_frame[30]["tz"]], [3, 0, 0], atol=1e-9)
+    mid_angle = 2.0 * np.degrees(np.arccos(min(1.0, abs(by_frame[30]["qw"]))))
+    assert abs(mid_angle - 30.0) < 1e-6
+
+
+def test_interpolate_poses_empty_registered():
+    assert sfm.interpolate_poses([], [{"frame_idx": 0, "t_video_s": 0.0}]) == []
+
+
+# ===========================================================================
+# frames.py — sharpness + windowed sampling
+# ===========================================================================
+def test_sharpness_sharp_beats_blurred():
+    import cv2
+
+    rng = np.random.default_rng(0)
+    noise = (rng.integers(0, 256, size=(120, 120), dtype=np.uint8))   # high-frequency
+    flat = np.full((120, 120), 128, dtype=np.uint8)                    # no edges
+    blurred = cv2.GaussianBlur(noise, (0, 0), sigmaX=5)
+    assert frames.sharpness(noise) > frames.sharpness(blurred) > frames.sharpness(flat)
+    # accepts BGR too
+    bgr = cv2.cvtColor(noise, cv2.COLOR_GRAY2BGR)
+    assert frames.sharpness(bgr) > 0
+
+
+def test_sample_frames_picks_sharpest_per_window(tmp_path):
+    import cv2
+
+    w = h = 128
+    fps = 30.0
+    n_frames = 30                       # 1.0 s -> two 0.5 s windows
+    vid_path = tmp_path / "clip.mp4"
+    writer = cv2.VideoWriter(str(vid_path), cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
+    if not writer.isOpened():
+        pytest.skip("no usable VideoWriter codec in this environment")
+    checker = np.indices((h, w)).sum(axis=0) % 2   # fine checkerboard -> sharp
+    sharp = (checker * 255).astype(np.uint8)
+    flat = np.full((h, w), 128, dtype=np.uint8)
+    sharp_frames = {7, 22}
+    for i in range(n_frames):
+        g = sharp if i in sharp_frames else flat
+        writer.write(cv2.cvtColor(g, cv2.COLOR_GRAY2BGR))
+    writer.release()
+
+    out = frames.sample_frames(vid_path, video_id=5, out_dir=tmp_path / "frames")
+    got_frames = sorted(int(Path(p).stem.split("_")[1]) for p in out)
+
+    # ground-truth: decode the written stream and take the per-window argmax ourselves
+    cap = cv2.VideoCapture(str(vid_path))
+    sh, idx = [], 0
+    while True:
+        ok, fr = cap.read()
+        if not ok:
+            break
+        sh.append(frames.sharpness(fr)); idx += 1
+    cap.release()
+    bucket = {}
+    for i, s in enumerate(sh):
+        win = int((i / fps) / 0.5)
+        if win not in bucket or s > bucket[win][0]:
+            bucket[win] = (s, i)
+    expected = sorted(v[1] for v in bucket.values())
+
+    assert got_frames == expected
+    assert all(Path(p).name.startswith("5_") for p in out)
+
+
+# ===========================================================================
+# run_reconstruct — graceful degradation + full export path (no real COLMAP)
+# ===========================================================================
+def _snapshot_poses():
+    return {
+        v.id: [(p.frame_idx, round(p.qw, 9), round(p.tx, 9), p.registered)
+               for p in db.get_poses(v.id)]
+        for v in db.get_videos()
+    }
+
+
+def _write_colmap_model(model_dir: Path, records: list[dict], cameras: dict,
+                        points, colors) -> None:
+    """Write a COLMAP TXT model (images/cameras/points3D) from pose records."""
+    model_dir.mkdir(parents=True, exist_ok=True)
+    with open(model_dir / "images.txt", "w") as f:
+        f.write("# image list\n")
+        for i, r in enumerate(records, start=1):
+            f.write(f"{i} {r['qw']} {r['qx']} {r['qy']} {r['qz']} "
+                    f"{r['tx']} {r['ty']} {r['tz']} {r['camera_id']} {r['name']}\n")
+            f.write("100.0 100.0 -1\n")  # dummy points2D line
+    with open(model_dir / "cameras.txt", "w") as f:
+        f.write("# camera list\n")
+        for cid, cam in cameras.items():
+            f.write(f"{cid} PINHOLE {cam['width']} {cam['height']} "
+                    f"{cam['fx']} {cam['fy']} {cam['cx']} {cam['cy']}\n")
+    with open(model_dir / "points3D.txt", "w") as f:
+        f.write("# point list\n")
+        for j, (p, c) in enumerate(zip(points, colors), start=1):
+            f.write(f"{j} {p[0]} {p[1]} {p[2]} {int(c[0])} {int(c[1])} {int(c[2])} 0.5 1 0\n")
+
+
+def test_reconstruct_no_videos():
+    db.init_engine()
+    db.reset_db()
+    result = sfm.run_reconstruct()
+    assert result["status"] == "no_videos"
+
+
+def test_reconstruct_skips_without_colmap(monkeypatch):
+    synthetic.build(run_ffmpeg=False)
+    monkeypatch.setattr(sfm, "colmap_available", lambda: False)
+    before = _snapshot_poses()
+    result = sfm.run_reconstruct()
+    assert result["status"] == "skipped_no_colmap"
+    assert _snapshot_poses() == before          # poses untouched
+
+
+def _fake_sampling(monkeypatch, frames_per_video):
+    """Make sampling hermetic: placeholder raw files + fake sample_frames output."""
+    for v in db.get_videos():
+        (config.RAW_DIR / v.filename).parent.mkdir(parents=True, exist_ok=True)
+        (config.RAW_DIR / v.filename).write_bytes(b"stub")
+
+    def fake_sample(video_path, video_id, out_dir, **kw):
+        return [Path(f"{video_id}_{f}.jpg") for f in frames_per_video]
+
+    monkeypatch.setattr(sfm.frames, "sample_frames", fake_sample)
+    monkeypatch.setattr(sfm, "colmap_available", lambda: True)
+
+
+def test_reconstruct_failed_no_model(monkeypatch):
+    synthetic.build(run_ffmpeg=False)
+    _fake_sampling(monkeypatch, [0, 30, 60])
+    monkeypatch.setattr(sfm, "run_colmap", lambda frames_dir, workspace: None)
+    before = _snapshot_poses()
+    result = sfm.run_reconstruct()
+    assert result["status"] == "failed_no_model"
+    assert _snapshot_poses() == before          # poses untouched
+
+
+def test_reconstruct_failed_weak(monkeypatch, tmp_path):
+    synthetic.build(run_ffmpeg=False)
+    _fake_sampling(monkeypatch, [0, 30, 60])     # 3 videos x 3 = 9 sampled
+    # model registers only one image of one video -> ~11%, < 60% and < 2 videos
+    track = synthetic.camera_track(0, 20.0, 30, 640, 360)
+    rec = dict(track[0]); rec["camera_id"] = 1; rec["name"] = "1_0.jpg"
+    cams = {1: {"width": 640, "height": 360, **synthetic.intrinsics(640, 360)}}
+    pts, cols = synthetic.generate_point_cloud()
+    model_dir = tmp_path / "weak"
+    _write_colmap_model(model_dir, [rec], cams, pts[:50], cols[:50])
+    monkeypatch.setattr(sfm, "run_colmap", lambda frames_dir, workspace: model_dir)
+    before = _snapshot_poses()
+    result = sfm.run_reconstruct()
+    assert result["status"] == "failed_weak"
+    assert _snapshot_poses() == before          # poses untouched
+
+
+def test_reconstruct_full_export_path(monkeypatch, tmp_path):
+    """The whole export path runs on injected known poses (spec M2 acceptance)."""
+    synthetic.build(run_ffmpeg=False)
+    videos = db.get_videos()
+    id_by_cam = {i: videos[i].id for i in range(len(videos))}   # cam index -> db id
+    sampled_frames = [0, 15, 30, 45, 60]
+    _fake_sampling(monkeypatch, sampled_frames)
+
+    # Build a model: videos for cam0/cam1 fully registered; cam2 registers a subset so
+    # interpolation must fill its gaps. All frame indices come from the synthetic tracks.
+    records, cameras = [], {}
+    reg_plan = {0: sampled_frames, 1: sampled_frames, 2: [0, 30, 60]}
+    for cam_idx, frame_list in reg_plan.items():
+        vid = id_by_cam[cam_idx]
+        cameras[vid] = {"width": 640, "height": 360, **synthetic.intrinsics(640, 360)}
+        track = {p["frame_idx"]: p for p in synthetic.camera_track(cam_idx, 20.0, 30, 640, 360)}
+        for f in frame_list:
+            p = track[f]
+            records.append({**p, "camera_id": vid, "name": f"{vid}_{f}.jpg"})
+    pts, cols = synthetic.generate_point_cloud()
+    model_dir = tmp_path / "good"
+    _write_colmap_model(model_dir, records, cameras, pts, cols)
+    monkeypatch.setattr(sfm, "run_colmap", lambda frames_dir, workspace: model_dir)
+
+    result = sfm.run_reconstruct()
+    assert result["status"] == "ok"
+    assert result["registered"] == 13           # 5 + 5 + 3
+    assert result["videos_in_model"] == 3
+    assert result["interpolated"] == 2           # cam2 frames 15 and 45
+
+    # cam2 poses: 5 total (3 registered + 2 interpolated), sorted, with interp flags
+    cam2_poses = db.get_poses(id_by_cam[2])
+    assert len(cam2_poses) == 5
+    assert sum(1 for p in cam2_poses if not p.registered) == 2
+    assert [p.frame_idx for p in cam2_poses] == [0, 15, 30, 45, 60]
+
+    # normalized point cloud written in the frozen PLY format; camera sphere radius ~ 10
+    assert config.POINTS_PLY.exists()
+    npoints, _ = synthetic.read_ply(config.POINTS_PLY)
+    assert len(npoints) == len(pts)
+    all_centers = []
+    for v in videos:
+        for p in db.get_poses(v.id):
+            R = quat_to_mat([p.qw, p.qx, p.qy, p.qz])
+            all_centers.append(-R.T @ np.array([p.tx, p.ty, p.tz]))
+    max_r = float(np.max(np.linalg.norm(np.array(all_centers), axis=1)))
+    assert abs(max_r - 10.0) < 0.5              # registered cams normalized to radius 10
+
+
+def test_colmap_model_parser_roundtrip(tmp_path):
+    """Writing then parsing a model recovers the poses/intrinsics/points."""
+    track = synthetic.camera_track(0, 20.0, 30, 640, 360)[:3]
+    records = [{**p, "camera_id": 1, "name": f"1_{p['frame_idx']}.jpg"} for p in track]
+    cams = {1: {"width": 640, "height": 360, **synthetic.intrinsics(640, 360)}}
+    pts, cols = synthetic.generate_point_cloud()
+    model_dir = tmp_path / "rt"
+    _write_colmap_model(model_dir, records, cams, pts[:20], cols[:20])
+
+    images = sfm.parse_images_txt(model_dir / "images.txt")
+    cameras = sfm.parse_cameras_txt(model_dir / "cameras.txt")
+    ppoints, pcolors = sfm.parse_points3d_txt(model_dir / "points3D.txt")
+    assert len(images) == 3
+    np.testing.assert_allclose(images[0]["qw"], track[0]["qw"], atol=1e-6)
+    assert cameras[1]["fx"] == pytest.approx(synthetic.intrinsics(640, 360)["fx"])
+    assert len(ppoints) == 20
+
+
+def test_parse_cameras_txt_truncated_params(tmp_path):
+    """A camera line with too few PARAMS gives a clean ValueError, not a raw IndexError."""
+    p = tmp_path / "cameras.txt"
+    p.write_text("# header\n7 PINHOLE 640 360 500.0 500.0 320.0\n")  # PINHOLE needs 4 params
+    with pytest.raises(ValueError, match="too few PARAMS"):
+        sfm.parse_cameras_txt(p)
+
+
+def test_largest_model_dir_picks_by_registered_count(tmp_path):
+    """Binary models are ranked by the images.bin header count, not file size."""
+    sparse = tmp_path / "sparse"
+    # model 0: 10 registered images but a LARGE images.bin (many keypoint observations)
+    (sparse / "0").mkdir(parents=True)
+    (sparse / "0" / "images.bin").write_bytes(struct.pack("<Q", 10) + b"\x00" * 5000)
+    # model 1: 15 registered images but a SMALL images.bin (few keypoints)
+    (sparse / "1").mkdir(parents=True)
+    (sparse / "1" / "images.bin").write_bytes(struct.pack("<Q", 15) + b"\x00" * 100)
+    assert sfm._registered_image_count(sparse / "1") == 15
+    assert sfm._largest_model_dir(sparse).name == "1"   # more images wins despite smaller file
+
+
+def test_interpolate_poses_drops_before_first_registered():
+    """Lower no-extrapolation guard: a sampled frame before the first registered one is dropped."""
+    q = [1.0, 0.0, 0.0, 0.0]
+    registered = [_pose(30, 1.0, q, [3, 0, 0]), _pose(60, 2.0, q, [6, 0, 0])]
+    sampled = [{"frame_idx": f, "t_video_s": f / 30.0} for f in (0, 30, 45, 60)]
+    out = sfm.interpolate_poses(registered, sampled)
+    frames_out = {p["frame_idx"] for p in out}
+    assert 0 not in frames_out                # tv=0 < t_first=1.0 -> dropped, no extrapolation
+    assert frames_out == {30, 45, 60}
+    by = {p["frame_idx"]: p for p in out}
+    assert by[45]["registered"] is False      # interior gap still interpolated
+
+
+def test_reconstruct_subset_leaves_others_untouched(monkeypatch, tmp_path):
+    """When only a subset of videos registers, the rest keep their existing poses (DB safety)."""
+    synthetic.build(run_ffmpeg=False)
+    videos = db.get_videos()
+    id_by_cam = {i: videos[i].id for i in range(len(videos))}
+    sampled_frames = [0, 15, 30, 45, 60]
+    _fake_sampling(monkeypatch, sampled_frames)
+
+    # Register cam0 + cam1 fully (10/15 = 67% >= 60%, 2 videos -> ok); cam2 NOT in the model.
+    records, cameras = [], {}
+    for cam_idx in (0, 1):
+        vid = id_by_cam[cam_idx]
+        cameras[vid] = {"width": 640, "height": 360, **synthetic.intrinsics(640, 360)}
+        track = {p["frame_idx"]: p for p in synthetic.camera_track(cam_idx, 20.0, 30, 640, 360)}
+        for f in sampled_frames:
+            records.append({**track[f], "camera_id": vid, "name": f"{vid}_{f}.jpg"})
+    pts, cols = synthetic.generate_point_cloud()
+    model_dir = tmp_path / "subset"
+    _write_colmap_model(model_dir, records, cameras, pts, cols)
+    monkeypatch.setattr(sfm, "run_colmap", lambda frames_dir, workspace: model_dir)
+
+    cam2_before = [(p.frame_idx, p.qw, p.tx, p.registered) for p in db.get_poses(id_by_cam[2])]
+    result = sfm.run_reconstruct()
+    assert result["status"] == "ok"
+    assert result["videos_in_model"] == 2
+    # cam2 absent from the model: its rows must be byte-identical afterward
+    cam2_after = [(p.frame_idx, p.qw, p.tx, p.registered) for p in db.get_poses(id_by_cam[2])]
+    assert cam2_after == cam2_before
+    assert len(cam2_before) == 41                       # original synthetic poses intact
+    assert len(db.get_poses(id_by_cam[0])) == 5         # cam0 replaced with sampled frames
+
+
+def test_reconstruct_no_frames_when_videos_missing(monkeypatch):
+    """All raw files absent -> videos skipped -> no_frames, poses untouched."""
+    synthetic.build(run_ffmpeg=False)
+    for f in config.RAW_DIR.glob("*.mp4"):              # clear stubs left by other tests
+        f.unlink()
+    monkeypatch.setattr(sfm, "colmap_available", lambda: True)
+    before = _snapshot_poses()
+    result = sfm.run_reconstruct()
+    assert result["status"] == "no_frames"
+    assert _snapshot_poses() == before
+
+
+def test_reconstruct_no_frames_empty_sampling(monkeypatch):
+    """Raw files exist but sampling yields nothing -> no_frames, poses untouched."""
+    synthetic.build(run_ffmpeg=False)
+    _fake_sampling(monkeypatch, [])                     # sample_frames returns []
+    before = _snapshot_poses()
+    result = sfm.run_reconstruct()
+    assert result["status"] == "no_frames"
+    assert _snapshot_poses() == before

backend/tests/test_synthetic.py

@@ -0,0 +1,118 @@
+"""Fixture tests for synthetic.py (foundation-owned; synthetic.py is a frozen contract).
+
+These lock the synthetic fixture's contract: known audio offsets are recoverable, bang
+times stand out for M7 candidate detection, the PLY round-trips, generated poses face the
+stage under the frozen conversion, and ``build`` populates the DB + files as ground truth.
+"""
+
+from __future__ import annotations
+
+import json
+import shutil
+
+import numpy as np
+import pytest
+from scipy.signal import correlate
+
+from festival4d import config, db, synthetic
+from festival4d.geometry import colmap_to_threejs
+
+
+def _estimate_offset_ms(clip_ref, clip_v, sr):
+    """Recover clip_v's offset vs the reference via cross-correlation (see derivation)."""
+    corr = correlate(clip_v, clip_ref, mode="full")
+    lag = int(np.argmax(corr)) - (len(clip_ref) - 1)
+    return -lag / sr * 1000.0
+
+
+def test_audio_slices_recover_known_offsets():
+    sr = config.SYNTH_AUDIO_SR
+    master, _ = synthetic.synth_master_audio(sr)
+    ref = synthetic.slice_for_offset(master, sr, 0.0, config.SYNTH_DURATION_S)
+    for offset_ms in config.SYNTH_OFFSETS_MS:
+        clip = synthetic.slice_for_offset(master, sr, offset_ms, config.SYNTH_DURATION_S)
+        recovered = _estimate_offset_ms(ref, clip, sr)
+        assert abs(recovered - offset_ms) < 1.0, (offset_ms, recovered)
+
+
+def test_bang_times_dominate_energy():
+    """Loud bangs (M7 candidates) must clearly exceed the regular-beat energy floor."""
+    sr = config.SYNTH_AUDIO_SR
+    master, gt = synthetic.synth_master_audio(sr)
+
+    def window_rms(center_s, half=0.05):
+        i0 = int((center_s - synthetic.G_START - half) * sr)
+        i1 = int((center_s - synthetic.G_START + half) * sr)
+        return float(np.sqrt(np.mean(master[i0:i1] ** 2)))
+
+    bang_rms = [window_rms(t) for t in gt["bang_times_global_s"]]
+    # a spread of non-bang beat times for the baseline
+    beat_rms = [window_rms(t) for t in (1.0, 4.5, 7.0, 11.5, 14.0)]
+    assert min(bang_rms) > 3.0 * np.median(beat_rms)
+
+
+def test_ply_roundtrip(tmp_path):
+    pts, cols = synthetic.generate_point_cloud()
+    path = tmp_path / "points.ply"
+    synthetic.write_ply(path, pts, cols)
+    pts2, cols2 = synthetic.read_ply(path)
+    assert pts2.shape == pts.shape and cols2.shape == cols.shape
+    np.testing.assert_allclose(pts2, pts, atol=1e-6)
+    np.testing.assert_array_equal(cols2, cols)
+
+
+def test_generated_poses_face_the_stage():
+    """Each synthetic pose, run through the frozen conversion, must look at the stage."""
+    poses = synthetic.camera_track(1, config.SYNTH_DURATION_S, config.SYNTH_FPS,
+                                   config.SYNTH_VIDEO_W, config.SYNTH_VIDEO_H)
+    target = np.array(synthetic.STAGE_TARGET)
+    for p in poses:
+        q = [p["qw"], p["qx"], p["qy"], p["qz"]]
+        t = [p["tx"], p["ty"], p["tz"]]
+        position, R = colmap_to_threejs(q, t)
+        look = R @ np.array([0.0, 0.0, -1.0])          # Three.js camera looks down -z
+        expected = target - position
+        expected /= np.linalg.norm(expected)
+        np.testing.assert_allclose(look, expected, atol=1e-9)
+
+
+def test_build_no_ffmpeg_populates_db_and_files(tmp_path):
+    summary = synthetic.build(base_dir=tmp_path, duration_s=5.0, run_ffmpeg=False)
+
+    videos = db.get_videos()
+    assert len(videos) == 3
+    by_name = {v.filename: v for v in videos}
+    assert by_name["cam0.mp4"].offset_ms == 0.0
+    assert by_name["cam1.mp4"].offset_ms == 1370.0
+    assert by_name["cam2.mp4"].offset_ms == -842.0
+    for v in videos:
+        assert len(db.get_poses(v.id)) > 0
+        assert db.get_poses(v.id)[0].registered is True
+
+    assert db.has_poses()
+    assert len(db.get_anchors()) == 4
+    assert len(db.get_events()) == len(synthetic.SEED_EVENTS)
+
+    ply = tmp_path / "work" / "points.ply"
+    assert ply.exists()
+    pts, _ = synthetic.read_ply(ply)
+    assert len(pts) == summary["point_count"] > 1000
+
+    gt = json.loads((tmp_path / "work" / "ground_truth.json").read_text())
+    assert [vv["offset_ms"] for vv in gt["videos"]] == [0.0, 1370.0, -842.0]
+    assert gt["reference_filename"] == "cam0.mp4"
+
+
+@pytest.mark.skipif(shutil.which("ffmpeg") is None or shutil.which("ffprobe") is None,
+                    reason="ffmpeg/ffprobe required for the full fixture build")
+def test_build_full_ffmpeg_renders_seekable_videos(tmp_path):
+    synthetic.build(base_dir=tmp_path, duration_s=4.0, run_ffmpeg=True)
+    raw = tmp_path / "raw"
+    for name in ("cam0.mp4", "cam1.mp4", "cam2.mp4"):
+        path = raw / name
+        assert path.exists() and path.stat().st_size > 10_000
+    videos = db.get_videos()
+    assert len(videos) == 3
+    for v in videos:
+        assert 3.5 < v.duration_s < 4.6         # ~4 s clips
+        assert v.width == 640 and v.height == 360

frontend/index.html

@@ -0,0 +1,210 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="UTF-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <title>Festival 4D</title>
+    <style>
+      :root {
+        color-scheme: dark;
+        --bg: #0b0d12;
+        --panel: #141824;
+        --panel-2: #1c2233;
+        --border: #232838;
+        --text: #e6e8ee;
+        --dim: #8a92a6;
+        --ok: #59d499;
+        --bad: #ff6b6b;
+        --accent: #59d499;
+      }
+      * { box-sizing: border-box; }
+      html, body { height: 100%; }
+      body {
+        margin: 0;
+        font: 14px/1.45 -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
+        background: var(--bg);
+        color: var(--text);
+        overflow: hidden;
+      }
+      #app { display: flex; flex-direction: column; height: 100vh; }
+
+      /* Top bar */
+      #topbar {
+        display: flex; align-items: baseline; gap: 0.6rem;
+        padding: 0.5rem 0.9rem; border-bottom: 1px solid var(--border); flex: 0 0 auto;
+      }
+      #topbar h1 { margin: 0; font-size: 1.05rem; font-weight: 700; letter-spacing: 0.2px; }
+      #topbar .sub { color: var(--dim); font-size: 0.8rem; }
+
+      /* Stage: 3D on the left, video grid on the right */
+      #stage { flex: 1 1 auto; display: flex; min-height: 0; }
+      #scene-pane { flex: 1 1 55%; position: relative; min-width: 0; background: #0a0c11; }
+      #scene3d { position: absolute; inset: 0; width: 100%; height: 100%; display: block; }
+      #scene-controls {
+        position: absolute; top: 0.6rem; left: 0.6rem; display: flex; gap: 0.4rem; z-index: 5;
+      }
+      #scene-hint {
+        position: absolute; bottom: 0.5rem; left: 0.6rem; color: var(--dim);
+        font-size: 0.72rem; z-index: 5; pointer-events: none;
+        background: rgba(10,12,17,0.6); padding: 0.2rem 0.45rem; border-radius: 5px;
+      }
+      #grid-pane {
+        flex: 1 1 45%; min-width: 0; border-left: 1px solid var(--border);
+        overflow: auto; background: #0d1017;
+      }
+      #video-grid {
+        display: grid; gap: 6px; padding: 6px;
+        grid-template-columns: repeat(auto-fit, minmax(240px, 1fr));
+      }
+
+      .cell {
+        position: relative; aspect-ratio: 16 / 9; background: #000;
+        border: 1px solid var(--border); border-radius: 8px; overflow: hidden;
+        transition: opacity 0.15s ease, border-color 0.15s ease;
+      }
+      .cell.out { opacity: 0.32; }
+      .cell.audio-on { border-color: var(--accent); }
+      .cell.followed { box-shadow: inset 0 0 0 2px var(--accent); }
+      .cell-video { position: absolute; inset: 0; width: 100%; height: 100%; object-fit: contain; }
+      .cell-overlay { position: absolute; inset: 0; pointer-events: none; }
+      .cell-bar {
+        position: absolute; left: 0; right: 0; bottom: 0; display: flex; align-items: center;
+        gap: 0.35rem; padding: 0.25rem 0.4rem; font-size: 0.72rem;
+        background: linear-gradient(transparent, rgba(6,8,14,0.85));
+      }
+      .cell-label { font-weight: 600; }
+      .cell-status { color: var(--dim); margin-left: 0.1rem; }
+      .cell-status.ok { color: var(--ok); }
+      .cell-status.bad { color: var(--bad); }
+      .cell-btn {
+        margin-left: auto; background: rgba(255,255,255,0.08); color: var(--text);
+        border: 1px solid var(--border); border-radius: 5px; cursor: pointer;
+        width: 22px; height: 20px; font-size: 0.72rem; line-height: 1; padding: 0;
+      }
+      .cell-btn + .cell-btn { margin-left: 0.25rem; }
+      .cell-btn:hover { background: rgba(255,255,255,0.16); }
+      .cell.audio-on .audio-btn { background: var(--accent); color: #071018; border-color: var(--accent); }
+
+      /* Bottom bar: transport + timeline */
+      #bottombar {
+        flex: 0 0 auto; border-top: 1px solid var(--border); padding: 0.5rem 0.9rem 0.7rem;
+        background: var(--panel);
+      }
+      #controls { display: flex; align-items: center; gap: 0.5rem; margin-bottom: 0.5rem; }
+      .btn {
+        background: var(--panel-2); color: var(--text); border: 1px solid var(--border);
+        border-radius: 6px; cursor: pointer; padding: 0.3rem 0.6rem; font-size: 0.85rem;
+      }
+      .btn:hover { background: #262d42; }
+      .btn.active { background: var(--accent); color: #071018; border-color: var(--accent); }
+      #btn-play { min-width: 2.4rem; font-size: 1rem; }
+      #time-display { font-variant-numeric: tabular-nums; color: var(--dim); margin-left: 0.2rem; }
+      select.btn { padding-right: 0.4rem; }
+      .spacer { flex: 1 1 auto; }
+      .ctl-label { color: var(--dim); font-size: 0.78rem; }
+
+      /* Timeline */
+      #timeline { position: relative; }
+      .tl-track {
+        position: relative; height: 26px; background: var(--panel-2);
+        border: 1px solid var(--border); border-radius: 6px; cursor: pointer; overflow: visible;
+        touch-action: none;
+      }
+      .tl-fill { position: absolute; left: 0; top: 0; bottom: 0; width: 0;
+        background: rgba(89,212,153,0.14); border-radius: 6px 0 0 6px; }
+      .tl-markers { position: absolute; inset: 0; }
+      .tl-marker {
+        position: absolute; top: 0; bottom: 0; width: 3px; margin-left: -1.5px;
+        border-radius: 2px; cursor: pointer; box-shadow: 0 0 0 1px rgba(0,0,0,0.35);
+      }
+      .tl-marker:hover { width: 5px; margin-left: -2.5px; }
+      .tl-playhead {
+        position: absolute; top: -3px; bottom: -3px; width: 2px; margin-left: -1px;
+        background: #fff; box-shadow: 0 0 6px rgba(255,255,255,0.6); pointer-events: none;
+      }
+      .tl-tooltip {
+        position: absolute; bottom: 34px; transform: translateX(-50%);
+        background: #0f1420; border: 1px solid var(--border); border-radius: 7px;
+        padding: 0.4rem 0.55rem; font-size: 0.78rem; max-width: 260px; z-index: 20;
+        pointer-events: none; box-shadow: 0 6px 20px rgba(0,0,0,0.5);
+      }
+      .tl-tooltip strong { text-transform: capitalize; }
+      .tl-conf { color: var(--dim); font-size: 0.72rem; }
+      .tl-legend {
+        display: flex; flex-wrap: wrap; gap: 0.5rem 0.9rem; margin-top: 0.45rem;
+        font-size: 0.72rem; color: var(--dim);
+      }
+      .tl-legend-item { display: inline-flex; align-items: center; gap: 0.3rem; text-transform: capitalize; }
+      .tl-legend-item i { width: 10px; height: 10px; border-radius: 3px; display: inline-block; }
+
+      /* Dev sync overlay */
+      #dev-overlay {
+        position: fixed; top: 0.6rem; right: 0.6rem; z-index: 30;
+        background: rgba(15,20,32,0.9); border: 1px solid var(--border); border-radius: 8px;
+        padding: 0.45rem 0.6rem; font-size: 0.72rem; min-width: 132px;
+        backdrop-filter: blur(4px);
+      }
+      #dev-overlay .hd { color: var(--dim); font-weight: 600; margin-bottom: 0.3rem; }
+      .dev-row { display: flex; justify-content: space-between; gap: 0.8rem; font-variant-numeric: tabular-nums; }
+      .ok { color: var(--ok); }
+      .bad { color: var(--bad); }
+      .dim { color: var(--dim); }
+
+      /* Loading / error */
+      #loading {
+        position: fixed; inset: 0; display: flex; align-items: center; justify-content: center;
+        background: var(--bg); z-index: 100; color: var(--dim);
+      }
+      #loading .err { text-align: center; color: var(--text); }
+      code { background: var(--panel-2); padding: 0.1rem 0.35rem; border-radius: 4px; }
+    </style>
+  </head>
+  <body>
+    <div id="app">
+      <div id="topbar">
+        <h1>Festival 4D</h1>
+        <span class="sub">synchronized multi-cam replay · 3D viewer · x-ray overlays</span>
+      </div>
+
+      <div id="stage">
+        <div id="scene-pane">
+          <canvas id="scene3d"></canvas>
+          <div id="scene-controls">
+            <button class="btn active" id="btn-roam" title="Free roam (Esc)">Free roam</button>
+          </div>
+          <div id="scene-hint">drag to orbit · click a camera or press 1–9 to snap · Esc to detach</div>
+        </div>
+        <div id="grid-pane">
+          <div id="video-grid"></div>
+        </div>
+      </div>
+
+      <div id="bottombar">
+        <div id="controls">
+          <button class="btn" id="btn-back" title="Back 1s (←)">⏪</button>
+          <button class="btn" id="btn-play" title="Play/Pause (Space)">▶</button>
+          <button class="btn" id="btn-fwd" title="Forward 1s (→)">⏩</button>
+          <span id="time-display">0:00.0 / 0:00.0</span>
+          <span class="spacer"></span>
+          <span class="ctl-label">speed</span>
+          <select class="btn" id="rate-select">
+            <option value="0.25">0.25×</option>
+            <option value="0.5">0.5×</option>
+            <option value="1" selected>1×</option>
+            <option value="2">2×</option>
+          </select>
+        </div>
+        <div id="timeline"></div>
+      </div>
+    </div>
+
+    <div id="dev-overlay">
+      <div class="hd">sync error</div>
+      <div id="dev-rows"></div>
+    </div>
+
+    <div id="loading">Loading project…</div>
+
+    <script type="module" src="/src/main.js"></script>
+  </body>
+</html>

frontend/package.json

@@ -0,0 +1,17 @@
+{
+  "name": "festival4d-frontend",
+  "private": true,
+  "version": "0.1.0",
+  "type": "module",
+  "scripts": {
+    "dev": "vite",
+    "build": "vite build",
+    "preview": "vite preview"
+  },
+  "dependencies": {
+    "three": "^0.170.0"
+  },
+  "devDependencies": {
+    "vite": "^6.0.0"
+  }
+}

frontend/src/annotate.js

@@ -0,0 +1,6 @@
+// bbox drawing + submission → POST /api/annotations (spec M8).
+// STUB — this is phase-3 (integration) work; lane C leaves it as a clean seam. Drag a
+// rectangle on a video's overlay canvas → normalized bbox → POST /api/annotations; the
+// resolved anchor then appears live in overlays + the 3D scene.
+
+export {};

frontend/src/camPath.js

@@ -0,0 +1,6 @@
+// Keyframed camera paths (spec M9).
+// STUB — this is phase-3 (integration) work; lane C leaves it as a clean seam. "Add
+// keyframe" captures the current free-roam pose at the current t_global; playback
+// interpolates position (Catmull-Rom) + orientation (slerp); export/import path as JSON.
+
+export {};

frontend/src/lib/pose.js

@@ -0,0 +1,143 @@
+// COLMAP world->camera pose  ->  Three.js camera pose (spec M5).
+//
+// FROZEN CONTRACT. This is the JavaScript mirror of `backend/festival4d/geometry.py`
+// (`colmap_to_threejs`, `quat_to_mat`). The math and the POSE_TEST_VECTORS below are
+// duplicated verbatim on the Python side (`backend/tests/test_geometry.py`) so both agree.
+// Lanes B and C CONSUME this; do not reimplement the conversion elsewhere. Changing it
+// requires a change request and a synchronized edit to both files + both vector sets.
+//
+// Conventions:
+//   - Quaternions are COLMAP order [w, x, y, z] (scalar first), unit norm.
+//   - Pose (q, t) is world->camera: x_cam = R(q) * x_world + t.
+//   - COLMAP camera axes: +x right, +y down, +z forward.
+//   - Three.js cameras look down -z with +y up; hence the diag(1, -1, -1) flip.
+
+const SQRT1_2 = 0.7071067811865476;
+
+// Camera-axis flip: COLMAP camera-local (x right, y down, z forward) -> Three.js (x right,
+// y up, z backward). Same _FLIP_YZ as the Python side.
+const FLIP_YZ = [
+  [1, 0, 0],
+  [0, -1, 0],
+  [0, 0, -1],
+];
+
+/**
+ * Convert a unit quaternion [w, x, y, z] to a 3x3 rotation matrix (row-major nested array).
+ * Hamilton convention, right-handed, active rotation (COLMAP world->camera when q is a
+ * COLMAP pose quaternion).
+ * @param {number[]} q - [w, x, y, z]
+ * @returns {number[][]} 3x3 rotation matrix
+ */
+export function quatToMat(q) {
+  const [w, x, y, z] = q;
+  const n = w * w + x * x + y * y + z * z;
+  if (n < 1e-12) throw new Error("quaternion has near-zero norm");
+  const s = 2.0 / n;
+  const wx = s * w * x, wy = s * w * y, wz = s * w * z;
+  const xx = s * x * x, xy = s * x * y, xz = s * x * z;
+  const yy = s * y * y, yz = s * y * z, zz = s * z * z;
+  return [
+    [1.0 - (yy + zz), xy - wz, xz + wy],
+    [xy + wz, 1.0 - (xx + zz), yz - wx],
+    [xz - wy, yz + wx, 1.0 - (xx + yy)],
+  ];
+}
+
+function transpose3(m) {
+  return [
+    [m[0][0], m[1][0], m[2][0]],
+    [m[0][1], m[1][1], m[2][1]],
+    [m[0][2], m[1][2], m[2][2]],
+  ];
+}
+
+function matMul3(a, b) {
+  const out = [[0, 0, 0], [0, 0, 0], [0, 0, 0]];
+  for (let i = 0; i < 3; i++)
+    for (let j = 0; j < 3; j++)
+      out[i][j] = a[i][0] * b[0][j] + a[i][1] * b[1][j] + a[i][2] * b[2][j];
+  return out;
+}
+
+function matVec3(m, v) {
+  return [
+    m[0][0] * v[0] + m[0][1] * v[1] + m[0][2] * v[2],
+    m[1][0] * v[0] + m[1][1] * v[1] + m[1][2] * v[2],
+    m[2][0] * v[0] + m[2][1] * v[1] + m[2][2] * v[2],
+  ];
+}
+
+/**
+ * Convert a COLMAP world->camera pose to a Three.js camera pose (spec M5). FROZEN.
+ * @param {number[]} q - COLMAP world->camera quaternion [w, x, y, z]
+ * @param {number[]} t - COLMAP world->camera translation [tx, ty, tz]
+ * @returns {{position: number[], rotation: number[][], matrixWorld: number[]}}
+ *   position       - camera center in world coords, C = -R^T t (assign to camera.position)
+ *   rotation       - Three.js camera world rotation R_three = R^T * diag(1,-1,-1),
+ *                    3x3 row-major (feed to camera.setRotationFromMatrix via a Matrix4)
+ *   matrixWorld    - column-major 16-array [R_three | position], ready for
+ *                    THREE.Matrix4().fromArray(...) when driving the camera by matrix.
+ */
+export function colmapToThreejs(q, t) {
+  const R = quatToMat(q); // world -> cam
+  const Rc2w = transpose3(R); // cam -> world
+  const position = matVec3(Rc2w, [-t[0], -t[1], -t[2]]); // C = -R^T t
+  const rotation = matMul3(Rc2w, FLIP_YZ); // R_three
+
+  // Column-major Matrix4 with rotation in the upper-left 3x3 and translation = position.
+  const m = rotation;
+  const p = position;
+  const matrixWorld = [
+    m[0][0], m[1][0], m[2][0], 0,
+    m[0][1], m[1][1], m[2][1], 0,
+    m[0][2], m[1][2], m[2][2], 0,
+    p[0], p[1], p[2], 1,
+  ];
+  return { position, rotation, matrixWorld };
+}
+
+// --- FROZEN test vectors (must equal POSE_TEST_VECTORS in backend/tests/test_geometry.py).
+export const POSE_TEST_VECTORS = [
+  {
+    name: "identity",
+    q: [1.0, 0.0, 0.0, 0.0],
+    t: [0.0, 0.0, -10.0],
+    position: [0.0, 0.0, 10.0],
+    rotation: [[1.0, 0.0, 0.0], [0.0, -1.0, 0.0], [0.0, 0.0, -1.0]],
+  },
+  {
+    name: "yaw90",
+    q: [SQRT1_2, 0.0, SQRT1_2, 0.0],
+    t: [0.0, 0.0, 10.0],
+    position: [10.0, 0.0, 0.0],
+    rotation: [[0.0, 0.0, 1.0], [0.0, -1.0, 0.0], [1.0, 0.0, 0.0]],
+  },
+  {
+    name: "lookat",
+    q: [0.0, 1.0, 0.0, 0.0],
+    t: [0.0, 0.0, 8.0],
+    position: [0.0, 0.0, 8.0],
+    rotation: [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]],
+  },
+];
+
+/**
+ * Self-check the conversion against POSE_TEST_VECTORS. Returns true on success, throws on
+ * mismatch. Called by main.js so the frozen contract is verified live in the browser.
+ * @param {number} [atol=1e-9]
+ */
+export function selfTest(atol = 1e-9) {
+  for (const vec of POSE_TEST_VECTORS) {
+    const { position, rotation } = colmapToThreejs(vec.q, vec.t);
+    for (let i = 0; i < 3; i++) {
+      if (Math.abs(position[i] - vec.position[i]) > atol)
+        throw new Error(`pose.js selfTest[${vec.name}]: position mismatch at ${i}`);
+      for (let j = 0; j < 3; j++) {
+        if (Math.abs(rotation[i][j] - vec.rotation[i][j]) > atol)
+          throw new Error(`pose.js selfTest[${vec.name}]: rotation mismatch at ${i},${j}`);
+      }
+    }
+  }
+  return true;
+}

frontend/src/lib/poseTrack.js

@@ -0,0 +1,57 @@
+// Pose lookup + interpolation over a video's `camera_poses` track (from GET /api/videos/{id}/poses).
+//
+// Poses arrive keyed by frame_idx / t_video_s (every ~0.5 s on the synthetic fixture, all
+// registered). To render a smooth frustum / follow-cam at an arbitrary t_video we interpolate
+// BETWEEN stored poses in COLMAP space — slerp the world->camera quaternion, lerp the
+// translation — then hand the interpolated (q, t) to the FROZEN colmapToThreejs() helper.
+// We interpolate the CONTRACT INPUTS; we never reimplement the COLMAP->Three.js conversion.
+
+import * as THREE from "three";
+
+const _qa = new THREE.Quaternion();
+const _qb = new THREE.Quaternion();
+
+/**
+ * Interpolated pose at local video time `tv` (seconds). Returns a COLMAP-convention pose
+ * `{ q:[w,x,y,z], t:[x,y,z], intrinsics, registered, t_video_s }` ready for colmapToThreejs,
+ * or null if the track is empty. Clamps to the endpoints (no extrapolation).
+ * @param {Array} poses  ascending-by-t_video_s pose list
+ * @param {number} tv     local video time (seconds)
+ */
+export function poseAt(poses, tv) {
+  if (!poses || poses.length === 0) return null;
+  const n = poses.length;
+  if (tv <= poses[0].t_video_s) return poses[0];
+  if (tv >= poses[n - 1].t_video_s) return poses[n - 1];
+
+  // binary search for the bracketing pair [lo, hi] with poses[lo].t <= tv < poses[hi].t
+  let lo = 0;
+  let hi = n - 1;
+  while (hi - lo > 1) {
+    const mid = (lo + hi) >> 1;
+    if (poses[mid].t_video_s <= tv) lo = mid;
+    else hi = mid;
+  }
+  const a = poses[lo];
+  const b = poses[hi];
+  const span = b.t_video_s - a.t_video_s;
+  const f = span > 1e-9 ? (tv - a.t_video_s) / span : 0;
+
+  // COLMAP quaternion order is [w, x, y, z]; THREE.Quaternion is (x, y, z, w).
+  _qa.set(a.q[1], a.q[2], a.q[3], a.q[0]);
+  _qb.set(b.q[1], b.q[2], b.q[3], b.q[0]);
+  _qa.slerp(_qb, f);
+  const q = [_qa.w, _qa.x, _qa.y, _qa.z];
+  const t = [
+    a.t[0] + (b.t[0] - a.t[0]) * f,
+    a.t[1] + (b.t[1] - a.t[1]) * f,
+    a.t[2] + (b.t[2] - a.t[2]) * f,
+  ];
+  return {
+    q,
+    t,
+    intrinsics: a.intrinsics,
+    registered: a.registered && b.registered,
+    t_video_s: tv,
+  };
+}

frontend/src/lib/timebase.js

@@ -0,0 +1,18 @@
+// Timebase convention (FROZEN CONTRACT) — the JS mirror of backend/festival4d/config.py
+// (`t_video_from_global` / `t_global_from_video`). Keep these identical to the Python side.
+//
+//   t_global is the master timeline in seconds. For a video v:
+//     t_video = (t_global - offset_ms/1000) * (1 + drift_ppm * 1e-6)
+//   The reference video has offset_ms == 0 (and drift_ppm == 0). A positive offset_ms means
+//   the video started recording later than the master zero, so at a given t_global its local
+//   playhead is earlier. Never re-derive this algebra inline — call these helpers everywhere.
+
+/** Master-timeline seconds -> a video's local playhead seconds (FROZEN). */
+export function tVideoFromGlobal(tGlobal, offsetMs, driftPpm = 0) {
+  return (tGlobal - offsetMs / 1000) * (1 + driftPpm * 1e-6);
+}
+
+/** Inverse of tVideoFromGlobal (FROZEN). */
+export function tGlobalFromVideo(tVideo, offsetMs, driftPpm = 0) {
+  return tVideo / (1 + driftPpm * 1e-6) + offsetMs / 1000;
+}

frontend/src/main.js

@@ -0,0 +1,205 @@
+// Festival 4D viewer — composition root (spec M4/M5/M6 + timeline markers).
+//
+// Boots by loading the frozen synthetic API (manifest, poses, anchors, events), builds the
+// video grid + 3D scene + timeline, wires transport controls + keyboard shortcuts, and runs a
+// single master animation loop that: advances the master clock & corrects every video
+// (transport), redraws each overlay for its displayed frame, updates the 3D frusta / follow-cam,
+// moves the timeline playhead, and refreshes the dev sync overlay. The master clock lives in
+// transport.js and is derived from performance.now() — never from a <video> element.
+
+import { state, on, API_BASE, referenceVideoId } from "./state.js";
+import { transport } from "./transport.js";
+import { createVideoGrid } from "./videoGrid.js";
+import { drawOverlay, projectWorldToVideoPx } from "./overlays.js";
+import { poseAt } from "./lib/poseTrack.js";
+import { scene3d, videoColor } from "./scene3d.js";
+import { timeline } from "./timeline.js";
+
+async function getJSON(path) {
+  const resp = await fetch(API_BASE + path);
+  if (!resp.ok) throw new Error(`${path} -> HTTP ${resp.status}`);
+  return resp.json();
+}
+
+function fmtTime(s) {
+  s = Math.max(0, s);
+  const m = Math.floor(s / 60);
+  const sec = s - m * 60;
+  return `${m}:${sec.toFixed(1).padStart(4, "0")}`;
+}
+
+function el(id) {
+  return document.getElementById(id);
+}
+
+async function boot() {
+  const loading = el("loading");
+  try {
+    const manifest = await getJSON("/api/manifest");
+    state.manifest = manifest;
+    state.videos = manifest.videos;
+    state.tGlobalMax = manifest.t_global_max;
+    state.hasPoses = manifest.has_poses;
+    for (const v of state.videos) state.enabled[v.id] = true;
+    state.audioSourceId = referenceVideoId();
+
+    const [anchors, events] = await Promise.all([
+      getJSON("/api/anchors"),
+      getJSON("/api/events"),
+    ]);
+    state.anchors = anchors;
+    state.events = events;
+
+    if (state.hasPoses) {
+      const lists = await Promise.all(
+        state.videos.map((v) => getJSON(`/api/videos/${v.id}/poses`))
+      );
+      state.videos.forEach((v, i) => (state.poses[v.id] = lists[i]));
+    }
+
+    buildUI();
+    startLoop();
+    loading.style.display = "none";
+
+    // Dev-only hook so the transport/scene can be driven in tests (e.g. a setTimeout pump when
+    // a headless pane throttles rAF). Never present in a production build.
+    if (import.meta.env?.DEV) {
+      window.__f4d = {
+        state, transport, scene3d, timeline, cells: () => cells,
+        poseAt, projectWorldToVideoPx,
+      };
+    }
+  } catch (err) {
+    console.error("[festival4d] boot failed", err);
+    loading.innerHTML = `<div class="err">Failed to load the project.<br><code>${err.message}</code>` +
+      `<br><span class="dim">Is the backend running? <code>python -m festival4d serve</code></span></div>`;
+  }
+}
+
+let cells = [];
+
+function buildUI() {
+  // Video grid
+  const grid = el("video-grid");
+  cells = createVideoGrid(grid, state.videos, API_BASE, transport);
+  transport.register(cells);
+
+  // Per-video snap buttons -> scene3d (grid module leaves them unwired).
+  cells.forEach((cell, i) => {
+    cell.el.style.setProperty("--accent", videoColor(i));
+    cell.snapBtn.addEventListener("click", (e) => {
+      e.stopPropagation();
+      scene3d.snapTo(cell.id);
+    });
+  });
+
+  // 3D scene
+  scene3d.init(el("scene3d"));
+
+  // Timeline
+  timeline.init(el("timeline"), transport);
+
+  wireTransportControls();
+  wireKeyboard();
+  positionVideosWhenReady();
+
+  // Reflect follow/roam state on the free-roam button.
+  on("follow", (id) => {
+    el("btn-roam").classList.toggle("active", id == null);
+  });
+}
+
+function wireTransportControls() {
+  el("btn-play").addEventListener("click", () => transport.toggle());
+  el("btn-back").addEventListener("click", () => transport.seekBy(-1));
+  el("btn-fwd").addEventListener("click", () => transport.seekBy(1));
+  el("rate-select").addEventListener("change", (e) =>
+    transport.setRate(parseFloat(e.target.value))
+  );
+  el("btn-roam").addEventListener("click", () => scene3d.freeRoam());
+}
+
+function wireKeyboard() {
+  window.addEventListener("keydown", (e) => {
+    if (e.target && /^(INPUT|SELECT|TEXTAREA)$/.test(e.target.tagName)) return;
+    if (e.code === "Space") {
+      e.preventDefault();
+      transport.toggle();
+    } else if (e.code === "ArrowLeft") {
+      e.preventDefault();
+      transport.seekBy(e.shiftKey ? -5 : -1);
+    } else if (e.code === "ArrowRight") {
+      e.preventDefault();
+      transport.seekBy(e.shiftKey ? 5 : 1);
+    } else if (e.code === "Escape" || e.code === "Digit0") {
+      scene3d.freeRoam();
+    } else if (/^Digit[1-9]$/.test(e.code)) {
+      const n = parseInt(e.code.slice(5), 10) - 1;
+      const v = state.videos[n];
+      if (v) scene3d.snapTo(v.id);
+    }
+  });
+}
+
+function positionVideosWhenReady() {
+  let remaining = cells.length;
+  if (remaining === 0) return;
+  const done = () => {
+    if (--remaining === 0) transport.seek(0); // park every video on its t=0 frame
+  };
+  for (const { video } of cells) {
+    if (video.readyState >= 1) done();
+    else video.addEventListener("loadedmetadata", done, { once: true });
+  }
+}
+
+function startLoop() {
+  const frame = () => {
+    transport.tick();
+    for (const cell of cells) drawOverlay(cell);
+    scene3d.update();
+    timeline.update(state.tGlobal);
+    updateHud();
+    requestAnimationFrame(frame);
+  };
+  requestAnimationFrame(frame);
+}
+
+function updateHud() {
+  el("btn-play").textContent = state.playing ? "⏸" : "▶";
+  el("time-display").textContent = `${fmtTime(state.tGlobal)} / ${fmtTime(state.tGlobalMax)}`;
+
+  // Dev sync overlay + per-cell status.
+  let devRows = "";
+  for (const cell of cells) {
+    const { id, meta } = cell;
+    const err = state.syncErrorMs[id];
+    const inRange = state.inRange[id];
+    const enabled = state.enabled[id] !== false;
+    let label, cls;
+    if (!enabled) {
+      label = "disabled";
+      cls = "dim";
+    } else if (!inRange) {
+      label = "out of range";
+      cls = "dim";
+    } else if (err == null) {
+      label = "—";
+      cls = "dim";
+    } else {
+      label = `${err >= 0 ? "+" : ""}${err.toFixed(0)} ms`;
+      cls = Math.abs(err) < 50 ? "ok" : "bad";
+    }
+    devRows += `<div class="dev-row"><span>${meta.filename}</span><span class="${cls}">${label}</span></div>`;
+
+    // Cell overlay state (dim out-of-range / disabled, mark audio + follow).
+    cell.el.classList.toggle("out", !inRange || !enabled);
+    cell.el.classList.toggle("audio-on", state.audioSourceId === id);
+    cell.el.classList.toggle("followed", state.followCameraId === id);
+    cell.statusEl.textContent = enabled ? (inRange ? label : "waiting…") : "off";
+    cell.statusEl.className = `cell-status ${cls}`;
+  }
+  el("dev-rows").innerHTML = devRows;
+}
+
+boot();

frontend/src/overlays.js

@@ -0,0 +1,137 @@
+// 3D -> 2D anchor projection onto each video's overlay canvas — the "x-ray" HUD (spec M6).
+//
+// For each visible video we build a Three.js PerspectiveCamera at that video's CURRENT pose
+// (from the FROZEN colmapToThreejs helper, matrixWorld path) with an fov derived from the
+// stored intrinsics, project every anchor, drop anchors behind the camera (camera-space z >= 0),
+// convert NDC -> native video pixels -> canvas pixels through the letterbox content rect, and
+// draw a dot + halo + label. Anchors are drawn regardless of real-world occlusion — that IS the
+// x-ray feature. We consume pose.js; we never reimplement the COLMAP->Three.js conversion.
+
+import * as THREE from "three";
+import { colmapToThreejs } from "./lib/pose.js";
+import { poseAt } from "./lib/poseTrack.js";
+import { contentRect } from "./videoGrid.js";
+import { state } from "./state.js";
+
+// Reused across all videos/frames (every synthetic video shares intrinsics + resolution, and
+// we fully overwrite the matrices each call, so a single scratch camera is safe and cheap).
+const _cam = new THREE.PerspectiveCamera();
+_cam.matrixAutoUpdate = false;
+const _m4 = new THREE.Matrix4();
+const _v = new THREE.Vector3();
+
+function configureProjector(pose, W, H) {
+  // Centered-principal-point pinhole -> PerspectiveCamera (spec M5/M6 prototype assumption;
+  // the synthetic fixture has cx=W/2, cy=H/2 exactly).
+  const fy = pose.intrinsics.fy;
+  _cam.fov = (2 * Math.atan(H / (2 * fy)) * 180) / Math.PI;
+  _cam.aspect = W / H;
+  _cam.near = 0.01;
+  _cam.far = 2000;
+  _cam.updateProjectionMatrix();
+
+  const { matrixWorld } = colmapToThreejs(pose.q, pose.t);
+  _m4.fromArray(matrixWorld);
+  _cam.matrixWorld.copy(_m4);
+  _cam.matrixWorldInverse.copy(_m4).invert();
+}
+
+/** Project a world point to native video pixels, or null if behind the camera. */
+function projectToVideoPx(x, y, z, W, H) {
+  _v.set(x, y, z);
+  // Behind-camera test in camera space: Three.js cameras look down -z, so a visible point has
+  // camera-space z < 0. Do this BEFORE .project() (which mutates _v into NDC).
+  _v.applyMatrix4(_cam.matrixWorldInverse);
+  if (_v.z >= 0) return null;
+  _v.applyMatrix4(_cam.projectionMatrix); // perspective divide happens inside .project(); do it manually
+  // _v is now clip space already divided (applyMatrix4 on a Vector3 divides by w). NDC in [-1,1].
+  const u = (_v.x * 0.5 + 0.5) * W;
+  const vpx = (1 - (_v.y * 0.5 + 0.5)) * H; // flip Y: NDC +y is up, pixels grow downward
+  return { u, v: vpx };
+}
+
+/**
+ * Project a world point to native video pixels through the FROZEN pose helper for a given
+ * COLMAP pose (as returned by poseAt) and video resolution. Returns {u, v} or null (behind).
+ * Single source of truth for overlay projection — used by drawOverlay and by tests.
+ */
+export function projectWorldToVideoPx(pose, x, y, z, W, H) {
+  configureProjector(pose, W, H);
+  return projectToVideoPx(x, y, z, W, H);
+}
+
+function drawMarker(ctx, x, y, anchor) {
+  const color = anchor.color || "#59d499";
+  // halo
+  ctx.beginPath();
+  ctx.arc(x, y, 5.5, 0, Math.PI * 2);
+  ctx.fillStyle = "rgba(255,255,255,0.85)";
+  ctx.fill();
+  // dot
+  ctx.beginPath();
+  ctx.arc(x, y, 4, 0, Math.PI * 2);
+  ctx.fillStyle = color;
+  ctx.fill();
+  // label
+  const text = anchor.label || "";
+  if (text) {
+    ctx.font = "600 11px -apple-system, system-ui, sans-serif";
+    ctx.textBaseline = "middle";
+    const tx = x + 9;
+    const ty = y;
+    const w = ctx.measureText(text).width;
+    ctx.fillStyle = "rgba(6,8,14,0.72)";
+    ctx.fillRect(tx - 3, ty - 8, w + 6, 16);
+    ctx.fillStyle = "#e6e8ee";
+    ctx.fillText(text, tx, ty + 0.5);
+  }
+}
+
+/** Redraw one cell's overlay for the frame currently displayed by its <video>. */
+export function drawOverlay(cell) {
+  const { video, canvas } = cell;
+  const ctx = canvas.getContext("2d");
+  const dpr = window.devicePixelRatio || 1;
+  const elW = video.clientWidth;
+  const elH = video.clientHeight;
+  if (elW === 0 || elH === 0) return;
+
+  // Resolution-match the canvas backing store to the element (dpr-aware).
+  const bw = Math.round(elW * dpr);
+  const bh = Math.round(elH * dpr);
+  if (canvas.width !== bw || canvas.height !== bh) {
+    canvas.width = bw;
+    canvas.height = bh;
+    canvas.style.width = elW + "px";
+    canvas.style.height = elH + "px";
+  }
+  ctx.setTransform(dpr, 0, 0, dpr, 0, 0);
+  ctx.clearRect(0, 0, elW, elH);
+
+  if (!state.hasPoses || !state.inRange[cell.id]) return;
+  if (video.videoWidth === 0) return;
+  const poses = state.poses[cell.id];
+  if (!poses || poses.length === 0 || state.anchors.length === 0) return;
+
+  // Use the frame the <video> is ACTUALLY showing (currentTime), not the master target, so the
+  // overlay tracks the displayed picture even mid-correction.
+  const pose = poseAt(poses, video.currentTime);
+  if (!pose) return;
+
+  const W = video.videoWidth;
+  const H = video.videoHeight;
+  configureProjector(pose, W, H);
+  const rect = contentRect(video);
+
+  for (const a of state.anchors) {
+    const p = projectToVideoPx(a.x, a.y, a.z, W, H);
+    if (!p) continue;
+    const cx = rect.ox + (p.u / W) * rect.cw;
+    const cy = rect.oy + (p.v / H) * rect.ch;
+    // Skip anchors that fall well outside the visible content rect.
+    const m = 24;
+    if (cx < rect.ox - m || cx > rect.ox + rect.cw + m) continue;
+    if (cy < rect.oy - m || cy > rect.oy + rect.ch + m) continue;
+    drawMarker(ctx, cx, cy, a);
+  }
+}

frontend/src/scene3d.js

@@ -0,0 +1,300 @@
+// Three.js scene (spec M5): point cloud (/api/pointcloud via PLYLoader), each video's camera
+// path as a line + current-pose frustum wireframe (colored per video), OrbitControls free roam,
+// and snap-to-camera. ALL COLMAP->Three.js conversion goes through the FROZEN lib/pose.js
+// helper (colmapToThreejs) — the diag(1,-1,-1) math is never reimplemented here.
+
+import * as THREE from "three";
+import { OrbitControls } from "three/addons/controls/OrbitControls.js";
+import { PLYLoader } from "three/addons/loaders/PLYLoader.js";
+import { colmapToThreejs } from "./lib/pose.js";
+import { poseAt } from "./lib/poseTrack.js";
+import { tVideoFromGlobal } from "./lib/timebase.js";
+import { state, emit } from "./state.js";
+
+// Per-video colors (match the frustum, path line, and grid accents).
+export const VIDEO_COLORS = ["#ff5d73", "#4dd0ff", "#c9a2ff", "#ffcf5d", "#7dffb0", "#ff9d5d"];
+export function videoColor(index) {
+  return VIDEO_COLORS[index % VIDEO_COLORS.length];
+}
+
+const FRUSTUM_DEPTH = 1.2;
+const DEFAULT_FOV = 50;
+const STAGE_TARGET = new THREE.Vector3(0, 0.5, 0);
+
+const _m4 = new THREE.Matrix4();
+const _pos = new THREE.Vector3();
+const _quat = new THREE.Quaternion();
+const _scale = new THREE.Vector3();
+
+export class Scene3D {
+  constructor() {
+    this.videoIndex = {}; // id -> index (for stable colors)
+    this.rigs = {}; // id -> { frustum, marker, pick, path }
+    this._tween = null;
+    this._raf = null;
+  }
+
+  init(canvas) {
+    this.canvas = canvas;
+    const parent = canvas.parentElement;
+    const w = parent.clientWidth || 1;
+    const h = parent.clientHeight || 1;
+
+    this.renderer = new THREE.WebGLRenderer({ canvas, antialias: true, alpha: false });
+    this.renderer.setPixelRatio(Math.min(window.devicePixelRatio || 1, 2));
+    this.renderer.setSize(w, h, false);
+    this.renderer.setClearColor(0x0a0c11, 1);
+
+    this.scene = new THREE.Scene();
+    this.camera = new THREE.PerspectiveCamera(DEFAULT_FOV, w / h, 0.05, 3000);
+    this.camera.position.set(12, 9, 15);
+    this.camera.lookAt(STAGE_TARGET);
+
+    this.controls = new OrbitControls(this.camera, this.renderer.domElement);
+    this.controls.enableDamping = true;
+    this.controls.dampingFactor = 0.08;
+    this.controls.target.copy(STAGE_TARGET);
+    this.controls.update();
+
+    // Lights (points ignore lighting, but frusta/markers use MeshBasic so this is mostly cosmetic).
+    this.scene.add(new THREE.AmbientLight(0xffffff, 0.9));
+    const grid = new THREE.GridHelper(40, 40, 0x22384a, 0x161c26);
+    grid.position.y = 0;
+    this.scene.add(grid);
+    const axes = new THREE.AxesHelper(1.5);
+    this.scene.add(axes);
+
+    this._buildRigs();
+    this._loadPointCloud();
+    this._wirePicking();
+    this._wireResize(parent);
+  }
+
+  _buildRigs() {
+    state.videos.forEach((v, i) => {
+      this.videoIndex[v.id] = i;
+      const color = new THREE.Color(videoColor(i));
+      const group = new THREE.Group();
+
+      // Frustum wireframe (built in Three.js camera-local space: looks down -z, +y up).
+      const poses = state.poses[v.id] || [];
+      const intr = poses[0]?.intrinsics;
+      const W = v.width;
+      const H = v.height;
+      const hx = intr ? (FRUSTUM_DEPTH * W) / (2 * intr.fx) : FRUSTUM_DEPTH * 0.6;
+      const hy = intr ? (FRUSTUM_DEPTH * H) / (2 * intr.fy) : FRUSTUM_DEPTH * 0.34;
+      const d = FRUSTUM_DEPTH;
+      const c = [
+        [-hx, -hy, -d],
+        [hx, -hy, -d],
+        [hx, hy, -d],
+        [-hx, hy, -d],
+      ];
+      const apex = [0, 0, 0];
+      const segs = [
+        apex, c[0], apex, c[1], apex, c[2], apex, c[3],
+        c[0], c[1], c[1], c[2], c[2], c[3], c[3], c[0],
+      ];
+      const fg = new THREE.BufferGeometry();
+      fg.setAttribute("position", new THREE.Float32BufferAttribute(segs.flat(), 3));
+      const frustum = new THREE.LineSegments(
+        fg,
+        new THREE.LineBasicMaterial({ color })
+      );
+      frustum.matrixAutoUpdate = true;
+      group.add(frustum);
+
+      // Camera-center marker (visible) + an invisible larger sphere for click picking.
+      const marker = new THREE.Mesh(
+        new THREE.SphereGeometry(0.12, 12, 12),
+        new THREE.MeshBasicMaterial({ color })
+      );
+      group.add(marker);
+      this.scene.add(group);
+
+      const pick = new THREE.Mesh(
+        new THREE.SphereGeometry(0.55, 8, 8),
+        new THREE.MeshBasicMaterial({ visible: false })
+      );
+      pick.userData.videoId = v.id;
+      group.add(pick);
+
+      // Camera path line (one vertex per stored pose center).
+      let path = null;
+      if (poses.length > 1) {
+        const pts = [];
+        for (const p of poses) {
+          const { position } = colmapToThreejs(p.q, p.t);
+          pts.push(position[0], position[1], position[2]);
+        }
+        const pg = new THREE.BufferGeometry();
+        pg.setAttribute("position", new THREE.Float32BufferAttribute(pts, 3));
+        path = new THREE.Line(
+          pg,
+          new THREE.LineBasicMaterial({ color, transparent: true, opacity: 0.5 })
+        );
+        this.scene.add(path);
+      }
+
+      this.rigs[v.id] = { group, frustum, marker, pick, path };
+    });
+  }
+
+  _loadPointCloud() {
+    const loader = new PLYLoader();
+    loader.load(
+      state.apiBase + "/api/pointcloud",
+      (geometry) => {
+        geometry.computeBoundingBox();
+        const hasColor = !!geometry.getAttribute("color");
+        const material = new THREE.PointsMaterial({
+          size: 0.05,
+          sizeAttenuation: true,
+          vertexColors: hasColor,
+          color: hasColor ? 0xffffff : 0x88ccff,
+        });
+        this.points = new THREE.Points(geometry, material);
+        this.scene.add(this.points);
+        emit("pointcloud-loaded", geometry.getAttribute("position")?.count ?? 0);
+      },
+      undefined,
+      (err) => {
+        console.warn("[scene3d] point cloud load failed", err);
+      }
+    );
+  }
+
+  // Live pose (COLMAP convention) for a video at the current master time, or null.
+  _livePose(videoId) {
+    const poses = state.poses[videoId];
+    if (!poses || poses.length === 0) return null;
+    const meta = state.videos.find((v) => v.id === videoId);
+    const tv = tVideoFromGlobal(state.tGlobal, meta.offset_ms || 0, meta.drift_ppm || 0);
+    return poseAt(poses, tv);
+  }
+
+  /** Snap the viewer camera to a video's pose and follow it while playing. */
+  snapTo(videoId) {
+    const pose = this._livePose(videoId);
+    if (!pose) return;
+    state.followCameraId = videoId;
+    this.controls.enabled = false;
+    const targetFov = (2 * Math.atan(state.videos.find((v) => v.id === videoId).height /
+      (2 * pose.intrinsics.fy)) * 180) / Math.PI;
+    this._tween = {
+      start: performance.now(),
+      dur: 600,
+      fromPos: this.camera.position.clone(),
+      fromQuat: this.camera.quaternion.clone(),
+      fromFov: this.camera.fov,
+      toFov: targetFov,
+    };
+    emit("follow", videoId);
+  }
+
+  /** Detach to free roam (OrbitControls). */
+  freeRoam() {
+    if (state.followCameraId == null) return;
+    state.followCameraId = null;
+    this._tween = null;
+    this.camera.fov = DEFAULT_FOV;
+    this.camera.updateProjectionMatrix();
+    // Re-anchor OrbitControls to the stage, orbiting from the current position.
+    this.controls.target.copy(STAGE_TARGET);
+    this.controls.enabled = true;
+    this.controls.update();
+    emit("follow", null);
+  }
+
+  _applyPoseToCamera(pose, targetFov) {
+    const { matrixWorld } = colmapToThreejs(pose.q, pose.t);
+    _m4.fromArray(matrixWorld);
+    _m4.decompose(_pos, _quat, _scale);
+    const t = this._tween;
+    if (t) {
+      const k = Math.min(1, (performance.now() - t.start) / t.dur);
+      const e = k * k * (3 - 2 * k); // smoothstep
+      this.camera.position.lerpVectors(t.fromPos, _pos, e);
+      this.camera.quaternion.copy(t.fromQuat).slerp(_quat, e);
+      this.camera.fov = t.fromFov + (t.toFov - t.fromFov) * e;
+      this.camera.updateProjectionMatrix();
+      if (k >= 1) this._tween = null;
+    } else {
+      this.camera.position.copy(_pos);
+      this.camera.quaternion.copy(_quat);
+      if (Math.abs(this.camera.fov - targetFov) > 1e-3) {
+        this.camera.fov = targetFov;
+        this.camera.updateProjectionMatrix();
+      }
+    }
+  }
+
+  /** Per-frame update: move frusta to current poses, drive follow-cam, render. */
+  update() {
+    for (const v of state.videos) {
+      const rig = this.rigs[v.id];
+      if (!rig) continue;
+      const pose = this._livePose(v.id);
+      const on = pose && state.enabled[v.id] !== false;
+      rig.group.visible = !!on;
+      if (rig.path) rig.path.visible = state.enabled[v.id] !== false;
+      if (!on) continue;
+      const { matrixWorld } = colmapToThreejs(pose.q, pose.t);
+      _m4.fromArray(matrixWorld);
+      _m4.decompose(rig.group.position, rig.group.quaternion, rig.group.scale);
+    }
+
+    if (state.followCameraId != null) {
+      const pose = this._livePose(state.followCameraId);
+      if (pose) {
+        const meta = state.videos.find((v) => v.id === state.followCameraId);
+        const targetFov =
+          (2 * Math.atan(meta.height / (2 * pose.intrinsics.fy)) * 180) / Math.PI;
+        this._applyPoseToCamera(pose, targetFov);
+      }
+    } else {
+      this.controls.update();
+    }
+
+    this.renderer.render(this.scene, this.camera);
+  }
+
+  _wirePicking() {
+    const el = this.renderer.domElement;
+    const ray = new THREE.Raycaster();
+    const ndc = new THREE.Vector2();
+    let downX = 0;
+    let downY = 0;
+    el.addEventListener("pointerdown", (e) => {
+      downX = e.clientX;
+      downY = e.clientY;
+    });
+    el.addEventListener("pointerup", (e) => {
+      if (Math.hypot(e.clientX - downX, e.clientY - downY) > 5) return; // was a drag, not a click
+      const rect = el.getBoundingClientRect();
+      ndc.x = ((e.clientX - rect.left) / rect.width) * 2 - 1;
+      ndc.y = -((e.clientY - rect.top) / rect.height) * 2 + 1;
+      ray.setFromCamera(ndc, this.camera);
+      const picks = Object.values(this.rigs)
+        .map((r) => r.pick)
+        .filter((m) => m.parent && m.parent.visible);
+      const hits = ray.intersectObjects(picks, false);
+      if (hits.length) this.snapTo(hits[0].object.userData.videoId);
+    });
+  }
+
+  _wireResize(parent) {
+    const ro = new ResizeObserver(() => {
+      const w = parent.clientWidth || 1;
+      const h = parent.clientHeight || 1;
+      this.renderer.setSize(w, h, false);
+      // Aspect always follows the 3D canvas (never the video) so nothing stretches; snap-to-
+      // camera only overrides the vertical fov + pose, matching the video's vertical framing.
+      this.camera.aspect = w / h;
+      this.camera.updateProjectionMatrix();
+    });
+    ro.observe(parent);
+  }
+}
+
+export const scene3d = new Scene3D();

frontend/src/state.js

@@ -0,0 +1,53 @@
+// Global store: manifest, playhead (t_global), playing, rate, selected camera, audio source,
+// plus the loaded poses / anchors / events and per-video live playback status. Single source
+// of truth: the transport writes `tGlobal` every animation frame; every module reads it.
+// Discrete changes (play/pause/seek/selection) go through the tiny pub/sub below.
+
+export const API_BASE = "http://localhost:8000";
+
+export const state = {
+  apiBase: API_BASE,
+
+  // loaded from the API (see main.js boot)
+  manifest: null,
+  videos: [], // [{id, filename, url, duration_s, fps, width, height, offset_ms, drift_ppm}]
+  tGlobalMax: 0,
+  hasPoses: false,
+  poses: {}, // id -> [{frame_idx, t_video_s, q:[w,x,y,z], t:[x,y,z], intrinsics, registered}]
+  anchors: [], // [{id, label, x, y, z, color}]
+  events: [], // [{id, t_global_s, duration_s, event_type, confidence, description, source}]
+
+  // transport / playback
+  tGlobal: 0,
+  playing: false,
+  rate: 1,
+  audioSourceId: null, // video id whose audio is unmuted (exactly one)
+  enabled: {}, // id -> bool (per-video enable; disabled videos pause + dim + drop from sync)
+  inRange: {}, // id -> bool (t_video within [0, duration] and enabled)
+  syncErrorMs: {}, // id -> number | null (currentTime - target, ms; null when out of range)
+
+  // 3D viewer
+  followCameraId: null, // video id the viewer camera is snapped to, or null for free roam
+};
+
+// --- tiny pub/sub for discrete events -------------------------------------------------------
+const listeners = new Map();
+
+/** Subscribe to an event; returns an unsubscribe fn. */
+export function on(evt, fn) {
+  if (!listeners.has(evt)) listeners.set(evt, new Set());
+  listeners.get(evt).add(fn);
+  return () => listeners.get(evt)?.delete(fn);
+}
+
+/** Emit an event to all subscribers. */
+export function emit(evt, payload) {
+  const set = listeners.get(evt);
+  if (set) for (const fn of [...set]) fn(payload);
+}
+
+/** The reference video (offset 0) if present, else the first video. Used as default audio source. */
+export function referenceVideoId() {
+  const ref = state.videos.find((v) => (v.offset_ms ?? 0) === 0);
+  return ref ? ref.id : state.videos[0]?.id ?? null;
+}

frontend/src/timeline.js

@@ -0,0 +1,149 @@
+// Timeline scrubber + event markers (spec M4 transport + M7 markers).
+//
+// Renders a draggable scrubber over [0, tGlobalMax] with the playhead, plus GET /api/events as
+// colored ticks (color by event_type, legend in the corner). Hover a marker -> description
+// tooltip; click a marker -> jump the playhead there. Event data is already seeded
+// synthetically; classification QUALITY is lane D's concern, not this renderer's.
+
+import { state } from "./state.js";
+
+export const EVENT_COLORS = {
+  bass_drop: "#ff3b6b",
+  pyro: "#ff8c1a",
+  confetti: "#ffd23b",
+  crowd_wave: "#3bc9ff",
+  artist_moment: "#c77dff",
+  light_show: "#59d499",
+  quiet_moment: "#8a92a6",
+  candidate: "#c0c0c0",
+  other: "#9aa3b2",
+};
+export function eventColor(type) {
+  return EVENT_COLORS[type] || EVENT_COLORS.other;
+}
+
+function fmtTime(s) {
+  s = Math.max(0, s);
+  const m = Math.floor(s / 60);
+  const sec = s - m * 60;
+  return `${m}:${sec.toFixed(1).padStart(4, "0")}`;
+}
+
+export class Timeline {
+  constructor() {
+    this.max = 1;
+  }
+
+  init(root, transport) {
+    this.transport = transport;
+    this.max = state.tGlobalMax || 1;
+    root.innerHTML = "";
+
+    const track = document.createElement("div");
+    track.className = "tl-track";
+
+    const fill = document.createElement("div");
+    fill.className = "tl-fill";
+
+    const markers = document.createElement("div");
+    markers.className = "tl-markers";
+
+    const playhead = document.createElement("div");
+    playhead.className = "tl-playhead";
+
+    track.append(fill, markers, playhead);
+
+    const tip = document.createElement("div");
+    tip.className = "tl-tooltip";
+    tip.style.display = "none";
+
+    root.append(track, tip);
+    this.track = track;
+    this.fill = fill;
+    this.playhead = playhead;
+    this.tip = tip;
+
+    this._buildMarkers(markers, tip);
+    this._buildLegend(root);
+    this._wireScrub(track);
+  }
+
+  _buildMarkers(container, tip) {
+    for (const ev of state.events) {
+      const m = document.createElement("div");
+      m.className = "tl-marker";
+      m.style.left = `${(ev.t_global_s / this.max) * 100}%`;
+      m.style.background = eventColor(ev.event_type);
+      m.addEventListener("click", (e) => {
+        e.stopPropagation();
+        this.transport.seek(ev.t_global_s);
+      });
+      const show = (e) => {
+        tip.innerHTML =
+          `<strong>${ev.event_type.replace(/_/g, " ")}</strong> · ${fmtTime(ev.t_global_s)}` +
+          `<br>${ev.description || ""}` +
+          (ev.confidence != null
+            ? `<br><span class="tl-conf">confidence ${(ev.confidence * 100).toFixed(0)}% · ${ev.source}</span>`
+            : "");
+        tip.style.display = "block";
+        const rect = this.track.getBoundingClientRect();
+        const x = (ev.t_global_s / this.max) * rect.width;
+        tip.style.left = `${Math.max(4, Math.min(rect.width - 4, x))}px`;
+      };
+      m.addEventListener("mouseenter", show);
+      m.addEventListener("mousemove", show);
+      m.addEventListener("mouseleave", () => (tip.style.display = "none"));
+      container.append(m);
+    }
+  }
+
+  _buildLegend(root) {
+    const types = [...new Set(state.events.map((e) => e.event_type))];
+    if (types.length === 0) return;
+    const legend = document.createElement("div");
+    legend.className = "tl-legend";
+    for (const t of types) {
+      const item = document.createElement("span");
+      item.className = "tl-legend-item";
+      item.innerHTML = `<i style="background:${eventColor(t)}"></i>${t.replace(/_/g, " ")}`;
+      legend.append(item);
+    }
+    root.append(legend);
+  }
+
+  _wireScrub(track) {
+    let dragging = false;
+    const seekTo = (clientX) => {
+      const rect = track.getBoundingClientRect();
+      const f = Math.max(0, Math.min(1, (clientX - rect.left) / rect.width));
+      this.transport.seek(f * this.max);
+    };
+    track.addEventListener("pointerdown", (e) => {
+      dragging = true;
+      track.setPointerCapture(e.pointerId);
+      seekTo(e.clientX);
+    });
+    track.addEventListener("pointermove", (e) => {
+      if (dragging) seekTo(e.clientX);
+    });
+    const end = (e) => {
+      dragging = false;
+      try {
+        track.releasePointerCapture(e.pointerId);
+      } catch {
+        /* ignore */
+      }
+    };
+    track.addEventListener("pointerup", end);
+    track.addEventListener("pointercancel", end);
+  }
+
+  /** Move the playhead + fill to the current master time. */
+  update(tGlobal) {
+    const f = this.max > 0 ? Math.max(0, Math.min(1, tGlobal / this.max)) : 0;
+    this.playhead.style.left = `${f * 100}%`;
+    this.fill.style.width = `${f * 100}%`;
+  }
+}
+
+export const timeline = new Timeline();

frontend/src/transport.js

@@ -0,0 +1,274 @@
+// Master clock + per-video synchronization (spec M4 — the heart of the app).
+//
+// The master clock is derived from performance.now(); we NEVER trust a <video> element as the
+// clock (HTML5 video is not frame-accurate — pitfall #2). Correction is continuous:
+//   |err| > 150 ms          -> hard seek (fastSeek where available)
+//   20 ms < |err| <= 150 ms -> nudge playbackRate within master rate x[0.95, 1.05]
+//   |err| <= 20 ms          -> locked: playbackRate = master rate
+//
+// While PLAYING we measure error per presented frame via requestVideoFrameCallback: `mediaTime`
+// is the exact timestamp of the frame on screen and `expectedDisplayTime` is when it shows, so
+// err = mediaTime - target(clock @ expectedDisplayTime) is the true on-screen sync error (far
+// better than sampling the frame-quantized currentTime in rAF). Browsers without rVFC fall back
+// to a currentTime measurement inside the rAF tick(). While PAUSED / seeking, videos are parked
+// on their target frame via currentTime. Videos whose target t_video falls outside [0, duration]
+// pause and are marked out-of-range (the grid dims them). Exactly one video's audio is unmuted.
+
+import { state, emit } from "./state.js";
+import { tVideoFromGlobal } from "./lib/timebase.js";
+
+const HARD_SEEK_S = 0.15; // > this error => hard seek
+const NUDGE_MIN_S = 0.02; // > this (and <= HARD_SEEK) => trim playbackRate
+const RATE_TRIM = 0.05; // max ±5% playbackRate trim while nudging
+
+export class Transport {
+  constructor() {
+    this.cells = []; // [{ id, meta, video, ... }] from videoGrid
+    this._t0 = 0; // master-timeline anchor (seconds)
+    this._wall0 = performance.now(); // wall-clock anchor (ms) matching _t0
+    this._visWired = false;
+  }
+
+  /** Register the video cells the transport drives and start their frame loops. */
+  register(cells) {
+    this.cells = cells;
+    this._applyAudio();
+    for (const cell of this.cells) this._startFrameLoop(cell);
+    if (!this._visWired) {
+      // A throttled/backgrounded tab stalls rAF while decoders free-run; re-lock on return.
+      document.addEventListener("visibilitychange", () => {
+        if (!document.hidden) this._resync();
+      });
+      this._visWired = true;
+    }
+  }
+
+  /** Master-timeline seconds at a given wall-clock (ms) timestamp, clamped to [0, tGlobalMax]. */
+  _clockAt(wallMs) {
+    if (!state.playing) return this._t0;
+    const t = this._t0 + ((wallMs - this._wall0) / 1000) * state.rate;
+    if (t <= 0) return 0;
+    if (t >= state.tGlobalMax) return state.tGlobalMax;
+    return t;
+  }
+
+  computeTGlobal() {
+    return this._clockAt(performance.now());
+  }
+
+  _targetFor(meta, tGlobal) {
+    return tVideoFromGlobal(tGlobal, meta.offset_ms || 0, meta.drift_ppm || 0);
+  }
+
+  play() {
+    if (state.playing) return;
+    let t = state.tGlobal;
+    if (t >= state.tGlobalMax - 1e-3) t = 0; // restart from the top if parked at the end
+    this._t0 = t;
+    this._wall0 = performance.now();
+    state.tGlobal = t;
+    state.playing = true;
+    this._applyAudio(); // first play() is a user gesture -> browser lets us unmute
+    emit("play");
+  }
+
+  pause() {
+    if (!state.playing) return;
+    state.tGlobal = this.computeTGlobal();
+    this._t0 = state.tGlobal;
+    state.playing = false;
+    for (const { video } of this.cells) this._safePause(video);
+    emit("pause");
+  }
+
+  toggle() {
+    if (state.playing) this.pause();
+    else this.play();
+  }
+
+  /** Jump the master timeline; parks every in-range video on its target frame immediately. */
+  seek(t) {
+    t = Math.max(0, Math.min(state.tGlobalMax, t));
+    this._t0 = t;
+    this._wall0 = performance.now();
+    state.tGlobal = t;
+    for (const cell of this.cells) {
+      const target = this._targetFor(cell.meta, t);
+      const inRange =
+        state.enabled[cell.id] !== false && target >= 0 && target <= cell.meta.duration_s;
+      state.inRange[cell.id] = inRange;
+      if (inRange) {
+        this._hardSeek(cell.video, target);
+        cell.video.playbackRate = state.rate;
+        state.syncErrorMs[cell.id] = 0;
+      } else {
+        state.syncErrorMs[cell.id] = null;
+        this._safePause(cell.video);
+      }
+    }
+    emit("seek", t);
+  }
+
+  /** Nudge the playhead by a delta (keyboard ±1 s). */
+  seekBy(dt) {
+    this.seek((state.playing ? this.computeTGlobal() : this._t0) + dt);
+  }
+
+  /** Set playback speed, rebasing the clock so t_global stays continuous. */
+  setRate(r) {
+    state.tGlobal = this.computeTGlobal();
+    this._t0 = state.tGlobal;
+    this._wall0 = performance.now();
+    state.rate = r;
+    for (const { video } of this.cells) video.playbackRate = r; // re-trimmed by correction
+    emit("rate", r);
+  }
+
+  /** Choose which video's audio is unmuted (all others muted — avoids phasing chaos). */
+  setAudioSource(id) {
+    state.audioSourceId = id;
+    this._applyAudio();
+    emit("audiosource", id);
+  }
+
+  _applyAudio() {
+    for (const { id, video } of this.cells) {
+      const isSrc = id === state.audioSourceId;
+      video.muted = !isSrc;
+      if (isSrc) video.volume = 1;
+    }
+  }
+
+  // Per-presented-frame correction via requestVideoFrameCallback (accurate). Self-sustaining:
+  // each callback re-registers the next. No-ops while paused / out of range; rAF tick() handles
+  // the fallback path for browsers without rVFC.
+  _startFrameLoop(cell) {
+    const { video } = cell;
+    if (typeof video.requestVideoFrameCallback !== "function") {
+      cell._useRvfc = false;
+      return;
+    }
+    cell._useRvfc = true;
+    cell._lastRvfc = -Infinity;
+    const cb = (now, meta) => {
+      cell._rvfcId = video.requestVideoFrameCallback(cb);
+      cell._lastRvfc = performance.now();
+      if (!state.playing || !state.inRange[cell.id]) return;
+      const displayWall = meta.expectedDisplayTime ?? now;
+      const target = this._targetFor(cell.meta, this._clockAt(displayWall));
+      // mediaTime is the exact PTS of the on-screen frame — no quantization compensation needed.
+      this._correct(cell, meta.mediaTime, target, true);
+    };
+    cell._rvfcId = video.requestVideoFrameCallback(cb);
+  }
+
+  _correct(cell, measured, target, allowSeek) {
+    const { id, video } = cell;
+    const err = measured - target; // + => video is ahead of where it should be
+    state.syncErrorMs[id] = err * 1000;
+    if (allowSeek && Math.abs(err) > HARD_SEEK_S) {
+      this._hardSeek(video, target);
+      video.playbackRate = state.rate;
+    } else if (Math.abs(err) > NUDGE_MIN_S) {
+      // Ahead (err>0) => slow down; behind (err<0) => speed up. Proportional, clamped ±5%.
+      const trim = Math.max(-1, Math.min(1, err / HARD_SEEK_S)) * RATE_TRIM;
+      video.playbackRate = state.rate * (1 - trim);
+    } else {
+      video.playbackRate = state.rate;
+    }
+  }
+
+  _hardSeek(video, target) {
+    if (typeof video.fastSeek === "function") {
+      try {
+        video.fastSeek(target);
+        return;
+      } catch {
+        /* fall through */
+      }
+    }
+    video.currentTime = target;
+  }
+
+  /** Re-lock every in-range video to the current master time (after a tab-visibility stall). */
+  _resync() {
+    const t = state.tGlobal;
+    for (const cell of this.cells) {
+      const target = this._targetFor(cell.meta, t);
+      if (state.enabled[cell.id] !== false && target >= 0 && target <= cell.meta.duration_s) {
+        this._hardSeek(cell.video, target);
+        cell.video.playbackRate = state.rate;
+      }
+    }
+  }
+
+  /** Drive one animation-frame tick: advance the clock and manage each video. */
+  tick() {
+    const t = this.computeTGlobal();
+    state.tGlobal = t;
+    if (state.playing && t >= state.tGlobalMax) {
+      this.pause();
+      return;
+    }
+    for (const cell of this.cells) this._manage(cell, t);
+  }
+
+  // Play/pause + range management every rAF. For non-rVFC browsers this also runs correction
+  // from the (frame-quantized) currentTime; with rVFC, correction happens in the frame loop.
+  _manage(cell, tGlobal) {
+    const { id, meta, video } = cell;
+    const target = this._targetFor(meta, tGlobal);
+    const enabled = state.enabled[id] !== false;
+    const inRange = enabled && target >= 0 && target <= meta.duration_s;
+    state.inRange[id] = inRange;
+
+    if (!inRange) {
+      state.syncErrorMs[id] = null;
+      this._safePause(video);
+      return;
+    }
+
+    if (state.playing) {
+      this._ensurePlaying(video);
+      // rVFC drives correction when it's firing. Fall back to a currentTime measurement when
+      // rVFC is unavailable OR stale (>100 ms) — e.g. a tab where rVFC is throttled but rAF runs.
+      const rvfcStale = !cell._useRvfc || performance.now() - (cell._lastRvfc ?? -Infinity) > 100;
+      if (rvfcStale) this._correct(cell, this._measureFallback(cell), target, true);
+    } else {
+      this._safePause(video);
+      state.syncErrorMs[id] = (this._measureFallback(cell) - target) * 1000;
+    }
+  }
+
+  // Estimate a playing video's true position from currentTime. currentTime is quantized to the
+  // displayed frame (whose PTS is <= the true position), so add half a frame to de-bias it. The
+  // rVFC path uses exact mediaTime instead and needs no such correction.
+  _measureFallback(cell) {
+    const fps = cell.meta.fps || 30;
+    const halfFrame = state.playing ? (0.5 / fps) * state.rate : 0;
+    return cell.video.currentTime + halfFrame;
+  }
+
+  _ensurePlaying(video) {
+    if (video.paused && !video._playPending && video.readyState >= 2) {
+      video._playPending = true;
+      video.play().then(
+        () => (video._playPending = false),
+        () => (video._playPending = false)
+      );
+    }
+  }
+
+  _safePause(video) {
+    if (video._playPending) return; // let the pending play() settle; next tick re-evaluates
+    if (!video.paused) {
+      try {
+        video.pause();
+      } catch {
+        /* ignore */
+      }
+    }
+  }
+}
+
+export const transport = new Transport();

frontend/src/videoGrid.js

@@ -0,0 +1,114 @@
+// <video> grid + per-video transparent overlay canvases (spec M4 / M6).
+//
+// Each cell stacks a <video> (object-fit: contain) under a transparent <canvas> that exactly
+// covers the element box. Because object-fit letterboxes, the video's CONTENT rect is smaller
+// than the element (pitfall #4) — contentRect() computes it so overlays.js can map projected
+// video-pixel coordinates to the right place on the canvas.
+
+import { state, emit } from "./state.js";
+
+/**
+ * Build the grid inside `container`. Returns cell descriptors:
+ *   { id, meta, el, video, canvas, statusEl, audioBtn, snapBtn }
+ */
+export function createVideoGrid(container, videos, apiBase, transport) {
+  container.innerHTML = "";
+  const cells = [];
+
+  for (const meta of videos) {
+    const el = document.createElement("div");
+    el.className = "cell";
+    el.dataset.id = String(meta.id);
+
+    const video = document.createElement("video");
+    video.className = "cell-video";
+    video.src = apiBase + meta.url;
+    video.muted = true;
+    video.loop = false;
+    video.playsInline = true;
+    video.setAttribute("playsinline", "");
+    video.setAttribute("webkit-playsinline", "");
+    video.preload = "auto";
+    // No crossOrigin: we never read video pixels (overlays are a separate canvas), so we avoid
+    // making playback depend on CORS headers for the /media mount.
+
+    const canvas = document.createElement("canvas");
+    canvas.className = "cell-overlay";
+
+    const bar = document.createElement("div");
+    bar.className = "cell-bar";
+
+    const label = document.createElement("span");
+    label.className = "cell-label";
+    label.textContent = meta.filename;
+
+    const audioBtn = document.createElement("button");
+    audioBtn.className = "cell-btn audio-btn";
+    audioBtn.title = "Use this video's audio";
+    audioBtn.textContent = "🔈";
+    audioBtn.addEventListener("click", (e) => {
+      e.stopPropagation();
+      transport.setAudioSource(meta.id);
+    });
+
+    const snapBtn = document.createElement("button");
+    snapBtn.className = "cell-btn snap-btn";
+    snapBtn.title = "Snap 3D view to this camera";
+    snapBtn.textContent = "⛶";
+    // wired in main.js (needs scene3d)
+
+    const enableBtn = document.createElement("button");
+    enableBtn.className = "cell-btn enable-btn";
+    enableBtn.title = "Enable / disable this video";
+    enableBtn.textContent = "◉";
+    enableBtn.addEventListener("click", (e) => {
+      e.stopPropagation();
+      const now = state.enabled[meta.id] !== false;
+      state.enabled[meta.id] = !now;
+      emit("enabled", { id: meta.id, enabled: !now });
+    });
+
+    const status = document.createElement("span");
+    status.className = "cell-status";
+
+    bar.append(label, status, enableBtn, snapBtn, audioBtn);
+    el.append(video, canvas, bar);
+    container.append(el);
+
+    // Clicking the video frame selects its audio (natural "listen to this one" gesture).
+    video.addEventListener("click", () => transport.setAudioSource(meta.id));
+
+    cells.push({ id: meta.id, meta, el, video, canvas, statusEl: status, audioBtn, snapBtn });
+  }
+
+  return cells;
+}
+
+/**
+ * The video's displayed content rectangle inside its element, in CSS pixels, accounting for
+ * object-fit: contain letterboxing. Maps native video pixels -> element pixels via:
+ *   x = ox + (u / vw) * cw ,  y = oy + (v / vh) * ch
+ */
+export function contentRect(video) {
+  const elW = video.clientWidth;
+  const elH = video.clientHeight;
+  const vw = video.videoWidth || 1;
+  const vh = video.videoHeight || 1;
+  const elAspect = elW / elH;
+  const vidAspect = vw / vh;
+  let cw, ch, ox, oy;
+  if (vidAspect > elAspect) {
+    // wider than the box -> pillarbox: full width, bars top/bottom
+    cw = elW;
+    ch = elW / vidAspect;
+    ox = 0;
+    oy = (elH - ch) / 2;
+  } else {
+    // taller than the box -> letterbox: full height, bars left/right
+    ch = elH;
+    cw = elH * vidAspect;
+    oy = 0;
+    ox = (elW - cw) / 2;
+  }
+  return { ox, oy, cw, ch, elW, elH, vw, vh };
+}

plan/CHANGE_REQUESTS.md

@@ -0,0 +1,35 @@
+# Change Requests
+
+Lanes that believe a **frozen** file must change (`db.py` schema, `api.py` routes/shapes,
+`cli.py`, `config.py`, `synthetic.py`, `pyproject.toml`, the pose contract) do **not** edit it.
+Instead append an entry here and work around it locally; the integration agent adjudicates.
+
+**Format** (one entry per request, newest at the bottom):
+
+```
+### CR-<n> — <lane> — <date> — <one-line title>
+- **File:** <frozen file + symbol>
+- **Problem:** why the current contract blocks you
+- **Proposed change:** the minimal change you need
+- **Workaround in place:** what you did locally so you're not blocked
+- **Decision:** (integration fills this: applied | rejected — rationale)
+```
+
+---
+
+<!-- entries below -->
+
+### CR-1 — lane D — 2026-07-16 — Update obsolete stub assertion in test_api.py
+- **File:** `backend/tests/test_api.py::test_detect_degrades_gracefully`
+- **Problem:** That foundation test asserted the *stub* contract of `POST /api/events/detect`
+  (`"note" in data`), which was correct only while lane D was unimplemented. With M7 landed,
+  `run_events` no longer raises `NotImplementedError`, so `api.py` returns the real
+  `{result, events}` shape (spec M3) and the stub assertion fails.
+- **Proposed change:** rename to `test_detect_events_endpoint` and assert the real landed
+  contract (200; `result` is the run_events summary with candidates/classified/… keys; `events`
+  is a list; idempotent on repeat).
+- **Workaround in place:** applied the minimal edit directly (test file, not a frozen contract
+  file). The frozen `api.py` app is **unchanged**. Only lane D's owned modules + this obsolete
+  stub assertion were touched. Full suite green (89 passed).
+- **Decision:** (integration) — informational; the change is a direct, unavoidable consequence
+  of lane D landing. Confirm the assertion matches the intended M3 detect-response shape.

plan/DIRECTIVES.md

@@ -4,6 +4,20 @@ Written **only** by the coordinator (the human's planning session). Agents: read
 
 ---
 
+## Round 1 — 2026-07-16 — All lanes merged & verified; integration begins
+
+**Coordinator review result:** foundation + all four lanes are merged to `main` (merge commits `e28dc78` A, `2cad2f7` B, `9d17b47` C; D fast-forwarded earlier at `5c2d7c6`). Independently verified on the merged tree: **96 backend tests pass**, synthetic `ingest → sync` recovers ground-truth offsets exactly (+0 / +1370 / −842 ms, drift 0), `events` finds 3 candidates and degrades gracefully unconfigured, frontend production build clean. Excellent evidence discipline across all lanes — keep it.
+
+**Directives:**
+
+1. **Integration phase starts now** — one agent, branch `integration` from `main`, per `plan/10-integration.md`. Maintain `plan/status/integration.md`.
+2. **CR-1 (lane D, test_api.py stub assertion): APPROVED by coordinator.** The edit was a direct consequence of M7 landing and touched no frozen contract file. Integration agent: mark the Decision line in `plan/CHANGE_REQUESTS.md` and confirm the new assertion matches the spec M3 `POST /api/events/detect` response shape (`{result, events}`).
+3. **Integration additions to the plan/10-integration.md scope** (small, discovered in review):
+   - The shared-worktree collision (see lane A/C status round 0) left a coordinator-stashed debris stash in the main working dir (`git stash list` — "stale lane A/B debris"). After confirming nothing of value remains, drop it, and remove the now-merged worktrees `../festifun-laneA` and `../festifun-laneB` (`git worktree remove`).
+   - Lane C's env note: continuous-playback sync was verified deterministically because the automated browser tab was backgrounded. During your full-system pass, verify the M4 sync overlay (<50 ms) once in a real focused browser window.
+   - Test count sanity: suite is 96 passed on merge day; it must never drop below that.
+4. **Worktree protocol lesson (standing):** integration runs single-agent, so use the main working directory directly on branch `integration`. If parallel agents are ever launched again, each MUST create its own `git worktree` before its first edit — never share a working directory.
+
 ## Round 0 — 2026-07-16 — Run order & protocol
 
 **Execution order:**

plan/status/foundation.md

@@ -0,0 +1,34 @@
+# Status — foundation
+
+## Round 0 — 2026-07-16 — STATUS: merged
+
+**Merged:** `foundation` → `main` (fast-forward, commit cf6a6fb) after all acceptance
+below passed. Local only — not yet pushed to `origin` (awaiting go-ahead).
+
+**Directives acknowledged:** round 0 of plan/DIRECTIVES.md (run `foundation` only; contracts
+correctness beats speed; lanes hard-depend on API shapes, schema, pose math, synthetic fixtures).
+
+**Acceptance checklist** (plan/00-foundation.md + spec M0/M3) — all pass:
+- [x] `python -m festival4d synthetic` → 3 fixture videos + populated DB + `points.ply` (3106 pts) + `ground_truth.json`. Evidence: CLI run logs "3 videos, 3106 points, 7 events, 4 anchors".
+- [x] `python -m festival4d serve` → every M3 endpoint returns synthetic data; videos seekable. Evidence: curl of manifest/poses/anchors/events/pointcloud; `curl -H "Range: bytes=0-100" /media/cam0.mp4` → **206** with `content-range`; browser `<video>` seeked to 15s, `seekable=[0,20]`, readyState 4.
+- [x] `pytest` green — **24 passed** (test_geometry, test_synthetic, test_api).
+- [x] `npm run dev` serves the hello page; browser fetched `/api/manifest` cross-origin (CORS ok), pose.js self-test passed, **0 console errors**. (Vite fell back to :5174 — an unrelated dev server holds :5173; CORS now allows any localhost port.)
+- [x] All lane-owned modules stubbed with frozen signatures; `cli.py` registers all 6 subcommands (dispatch + graceful NotImplementedError → exit 2).
+- [x] Frozen geometry contract: `colmap_to_threejs` + `test_geometry.py` (3 known vectors, random round-trip, scipy oracle) + mirrored `frontend/src/lib/pose.js` with identical `POSE_TEST_VECTORS` (node self-test passes).
+- [x] Classifier contract: `MomentClassification` (Pydantic) + `MomentClassifier` (runtime Protocol) + Gemini/Claude/Local provider stubs.
+- [x] `plan/CHANGE_REQUESTS.md` created.
+
+**Done this round:** full M0 + M3 + all frozen contracts. Env: Python 3.12 venv via uv, all
+spec deps installed. DB schema per spec §2 with CRUD helpers. Timebase helpers in `config.py`.
+API is Range-capable and CORS-enabled. Frontend scaffolds every layout file (lane-C modules
+are stubs; `lib/pose.js` is the live frozen contract).
+
+**Frozen-contract note for lanes:** CORS was widened from a single `:5173` origin to any
+`http://localhost|127.0.0.1:<port>` (local single-user tool; robust to Vite port fallback).
+Explicit `:5173` still included. If a lane needs a different API shape/schema/pose math,
+file a CR in `plan/CHANGE_REQUESTS.md` — do not edit `db.py`/`api.py`/`cli.py`/`config.py`/
+`synthetic.py`/`pyproject.toml`.
+
+**Blockers / questions for coordinator:** none.
+
+**Next:** merge `foundation` → `main`; lanes may start (per directive: C, then A, D, B).

plan/status/lane-A.md

@@ -0,0 +1,24 @@
+# Status — lane-A
+
+## Round 0 — 2026-07-16 — STATUS: ready_to_merge (committed on isolated worktree)
+
+**Directives acknowledged:** round 0 of plan/DIRECTIVES.md (run order & protocol; Lane A = spec M1).
+
+**Acceptance checklist** (spec M1 + lane brief) — ALL PASSING:
+- [x] `ingest`: ffprobe each file in `data/raw`, register in `videos`, extract mono 16 kHz WAV into `data/work/audio/`.
+- [x] `sync`: PHAT-whitened GCC-PHAT pairwise offsets, scored by peak-to-second-peak ratio; sub-sample parabolic peak refinement.
+- [x] Global confidence-weighted least-squares solve; cycle-consistency rejection (>50 ms residual).
+- [x] Drift: windowed GCC-PHAT + line fit → `drift_ppm`; store 0 if `|drift| < 5 ppm`.
+- [x] Disconnected components → `offset_ms = None` (spec pitfall #5).
+- [x] Persist via `db.update_video_sync`; export `config.SYNC_JSON`.
+- [x] `synthetic → ingest → sync` recovers GT offsets (err < 0.001 ms) and drift 0 ppm — inside ±10 ms / ±3 ppm.
+- [x] `pytest` green in isolation: **43 passed** (24 foundation + 19 new) on pristine foundation + Lane A only.
+
+**Done this round:** implemented `ingest.py` + `audio_sync.py`; wrote `test_ingest.py` (6) + `test_audio_sync.py` (13). Committed as `157c783` on `lane/a-media`, based directly on `main`/foundation (`5fa7301`) — exactly the 5 owned files, no other lane's changes. Evidence (run in the isolated `festifun-laneA` worktree):
+- `pytest` → `43 passed, 4 warnings` (warnings are librosa/audioread deprecations only).
+- `synthetic → ingest → sync` → DB cam0/1/2 = +0.000 / +1370.000 / −842.000 ms, drift 0, err ≤ 0.0002 ms vs `ground_truth.json`.
+- `sync` exported `data/work/sync.json` (reference id, per-video offset/drift/confidence, raw edges, components).
+
+**Coordination note (was a blocker; resolved for Lane A):** Lanes A/C/D were sharing one working directory (`/Users/m3ultra/Documents/festifun`) and colliding — HEAD there was moved to `lane/d-events`, and the shared tree held four lanes' uncommitted changes intermixed. Per coordinator sign-off, Lane A was landed into its own `git worktree` (`festifun-laneA`, branch `lane/a-media` reset to `main`) and committed there, disturbing no other worktree. **Still open for coordinator:** lanes C and D also need their own worktrees (lane B already has one); the shared dir still contains their intermixed uncommitted work.
+
+**Next:** awaiting coordinator go-ahead to `git push origin lane/a-media` and merge to `main` (merge is coordination-sensitive given the collision; branch is ready).

plan/status/lane-B.md

@@ -0,0 +1,59 @@
+# Status — lane-B
+
+## Round 0 — 2026-07-16 — STATUS: ready_to_merge
+
+**Directives acknowledged:** round 0 of plan/DIRECTIVES.md (run order C→A→D→B; merge when
+acceptance passes, don't wait on siblings; maintain this status file; evidence discipline).
+
+**Environment:** COLMAP 4.1.0 (CPU-only) + ffmpeg both installed, so the pipeline was
+exercised against **real COLMAP end-to-end**, not just the synthetic export path.
+
+**Isolation note (see blocker):** worked in a dedicated git worktree
+(`/Users/m3ultra/Documents/festifun-laneB` on `lane/b-recon`) because all lane agents were
+sharing one working directory. Only the 6 lane-B-owned files were touched.
+
+**Acceptance checklist** (spec M2 + M8 geometry + lane brief) — all pass:
+- [x] `geometry.py` M8 stubs implemented: `slerp_pose` (shortest-arc, double-cover), `ray_from_pixel`
+      (COLMAP +y-down back-projection), `triangulate_rays` (closest-point + parallel guard),
+      `nearest_point_on_ray` (in-front radius cylinder). Frozen `colmap_to_threejs`/`quat_to_mat`/
+      `mat_to_quat` and the 3 frozen POSE_TEST_VECTORS untouched (`git diff 5fa7301` shows 0 removed
+      lines in test_geometry.py; the frozen funcs are unchanged).
+- [x] `frames.py`: `sharpness` (variance of Laplacian) + windowed sharpest-frame `sample_frames`.
+- [x] COLMAP TXT parsers (images/cameras/points3D) with hand-written-snippet tests + a truncated-PARAMS
+      guard (clean ValueError).
+- [x] `normalize_scene`: centroid≈0, camera-sphere radius≈10, up≈+Y — verified by `test_normalize_scene_invariants`
+      and a projection-invariance test (`test_normalize_scene_preserves_projection`).
+- [x] pose interpolation (slerp+lerp, `registered=False`, NO extrapolation past first/last registered) —
+      both guard directions tested.
+- [x] PLY export via frozen `synthetic.write_ply`; poses written via atomic `db.set_poses`.
+- [x] graceful degradation: COLMAP absent / <60% frames / <2 videos / no frames / missing files →
+      diagnostic, existing poses UNTOUCHED, DB uncorrupted. Mixed-subset case (some videos registered,
+      others kept) tested for the DB-safety invariant.
+- [x] real end-to-end: `synthetic` → `reconstruct` ran the full COLMAP CLI (feature_extractor →
+      exhaustive_matcher → mapper → image_undistorter → model_converter → my parsers), produced 6
+      components (2/15/11/16/17/37 imgs), correctly selected the largest by `images.bin` header count,
+      judged 37/120 (31%) weak, degraded gracefully, **left all 41 synthetic poses/video untouched, exit 0.**
+- [x] `pytest` green — **61 passed** (24 foundation + 37 lane-B; test_geometry.py + test_sfm.py).
+- [x] no edits outside owned files (`git diff 5fa7301 --name-only` = frames/geometry/sfm/test_geometry only;
+      untracked: test_sfm.py, this status file); no new deps.
+
+**Bugs found & fixed while validating on real COLMAP:**
+- COLMAP 4.x renamed `SiftExtraction`/`SiftMatching` → `FeatureExtraction`/`FeatureMatching`; option
+  detected from `--help` (which COLMAP prints to **stderr**) so the pipeline runs on 3.x and 4.x.
+- `_largest_model_dir` ranked binary models by file size (tracks keypoints, not image count) → now reads
+  the exact registered-image count from the `images.bin` uint64 header (validated on the real 6-component run).
+- workspace cleared before each run so a stale `database.db` can't fail reruns.
+
+**Adversarial review:** 5-lens review + verify pass (23 agents) → 18 raw findings, 5 CONFIRMED (2 code, 3
+test-coverage), all applied; 13 refuted (incl. triangulate/nearest-point "behind origin" — correct-as-written).
+
+**Blockers / questions for coordinator:**
+- **Shared working tree collision.** All four lane agents are operating in the *same* working directory
+  (`/Users/m3ultra/Documents/festifun`), not separate clones/worktrees as `plan/README.md` requires.
+  Mid-session the shared branch was switched to `lane/d-events` and the tree accumulated uncommitted edits
+  from lanes A/B/C/D at once. I isolated lane B in its own worktree so nothing is clobbered. **Recommend the
+  coordinator have each lane use `git worktree`/separate clones, and adjudicate merges centrally**, since a
+  naive `git branch -f main` from any lane could silently drop a sibling's merge. I did NOT auto-merge to
+  `main` for this reason — lane B is ready and awaiting a safe merge.
+
+**Next:** commit lane B to `lane/b-recon`; coordinator to merge to `main` (safely, given the shared-tree issue).

plan/status/lane-C.md

@@ -0,0 +1,53 @@
+# Status — lane-C
+
+## Round 1 — 2026-07-16 — STATUS: ready_to_merge
+
+**Directives acknowledged:** round 0 of plan/DIRECTIVES.md (run order C→A→D→B; C starts first
+as the longest lane + earliest API-contract validation; merge when acceptance passes; maintain
+this file; evidence discipline). No new rounds present on origin at time of writing.
+
+**Acceptance checklist (lane brief M4/M5/M6 + timeline markers) — all pass:**
+- [x] **M4 synchronized playback.** Master clock from `performance.now()` (never a `<video>`).
+      Timebase mapping proven exact by seek: at t_global=8.0 → cam0=8.000, cam1=6.630,
+      cam2=8.842 s, err=0 ms each. Per-video correction = hard-seek >150 ms / nudge
+      playbackRate ±5% 20–150 ms / lock <20 ms, driven by `requestVideoFrameCallback`
+      (exact `mediaTime`) with a currentTime+½-frame fallback when rVFC is unavailable/stale.
+      Out-of-range videos pause+dim (cam1 at t=0, target −1.37 s). One audio source; dev
+      overlay shows per-video sync ms. Measured continuous-playback **inter-video desync =
+      13 ms mean / 46 ms max** over 6 s (< 50 ms), driven by a setTimeout pump (see env note).
+- [x] **M5 3D viewer.** PLY point cloud (3106 pts, vertex colors); per-video camera-path lines
+      + current-pose frusta (colored); OrbitControls; snap-to-camera tweens viewer to the pose
+      (following cam1: camPos≈(1.78,2.63,9.25)≈true center (0,2.4,9.4); fov=36°≈2·atan(H/2fy);
+      orbit disabled) and free-roam restores (fov→50°, orbit on). All COLMAP→Three via frozen
+      `lib/pose.js` — never reimplemented.
+- [x] **M6 anchor overlays.** Per-video letterbox-correct transparent canvas; anchors projected
+      via pose.js camera. Overlay projection agrees with an independent direct COLMAP pinhole to
+      **1.45e-13 px** across 3 cams × 3 times × 4 anchors; corners match a standalone Python
+      pinhole. Behind-camera cull (z_cam<0). Stage corners render/track on all in-range videos.
+- [x] **Timeline event markers.** 7 events → colored markers + legend; hover→description
+      tooltip; click→jump (pyro marker → t_global 10.00). Scrubber seeks/parks all videos.
+- [x] **No console errors** across load/play/seek/snap/timeline; production build clean
+      (`npm run build`, 16 modules).
+
+**Env note (not a blocker):** the automated browser pane reports `document.hidden=true`, so rAF
+and rVFC only fire during screenshots — continuous playback can't be observed by polling here.
+Verified deterministically instead (seek exactness to 0 ms; projection to 1e-13 px; a
+setTimeout-driven run of the real correction code → 13 ms mean inter-video desync). A real
+focused browser runs the rAF loop at 60 fps and rVFC drives even tighter absolute sync.
+
+**Owned-files discipline:** committed only `frontend/**` + this status file. Left untouched the
+other lanes' in-tree changes (A: audio_sync/ingest; B: frames/geometry/sfm; D: events_ai).
+Added `frontend/src/lib/timebase.js` + `poseTrack.js` (new files inside lane C's ownership).
+No frozen files edited; no new deps (three was already present); **no change requests.**
+
+**Repo note for coordinator:** this working directory is shared/contended — `lane/c-viewer`
+was reset to foundation (5fa7301) and HEAD moved to `lane/d-events` between my session-start
+commit and now, and `main` has since advanced to lane D's self-merge (6ec55c7). I re-committed
+lane C's frontend cleanly on `lane/c-viewer` off the foundation base (commit cd88204, 1 ahead /
+3 behind main). lane C→main is **conflict-free** (frontend/** is disjoint from the backend
+changes lanes A/B/D put in main) but is a merge, not a fast-forward. I did **not** self-merge
+because the required `git checkout main` in this shared tree would collide with other lanes'
+**uncommitted** backend changes living here — that's for a clean clone/worktree to do.
+
+**Next:** coordinator to merge `lane/c-viewer` (cd88204) → `main` from a clean tree (frontend-
+only, conflict-free). Phase-3 seams left as clean stubs: `annotate.js` (M8), `camPath.js` (M9).

plan/status/lane-D.md

@@ -0,0 +1,65 @@
+# Status — lane-D
+
+## Round 1 — 2026-07-16 — STATUS: merged
+
+**Merged:** `lane/d-events` → `main` (fast-forward, commit `5c2d7c6`) after acceptance below
+passed. Local only (not pushed to `origin` — shared working tree; coordinator drives origin).
+
+
+**Adversarial review (self-run, 5 lenses → verify):** 3 findings, all confirmed & fixed —
+(1) `detect_candidates` fabricated ~1 candidate/s on silent/DC/noise audio → added an AC-RMS
+(std) floor `MIN_SIGNAL_STD`; silence/DC/noise now return `[]`, fixture unchanged [3,10,17];
+(2) degradation test's raised tripwire was swallowed by `except Exception` → switched to a
+call-recorder (`assert prepared == []`); (3) windowed test ran on a fresh DB → now seeds an
+out-of-window `ai` event and asserts it survives. `pytest` **90 passed** after fixes.
+
+
+**Directives acknowledged:** round 0 of plan/DIRECTIVES.md (foundation merged; lanes may start;
+lane D is small + fully independent; merge when acceptance passes — no waiting on siblings).
+
+**Acceptance checklist** (plan/lane-D-events.md + spec M7) — implementation complete, verified:
+- [x] `detect_candidates` finds candidates within ±0.5 s of fixture ground-truth pulses
+      [3.0, 10.0, 17.0]. Evidence: `test_events_ai.py::test_detect_candidates_matches_ground_truth`
+      (recall+precision+count). RMS×spectral-flux, centered-10 s-window local-max, ≥ P90.
+- [x] `MomentClassification` + `MomentClassifier` frozen contract preserved verbatim; import
+      of events_ai pulls in ZERO heavy libs/SDKs (all lazy). Evidence: import-cost probe → `[]`.
+- [x] Three providers implemented against the installed SDKs (verified surfaces): Gemini
+      (google-genai 2.11.0, `Part.from_bytes(data,mime_type)` + `generate_content` structured
+      output), Claude (anthropic 0.116.0, `messages.parse(output_format=…, thinking=adaptive,
+      max_tokens)` → `parsed_output`, NO temperature, model `claude-opus-4-8`), Local
+      (openai 2.45.0, `chat.completions.create` json_object, 6 image_url parts, 1 retry).
+- [x] Provider selection `FESTIVAL4D_CLASSIFIER=gemini|claude|local` (default gemini);
+      unconfigured → candidates-only + log line. Evidence: `test_get_classifier_*` (5 tests).
+- [x] Per-candidate exception isolation. Evidence: `test_run_events_isolates_per_candidate_failure`
+      (2 ai + 1 fallback candidate; batch of 3 completes despite one raising).
+- [x] `run_events` fills the frozen entrypoint (cli.py + api.py dispatch); writes via db helpers
+      only (insert-once in final state — db.py has no update-event helper; end state matches spec).
+- [x] `pytest` green with NO network: **90 passed** across consecutive full-suite runs
+      (16 lane-D tests; failures observed once were a transient lane-A shared-`AUDIO_DIR` race).
+- [x] Real-path end-to-end (ffmpeg-decoded AAC audio, not in-memory): isolated 20 s fixture →
+      `run_events()` degraded → candidates at 3.008/10.016/17.013 s (≤0.016 s error), exactly 3,
+      no spurious. Evidence: scratchpad/e2e_verify.py → RESULT PASS.
+
+**Done this round:** implemented `backend/festival4d/events_ai.py` (candidate detection + 3
+providers + get_classifier + prepare_inputs + run_events orchestration) and
+`backend/tests/test_events_ai.py` (16 tests). Detection algorithm locked via a scratch
+experiment before coding. Confirmed Claude call shape via claude-api skill and all three SDK
+surfaces via introspection. Ran a 5-lens adversarial review (8 agents: spec-compliance /
+provider-SDK / detection-orchestration / contract-ownership / test-quality → verify); 3 findings
+confirmed and all fixed (see the review summary at the top of this round).
+
+**Coordinator note (out-of-strict-ownership edit):** updated `backend/tests/test_api.py`
+`test_detect_degrades_gracefully` → `test_detect_events_endpoint`. That foundation test asserted
+the STUB behavior (`"note" in data`) of `POST /api/events/detect`; my landing replaces the stub
+with real detection returning `{result, events}`. The frozen `api.py` app is UNCHANGED — only the
+now-obsolete stub assertion was updated. Flagged here for transparency; see plan/CHANGE_REQUESTS.md CR-1.
+
+**Note on shared tree:** this working directory holds all four lanes' uncommitted work
+concurrently (A/B/C/D). I will stage and commit ONLY my owned files. One transient
+`test_audio_sync.py` (lane A) failure was observed once mid-run (concurrent write) but the suite
+is deterministically green (89/89 ×5); not caused by lane D.
+
+**Blockers / questions for coordinator:** none.
+
+**Next:** Lane D (M7) is done and merged to `main` (`5c2d7c6`). No follow-up needed in-lane.
+Coordinator: proceed to integration (phase 3) once lanes A/B/C also merge.

pyproject.toml

@@ -0,0 +1,38 @@
+[project]
+name = "festival4d"
+version = "0.1.0"
+description = "Turn multiple fan-shot smartphone concert videos into a synchronized, explorable 4D experience."
+readme = "README.md"
+requires-python = ">=3.11"
+dependencies = [
+    "fastapi>=0.110",
+    "uvicorn[standard]>=0.29",
+    "numpy>=1.26",
+    "scipy>=1.11",
+    "librosa>=0.10",
+    "soundfile>=0.12",
+    "sqlalchemy>=2.0",
+    "pydantic>=2.6",
+    "opencv-python-headless>=4.9",
+    # classifier providers (lane D) — imported lazily, never at module load
+    "google-genai>=0.3",
+    "anthropic>=0.40",
+    "openai>=1.30",
+]
+
+[project.optional-dependencies]
+dev = ["pytest>=8.0"]
+
+[project.scripts]
+festival4d = "festival4d.cli:main"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["backend/festival4d"]
+
+[tool.pytest.ini_options]
+testpaths = ["backend/tests"]
+addopts = "-q"