diff --git a/.gitignore b/.gitignore
index 74b3d59..e98ece7 100644
--- a/.gitignore
+++ b/.gitignore
@@ -5,3 +5,4 @@ node_modules/
__pycache__/
.venv/
.DS_Store
+frontend/dist/
diff --git a/README.md b/README.md
index 6658709..260aa3d 100644
--- a/README.md
+++ b/README.md
@@ -3,55 +3,187 @@
Turn multiple fan-shot smartphone videos of the same concert into a synchronized,
explorable **4D experience**: time-aligned multi-video playback, 3D scene reconstruction
with camera poses, a free-roam "god's eye" viewer, AR-style overlays projected onto each
-video, and AI-tagged moments on a shared timeline.
+video, AI-tagged moments on a shared timeline, click-to-place 3D annotations, and keyframed
+cinematic fly-throughs.
-> **Status: foundation phase.** The scaffold, database, frozen contracts (API, pose math,
-> DB schema), and a synthetic fixture generator are in place. Feature lanes (media sync,
-> reconstruction, viewer, AI events) build on top. The full README with the real-footage
-> workflow is written in the integration phase (M9). See
-> [`OPUS_BUILD_INSTRUCTIONS.md`](OPUS_BUILD_INSTRUCTIONS.md) for the canonical spec and
-> [`plan/`](plan/) for the execution plan.
+Everything works **from pixels and audio alone** — no depth sensors, no IMU logs. It runs
+fully offline and local; the only optional cloud piece is moment classification.
+
+---
+
+## What you can do
+
+- **Scrub a shared timeline** and watch every camera stay locked to the same instant, aligned
+ by their audio (no manual frame-matching).
+- **Fly through the reconstructed scene** in 3D — orbit freely, snap to any real camera, or
+ play a keyframed cinematic path.
+- **See an "x-ray" HUD**: 3D anchor points (stage corners, tagged objects) projected onto every
+ video, tracking even when occluded by crowd or scaffolding.
+- **Jump between AI-tagged moments** (bass drops, pyro, crowd waves…) on the timeline, and
+ correct any misclassification.
+- **Click to locate things in 3D**: draw a box around an object in one or two videos and the
+ app triangulates its 3D position, dropping an anchor that appears everywhere at once.
+
+---
## Prerequisites
-- **Python 3.11+** and [`uv`](https://docs.astral.sh/uv/) (or venv + pip)
-- **ffmpeg** / **ffprobe** on your `PATH` (required)
-- **COLMAP** (optional — reconstruction degrades gracefully without it)
-- A classifier API key (optional — `GEMINI_API_KEY` by default; see `events_ai.py`)
-- **Node 18+** for the frontend
+| Tool | Required? | Notes |
+|---|---|---|
+| **Python 3.11+** + [`uv`](https://docs.astral.sh/uv/) (or venv+pip) | yes | backend |
+| **ffmpeg** / **ffprobe** on `PATH` | yes | audio extraction, frame sampling, clip cutting |
+| **Node 18+** | yes | frontend (Vite + Three.js) |
+| **COLMAP** on `PATH` | optional | 3D reconstruction. Without it the app still runs — you get synced videos + timeline, just no 3D scene. `brew install colmap` on macOS. |
+| A classifier API key | optional | AI moment labels. Default **Gemini 2.5 Flash** (`GEMINI_API_KEY`); also Claude or any OpenAI-compatible/local/OpenRouter endpoint via `FESTIVAL4D_CLASSIFIER`. Without a key you still get audio-detected candidate moments, just unlabeled. |
+
+---
## Quickstart (synthetic demo — no footage needed)
+The synthetic fixture generates three fake camera videos of the same fake stage, with known
+audio offsets and known 3D geometry — so you can see the whole app working end-to-end before
+you have any real footage.
+
```bash
# 1. backend env
uv venv --python 3.12
uv pip install -e ".[dev]"
-# 2. generate the synthetic fixture project (fake videos + poses + point cloud + events)
+# 2. generate the synthetic project (fake videos + poses + point cloud + seeded events)
uv run python -m festival4d synthetic
# 3. serve the API (http://127.0.0.1:8000)
uv run python -m festival4d serve
-# 4. in another terminal, the frontend
+# 4. in another terminal, run the frontend
cd frontend
npm install
-npm run dev # http://localhost:5173
+npm run dev # opens http://localhost:5173 (or the next free port)
```
-## Backend CLI
+Open the printed URL. You should see the 3D scene with three camera frusta around a stage box,
+the three synced video players, stage-corner anchors projected onto each video, and colored
+moment markers on the timeline.
+
+---
+
+## Real-footage workflow
+
+1. **Shoot / gather** 2–4 videos of the same performance from different positions (see
+ *Shooting tips* below) and drop the files into `data/raw/`.
+2. **Run the pipeline:**
+
+ ```bash
+ uv run python -m festival4d ingest # probe videos, extract 16 kHz mono audio
+ uv run python -m festival4d sync # GCC-PHAT audio alignment (offsets + drift)
+ uv run python -m festival4d reconstruct # COLMAP structure-from-motion → camera poses + point cloud
+ uv run python -m festival4d events # detect audio moments + AI-classify them
+ uv run python -m festival4d serve # serve it
+ ```
+
+ Each step is independent and re-runnable. `sync` alone already gives you locked multi-video
+ playback; `reconstruct` adds the 3D scene and overlays; `events` adds the tagged timeline.
+3. **Explore** in the browser (`cd frontend && npm run dev`).
+
+If COLMAP can't reconstruct your footage (common with dark, motion-blurred, or low-overlap
+clips), the pipeline says so and leaves you with the synced-video experience — it never
+corrupts existing data.
+
+To enable AI moment labels, set a key before `events`:
+
+```bash
+export GEMINI_API_KEY=... # default provider (Gemini 2.5 Flash, native video)
+# or: export FESTIVAL4D_CLASSIFIER=claude ANTHROPIC_API_KEY=...
+# or: export FESTIVAL4D_CLASSIFIER=local FESTIVAL4D_OPENAI_BASE_URL=... FESTIVAL4D_OPENAI_MODEL=...
+```
+
+---
+
+## Using the viewer
+
+| Action | How |
+|---|---|
+| Play / pause | `Space` or the ⏵ button |
+| Nudge time ±1 s (±5 s) | `←` / `→` (hold `Shift`) |
+| Scrub | drag the timeline |
+| Pick the audio you hear | click a video (only one is unmuted at a time) |
+| Enable/disable a video | ◉ on the video |
+| Orbit the 3D scene | drag in the 3D pane |
+| Snap the 3D view to a camera | click its frustum, its ⛶ button, or press `1`–`9` |
+| Free roam again | `Esc` / `0` / **Free roam** |
+| Correct a moment / annotate | click a timeline marker → panel |
+| Place a 3D anchor | in the panel, **Annotate location**, then drag a box on a video (repeat on a 2nd video to triangulate) |
+| Add a camera keyframe | **＋ Key** or `K` (captures the current free-roam view at the current time) |
+| Play the keyframed path | **▶ Path** · export/import with ⤓ / ⤒ |
+
+The top-right **sync error** panel shows each video's alignment error in ms while playing —
+a dev aid; it should stay green (<50 ms) once footage is well-synced.
+
+---
+
+## Architecture
```
-python -m festival4d synthetic # generate the synthetic fixture (M0)
-python -m festival4d ingest # probe videos + extract audio (lane A / M1)
-python -m festival4d sync # GCC-PHAT audio alignment (lane A / M1)
-python -m festival4d reconstruct # COLMAP SfM + pose export (lane B / M2)
-python -m festival4d events # audio candidates + AI classify (lane D / M7)
-python -m festival4d serve # FastAPI app (M3)
+data/raw/ ──ingest──▶ data/work/audio ──sync────▶ videos.offset_ms / drift_ppm ┐
+ │
+ ──frames──▶ data/work/frames ──reconstruct──▶ camera_poses + points.ply
+ │
+ reference audio ──events──▶ events (audio candidates → AI labels)
+ │
+ SQLite (data/project.db) ◀─────────┘
+ │
+ FastAPI (backend/festival4d/api.py)
+ │ /api/manifest, /poses, /pointcloud,
+ │ /events, /anchors, /annotations
+ ▼
+ Vite + Three.js SPA (frontend/src)
+ transport (master clock) · videoGrid + overlays · scene3d · timeline
+ annotate (M8) · camPath (M9)
```
+- **Backend** — Python: `ingest`/`audio_sync` (media + GCC-PHAT sync), `frames`/`sfm`/`geometry`
+ (COLMAP orchestration + pose math + triangulation), `events_ai` (moment detection + pluggable
+ classifiers), `resolve` (annotation→3D), `api` (FastAPI, Range-capable video serving), `db`
+ (SQLAlchemy/SQLite), `synthetic` (the test fixture).
+- **Frontend** — vanilla JS + Three.js. A single master clock (`transport.js`) derives each
+ video's local time from the shared timeline and continuously corrects playback; `lib/pose.js`
+ is the one COLMAP→Three.js pose conversion, mirrored from `geometry.py` and locked by a shared
+ test.
+- **Timebase contract:** `t_video = (t_global − offset_ms/1000) · (1 + drift_ppm·1e−6)`; the
+ reference video has offset 0.
+
+Full design + milestone history: [`OPUS_BUILD_INSTRUCTIONS.md`](OPUS_BUILD_INSTRUCTIONS.md);
+the parallel build plan and per-lane status live in [`plan/`](plan/).
+
+---
+
+## Shooting tips (for good reconstructions)
+
+Concert footage is genuinely hard for structure-from-motion. To give COLMAP a chance:
+
+- **Spread the cameras out** but keep **overlapping views** — each pair of cameras should see
+ some of the same stage/structure. No overlap → they can't be related in 3D.
+- **Keep some static structure in frame** (stage edges, truss, speaker stacks). A frame that's
+ all moving crowd and lights has nothing stable to triangulate.
+- **Avoid pure zoom** — physically moving parallax reconstructs far better than zooming.
+- **Brighter, sharper is better** — motion blur and near-dark frames are the main failure cause.
+- **2–4 phones is the sweet spot** for a first reconstruction; more is fine but slower.
+
+Audio sync is far more forgiving: any clips that share audible sound (the same music/claps)
+will align, even across otherwise-unrelated angles.
+
+---
+
## Tests
```bash
-uv run pytest
+uv run pytest # backend
+cd frontend && npm run build # frontend typecheck/build
```
+
+---
+
+## Non-goals (this prototype)
+
+No accounts/auth, no cloud storage, no realtime ingest, no NeRF/Gaussian splatting, no mobile
+UI, no Docker. Single local user, one project at a time.
diff --git a/backend/festival4d/api.py b/backend/festival4d/api.py
index dbaf02b..316a6f4 100644
--- a/backend/festival4d/api.py
+++ b/backend/festival4d/api.py
@@ -73,6 +73,14 @@ class AnnotationIn(BaseModel):
video_id: int
t_video_s: float
bbox: list[float] = Field(min_length=4, max_length=4) # [x0, y0, x1, y1] normalized
+ event_id: int | None = None # CR-2: link to the event this annotation locates (M8)
+
+
+class EventPatch(BaseModel):
+ event_type: str | None = None
+ source: str | None = None
+ description: str | None = None
+ confidence: float | None = None
# ---------------------------------------------------------------------------
@@ -191,19 +199,57 @@ def detect_events(payload: DetectIn) -> dict:
return {"result": result, "events": [_event_dict(e) for e in db.get_events()]}
+@app.patch("/api/events/{event_id}")
+def patch_event(event_id: int, payload: EventPatch) -> dict:
+ """Correct an event in place (M8). A user correction defaults ``source`` to ``'user'``."""
+ if db.get_event(event_id) is None:
+ raise HTTPException(status_code=404, detail=f"no event with id={event_id}")
+ source = payload.source
+ if source is None and (payload.event_type is not None or payload.description is not None):
+ source = "user" # any user-driven correction is attributed to the user
+ event = db.update_event(
+ event_id,
+ event_type=payload.event_type,
+ source=source,
+ description=payload.description,
+ confidence=payload.confidence,
+ )
+ return _event_dict(event)
+
+
@app.post("/api/annotations")
def create_annotation(payload: AnnotationIn) -> dict:
- x0, y0, x1, y1 = payload.bbox
- annotation = db.add_annotation(payload.video_id, payload.t_video_s, x0, y0, x1, y1)
+ from festival4d import resolve
+
+ x0, y0, x1, y1 = payload.bbox
+ if db.get_video(payload.video_id) is None:
+ raise HTTPException(status_code=404, detail=f"no video with id={payload.video_id}")
+ annotation = db.add_annotation(
+ payload.video_id, payload.t_video_s, x0, y0, x1, y1, event_id=payload.event_id
+ )
+
+ # M8: cast a ray through the bbox center; triangulate against another view of the same
+ # event, else fall back to the nearest point-cloud point / centroid depth.
+ result = resolve.resolve_annotation(
+ payload.video_id, payload.t_video_s, payload.bbox, event_id=payload.event_id
+ )
+
+ anchor = None
+ if result.point is not None:
+ label = "annotation"
+ color = None
+ if payload.event_id is not None:
+ ev = db.get_event(payload.event_id)
+ if ev is not None:
+ label = ev.description or ev.event_type or label
+ anchor = db.add_anchor(label, result.point[0], result.point[1], result.point[2], color)
+ db.set_annotation_anchor(annotation.id, anchor.id)
- # M8 resolution (triangulate / nearest-point) is integration work; geometry stubs raise
- # until lane B lands. Store the annotation now; resolve to an anchor later.
- anchor_id = None
- point = None
- note = "annotation stored; 3D resolution is M8/integration work"
return {
"annotation_id": annotation.id,
- "anchor_id": anchor_id,
- "point": point,
- "note": note,
+ "anchor_id": anchor.id if anchor else None,
+ "point": result.point,
+ "method": result.method,
+ "gap": result.gap,
+ "anchor": _anchor_dict(anchor) if anchor else None,
}
diff --git a/backend/festival4d/db.py b/backend/festival4d/db.py
index 171cacf..ecd299e 100644
--- a/backend/festival4d/db.py
+++ b/backend/festival4d/db.py
@@ -120,6 +120,9 @@ class Annotation(Base):
y0: Mapped[float] = mapped_column(Float, nullable=False)
x1: Mapped[float] = mapped_column(Float, nullable=False)
y1: Mapped[float] = mapped_column(Float, nullable=False)
+ # Nullable link to the event this annotation locates (CR-2, M8). Two annotations sharing an
+ # event_id from different videos triangulate to a 3D anchor. Nullable => additive/back-compat.
+ event_id: Mapped[int | None] = mapped_column(ForeignKey("events.id"), nullable=True)
resolved_anchor_id: Mapped[int | None] = mapped_column(
ForeignKey("anchors.id"), nullable=True
)
@@ -356,6 +359,39 @@ def clear_events(source: str | None = None) -> int:
return int(result.rowcount or 0)
+def get_event(event_id: int) -> Event | None:
+ with session_scope() as s:
+ return s.get(Event, event_id)
+
+
+def update_event(
+ event_id: int,
+ event_type: str | None = None,
+ source: str | None = None,
+ description: str | None = None,
+ confidence: float | None = None,
+) -> Event:
+ """Patch an existing event in place (M8 correction). Only provided fields change.
+
+ Returns the updated row; raises ``KeyError`` if the id is unknown.
+ """
+ with session_scope() as s:
+ event = s.get(Event, event_id)
+ if event is None:
+ raise KeyError(f"no event with id={event_id}")
+ if event_type is not None:
+ event.event_type = event_type
+ if source is not None:
+ event.source = source
+ if description is not None:
+ event.description = description
+ if confidence is not None:
+ event.confidence = confidence
+ s.flush()
+ s.refresh(event)
+ return event
+
+
# ---------------------------------------------------------------------------
# Annotations
# ---------------------------------------------------------------------------
@@ -366,6 +402,7 @@ def add_annotation(
y0: float,
x1: float,
y1: float,
+ event_id: int | None = None,
resolved_anchor_id: int | None = None,
) -> Annotation:
with session_scope() as s:
@@ -373,6 +410,7 @@ def add_annotation(
video_id=video_id,
t_video_s=t_video_s,
x0=x0, y0=y0, x1=x1, y1=y1,
+ event_id=event_id,
resolved_anchor_id=resolved_anchor_id,
)
s.add(annotation)
diff --git a/backend/festival4d/resolve.py b/backend/festival4d/resolve.py
new file mode 100644
index 0000000..bfd393f
--- /dev/null
+++ b/backend/festival4d/resolve.py
@@ -0,0 +1,152 @@
+"""Annotation -> 3D point resolution (spec M8, integration phase).
+
+Given a 2D bounding box drawn on one video at a local video time, recover a 3D scene point:
+
+ 1. Cast a ray from that camera through the bbox center, using the pose interpolated at
+ ``t_video_s`` and the stored pinhole intrinsics.
+ 2. If another annotation for the **same event** exists from a **different video**,
+ triangulate the two rays (reject near-parallel rays or a mutual gap > 0.5 scene units).
+ 3. Else take the nearest point-cloud point to the ray (within a 0.3-unit cylinder).
+ 4. Else return the ray at depth = distance from the camera to the point-cloud centroid.
+
+This module is pure orchestration over lane B's frozen geometry primitives
+(:func:`geometry.ray_from_pixel`, :func:`geometry.triangulate_rays`,
+:func:`geometry.nearest_point_on_ray`, :func:`geometry.slerp_pose`) — it never reimplements
+the pose math. It reads the point cloud via :func:`synthetic.read_ply`.
+"""
+
+from __future__ import annotations
+
+import logging
+from dataclasses import dataclass
+
+import numpy as np
+
+from festival4d import config, db, geometry
+from festival4d.synthetic import read_ply
+
+log = logging.getLogger("festival4d.resolve")
+
+TRIANGULATION_MAX_GAP = 0.5 # scene units — reject looser two-view intersections (spec M8)
+NEAREST_RADIUS = 0.3 # scene units — cylinder for the point-cloud fallback (spec M8)
+PARALLEL_DOT = 0.9995 # |cos(angle)| above this => rays too parallel to triangulate
+
+
+@dataclass
+class Resolution:
+ """Outcome of resolving one annotation to a 3D point."""
+
+ point: list[float] | None
+ method: str # 'triangulated' | 'nearest_point' | 'ray_depth' | 'no_geometry'
+ gap: float | None = None # mutual-approach distance when method == 'triangulated'
+
+
+def _pose_tuple(p) -> tuple[list[float], list[float], tuple[float, float, float, float]]:
+ return (
+ [p.qw, p.qx, p.qy, p.qz],
+ [p.tx, p.ty, p.tz],
+ (p.fx, p.fy, p.cx, p.cy),
+ )
+
+
+def _pose_at(poses, t_video_s: float):
+ """Interpolate a COLMAP pose ``(q, t, (fx,fy,cx,cy))`` at ``t_video_s``.
+
+ ``poses`` is ascending by ``t_video_s`` (as returned by :func:`db.get_poses`). Clamps to
+ the endpoints — no extrapolation, matching the frontend ``poseAt`` and spec M2.
+ """
+ if not poses:
+ return None
+ if t_video_s <= poses[0].t_video_s:
+ return _pose_tuple(poses[0])
+ if t_video_s >= poses[-1].t_video_s:
+ return _pose_tuple(poses[-1])
+
+ lo, hi = 0, len(poses) - 1
+ while hi - lo > 1:
+ mid = (lo + hi) // 2
+ if poses[mid].t_video_s <= t_video_s:
+ lo = mid
+ else:
+ hi = mid
+ a, b = poses[lo], poses[hi]
+ span = b.t_video_s - a.t_video_s
+ alpha = (t_video_s - a.t_video_s) / span if span > 1e-9 else 0.0
+ q, t = geometry.slerp_pose(
+ [a.qw, a.qx, a.qy, a.qz], [a.tx, a.ty, a.tz],
+ [b.qw, b.qx, b.qy, b.qz], [b.tx, b.ty, b.tz],
+ alpha,
+ )
+ # Intrinsics don't interpolate meaningfully across a fixed camera; use the earlier pose's.
+ return list(q), list(t), (a.fx, a.fy, a.cx, a.cy)
+
+
+def _ray_for_annotation(video_id: int, t_video_s: float, bbox):
+ """World-space ray (origin, unit direction) through the bbox center, or ``None``.
+
+ Returns ``None`` when the video has no poses (reconstruction absent for that camera).
+ """
+ video = db.get_video(video_id)
+ if video is None:
+ raise KeyError(f"no video with id={video_id}")
+ pose = _pose_at(db.get_poses(video_id), t_video_s)
+ if pose is None:
+ return None
+ q, t, (fx, fy, cx, cy) = pose
+ x0, y0, x1, y1 = bbox
+ px = (x0 + x1) / 2.0 * video.width
+ py = (y0 + y1) / 2.0 * video.height
+ return geometry.ray_from_pixel(q, t, fx, fy, cx, cy, px, py)
+
+
+def _load_points() -> np.ndarray | None:
+ if not config.POINTS_PLY.exists():
+ return None
+ try:
+ pts, _colors = read_ply(config.POINTS_PLY)
+ except Exception as exc: # pragma: no cover - defensive
+ log.warning("resolve: failed to read point cloud: %s", exc)
+ return None
+ return np.asarray(pts, dtype=np.float64).reshape(-1, 3)
+
+
+def resolve_annotation(video_id: int, t_video_s: float, bbox, event_id: int | None = None) -> Resolution:
+ """Resolve one annotation to a 3D point (see module docstring for the cascade)."""
+ ray = _ray_for_annotation(video_id, t_video_s, bbox)
+ if ray is None:
+ return Resolution(point=None, method="no_geometry")
+ origin, direction = ray
+
+ # 1. Two-view triangulation against another annotation of the same event.
+ if event_id is not None:
+ for other in reversed(db.get_annotations()): # most recent first
+ if other.event_id != event_id or other.video_id == video_id:
+ continue
+ other_ray = _ray_for_annotation(
+ other.video_id, other.t_video_s, [other.x0, other.y0, other.x1, other.y1]
+ )
+ if other_ray is None:
+ continue
+ o2, d2 = other_ray
+ if abs(float(np.dot(direction, d2))) >= PARALLEL_DOT:
+ continue # near-parallel: no reliable intersection
+ point, gap = geometry.triangulate_rays(origin, direction, o2, d2)
+ if gap <= TRIANGULATION_MAX_GAP:
+ return Resolution(
+ point=[float(v) for v in point], method="triangulated", gap=float(gap)
+ )
+
+ # 2/3. Point-cloud fallbacks.
+ pts = _load_points()
+ if pts is not None and len(pts):
+ near = geometry.nearest_point_on_ray(origin, direction, pts, NEAREST_RADIUS)
+ if near is not None:
+ return Resolution(point=[float(v) for v in near], method="nearest_point")
+ centroid = pts.mean(axis=0)
+ depth = float(np.linalg.norm(centroid - origin))
+ pt = origin + direction * depth
+ return Resolution(point=[float(v) for v in pt], method="ray_depth")
+
+ # 4. No point cloud at all: park at unit depth so a usable anchor still comes back.
+ pt = origin + direction * 1.0
+ return Resolution(point=[float(v) for v in pt], method="ray_depth")
diff --git a/backend/tests/test_api.py b/backend/tests/test_api.py
index 2aea65e..5eff676 100644
--- a/backend/tests/test_api.py
+++ b/backend/tests/test_api.py
@@ -103,10 +103,26 @@ def test_detect_events_endpoint(client):
assert client.post("/api/events/detect", json={}).status_code == 200 # idempotent
-def test_annotation_stored(client):
+def test_annotation_resolves_to_anchor(client):
+ # M8 (integration): a stored annotation now resolves to a 3D anchor. Single-view (no
+ # event_id) falls back to the nearest point-cloud point / centroid-depth ray, so an
+ # anchor + point come back. (Foundation asserted the stub contract anchor_id is None;
+ # superseded by M8 landing — same situation as CR-1.)
data = client.post(
"/api/annotations",
json={"video_id": 1, "t_video_s": 0.5, "bbox": [0.4, 0.4, 0.6, 0.6]},
).json()
assert data["annotation_id"] > 0
- assert data["anchor_id"] is None # M8 resolution is integration work
+ assert data["anchor_id"] is not None
+ assert data["point"] is not None and len(data["point"]) == 3
+ assert data["method"] in {"nearest_point", "ray_depth", "triangulated"}
+ assert data["anchor"]["id"] == data["anchor_id"]
+
+
+def test_event_patch_sets_user_source(client):
+ events = client.get("/api/events").json()
+ eid = events[0]["id"]
+ patched = client.patch(f"/api/events/{eid}", json={"event_type": "pyro"}).json()
+ assert patched["event_type"] == "pyro"
+ assert patched["source"] == "user" # user correction attribution
+ assert client.patch("/api/events/99999", json={"event_type": "x"}).status_code == 404
diff --git a/backend/tests/test_resolve.py b/backend/tests/test_resolve.py
new file mode 100644
index 0000000..3d40933
--- /dev/null
+++ b/backend/tests/test_resolve.py
@@ -0,0 +1,127 @@
+"""M8 annotation -> 3D resolution tests (integration phase).
+
+The headline acceptance (spec M8): annotating the *same* stage corner from two videos
+produces an anchor within 0.2 scene units of the true corner. We build the synthetic fixture
+without ffmpeg (``run_ffmpeg=False`` populates the DB + point cloud only), forward-project a
+known corner into each camera to get faithful bbox centers, then resolve and check the error.
+
+Also covers: the single-view nearest-point fallback, the near-parallel/gap rejection path,
+and the ``update_event`` correction helper.
+"""
+
+from __future__ import annotations
+
+import numpy as np
+import pytest
+
+from festival4d import config, db, resolve, synthetic
+from festival4d.geometry import quat_to_mat
+
+
+@pytest.fixture(scope="module")
+def fixture():
+ # No ffmpeg: this writes videos/poses/anchors/events into the DB and points.ply to WORK_DIR.
+ synthetic.build(run_ffmpeg=False)
+ return synthetic.STAGE_CORNERS
+
+
+def _project(video, pose, world_pt):
+ """Forward-project a world point into a video, returning (px, py, z_cam).
+
+ Mirrors the COLMAP pinhole the fixture uses: x_cam = R@X + t; px = fx*x/z + cx (y-down).
+ """
+ q = [pose.qw, pose.qx, pose.qy, pose.qz]
+ t = np.array([pose.tx, pose.ty, pose.tz])
+ R = quat_to_mat(q)
+ x_cam = R @ np.asarray(world_pt, dtype=float) + t
+ z = x_cam[2]
+ px = pose.fx * x_cam[0] / z + pose.cx
+ py = pose.fy * x_cam[1] / z + pose.cy
+ return px, py, z
+
+
+def _bbox_center(px, py, W, H, half=0.02):
+ """A small normalized bbox centered on the projected pixel (clamped to [0,1])."""
+ cx, cy = px / W, py / H
+ return [
+ max(0.0, cx - half), max(0.0, cy - half),
+ min(1.0, cx + half), min(1.0, cy + half),
+ ]
+
+
+def _views_seeing(corner):
+ """Videos whose first pose projects `corner` in front of the camera and inside the frame."""
+ seen = []
+ for v in db.get_videos():
+ poses = db.get_poses(v.id)
+ if not poses:
+ continue
+ p0 = poses[0]
+ px, py, z = _project(v, p0, corner)
+ if z > 0 and 0 <= px < v.width and 0 <= py < v.height:
+ seen.append((v, p0, px, py))
+ return seen
+
+
+def test_two_view_triangulation_recovers_corner(fixture):
+ corners = fixture
+ # Pick a corner visible in at least two cameras.
+ chosen = None
+ for label, world in corners.items():
+ views = _views_seeing(world)
+ if len(views) >= 2:
+ chosen = (label, np.asarray(world, dtype=float), views)
+ break
+ assert chosen is not None, "no stage corner is visible in >=2 synthetic cameras"
+ label, world, views = chosen
+
+ event = db.add_event(t_global_s=5.0, event_type="artist_moment", source="ai",
+ description=f"located {label}")
+
+ # Annotate the corner from the first two cameras that see it, then resolve the second one
+ # (which triangulates against the first).
+ (v_a, pose_a, pxa, pya), (v_b, pose_b, pxb, pyb) = views[0], views[1]
+ bbox_a = _bbox_center(pxa, pya, v_a.width, v_a.height)
+ bbox_b = _bbox_center(pxb, pyb, v_b.width, v_b.height)
+
+ db.add_annotation(v_a.id, pose_a.t_video_s, *bbox_a, event_id=event.id)
+ res = resolve.resolve_annotation(v_b.id, pose_b.t_video_s, bbox_b, event_id=event.id)
+
+ assert res.method == "triangulated", f"expected triangulation, got {res.method}"
+ err = float(np.linalg.norm(np.asarray(res.point) - world))
+ assert err < 0.2, f"triangulated {label} off by {err:.3f} units (>0.2)"
+ # Sanity: the mutual-approach gap is tight for exact projections.
+ assert res.gap is not None and res.gap < 0.2
+
+
+def test_single_view_falls_back_to_point_cloud(fixture):
+ corners = fixture
+ world = np.asarray(corners["Stage FL"], dtype=float)
+ views = _views_seeing(world)
+ assert views, "Stage FL should be visible in at least one camera"
+ v, pose, px, py = views[0]
+ bbox = _bbox_center(px, py, v.width, v.height)
+
+ # No event_id => no other view to triangulate => nearest point-cloud point (corner is in
+ # the cloud) or centroid-depth ray. Either way a point comes back.
+ res = resolve.resolve_annotation(v.id, pose.t_video_s, bbox, event_id=None)
+ assert res.method in {"nearest_point", "ray_depth"}
+ assert res.point is not None and len(res.point) == 3
+ if res.method == "nearest_point":
+ # The exact corner is seeded into the cloud, so the nearest hit should be very close.
+ assert float(np.linalg.norm(np.asarray(res.point) - world)) < resolve.NEAREST_RADIUS
+
+
+def test_update_event_sets_user_source(fixture):
+ ev = db.add_event(t_global_s=9.0, event_type="candidate", source="audio_auto",
+ description="unlabeled")
+ updated = db.update_event(ev.id, event_type="pyro", source="user")
+ assert updated.event_type == "pyro"
+ assert updated.source == "user"
+ # Unspecified fields are preserved.
+ assert updated.description == "unlabeled"
+
+
+def test_missing_video_raises(fixture):
+ with pytest.raises(KeyError):
+ resolve.resolve_annotation(9999, 0.0, [0.4, 0.4, 0.6, 0.6], event_id=None)
diff --git a/frontend/index.html b/frontend/index.html
index 6df6c84..c2f70cb 100644
--- a/frontend/index.html
+++ b/frontend/index.html
@@ -42,7 +42,11 @@
#scene3d { position: absolute; inset: 0; width: 100%; height: 100%; display: block; }
#scene-controls {
position: absolute; top: 0.6rem; left: 0.6rem; display: flex; gap: 0.4rem; z-index: 5;
+ align-items: center; flex-wrap: wrap; max-width: calc(100% - 260px);
}
+ #scene-controls .btn { padding: 0.25rem 0.5rem; font-size: 0.78rem; }
+ #key-count { color: var(--accent); font-variant-numeric: tabular-nums; }
+ .ctl-sep { width: 1px; height: 18px; background: var(--border); margin: 0 0.15rem; }
#scene-hint {
position: absolute; bottom: 0.5rem; left: 0.6rem; color: var(--dim);
font-size: 0.72rem; z-index: 5; pointer-events: none;
@@ -150,6 +154,37 @@
.bad { color: var(--bad); }
.dim { color: var(--dim); }
+ /* M8 annotation correction panel */
+ #annot-panel {
+ position: absolute; top: 0.6rem; right: 0.6rem; width: 232px; z-index: 8;
+ background: rgba(15,20,32,0.94); border: 1px solid var(--border); border-radius: 9px;
+ font-size: 0.78rem; box-shadow: 0 8px 26px rgba(0,0,0,0.5); backdrop-filter: blur(5px);
+ }
+ .ap-head {
+ display: flex; align-items: center; gap: 0.4rem; padding: 0.45rem 0.55rem;
+ border-bottom: 1px solid var(--border);
+ }
+ .ap-dot { width: 9px; height: 9px; border-radius: 3px; flex: 0 0 auto; }
+ .ap-head strong { flex: 1 1 auto; font-size: 0.8rem; }
+ .ap-x {
+ background: none; border: none; color: var(--dim); cursor: pointer; font-size: 0.85rem;
+ padding: 0 0.2rem;
+ }
+ .ap-x:hover { color: var(--text); }
+ .ap-body { padding: 0.5rem 0.55rem 0.6rem; display: flex; flex-direction: column; gap: 0.45rem; }
+ .ap-row { display: flex; align-items: center; justify-content: space-between; gap: 0.4rem; color: var(--dim); }
+ .ap-type { flex: 1 1 auto; margin-left: 0.4rem; background: var(--panel-2); color: var(--text);
+ border: 1px solid var(--border); border-radius: 5px; padding: 0.2rem 0.3rem; text-transform: capitalize; }
+ .ap-desc { color: var(--text); line-height: 1.35; }
+ .ap-meta { text-transform: capitalize; }
+ .ap-annotate { width: 100%; text-align: center; }
+ .ap-hint { line-height: 1.35; min-height: 1.1em; }
+
+ .annot-rect {
+ position: absolute; border: 1.5px solid var(--accent); background: rgba(89,212,153,0.14);
+ pointer-events: none; z-index: 6; border-radius: 2px;
+ }
+
/* Loading / error */
#loading {
position: fixed; inset: 0; display: flex; align-items: center; justify-content: center;
@@ -171,8 +206,16 @@
+
+
+
+
+
+
+
-
drag to orbit · click a camera or press 1–9 to snap · Esc to detach
+
drag to orbit · 1–9 snap to camera · K add keyframe · Esc detach
+
diff --git a/frontend/src/annotate.js b/frontend/src/annotate.js
index 3aab307..c5fec8f 100644
--- a/frontend/src/annotate.js
+++ b/frontend/src/annotate.js
@@ -1,6 +1,262 @@
-// bbox drawing + submission → POST /api/annotations (spec M8).
-// STUB — this is phase-3 (integration) work; lane C leaves it as a clean seam. Drag a
-// rectangle on a video's overlay canvas → normalized bbox → POST /api/annotations; the
-// resolved anchor then appears live in overlays + the 3D scene.
+// Correction panel + bbox annotation → 3D (spec M8, integration phase).
+//
+// Clicking a timeline event opens a panel: it shows the classification, a dropdown to correct
+// the event_type (PATCH /api/events/{id}, source→'user'), and an "annotate location" mode.
+// In annotate mode the user drags a rectangle on ANY video's overlay; the normalized bbox is
+// POSTed to /api/annotations. The backend casts a ray through the bbox center and resolves a
+// 3D point (triangulate across two views of the same event, else nearest point-cloud point /
+// centroid-depth ray). The resolved anchor is pushed live into state so overlays + the 3D
+// scene show it immediately.
-export {};
+import { state, emit, API_BASE } from "./state.js";
+import { contentRect } from "./videoGrid.js";
+import { eventColor } from "./timeline.js";
+
+const EVENT_TYPES = [
+ "bass_drop", "pyro", "confetti", "crowd_wave",
+ "artist_moment", "light_show", "quiet_moment", "candidate", "other",
+];
+
+export class Annotator {
+ constructor() {
+ this.panel = null;
+ this.cells = [];
+ this.currentEvent = null;
+ this.annotating = false;
+ this._rubber = null; // { cell, div, sx, sy }
+ this._cleanup = [];
+ }
+
+ init(panelEl, cells) {
+ this.panel = panelEl;
+ this.cells = cells;
+ this.panel.style.display = "none";
+ }
+
+ /** Open the correction panel for a timeline event. */
+ openForEvent(ev) {
+ this.currentEvent = ev;
+ this.stopAnnotateMode();
+ this._render();
+ this.panel.style.display = "block";
+ }
+
+ close() {
+ this.stopAnnotateMode();
+ this.currentEvent = null;
+ if (this.panel) this.panel.style.display = "none";
+ }
+
+ _render() {
+ const ev = this.currentEvent;
+ if (!ev) return;
+ const options = EVENT_TYPES.map(
+ (t) => ``
+ ).join("");
+ this.panel.innerHTML = `
+