From e5d6b2c47fe409a41d4531136af92e32973b2745 Mon Sep 17 00:00:00 2001 From: m3ultra Date: Thu, 16 Jul 2026 09:59:51 +1000 Subject: [PATCH] =?UTF-8?q?Integration=20(M8=20+=20M9):=20annotation?= =?UTF-8?q?=E2=86=923D,=20camera=20paths,=20README,=20v0.1.0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit M8: POST /api/annotations resolves a bbox to a 3D point (ray → two-view triangulation / nearest-point / centroid-depth) via lane B's geometry primitives; correction panel + bbox-drawing UI; resolved anchors render live in overlays and the 3D scene. CR-2 (additive): annotations.event_id + PATCH /api/events/{id} + db.update_event. M9: keyframed god's-eye camera paths (Catmull-Rom position + slerp orientation, export/import), gizmos, path-mode camera hooks. Verified: 101 backend tests pass (two-view corner recovery err 0.00000); frontend build clean; live in-browser M8 triangulation (gap 0.21) + M9 path (camera on keyframe at t) + M4 sync ~16ms. Cleanup: worktrees removed, debris stash dropped, pristine demo DB. Co-Authored-By: Claude Opus 4.8 --- .gitignore | 1 + README.md | 178 ++++++++++++++++++++--- backend/festival4d/api.py | 66 +++++++-- backend/festival4d/db.py | 38 +++++ backend/festival4d/resolve.py | 152 +++++++++++++++++++ backend/tests/test_api.py | 20 ++- backend/tests/test_resolve.py | 127 ++++++++++++++++ frontend/index.html | 45 +++++- frontend/src/annotate.js | 266 +++++++++++++++++++++++++++++++++- frontend/src/camPath.js | 170 +++++++++++++++++++++- frontend/src/main.js | 78 +++++++++- frontend/src/scene3d.js | 99 ++++++++++++- frontend/src/timeline.js | 14 +- plan/CHANGE_REQUESTS.md | 27 +++- plan/status/integration.md | 66 +++++++++ 15 files changed, 1295 insertions(+), 52 deletions(-) create mode 100644 backend/festival4d/resolve.py create mode 100644 backend/tests/test_resolve.py create mode 100644 plan/status/integration.md diff --git a/.gitignore b/.gitignore index 74b3d59..e98ece7 100644 --- a/.gitignore +++ b/.gitignore @@ -5,3 +5,4 @@ node_modules/ __pycache__/ .venv/ .DS_Store +frontend/dist/ diff --git a/README.md b/README.md index 6658709..260aa3d 100644 --- a/README.md +++ b/README.md @@ -3,55 +3,187 @@ Turn multiple fan-shot smartphone videos of the same concert into a synchronized, explorable **4D experience**: time-aligned multi-video playback, 3D scene reconstruction with camera poses, a free-roam "god's eye" viewer, AR-style overlays projected onto each -video, and AI-tagged moments on a shared timeline. +video, AI-tagged moments on a shared timeline, click-to-place 3D annotations, and keyframed +cinematic fly-throughs. -> **Status: foundation phase.** The scaffold, database, frozen contracts (API, pose math, -> DB schema), and a synthetic fixture generator are in place. Feature lanes (media sync, -> reconstruction, viewer, AI events) build on top. The full README with the real-footage -> workflow is written in the integration phase (M9). See -> [`OPUS_BUILD_INSTRUCTIONS.md`](OPUS_BUILD_INSTRUCTIONS.md) for the canonical spec and -> [`plan/`](plan/) for the execution plan. +Everything works **from pixels and audio alone** — no depth sensors, no IMU logs. It runs +fully offline and local; the only optional cloud piece is moment classification. + +--- + +## What you can do + +- **Scrub a shared timeline** and watch every camera stay locked to the same instant, aligned + by their audio (no manual frame-matching). +- **Fly through the reconstructed scene** in 3D — orbit freely, snap to any real camera, or + play a keyframed cinematic path. +- **See an "x-ray" HUD**: 3D anchor points (stage corners, tagged objects) projected onto every + video, tracking even when occluded by crowd or scaffolding. +- **Jump between AI-tagged moments** (bass drops, pyro, crowd waves…) on the timeline, and + correct any misclassification. +- **Click to locate things in 3D**: draw a box around an object in one or two videos and the + app triangulates its 3D position, dropping an anchor that appears everywhere at once. + +--- ## Prerequisites -- **Python 3.11+** and [`uv`](https://docs.astral.sh/uv/) (or venv + pip) -- **ffmpeg** / **ffprobe** on your `PATH` (required) -- **COLMAP** (optional — reconstruction degrades gracefully without it) -- A classifier API key (optional — `GEMINI_API_KEY` by default; see `events_ai.py`) -- **Node 18+** for the frontend +| Tool | Required? | Notes | +|---|---|---| +| **Python 3.11+** + [`uv`](https://docs.astral.sh/uv/) (or venv+pip) | yes | backend | +| **ffmpeg** / **ffprobe** on `PATH` | yes | audio extraction, frame sampling, clip cutting | +| **Node 18+** | yes | frontend (Vite + Three.js) | +| **COLMAP** on `PATH` | optional | 3D reconstruction. Without it the app still runs — you get synced videos + timeline, just no 3D scene. `brew install colmap` on macOS. | +| A classifier API key | optional | AI moment labels. Default **Gemini 2.5 Flash** (`GEMINI_API_KEY`); also Claude or any OpenAI-compatible/local/OpenRouter endpoint via `FESTIVAL4D_CLASSIFIER`. Without a key you still get audio-detected candidate moments, just unlabeled. | + +--- ## Quickstart (synthetic demo — no footage needed) +The synthetic fixture generates three fake camera videos of the same fake stage, with known +audio offsets and known 3D geometry — so you can see the whole app working end-to-end before +you have any real footage. + ```bash # 1. backend env uv venv --python 3.12 uv pip install -e ".[dev]" -# 2. generate the synthetic fixture project (fake videos + poses + point cloud + events) +# 2. generate the synthetic project (fake videos + poses + point cloud + seeded events) uv run python -m festival4d synthetic # 3. serve the API (http://127.0.0.1:8000) uv run python -m festival4d serve -# 4. in another terminal, the frontend +# 4. in another terminal, run the frontend cd frontend npm install -npm run dev # http://localhost:5173 +npm run dev # opens http://localhost:5173 (or the next free port) ``` -## Backend CLI +Open the printed URL. You should see the 3D scene with three camera frusta around a stage box, +the three synced video players, stage-corner anchors projected onto each video, and colored +moment markers on the timeline. + +--- + +## Real-footage workflow + +1. **Shoot / gather** 2–4 videos of the same performance from different positions (see + *Shooting tips* below) and drop the files into `data/raw/`. +2. **Run the pipeline:** + + ```bash + uv run python -m festival4d ingest # probe videos, extract 16 kHz mono audio + uv run python -m festival4d sync # GCC-PHAT audio alignment (offsets + drift) + uv run python -m festival4d reconstruct # COLMAP structure-from-motion → camera poses + point cloud + uv run python -m festival4d events # detect audio moments + AI-classify them + uv run python -m festival4d serve # serve it + ``` + + Each step is independent and re-runnable. `sync` alone already gives you locked multi-video + playback; `reconstruct` adds the 3D scene and overlays; `events` adds the tagged timeline. +3. **Explore** in the browser (`cd frontend && npm run dev`). + +If COLMAP can't reconstruct your footage (common with dark, motion-blurred, or low-overlap +clips), the pipeline says so and leaves you with the synced-video experience — it never +corrupts existing data. + +To enable AI moment labels, set a key before `events`: + +```bash +export GEMINI_API_KEY=... # default provider (Gemini 2.5 Flash, native video) +# or: export FESTIVAL4D_CLASSIFIER=claude ANTHROPIC_API_KEY=... +# or: export FESTIVAL4D_CLASSIFIER=local FESTIVAL4D_OPENAI_BASE_URL=... FESTIVAL4D_OPENAI_MODEL=... +``` + +--- + +## Using the viewer + +| Action | How | +|---|---| +| Play / pause | `Space` or the ⏵ button | +| Nudge time ±1 s (±5 s) | `←` / `→` (hold `Shift`) | +| Scrub | drag the timeline | +| Pick the audio you hear | click a video (only one is unmuted at a time) | +| Enable/disable a video | ◉ on the video | +| Orbit the 3D scene | drag in the 3D pane | +| Snap the 3D view to a camera | click its frustum, its ⛶ button, or press `1`–`9` | +| Free roam again | `Esc` / `0` / **Free roam** | +| Correct a moment / annotate | click a timeline marker → panel | +| Place a 3D anchor | in the panel, **Annotate location**, then drag a box on a video (repeat on a 2nd video to triangulate) | +| Add a camera keyframe | **＋ Key** or `K` (captures the current free-roam view at the current time) | +| Play the keyframed path | **▶ Path** · export/import with ⤓ / ⤒ | + +The top-right **sync error** panel shows each video's alignment error in ms while playing — +a dev aid; it should stay green (<50 ms) once footage is well-synced. + +--- + +## Architecture ``` -python -m festival4d synthetic # generate the synthetic fixture (M0) -python -m festival4d ingest # probe videos + extract audio (lane A / M1) -python -m festival4d sync # GCC-PHAT audio alignment (lane A / M1) -python -m festival4d reconstruct # COLMAP SfM + pose export (lane B / M2) -python -m festival4d events # audio candidates + AI classify (lane D / M7) -python -m festival4d serve # FastAPI app (M3) +data/raw/ ──ingest──▶ data/work/audio ──sync────▶ videos.offset_ms / drift_ppm ┐ + │ + ──frames──▶ data/work/frames ──reconstruct──▶ camera_poses + points.ply + │ + reference audio ──events──▶ events (audio candidates → AI labels) + │ + SQLite (data/project.db) ◀─────────┘ + │ + FastAPI (backend/festival4d/api.py) + │ /api/manifest, /poses, /pointcloud, + │ /events, /anchors, /annotations + ▼ + Vite + Three.js SPA (frontend/src) + transport (master clock) · videoGrid + overlays · scene3d · timeline + annotate (M8) · camPath (M9) ``` +- **Backend** — Python: `ingest`/`audio_sync` (media + GCC-PHAT sync), `frames`/`sfm`/`geometry` + (COLMAP orchestration + pose math + triangulation), `events_ai` (moment detection + pluggable + classifiers), `resolve` (annotation→3D), `api` (FastAPI, Range-capable video serving), `db` + (SQLAlchemy/SQLite), `synthetic` (the test fixture). +- **Frontend** — vanilla JS + Three.js. A single master clock (`transport.js`) derives each + video's local time from the shared timeline and continuously corrects playback; `lib/pose.js` + is the one COLMAP→Three.js pose conversion, mirrored from `geometry.py` and locked by a shared + test. +- **Timebase contract:** `t_video = (t_global − offset_ms/1000) · (1 + drift_ppm·1e−6)`; the + reference video has offset 0. + +Full design + milestone history: [`OPUS_BUILD_INSTRUCTIONS.md`](OPUS_BUILD_INSTRUCTIONS.md); +the parallel build plan and per-lane status live in [`plan/`](plan/). + +--- + +## Shooting tips (for good reconstructions) + +Concert footage is genuinely hard for structure-from-motion. To give COLMAP a chance: + +- **Spread the cameras out** but keep **overlapping views** — each pair of cameras should see + some of the same stage/structure. No overlap → they can't be related in 3D. +- **Keep some static structure in frame** (stage edges, truss, speaker stacks). A frame that's + all moving crowd and lights has nothing stable to triangulate. +- **Avoid pure zoom** — physically moving parallax reconstructs far better than zooming. +- **Brighter, sharper is better** — motion blur and near-dark frames are the main failure cause. +- **2–4 phones is the sweet spot** for a first reconstruction; more is fine but slower. + +Audio sync is far more forgiving: any clips that share audible sound (the same music/claps) +will align, even across otherwise-unrelated angles. + +--- + ## Tests ```bash -uv run pytest +uv run pytest # backend +cd frontend && npm run build # frontend typecheck/build ``` + +--- + +## Non-goals (this prototype) + +No accounts/auth, no cloud storage, no realtime ingest, no NeRF/Gaussian splatting, no mobile +UI, no Docker. Single local user, one project at a time. diff --git a/backend/festival4d/api.py b/backend/festival4d/api.py index dbaf02b..316a6f4 100644 --- a/backend/festival4d/api.py +++ b/backend/festival4d/api.py @@ -73,6 +73,14 @@ class AnnotationIn(BaseModel): video_id: int t_video_s: float bbox: list[float] = Field(min_length=4, max_length=4) # [x0, y0, x1, y1] normalized + event_id: int | None = None # CR-2: link to the event this annotation locates (M8) + + +class EventPatch(BaseModel): + event_type: str | None = None + source: str | None = None + description: str | None = None + confidence: float | None = None # --------------------------------------------------------------------------- @@ -191,19 +199,57 @@ def detect_events(payload: DetectIn) -> dict: return {"result": result, "events": [_event_dict(e) for e in db.get_events()]} +@app.patch("/api/events/{event_id}") +def patch_event(event_id: int, payload: EventPatch) -> dict: + """Correct an event in place (M8). A user correction defaults ``source`` to ``'user'``.""" + if db.get_event(event_id) is None: + raise HTTPException(status_code=404, detail=f"no event with id={event_id}") + source = payload.source + if source is None and (payload.event_type is not None or payload.description is not None): + source = "user" # any user-driven correction is attributed to the user + event = db.update_event( + event_id, + event_type=payload.event_type, + source=source, + description=payload.description, + confidence=payload.confidence, + ) + return _event_dict(event) + + @app.post("/api/annotations") def create_annotation(payload: AnnotationIn) -> dict: - x0, y0, x1, y1 = payload.bbox - annotation = db.add_annotation(payload.video_id, payload.t_video_s, x0, y0, x1, y1) + from festival4d import resolve + + x0, y0, x1, y1 = payload.bbox + if db.get_video(payload.video_id) is None: + raise HTTPException(status_code=404, detail=f"no video with id={payload.video_id}") + annotation = db.add_annotation( + payload.video_id, payload.t_video_s, x0, y0, x1, y1, event_id=payload.event_id + ) + + # M8: cast a ray through the bbox center; triangulate against another view of the same + # event, else fall back to the nearest point-cloud point / centroid depth. + result = resolve.resolve_annotation( + payload.video_id, payload.t_video_s, payload.bbox, event_id=payload.event_id + ) + + anchor = None + if result.point is not None: + label = "annotation" + color = None + if payload.event_id is not None: + ev = db.get_event(payload.event_id) + if ev is not None: + label = ev.description or ev.event_type or label + anchor = db.add_anchor(label, result.point[0], result.point[1], result.point[2], color) + db.set_annotation_anchor(annotation.id, anchor.id) - # M8 resolution (triangulate / nearest-point) is integration work; geometry stubs raise - # until lane B lands. Store the annotation now; resolve to an anchor later. - anchor_id = None - point = None - note = "annotation stored; 3D resolution is M8/integration work" return { "annotation_id": annotation.id, - "anchor_id": anchor_id, - "point": point, - "note": note, + "anchor_id": anchor.id if anchor else None, + "point": result.point, + "method": result.method, + "gap": result.gap, + "anchor": _anchor_dict(anchor) if anchor else None, } diff --git a/backend/festival4d/db.py b/backend/festival4d/db.py index 171cacf..ecd299e 100644 --- a/backend/festival4d/db.py +++ b/backend/festival4d/db.py @@ -120,6 +120,9 @@ class Annotation(Base): y0: Mapped[float] = mapped_column(Float, nullable=False) x1: Mapped[float] = mapped_column(Float, nullable=False) y1: Mapped[float] = mapped_column(Float, nullable=False) + # Nullable link to the event this annotation locates (CR-2, M8). Two annotations sharing an + # event_id from different videos triangulate to a 3D anchor. Nullable => additive/back-compat. + event_id: Mapped[int | None] = mapped_column(ForeignKey("events.id"), nullable=True) resolved_anchor_id: Mapped[int | None] = mapped_column( ForeignKey("anchors.id"), nullable=True ) @@ -356,6 +359,39 @@ def clear_events(source: str | None = None) -> int: return int(result.rowcount or 0) +def get_event(event_id: int) -> Event | None: + with session_scope() as s: + return s.get(Event, event_id) + + +def update_event( + event_id: int, + event_type: str | None = None, + source: str | None = None, + description: str | None = None, + confidence: float | None = None, +) -> Event: + """Patch an existing event in place (M8 correction). Only provided fields change. + + Returns the updated row; raises ``KeyError`` if the id is unknown. + """ + with session_scope() as s: + event = s.get(Event, event_id) + if event is None: + raise KeyError(f"no event with id={event_id}") + if event_type is not None: + event.event_type = event_type + if source is not None: + event.source = source + if description is not None: + event.description = description + if confidence is not None: + event.confidence = confidence + s.flush() + s.refresh(event) + return event + + # --------------------------------------------------------------------------- # Annotations # --------------------------------------------------------------------------- @@ -366,6 +402,7 @@ def add_annotation( y0: float, x1: float, y1: float, + event_id: int | None = None, resolved_anchor_id: int | None = None, ) -> Annotation: with session_scope() as s: @@ -373,6 +410,7 @@ def add_annotation( video_id=video_id, t_video_s=t_video_s, x0=x0, y0=y0, x1=x1, y1=y1, + event_id=event_id, resolved_anchor_id=resolved_anchor_id, ) s.add(annotation) diff --git a/backend/festival4d/resolve.py b/backend/festival4d/resolve.py new file mode 100644 index 0000000..bfd393f --- /dev/null +++ b/backend/festival4d/resolve.py @@ -0,0 +1,152 @@ +"""Annotation -> 3D point resolution (spec M8, integration phase). + +Given a 2D bounding box drawn on one video at a local video time, recover a 3D scene point: + + 1. Cast a ray from that camera through the bbox center, using the pose interpolated at + ``t_video_s`` and the stored pinhole intrinsics. + 2. If another annotation for the **same event** exists from a **different video**, + triangulate the two rays (reject near-parallel rays or a mutual gap > 0.5 scene units). + 3. Else take the nearest point-cloud point to the ray (within a 0.3-unit cylinder). + 4. Else return the ray at depth = distance from the camera to the point-cloud centroid. + +This module is pure orchestration over lane B's frozen geometry primitives +(:func:`geometry.ray_from_pixel`, :func:`geometry.triangulate_rays`, +:func:`geometry.nearest_point_on_ray`, :func:`geometry.slerp_pose`) — it never reimplements +the pose math. It reads the point cloud via :func:`synthetic.read_ply`. +""" + +from __future__ import annotations + +import logging +from dataclasses import dataclass + +import numpy as np + +from festival4d import config, db, geometry +from festival4d.synthetic import read_ply + +log = logging.getLogger("festival4d.resolve") + +TRIANGULATION_MAX_GAP = 0.5 # scene units — reject looser two-view intersections (spec M8) +NEAREST_RADIUS = 0.3 # scene units — cylinder for the point-cloud fallback (spec M8) +PARALLEL_DOT = 0.9995 # |cos(angle)| above this => rays too parallel to triangulate + + +@dataclass +class Resolution: + """Outcome of resolving one annotation to a 3D point.""" + + point: list[float] | None + method: str # 'triangulated' | 'nearest_point' | 'ray_depth' | 'no_geometry' + gap: float | None = None # mutual-approach distance when method == 'triangulated' + + +def _pose_tuple(p) -> tuple[list[float], list[float], tuple[float, float, float, float]]: + return ( + [p.qw, p.qx, p.qy, p.qz], + [p.tx, p.ty, p.tz], + (p.fx, p.fy, p.cx, p.cy), + ) + + +def _pose_at(poses, t_video_s: float): + """Interpolate a COLMAP pose ``(q, t, (fx,fy,cx,cy))`` at ``t_video_s``. + + ``poses`` is ascending by ``t_video_s`` (as returned by :func:`db.get_poses`). Clamps to + the endpoints — no extrapolation, matching the frontend ``poseAt`` and spec M2. + """ + if not poses: + return None + if t_video_s <= poses[0].t_video_s: + return _pose_tuple(poses[0]) + if t_video_s >= poses[-1].t_video_s: + return _pose_tuple(poses[-1]) + + lo, hi = 0, len(poses) - 1 + while hi - lo > 1: + mid = (lo + hi) // 2 + if poses[mid].t_video_s <= t_video_s: + lo = mid + else: + hi = mid + a, b = poses[lo], poses[hi] + span = b.t_video_s - a.t_video_s + alpha = (t_video_s - a.t_video_s) / span if span > 1e-9 else 0.0 + q, t = geometry.slerp_pose( + [a.qw, a.qx, a.qy, a.qz], [a.tx, a.ty, a.tz], + [b.qw, b.qx, b.qy, b.qz], [b.tx, b.ty, b.tz], + alpha, + ) + # Intrinsics don't interpolate meaningfully across a fixed camera; use the earlier pose's. + return list(q), list(t), (a.fx, a.fy, a.cx, a.cy) + + +def _ray_for_annotation(video_id: int, t_video_s: float, bbox): + """World-space ray (origin, unit direction) through the bbox center, or ``None``. + + Returns ``None`` when the video has no poses (reconstruction absent for that camera). + """ + video = db.get_video(video_id) + if video is None: + raise KeyError(f"no video with id={video_id}") + pose = _pose_at(db.get_poses(video_id), t_video_s) + if pose is None: + return None + q, t, (fx, fy, cx, cy) = pose + x0, y0, x1, y1 = bbox + px = (x0 + x1) / 2.0 * video.width + py = (y0 + y1) / 2.0 * video.height + return geometry.ray_from_pixel(q, t, fx, fy, cx, cy, px, py) + + +def _load_points() -> np.ndarray | None: + if not config.POINTS_PLY.exists(): + return None + try: + pts, _colors = read_ply(config.POINTS_PLY) + except Exception as exc: # pragma: no cover - defensive + log.warning("resolve: failed to read point cloud: %s", exc) + return None + return np.asarray(pts, dtype=np.float64).reshape(-1, 3) + + +def resolve_annotation(video_id: int, t_video_s: float, bbox, event_id: int | None = None) -> Resolution: + """Resolve one annotation to a 3D point (see module docstring for the cascade).""" + ray = _ray_for_annotation(video_id, t_video_s, bbox) + if ray is None: + return Resolution(point=None, method="no_geometry") + origin, direction = ray + + # 1. Two-view triangulation against another annotation of the same event. + if event_id is not None: + for other in reversed(db.get_annotations()): # most recent first + if other.event_id != event_id or other.video_id == video_id: + continue + other_ray = _ray_for_annotation( + other.video_id, other.t_video_s, [other.x0, other.y0, other.x1, other.y1] + ) + if other_ray is None: + continue + o2, d2 = other_ray + if abs(float(np.dot(direction, d2))) >= PARALLEL_DOT: + continue # near-parallel: no reliable intersection + point, gap = geometry.triangulate_rays(origin, direction, o2, d2) + if gap <= TRIANGULATION_MAX_GAP: + return Resolution( + point=[float(v) for v in point], method="triangulated", gap=float(gap) + ) + + # 2/3. Point-cloud fallbacks. + pts = _load_points() + if pts is not None and len(pts): + near = geometry.nearest_point_on_ray(origin, direction, pts, NEAREST_RADIUS) + if near is not None: + return Resolution(point=[float(v) for v in near], method="nearest_point") + centroid = pts.mean(axis=0) + depth = float(np.linalg.norm(centroid - origin)) + pt = origin + direction * depth + return Resolution(point=[float(v) for v in pt], method="ray_depth") + + # 4. No point cloud at all: park at unit depth so a usable anchor still comes back. + pt = origin + direction * 1.0 + return Resolution(point=[float(v) for v in pt], method="ray_depth") diff --git a/backend/tests/test_api.py b/backend/tests/test_api.py index 2aea65e..5eff676 100644 --- a/backend/tests/test_api.py +++ b/backend/tests/test_api.py @@ -103,10 +103,26 @@ def test_detect_events_endpoint(client): assert client.post("/api/events/detect", json={}).status_code == 200 # idempotent -def test_annotation_stored(client): +def test_annotation_resolves_to_anchor(client): + # M8 (integration): a stored annotation now resolves to a 3D anchor. Single-view (no + # event_id) falls back to the nearest point-cloud point / centroid-depth ray, so an + # anchor + point come back. (Foundation asserted the stub contract anchor_id is None; + # superseded by M8 landing — same situation as CR-1.) data = client.post( "/api/annotations", json={"video_id": 1, "t_video_s": 0.5, "bbox": [0.4, 0.4, 0.6, 0.6]}, ).json() assert data["annotation_id"] > 0 - assert data["anchor_id"] is None # M8 resolution is integration work + assert data["anchor_id"] is not None + assert data["point"] is not None and len(data["point"]) == 3 + assert data["method"] in {"nearest_point", "ray_depth", "triangulated"} + assert data["anchor"]["id"] == data["anchor_id"] + + +def test_event_patch_sets_user_source(client): + events = client.get("/api/events").json() + eid = events[0]["id"] + patched = client.patch(f"/api/events/{eid}", json={"event_type": "pyro"}).json() + assert patched["event_type"] == "pyro" + assert patched["source"] == "user" # user correction attribution + assert client.patch("/api/events/99999", json={"event_type": "x"}).status_code == 404 diff --git a/backend/tests/test_resolve.py b/backend/tests/test_resolve.py new file mode 100644 index 0000000..3d40933 --- /dev/null +++ b/backend/tests/test_resolve.py @@ -0,0 +1,127 @@ +"""M8 annotation -> 3D resolution tests (integration phase). + +The headline acceptance (spec M8): annotating the *same* stage corner from two videos +produces an anchor within 0.2 scene units of the true corner. We build the synthetic fixture +without ffmpeg (``run_ffmpeg=False`` populates the DB + point cloud only), forward-project a +known corner into each camera to get faithful bbox centers, then resolve and check the error. + +Also covers: the single-view nearest-point fallback, the near-parallel/gap rejection path, +and the ``update_event`` correction helper. +""" + +from __future__ import annotations + +import numpy as np +import pytest + +from festival4d import config, db, resolve, synthetic +from festival4d.geometry import quat_to_mat + + +@pytest.fixture(scope="module") +def fixture(): + # No ffmpeg: this writes videos/poses/anchors/events into the DB and points.ply to WORK_DIR. + synthetic.build(run_ffmpeg=False) + return synthetic.STAGE_CORNERS + + +def _project(video, pose, world_pt): + """Forward-project a world point into a video, returning (px, py, z_cam). + + Mirrors the COLMAP pinhole the fixture uses: x_cam = R@X + t; px = fx*x/z + cx (y-down). + """ + q = [pose.qw, pose.qx, pose.qy, pose.qz] + t = np.array([pose.tx, pose.ty, pose.tz]) + R = quat_to_mat(q) + x_cam = R @ np.asarray(world_pt, dtype=float) + t + z = x_cam[2] + px = pose.fx * x_cam[0] / z + pose.cx + py = pose.fy * x_cam[1] / z + pose.cy + return px, py, z + + +def _bbox_center(px, py, W, H, half=0.02): + """A small normalized bbox centered on the projected pixel (clamped to [0,1]).""" + cx, cy = px / W, py / H + return [ + max(0.0, cx - half), max(0.0, cy - half), + min(1.0, cx + half), min(1.0, cy + half), + ] + + +def _views_seeing(corner): + """Videos whose first pose projects `corner` in front of the camera and inside the frame.""" + seen = [] + for v in db.get_videos(): + poses = db.get_poses(v.id) + if not poses: + continue + p0 = poses[0] + px, py, z = _project(v, p0, corner) + if z > 0 and 0 <= px < v.width and 0 <= py < v.height: + seen.append((v, p0, px, py)) + return seen + + +def test_two_view_triangulation_recovers_corner(fixture): + corners = fixture + # Pick a corner visible in at least two cameras. + chosen = None + for label, world in corners.items(): + views = _views_seeing(world) + if len(views) >= 2: + chosen = (label, np.asarray(world, dtype=float), views) + break + assert chosen is not None, "no stage corner is visible in >=2 synthetic cameras" + label, world, views = chosen + + event = db.add_event(t_global_s=5.0, event_type="artist_moment", source="ai", + description=f"located {label}") + + # Annotate the corner from the first two cameras that see it, then resolve the second one + # (which triangulates against the first). + (v_a, pose_a, pxa, pya), (v_b, pose_b, pxb, pyb) = views[0], views[1] + bbox_a = _bbox_center(pxa, pya, v_a.width, v_a.height) + bbox_b = _bbox_center(pxb, pyb, v_b.width, v_b.height) + + db.add_annotation(v_a.id, pose_a.t_video_s, *bbox_a, event_id=event.id) + res = resolve.resolve_annotation(v_b.id, pose_b.t_video_s, bbox_b, event_id=event.id) + + assert res.method == "triangulated", f"expected triangulation, got {res.method}" + err = float(np.linalg.norm(np.asarray(res.point) - world)) + assert err < 0.2, f"triangulated {label} off by {err:.3f} units (>0.2)" + # Sanity: the mutual-approach gap is tight for exact projections. + assert res.gap is not None and res.gap < 0.2 + + +def test_single_view_falls_back_to_point_cloud(fixture): + corners = fixture + world = np.asarray(corners["Stage FL"], dtype=float) + views = _views_seeing(world) + assert views, "Stage FL should be visible in at least one camera" + v, pose, px, py = views[0] + bbox = _bbox_center(px, py, v.width, v.height) + + # No event_id => no other view to triangulate => nearest point-cloud point (corner is in + # the cloud) or centroid-depth ray. Either way a point comes back. + res = resolve.resolve_annotation(v.id, pose.t_video_s, bbox, event_id=None) + assert res.method in {"nearest_point", "ray_depth"} + assert res.point is not None and len(res.point) == 3 + if res.method == "nearest_point": + # The exact corner is seeded into the cloud, so the nearest hit should be very close. + assert float(np.linalg.norm(np.asarray(res.point) - world)) < resolve.NEAREST_RADIUS + + +def test_update_event_sets_user_source(fixture): + ev = db.add_event(t_global_s=9.0, event_type="candidate", source="audio_auto", + description="unlabeled") + updated = db.update_event(ev.id, event_type="pyro", source="user") + assert updated.event_type == "pyro" + assert updated.source == "user" + # Unspecified fields are preserved. + assert updated.description == "unlabeled" + + +def test_missing_video_raises(fixture): + with pytest.raises(KeyError): + resolve.resolve_annotation(9999, 0.0, [0.4, 0.4, 0.6, 0.6], event_id=None) diff --git a/frontend/index.html b/frontend/index.html index 6df6c84..c2f70cb 100644 --- a/frontend/index.html +++ b/frontend/index.html @@ -42,7 +42,11 @@ #scene3d { position: absolute; inset: 0; width: 100%; height: 100%; display: block; } #scene-controls { position: absolute; top: 0.6rem; left: 0.6rem; display: flex; gap: 0.4rem; z-index: 5; + align-items: center; flex-wrap: wrap; max-width: calc(100% - 260px); } + #scene-controls .btn { padding: 0.25rem 0.5rem; font-size: 0.78rem; } + #key-count { color: var(--accent); font-variant-numeric: tabular-nums; } + .ctl-sep { width: 1px; height: 18px; background: var(--border); margin: 0 0.15rem; } #scene-hint { position: absolute; bottom: 0.5rem; left: 0.6rem; color: var(--dim); font-size: 0.72rem; z-index: 5; pointer-events: none; @@ -150,6 +154,37 @@ .bad { color: var(--bad); } .dim { color: var(--dim); } + /* M8 annotation correction panel */ + #annot-panel { + position: absolute; top: 0.6rem; right: 0.6rem; width: 232px; z-index: 8; + background: rgba(15,20,32,0.94); border: 1px solid var(--border); border-radius: 9px; + font-size: 0.78rem; box-shadow: 0 8px 26px rgba(0,0,0,0.5); backdrop-filter: blur(5px); + } + .ap-head { + display: flex; align-items: center; gap: 0.4rem; padding: 0.45rem 0.55rem; + border-bottom: 1px solid var(--border); + } + .ap-dot { width: 9px; height: 9px; border-radius: 3px; flex: 0 0 auto; } + .ap-head strong { flex: 1 1 auto; font-size: 0.8rem; } + .ap-x { + background: none; border: none; color: var(--dim); cursor: pointer; font-size: 0.85rem; + padding: 0 0.2rem; + } + .ap-x:hover { color: var(--text); } + .ap-body { padding: 0.5rem 0.55rem 0.6rem; display: flex; flex-direction: column; gap: 0.45rem; } + .ap-row { display: flex; align-items: center; justify-content: space-between; gap: 0.4rem; color: var(--dim); } + .ap-type { flex: 1 1 auto; margin-left: 0.4rem; background: var(--panel-2); color: var(--text); + border: 1px solid var(--border); border-radius: 5px; padding: 0.2rem 0.3rem; text-transform: capitalize; } + .ap-desc { color: var(--text); line-height: 1.35; } + .ap-meta { text-transform: capitalize; } + .ap-annotate { width: 100%; text-align: center; } + .ap-hint { line-height: 1.35; min-height: 1.1em; } + + .annot-rect { + position: absolute; border: 1.5px solid var(--accent); background: rgba(89,212,153,0.14); + pointer-events: none; z-index: 6; border-radius: 2px; + } + /* Loading / error */ #loading { position: fixed; inset: 0; display: flex; align-items: center; justify-content: center; @@ -171,8 +206,16 @@
+ + + + + + +
-
drag to orbit · click a camera or press 1–9 to snap · Esc to detach
+
drag to orbit · 1–9 snap to camera · K add keyframe · Esc detach
+
diff --git a/frontend/src/annotate.js b/frontend/src/annotate.js index 3aab307..c5fec8f 100644 --- a/frontend/src/annotate.js +++ b/frontend/src/annotate.js @@ -1,6 +1,262 @@ -// bbox drawing + submission → POST /api/annotations (spec M8). -// STUB — this is phase-3 (integration) work; lane C leaves it as a clean seam. Drag a -// rectangle on a video's overlay canvas → normalized bbox → POST /api/annotations; the -// resolved anchor then appears live in overlays + the 3D scene. +// Correction panel + bbox annotation → 3D (spec M8, integration phase). +// +// Clicking a timeline event opens a panel: it shows the classification, a dropdown to correct +// the event_type (PATCH /api/events/{id}, source→'user'), and an "annotate location" mode. +// In annotate mode the user drags a rectangle on ANY video's overlay; the normalized bbox is +// POSTed to /api/annotations. The backend casts a ray through the bbox center and resolves a +// 3D point (triangulate across two views of the same event, else nearest point-cloud point / +// centroid-depth ray). The resolved anchor is pushed live into state so overlays + the 3D +// scene show it immediately. -export {}; +import { state, emit, API_BASE } from "./state.js"; +import { contentRect } from "./videoGrid.js"; +import { eventColor } from "./timeline.js"; + +const EVENT_TYPES = [ + "bass_drop", "pyro", "confetti", "crowd_wave", + "artist_moment", "light_show", "quiet_moment", "candidate", "other", +]; + +export class Annotator { + constructor() { + this.panel = null; + this.cells = []; + this.currentEvent = null; + this.annotating = false; + this._rubber = null; // { cell, div, sx, sy } + this._cleanup = []; + } + + init(panelEl, cells) { + this.panel = panelEl; + this.cells = cells; + this.panel.style.display = "none"; + } + + /** Open the correction panel for a timeline event. */ + openForEvent(ev) { + this.currentEvent = ev; + this.stopAnnotateMode(); + this._render(); + this.panel.style.display = "block"; + } + + close() { + this.stopAnnotateMode(); + this.currentEvent = null; + if (this.panel) this.panel.style.display = "none"; + } + + _render() { + const ev = this.currentEvent; + if (!ev) return; + const options = EVENT_TYPES.map( + (t) => `` + ).join(""); + this.panel.innerHTML = ` +
+ + Event @ ${ev.t_global_s.toFixed(1)}s + +
+
+ +
${ev.description || "no description"}
+
${ev.source}${ev.confidence != null ? ` · ${(ev.confidence * 100).toFixed(0)}%` : ""}
+ +
+
`; + + this.panel.querySelector(".ap-x").addEventListener("click", () => this.close()); + this.panel.querySelector(".ap-type").addEventListener("change", (e) => + this._correctType(e.target.value) + ); + this.panel.querySelector(".ap-annotate").addEventListener("click", () => + this.annotating ? this.stopAnnotateMode() : this.startAnnotateMode() + ); + } + + _hint(msg) { + const h = this.panel?.querySelector(".ap-hint"); + if (h) h.innerHTML = msg; + } + + async _correctType(newType) { + const ev = this.currentEvent; + if (!ev) return; + try { + const resp = await fetch(`${API_BASE}/api/events/${ev.id}`, { + method: "PATCH", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ event_type: newType }), + }); + if (!resp.ok) throw new Error(`HTTP ${resp.status}`); + const updated = await resp.json(); + // Reflect into state + timeline marker color. + const i = state.events.findIndex((e) => e.id === ev.id); + if (i >= 0) state.events[i] = updated; + this.currentEvent = updated; + emit("events-changed", updated); + this._hint(`corrected → ${newType.replace(/_/g, " ")} (source: user)`); + } catch (err) { + this._hint(`correction failed: ${err.message}`); + } + } + + startAnnotateMode() { + this.annotating = true; + const btn = this.panel?.querySelector(".ap-annotate"); + if (btn) { + btn.textContent = "✓ Done annotating"; + btn.classList.add("active"); + } + this._hint("drag a box on any video over the object. Repeat on a 2nd video to triangulate."); + for (const cell of this.cells) { + cell.canvas.style.pointerEvents = "auto"; + cell.canvas.style.cursor = "crosshair"; + const down = (e) => this._onDown(cell, e); + cell.canvas.addEventListener("pointerdown", down); + this._cleanup.push(() => cell.canvas.removeEventListener("pointerdown", down)); + } + } + + stopAnnotateMode() { + this.annotating = false; + const btn = this.panel?.querySelector(".ap-annotate"); + if (btn) { + btn.textContent = "📍 Annotate location"; + btn.classList.remove("active"); + } + for (const cell of this.cells) { + cell.canvas.style.pointerEvents = "none"; + cell.canvas.style.cursor = ""; + } + this._teardownRubber(); + for (const fn of this._cleanup) fn(); + this._cleanup = []; + } + + _onDown(cell, e) { + e.preventDefault(); + cell.canvas.setPointerCapture(e.pointerId); + const rect = cell.el.getBoundingClientRect(); + const sx = e.clientX - rect.left; + const sy = e.clientY - rect.top; + const div = document.createElement("div"); + div.className = "annot-rect"; + cell.el.appendChild(div); + this._rubber = { cell, div, sx, sy, rect }; + this._positionRubber(sx, sy); + + const move = (ev) => this._onMove(ev); + const up = (ev) => this._onUp(ev, move, up); + cell.canvas.addEventListener("pointermove", move); + cell.canvas.addEventListener("pointerup", up); + cell.canvas.addEventListener("pointercancel", up); + } + + _positionRubber(cx, cy) { + const r = this._rubber; + if (!r) return; + const x0 = Math.min(r.sx, cx); + const y0 = Math.min(r.sy, cy); + r.div.style.left = `${x0}px`; + r.div.style.top = `${y0}px`; + r.div.style.width = `${Math.abs(cx - r.sx)}px`; + r.div.style.height = `${Math.abs(cy - r.sy)}px`; + } + + _onMove(e) { + const r = this._rubber; + if (!r) return; + const rect = r.cell.el.getBoundingClientRect(); + this._positionRubber(e.clientX - rect.left, e.clientY - rect.top); + } + + async _onUp(e, move, up) { + const r = this._rubber; + if (!r) return; + const { cell } = r; + cell.canvas.removeEventListener("pointermove", move); + cell.canvas.removeEventListener("pointerup", up); + cell.canvas.removeEventListener("pointercancel", up); + + const rect = cell.el.getBoundingClientRect(); + const ex = e.clientX - rect.left; + const ey = e.clientY - rect.top; + const bbox = this._toNormalizedBbox(cell, r.sx, r.sy, ex, ey); + this._teardownRubber(); + if (!bbox) { + this._hint('draw the box over the video content area.'); + return; + } + await this._submit(cell, bbox); + } + + _teardownRubber() { + if (this._rubber?.div?.parentNode) this._rubber.div.parentNode.removeChild(this._rubber.div); + this._rubber = null; + } + + /** Element-px rectangle → normalized [0,1] bbox in the video's CONTENT space (letterbox-aware). */ + _toNormalizedBbox(cell, ax, ay, bx, by) { + const cr = contentRect(cell.video); + const nx = (v) => (v - cr.ox) / cr.cw; + const ny = (v) => (v - cr.oy) / cr.ch; + let x0 = nx(Math.min(ax, bx)); + let y0 = ny(Math.min(ay, by)); + let x1 = nx(Math.max(ax, bx)); + let y1 = ny(Math.max(ay, by)); + x0 = Math.max(0, Math.min(1, x0)); + y0 = Math.max(0, Math.min(1, y0)); + x1 = Math.max(0, Math.min(1, x1)); + y1 = Math.max(0, Math.min(1, y1)); + // A near-zero drag (a click) becomes a tiny box centered on the point. + if (x1 - x0 < 1e-4 && y1 - y0 < 1e-4) { + const cx = (x0 + x1) / 2; + const cy = (y0 + y1) / 2; + x0 = Math.max(0, cx - 0.01); x1 = Math.min(1, cx + 0.01); + y0 = Math.max(0, cy - 0.01); y1 = Math.min(1, cy + 0.01); + } + if (x1 <= x0 || y1 <= y0) return null; + return [x0, y0, x1, y1]; + } + + async _submit(cell, bbox) { + const ev = this.currentEvent; + this._hint("resolving 3D position…"); + try { + const resp = await fetch(`${API_BASE}/api/annotations`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + video_id: cell.id, + t_video_s: cell.video.currentTime, + bbox, + event_id: ev ? ev.id : null, + }), + }); + if (!resp.ok) throw new Error(`HTTP ${resp.status}`); + const res = await resp.json(); + if (res.anchor) { + state.anchors.push(res.anchor); + emit("anchors-changed", res.anchor); + const how = + res.method === "triangulated" + ? `triangulated (gap ${res.gap != null ? res.gap.toFixed(2) : "?"})` + : res.method === "nearest_point" + ? "nearest point-cloud point" + : "ray at scene depth"; + this._hint(`anchor placed via ${how}. Annotate another view to refine.`); + } else { + this._hint('could not resolve a 3D point (no poses?).'); + } + } catch (err) { + this._hint(`annotation failed: ${err.message}`); + } + } +} + +export const annotator = new Annotator(); diff --git a/frontend/src/camPath.js b/frontend/src/camPath.js index 9a17cc1..3d71f80 100644 --- a/frontend/src/camPath.js +++ b/frontend/src/camPath.js @@ -1,6 +1,166 @@ -// Keyframed camera paths (spec M9). -// STUB — this is phase-3 (integration) work; lane C leaves it as a clean seam. "Add -// keyframe" captures the current free-roam pose at the current t_global; playback -// interpolates position (Catmull-Rom) + orientation (slerp); export/import path as JSON. +// Keyframed "god's-eye" camera paths (spec M9, integration phase). +// +// "Add keyframe" captures the current free-roam camera pose at the current t_global. Keyframes +// render as octahedron gizmos with a smooth preview line. On playback (while the transport +// plays) the viewer camera follows the path: position is interpolated with a Catmull-Rom curve +// through the keyframe centers, orientation with quaternion slerp, both parameterized by +// t_global between the first and last keyframe. Paths export/import as JSON. -export {}; +import * as THREE from "three"; +import { state } from "./state.js"; +import { scene3d } from "./scene3d.js"; + +const _qa = new THREE.Quaternion(); +const _qb = new THREE.Quaternion(); +const _pos = new THREE.Vector3(); + +export class CamPath { + constructor() { + this.keyframes = []; // [{ t_global, pos:[x,y,z], quat:[x,y,z,w], fov }] + this.playing = false; + this.group = null; // gizmos + preview line + this._curve = null; + this.onChange = null; // optional UI callback (count) + } + + _ensureGroup() { + if (!this.group) { + this.group = new THREE.Group(); + scene3d.addObject(this.group); + } + } + + /** Capture the current viewer-camera pose at the current master time. */ + addKeyframe() { + const cam = scene3d.camera; + if (!cam) return; + this.keyframes.push({ + t_global: state.tGlobal, + pos: [cam.position.x, cam.position.y, cam.position.z], + quat: [cam.quaternion.x, cam.quaternion.y, cam.quaternion.z, cam.quaternion.w], + fov: cam.fov, + }); + this.keyframes.sort((a, b) => a.t_global - b.t_global); + this._rebuild(); + } + + clear() { + this.stop(); + this.keyframes = []; + this._rebuild(); + } + + count() { + return this.keyframes.length; + } + + _rebuild() { + this._ensureGroup(); + for (const child of [...this.group.children]) { + this.group.remove(child); + child.geometry?.dispose?.(); + child.material?.dispose?.(); + } + this._curve = null; + + for (const k of this.keyframes) { + const giz = new THREE.Mesh( + new THREE.OctahedronGeometry(0.16), + new THREE.MeshBasicMaterial({ color: 0xffd23b, wireframe: true }) + ); + giz.position.set(k.pos[0], k.pos[1], k.pos[2]); + this.group.add(giz); + } + + if (this.keyframes.length >= 2) { + this._curve = new THREE.CatmullRomCurve3( + this.keyframes.map((k) => new THREE.Vector3(k.pos[0], k.pos[1], k.pos[2])) + ); + const pts = this._curve.getPoints(Math.max(20, this.keyframes.length * 12)); + const line = new THREE.Line( + new THREE.BufferGeometry().setFromPoints(pts), + new THREE.LineBasicMaterial({ color: 0xffd23b, transparent: true, opacity: 0.6 }) + ); + this.group.add(line); + } + this.onChange?.(this.keyframes.length); + } + + /** Begin following the path (requires >= 2 keyframes). Caller should also start the transport. */ + play() { + if (this.keyframes.length < 2) return false; + this.playing = true; + scene3d.enterPathMode(); + return true; + } + + stop() { + if (!this.playing) return; + this.playing = false; + scene3d.exitPathMode(); + } + + /** Per-frame: if playing, drive the viewer camera from the path at the current master time. */ + update(tGlobal) { + if (!this.playing) return; + // If something else took the camera (snap-to-camera exits path mode), stop driving it. + if (!scene3d.pathMode) { + this.playing = false; + this.onChange?.(this.keyframes.length); + return; + } + if (!this._curve || this.keyframes.length < 2) return; + const kf = this.keyframes; + const n = kf.length; + const t0 = kf[0].t_global; + const t1 = kf[n - 1].t_global; + + // Locate the bracketing keyframes by time; clamp outside the path's time span. + let i = 0; + if (tGlobal <= t0) { + i = 0; + } else if (tGlobal >= t1) { + i = n - 2; + } else { + while (i < n - 2 && kf[i + 1].t_global <= tGlobal) i++; + } + const a = kf[i]; + const b = kf[i + 1]; + const span = b.t_global - a.t_global; + let f = span > 1e-6 ? (tGlobal - a.t_global) / span : 0; + f = Math.max(0, Math.min(1, f)); + + // Position: Catmull-Rom, parameterized by (segment index + local fraction) / (n - 1). + const u = Math.max(0, Math.min(1, (i + f) / (n - 1))); + this._curve.getPoint(u, _pos); + + // Orientation: slerp the two bracketing keyframe quaternions. + _qa.set(a.quat[0], a.quat[1], a.quat[2], a.quat[3]); + _qb.set(b.quat[0], b.quat[1], b.quat[2], b.quat[3]); + _qa.slerp(_qb, f); + + const fov = a.fov + (b.fov - a.fov) * f; + scene3d.applyCameraPose(_pos, _qa, fov); + } + + toJSON() { + return JSON.stringify({ version: 1, keyframes: this.keyframes }, null, 2); + } + + fromJSON(text) { + const data = typeof text === "string" ? JSON.parse(text) : text; + if (!data || !Array.isArray(data.keyframes)) throw new Error("invalid camera-path file"); + this.keyframes = data.keyframes + .filter((k) => Array.isArray(k.pos) && Array.isArray(k.quat)) + .map((k) => ({ + t_global: +k.t_global || 0, + pos: k.pos.slice(0, 3).map(Number), + quat: k.quat.slice(0, 4).map(Number), + fov: +k.fov || 50, + })) + .sort((a, b) => a.t_global - b.t_global); + this._rebuild(); + } +} + +export const camPath = new CamPath(); diff --git a/frontend/src/main.js b/frontend/src/main.js index d4e8f8e..fafe631 100644 --- a/frontend/src/main.js +++ b/frontend/src/main.js @@ -14,6 +14,8 @@ import { drawOverlay, projectWorldToVideoPx } from "./overlays.js"; import { poseAt } from "./lib/poseTrack.js"; import { scene3d, videoColor } from "./scene3d.js"; import { timeline } from "./timeline.js"; +import { annotator } from "./annotate.js"; +import { camPath } from "./camPath.js"; async function getJSON(path) { const resp = await fetch(API_BASE + path); @@ -99,13 +101,71 @@ function buildUI() { // Timeline timeline.init(el("timeline"), transport); + // M8: correction panel + bbox annotation → 3D anchors. + annotator.init(el("annot-panel"), cells); + on("event-selected", (ev) => annotator.openForEvent(ev)); + on("anchors-changed", () => scene3d.refreshAnchors()); + on("events-changed", () => timeline.refresh()); + wireTransportControls(); + wireCamPath(); wireKeyboard(); positionVideosWhenReady(); - // Reflect follow/roam state on the free-roam button. + // Reflect follow/roam state on the free-roam button; snapping cancels a playing path. on("follow", (id) => { el("btn-roam").classList.toggle("active", id == null); + if (id != null) resetPathButton(); + }); +} + +function resetPathButton() { + const b = el("btn-path"); + b.textContent = "▶ Path"; + b.classList.remove("active"); +} + +function wireCamPath() { + camPath.onChange = (n) => { + el("key-count").textContent = String(n); + if (!camPath.playing) resetPathButton(); + }; + el("btn-key").addEventListener("click", () => camPath.addKeyframe()); + el("btn-path").addEventListener("click", () => { + const b = el("btn-path"); + if (camPath.playing) { + camPath.stop(); + resetPathButton(); + } else if (camPath.play()) { + b.textContent = "⏹ Path"; + b.classList.add("active"); + if (!state.playing) transport.toggle(); // advance master time so the path animates + } + }); + el("btn-path-export").addEventListener("click", () => { + const blob = new Blob([camPath.toJSON()], { type: "application/json" }); + const a = document.createElement("a"); + a.href = URL.createObjectURL(blob); + a.download = "festival4d-campath.json"; + a.click(); + URL.revokeObjectURL(a.href); + }); + const fileInput = el("path-file"); + el("btn-path-import").addEventListener("click", () => fileInput.click()); + fileInput.addEventListener("change", async () => { + const f = fileInput.files?.[0]; + if (f) { + try { + camPath.fromJSON(await f.text()); + } catch (err) { + console.error("[festival4d] camera-path import failed", err); + } + } + fileInput.value = ""; + }); + el("btn-path-clear").addEventListener("click", () => { + camPath.clear(); + resetPathButton(); }); } @@ -116,7 +176,13 @@ function wireTransportControls() { el("rate-select").addEventListener("change", (e) => transport.setRate(parseFloat(e.target.value)) ); - el("btn-roam").addEventListener("click", () => scene3d.freeRoam()); + el("btn-roam").addEventListener("click", () => { + if (camPath.playing) { + camPath.stop(); + resetPathButton(); + } + scene3d.freeRoam(); + }); } function wireKeyboard() { @@ -132,7 +198,14 @@ function wireKeyboard() { e.preventDefault(); transport.seekBy(e.shiftKey ? 5 : 1); } else if (e.code === "Escape" || e.code === "Digit0") { + if (camPath.playing) { + camPath.stop(); + resetPathButton(); + } scene3d.freeRoam(); + } else if (e.code === "KeyK") { + e.preventDefault(); + camPath.addKeyframe(); } else if (/^Digit[1-9]$/.test(e.code)) { const n = parseInt(e.code.slice(5), 10) - 1; const v = state.videos[n]; @@ -157,6 +230,7 @@ function startLoop() { const frame = () => { transport.tick(); for (const cell of cells) drawOverlay(cell); + camPath.update(state.tGlobal); // M9: drive the viewer camera before the scene renders scene3d.update(); timeline.update(state.tGlobal); updateHud(); diff --git a/frontend/src/scene3d.js b/frontend/src/scene3d.js index f8b04f2..a5e990c 100644 --- a/frontend/src/scene3d.js +++ b/frontend/src/scene3d.js @@ -26,12 +26,42 @@ const _pos = new THREE.Vector3(); const _quat = new THREE.Quaternion(); const _scale = new THREE.Vector3(); +/** Build a camera-facing text label sprite (used for 3D anchor labels). */ +function makeLabelSprite(text) { + const font = 48; + const pad = 10; + const c = document.createElement("canvas"); + let ctx = c.getContext("2d"); + ctx.font = `600 ${font}px -apple-system, system-ui, sans-serif`; + const w = Math.ceil(ctx.measureText(text).width) + pad * 2; + const h = font + pad * 2; + c.width = w; + c.height = h; + ctx = c.getContext("2d"); + ctx.font = `600 ${font}px -apple-system, system-ui, sans-serif`; + ctx.fillStyle = "rgba(6,8,14,0.74)"; + ctx.fillRect(0, 0, w, h); + ctx.fillStyle = "#e6e8ee"; + ctx.textBaseline = "middle"; + ctx.fillText(text, pad, h / 2 + 1); + const tex = new THREE.CanvasTexture(c); + tex.minFilter = THREE.LinearFilter; + const sprite = new THREE.Sprite( + new THREE.SpriteMaterial({ map: tex, depthTest: false, transparent: true }) + ); + const s = 0.006; + sprite.scale.set(w * s, h * s, 1); + return sprite; +} + export class Scene3D { constructor() { this.videoIndex = {}; // id -> index (for stable colors) this.rigs = {}; // id -> { frustum, marker, pick, path } this._tween = null; this._raf = null; + this.anchorGroup = null; // 3D anchor spheres + labels (M6 anchors + M8 resolved) + this.pathMode = false; // when true, camPath (M9) drives the camera; we don't touch it } init(canvas) { @@ -64,12 +94,76 @@ export class Scene3D { const axes = new THREE.AxesHelper(1.5); this.scene.add(axes); + this.anchorGroup = new THREE.Group(); + this.scene.add(this.anchorGroup); + this._buildRigs(); this._loadPointCloud(); + this.refreshAnchors(); this._wirePicking(); this._wireResize(parent); } + /** Rebuild the 3D anchor markers from state.anchors (M6 seeded + M8 resolved). */ + refreshAnchors() { + if (!this.anchorGroup) return; + for (const child of [...this.anchorGroup.children]) { + this.anchorGroup.remove(child); + child.geometry?.dispose?.(); + if (child.material) { + child.material.map?.dispose?.(); + child.material.dispose(); + } + } + for (const a of state.anchors) { + const color = new THREE.Color(a.color || "#59d499"); + const sphere = new THREE.Mesh( + new THREE.SphereGeometry(0.13, 16, 16), + new THREE.MeshBasicMaterial({ color }) + ); + sphere.position.set(a.x, a.y, a.z); + this.anchorGroup.add(sphere); + if (a.label) { + const label = makeLabelSprite(a.label); + label.position.set(a.x, a.y + 0.3, a.z); + this.anchorGroup.add(label); + } + } + } + + /** Expose the scene graph for auxiliary overlays (e.g. camPath gizmos, M9). */ + addObject(obj) { + this.scene?.add(obj); + } + removeObject(obj) { + this.scene?.remove(obj); + } + + /** M9: hand camera control to camPath. update() then leaves the camera alone. */ + enterPathMode() { + this.pathMode = true; + this.controls.enabled = false; + state.followCameraId = null; + this._tween = null; + emit("follow", null); + } + exitPathMode() { + if (!this.pathMode) return; + this.pathMode = false; + this.controls.target.copy(STAGE_TARGET); + this.controls.enabled = true; + this.controls.update(); + } + /** M9: set the camera pose directly (called by camPath each frame while path-playing). */ + applyCameraPose(pos, quat, fov) { + this.camera.position.copy(pos); + this.camera.quaternion.copy(quat); + if (fov != null && Math.abs(this.camera.fov - fov) > 1e-3) { + this.camera.fov = fov; + this.camera.updateProjectionMatrix(); + } + } + _buildRigs() { state.videos.forEach((v, i) => { this.videoIndex[v.id] = i; @@ -177,6 +271,7 @@ export class Scene3D { snapTo(videoId) { const pose = this._livePose(videoId); if (!pose) return; + if (this.pathMode) this.exitPathMode(); state.followCameraId = videoId; this.controls.enabled = false; const targetFov = (2 * Math.atan(state.videos.find((v) => v.id === videoId).height / @@ -244,7 +339,9 @@ export class Scene3D { _m4.decompose(rig.group.position, rig.group.quaternion, rig.group.scale); } - if (state.followCameraId != null) { + if (this.pathMode) { + // camPath (M9) already set the camera pose this frame; don't fight it. + } else if (state.followCameraId != null) { const pose = this._livePose(state.followCameraId); if (pose) { const meta = state.videos.find((v) => v.id === state.followCameraId); diff --git a/frontend/src/timeline.js b/frontend/src/timeline.js index cee6682..53f44ce 100644 --- a/frontend/src/timeline.js +++ b/frontend/src/timeline.js @@ -5,7 +5,7 @@ // tooltip; click a marker -> jump the playhead there. Event data is already seeded // synthetically; classification QUALITY is lane D's concern, not this renderer's. -import { state } from "./state.js"; +import { state, emit } from "./state.js"; export const EVENT_COLORS = { bass_drop: "#ff3b6b", @@ -58,8 +58,10 @@ export class Timeline { tip.style.display = "none"; root.append(track, tip); + this.root = root; this.track = track; this.fill = fill; + this.markers = markers; this.playhead = playhead; this.tip = tip; @@ -68,6 +70,15 @@ export class Timeline { this._wireScrub(track); } + /** Rebuild markers + legend after events change (e.g. an M8 type correction). */ + refresh() { + if (!this.markers) return; + this.markers.innerHTML = ""; + this._buildMarkers(this.markers, this.tip); + this.root.querySelector(".tl-legend")?.remove(); + this._buildLegend(this.root); + } + _buildMarkers(container, tip) { for (const ev of state.events) { const m = document.createElement("div"); @@ -77,6 +88,7 @@ export class Timeline { m.addEventListener("click", (e) => { e.stopPropagation(); this.transport.seek(ev.t_global_s); + emit("event-selected", ev); // open the M8 correction panel }); const show = (e) => { tip.innerHTML = diff --git a/plan/CHANGE_REQUESTS.md b/plan/CHANGE_REQUESTS.md index 10c583e..66c407f 100644 --- a/plan/CHANGE_REQUESTS.md +++ b/plan/CHANGE_REQUESTS.md @@ -31,5 +31,28 @@ Instead append an entry here and work around it locally; the integration agent a - **Workaround in place:** applied the minimal edit directly (test file, not a frozen contract file). The frozen `api.py` app is **unchanged**. Only lane D's owned modules + this obsolete stub assertion were touched. Full suite green (89 passed). -- **Decision:** (integration) — informational; the change is a direct, unavoidable consequence - of lane D landing. Confirm the assertion matches the intended M3 detect-response shape. +- **Decision:** **APPLIED** (integration, 2026-07-16). Confirmed against the frozen api.py + docstring: `POST /api/events/detect` returns `{result, events}` (spec M3). The renamed + `test_detect_events_endpoint` asserts exactly that shape and is idempotent. No frozen file + changed; approval also recorded in DIRECTIVES round 1. + +### CR-2 — integration — 2026-07-16 — M8 needs annotation↔event link + event-type correction +- **File:** `db.py` (`Annotation` schema), `api.py` (`AnnotationIn`, events routes) +- **Problem:** Spec M8 requires (a) triangulating "another annotation **for the same event** + from a different video" — but the frozen `annotations` schema has no event reference, and + `AnnotationIn` carries no event id, so annotations can't be grouped per event; and (b) a + correction dropdown that "writes `source='user'`" to an event's type — but there is no + event-update helper or endpoint (only add/clear). +- **Proposed change (additive only — no existing shape altered):** + 1. `annotations.event_id` — new **nullable** `ForeignKey("events.id")` column. Nullable ⇒ + existing rows/inserts unaffected; the synthetic fixture creates annotations = none, so no + migration. Serves the two-view grouping. + 2. `AnnotationIn.event_id: int | None = None` — additive optional request field. + 3. `db.update_event(event_id, event_type=?, source=?, description=?)` helper + + `PATCH /api/events/{event_id}` — additive route; existing GET/POST/detect unchanged. +- **Workaround in place:** none needed — adjudicated and applied directly in this integration + phase (the phase where the ownership matrix ends and frozen contracts may change on an + approved CR). All changes are strictly additive; every pre-existing route/shape/column and + the timebase + pose contracts are byte-for-byte unchanged. Full suite stays green. +- **Decision:** **APPLIED** (integration, 2026-07-16). Additive, backward-compatible, and + required for M8 acceptance. Frontend and backend M8 both depend on it. diff --git a/plan/status/integration.md b/plan/status/integration.md new file mode 100644 index 0000000..7e0b0c8 --- /dev/null +++ b/plan/status/integration.md @@ -0,0 +1,66 @@ +# Status — integration + +## Round 1 — 2026-07-16 — STATUS: ready_to_merge + +**Directives acknowledged:** round 1 of plan/DIRECTIVES.md (all lanes merged & verified; +integration begins; CR-1 approved; drop debris stash + remove laneA/B worktrees; real-browser +sync check; test floor 96). + +**Scope done: M8 + M9 + README + cross-lane wiring + cleanup.** + +### Change requests adjudicated (plan/CHANGE_REQUESTS.md) +- **CR-1 (lane D): APPLIED** — confirmed `POST /api/events/detect` returns `{result, events}`; + renamed test asserts it. +- **CR-2 (integration, filed + APPLIED):** M8 needs an annotation↔event link and event-type + correction, neither of which the frozen contract had. Added — strictly additive: + - `annotations.event_id` (nullable FK) + `AnnotationIn.event_id` — two-view grouping. + - `db.update_event` + `db.get_event` + `PATCH /api/events/{id}` — the correction dropdown. + Every pre-existing route/shape/column and the timebase + pose contracts are unchanged. +- A second foundation stub assertion (`test_annotation_stored`: `anchor_id is None`) was, like + CR-1, only valid pre-M8; renamed to `test_annotation_resolves_to_anchor` for the landed shape. + +### M8 — correction + annotation → 3D +- New `backend/festival4d/resolve.py`: ray from bbox center (pose interpolated at t_video via + `geometry.slerp_pose`), then triangulate across two same-event views (reject near-parallel / + gap > 0.5), else nearest point-cloud point (0.3 cyl), else ray at centroid depth. Pure + orchestration over lane B's frozen primitives. +- `POST /api/annotations` now resolves + creates+links an anchor (label = event description). +- Frontend `annotate.js`: click a timeline marker → correction panel (type dropdown → PATCH, + source→user; live marker recolor); **Annotate location** → drag a letterbox-correct box on + any video → POST. Resolved anchors pushed live into overlays **and** the 3D scene (new anchor + spheres + labels in `scene3d.js`). +- **Acceptance (spec M8): PASS.** `tests/test_resolve.py::test_two_view_triangulation_recovers_corner` + — two-view stage-corner annotation lands **0.00000 units** from ground truth (< 0.2). Verified + live in-browser too: two-view annotation triangulated with gap 0.21 through the full + frontend→API→geometry path; correction changed pyro→confetti with source=user and recolored + the marker. + +### M9 — keyframed camera paths +- New `camPath.js`: **＋ Key**/`K` captures the free-roam pose at the current t_global; gizmos + + Catmull-Rom preview line; **▶ Path** follows position (Catmull-Rom) + orientation (slerp) by + t_global while the transport plays; ⤓/⤒ export/import JSON. `scene3d` gained + `enterPathMode`/`exitPathMode`/`applyCameraPose`; snap-to-camera and Free-roam/`Esc` cancel a + playing path. +- **Verified live:** 3 keyframes → path mode on → at t=6.0 s the viewer camera sat exactly on + keyframe #2 `[-10,6,12]` (Catmull-Rom driven). +- Polish present from lanes + this phase: loading state, per-video enable/disable, keyboard + (`Space`,`←/→`,`Shift`,`1–9`,`Esc/0`,`K`). + +### Full-system pass +- **Backend: 101 passed** (`../.venv/bin/python -m pytest`) — up from the 96 merge-day floor + (+5: 4 resolve + 1 event-patch; annotation test renamed). +- **Frontend: `npm run build` clean** (18 modules). +- End-to-end on the synthetic fixture: `synthetic → ingest → sync → events → serve` + frontend + loads with 0 console errors; M4 sync error ~16 ms at t=8 s (deterministic pump; half-frame + residual, < 50 ms). +- Cleanup: dropped the shared-worktree debris stash; removed `../festifun-laneA` and + `../festifun-laneB` worktrees; `frontend/dist/` gitignored; pristine demo DB regenerated. + +**Blockers / notes for coordinator:** +- **Real-focused-browser M4 check is the one item I could not fully close from here.** The + automated browser pane runs `document.hidden=true`, so rAF is throttled and continuous + playback can't be observed (same limitation lane C hit). I verified sync deterministically + (pump → ~16 ms) and every interaction via real handlers, but a human should open the app in a + normal focused window once to eyeball continuous multi-cam sync. Not code-blocking. + +**Next:** merge `integration` → `main`; tag `v0.1.0`.