Integration (M8 + M9): annotation→3D, camera paths, README, v0.1.0
M8: POST /api/annotations resolves a bbox to a 3D point (ray → two-view
triangulation / nearest-point / centroid-depth) via lane B's geometry
primitives; correction panel + bbox-drawing UI; resolved anchors render live
in overlays and the 3D scene. CR-2 (additive): annotations.event_id +
PATCH /api/events/{id} + db.update_event.
M9: keyframed god's-eye camera paths (Catmull-Rom position + slerp
orientation, export/import), gizmos, path-mode camera hooks.
Verified: 101 backend tests pass (two-view corner recovery err 0.00000);
frontend build clean; live in-browser M8 triangulation (gap 0.21) + M9 path
(camera on keyframe at t) + M4 sync ~16ms. Cleanup: worktrees removed, debris
stash dropped, pristine demo DB.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
m3ultra
parent
58c01559c1
commit
e5d6b2c47f
1
.gitignore
vendored
1
.gitignore
vendored
@ -5,3 +5,4 @@ node_modules/
|
||||
__pycache__/
|
||||
.venv/
|
||||
.DS_Store
|
||||
frontend/dist/
|
||||
|
||||
178
README.md
178
README.md
@ -3,55 +3,187 @@
|
||||
Turn multiple fan-shot smartphone videos of the same concert into a synchronized,
|
||||
explorable **4D experience**: time-aligned multi-video playback, 3D scene reconstruction
|
||||
with camera poses, a free-roam "god's eye" viewer, AR-style overlays projected onto each
|
||||
video, and AI-tagged moments on a shared timeline.
|
||||
video, AI-tagged moments on a shared timeline, click-to-place 3D annotations, and keyframed
|
||||
cinematic fly-throughs.
|
||||
|
||||
> **Status: foundation phase.** The scaffold, database, frozen contracts (API, pose math,
|
||||
> DB schema), and a synthetic fixture generator are in place. Feature lanes (media sync,
|
||||
> reconstruction, viewer, AI events) build on top. The full README with the real-footage
|
||||
> workflow is written in the integration phase (M9). See
|
||||
> [`OPUS_BUILD_INSTRUCTIONS.md`](OPUS_BUILD_INSTRUCTIONS.md) for the canonical spec and
|
||||
> [`plan/`](plan/) for the execution plan.
|
||||
Everything works **from pixels and audio alone** — no depth sensors, no IMU logs. It runs
|
||||
fully offline and local; the only optional cloud piece is moment classification.
|
||||
|
||||
---
|
||||
|
||||
## What you can do
|
||||
|
||||
- **Scrub a shared timeline** and watch every camera stay locked to the same instant, aligned
|
||||
by their audio (no manual frame-matching).
|
||||
- **Fly through the reconstructed scene** in 3D — orbit freely, snap to any real camera, or
|
||||
play a keyframed cinematic path.
|
||||
- **See an "x-ray" HUD**: 3D anchor points (stage corners, tagged objects) projected onto every
|
||||
video, tracking even when occluded by crowd or scaffolding.
|
||||
- **Jump between AI-tagged moments** (bass drops, pyro, crowd waves…) on the timeline, and
|
||||
correct any misclassification.
|
||||
- **Click to locate things in 3D**: draw a box around an object in one or two videos and the
|
||||
app triangulates its 3D position, dropping an anchor that appears everywhere at once.
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Python 3.11+** and [`uv`](https://docs.astral.sh/uv/) (or venv + pip)
|
||||
- **ffmpeg** / **ffprobe** on your `PATH` (required)
|
||||
- **COLMAP** (optional — reconstruction degrades gracefully without it)
|
||||
- A classifier API key (optional — `GEMINI_API_KEY` by default; see `events_ai.py`)
|
||||
- **Node 18+** for the frontend
|
||||
| Tool | Required? | Notes |
|
||||
|---|---|---|
|
||||
| **Python 3.11+** + [`uv`](https://docs.astral.sh/uv/) (or venv+pip) | yes | backend |
|
||||
| **ffmpeg** / **ffprobe** on `PATH` | yes | audio extraction, frame sampling, clip cutting |
|
||||
| **Node 18+** | yes | frontend (Vite + Three.js) |
|
||||
| **COLMAP** on `PATH` | optional | 3D reconstruction. Without it the app still runs — you get synced videos + timeline, just no 3D scene. `brew install colmap` on macOS. |
|
||||
| A classifier API key | optional | AI moment labels. Default **Gemini 2.5 Flash** (`GEMINI_API_KEY`); also Claude or any OpenAI-compatible/local/OpenRouter endpoint via `FESTIVAL4D_CLASSIFIER`. Without a key you still get audio-detected candidate moments, just unlabeled. |
|
||||
|
||||
---
|
||||
|
||||
## Quickstart (synthetic demo — no footage needed)
|
||||
|
||||
The synthetic fixture generates three fake camera videos of the same fake stage, with known
|
||||
audio offsets and known 3D geometry — so you can see the whole app working end-to-end before
|
||||
you have any real footage.
|
||||
|
||||
```bash
|
||||
# 1. backend env
|
||||
uv venv --python 3.12
|
||||
uv pip install -e ".[dev]"
|
||||
|
||||
# 2. generate the synthetic fixture project (fake videos + poses + point cloud + events)
|
||||
# 2. generate the synthetic project (fake videos + poses + point cloud + seeded events)
|
||||
uv run python -m festival4d synthetic
|
||||
|
||||
# 3. serve the API (http://127.0.0.1:8000)
|
||||
uv run python -m festival4d serve
|
||||
|
||||
# 4. in another terminal, the frontend
|
||||
# 4. in another terminal, run the frontend
|
||||
cd frontend
|
||||
npm install
|
||||
npm run dev # http://localhost:5173
|
||||
npm run dev # opens http://localhost:5173 (or the next free port)
|
||||
```
|
||||
|
||||
## Backend CLI
|
||||
Open the printed URL. You should see the 3D scene with three camera frusta around a stage box,
|
||||
the three synced video players, stage-corner anchors projected onto each video, and colored
|
||||
moment markers on the timeline.
|
||||
|
||||
---
|
||||
|
||||
## Real-footage workflow
|
||||
|
||||
1. **Shoot / gather** 2–4 videos of the same performance from different positions (see
|
||||
*Shooting tips* below) and drop the files into `data/raw/`.
|
||||
2. **Run the pipeline:**
|
||||
|
||||
```bash
|
||||
uv run python -m festival4d ingest # probe videos, extract 16 kHz mono audio
|
||||
uv run python -m festival4d sync # GCC-PHAT audio alignment (offsets + drift)
|
||||
uv run python -m festival4d reconstruct # COLMAP structure-from-motion → camera poses + point cloud
|
||||
uv run python -m festival4d events # detect audio moments + AI-classify them
|
||||
uv run python -m festival4d serve # serve it
|
||||
```
|
||||
|
||||
Each step is independent and re-runnable. `sync` alone already gives you locked multi-video
|
||||
playback; `reconstruct` adds the 3D scene and overlays; `events` adds the tagged timeline.
|
||||
3. **Explore** in the browser (`cd frontend && npm run dev`).
|
||||
|
||||
If COLMAP can't reconstruct your footage (common with dark, motion-blurred, or low-overlap
|
||||
clips), the pipeline says so and leaves you with the synced-video experience — it never
|
||||
corrupts existing data.
|
||||
|
||||
To enable AI moment labels, set a key before `events`:
|
||||
|
||||
```bash
|
||||
export GEMINI_API_KEY=... # default provider (Gemini 2.5 Flash, native video)
|
||||
# or: export FESTIVAL4D_CLASSIFIER=claude ANTHROPIC_API_KEY=...
|
||||
# or: export FESTIVAL4D_CLASSIFIER=local FESTIVAL4D_OPENAI_BASE_URL=... FESTIVAL4D_OPENAI_MODEL=...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Using the viewer
|
||||
|
||||
| Action | How |
|
||||
|---|---|
|
||||
| Play / pause | `Space` or the ⏵ button |
|
||||
| Nudge time ±1 s (±5 s) | `←` / `→` (hold `Shift`) |
|
||||
| Scrub | drag the timeline |
|
||||
| Pick the audio you hear | click a video (only one is unmuted at a time) |
|
||||
| Enable/disable a video | ◉ on the video |
|
||||
| Orbit the 3D scene | drag in the 3D pane |
|
||||
| Snap the 3D view to a camera | click its frustum, its ⛶ button, or press `1`–`9` |
|
||||
| Free roam again | `Esc` / `0` / **Free roam** |
|
||||
| Correct a moment / annotate | click a timeline marker → panel |
|
||||
| Place a 3D anchor | in the panel, **Annotate location**, then drag a box on a video (repeat on a 2nd video to triangulate) |
|
||||
| Add a camera keyframe | **＋ Key** or `K` (captures the current free-roam view at the current time) |
|
||||
| Play the keyframed path | **▶ Path** · export/import with ⤓ / ⤒ |
|
||||
|
||||
The top-right **sync error** panel shows each video's alignment error in ms while playing —
|
||||
a dev aid; it should stay green (<50 ms) once footage is well-synced.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
python -m festival4d synthetic # generate the synthetic fixture (M0)
|
||||
python -m festival4d ingest # probe videos + extract audio (lane A / M1)
|
||||
python -m festival4d sync # GCC-PHAT audio alignment (lane A / M1)
|
||||
python -m festival4d reconstruct # COLMAP SfM + pose export (lane B / M2)
|
||||
python -m festival4d events # audio candidates + AI classify (lane D / M7)
|
||||
python -m festival4d serve # FastAPI app (M3)
|
||||
data/raw/ ──ingest──▶ data/work/audio ──sync────▶ videos.offset_ms / drift_ppm ┐
|
||||
│
|
||||
──frames──▶ data/work/frames ──reconstruct──▶ camera_poses + points.ply
|
||||
│
|
||||
reference audio ──events──▶ events (audio candidates → AI labels)
|
||||
│
|
||||
SQLite (data/project.db) ◀─────────┘
|
||||
│
|
||||
FastAPI (backend/festival4d/api.py)
|
||||
│ /api/manifest, /poses, /pointcloud,
|
||||
│ /events, /anchors, /annotations
|
||||
▼
|
||||
Vite + Three.js SPA (frontend/src)
|
||||
transport (master clock) · videoGrid + overlays · scene3d · timeline
|
||||
annotate (M8) · camPath (M9)
|
||||
```
|
||||
|
||||
- **Backend** — Python: `ingest`/`audio_sync` (media + GCC-PHAT sync), `frames`/`sfm`/`geometry`
|
||||
(COLMAP orchestration + pose math + triangulation), `events_ai` (moment detection + pluggable
|
||||
classifiers), `resolve` (annotation→3D), `api` (FastAPI, Range-capable video serving), `db`
|
||||
(SQLAlchemy/SQLite), `synthetic` (the test fixture).
|
||||
- **Frontend** — vanilla JS + Three.js. A single master clock (`transport.js`) derives each
|
||||
video's local time from the shared timeline and continuously corrects playback; `lib/pose.js`
|
||||
is the one COLMAP→Three.js pose conversion, mirrored from `geometry.py` and locked by a shared
|
||||
test.
|
||||
- **Timebase contract:** `t_video = (t_global − offset_ms/1000) · (1 + drift_ppm·1e−6)`; the
|
||||
reference video has offset 0.
|
||||
|
||||
Full design + milestone history: [`OPUS_BUILD_INSTRUCTIONS.md`](OPUS_BUILD_INSTRUCTIONS.md);
|
||||
the parallel build plan and per-lane status live in [`plan/`](plan/).
|
||||
|
||||
---
|
||||
|
||||
## Shooting tips (for good reconstructions)
|
||||
|
||||
Concert footage is genuinely hard for structure-from-motion. To give COLMAP a chance:
|
||||
|
||||
- **Spread the cameras out** but keep **overlapping views** — each pair of cameras should see
|
||||
some of the same stage/structure. No overlap → they can't be related in 3D.
|
||||
- **Keep some static structure in frame** (stage edges, truss, speaker stacks). A frame that's
|
||||
all moving crowd and lights has nothing stable to triangulate.
|
||||
- **Avoid pure zoom** — physically moving parallax reconstructs far better than zooming.
|
||||
- **Brighter, sharper is better** — motion blur and near-dark frames are the main failure cause.
|
||||
- **2–4 phones is the sweet spot** for a first reconstruction; more is fine but slower.
|
||||
|
||||
Audio sync is far more forgiving: any clips that share audible sound (the same music/claps)
|
||||
will align, even across otherwise-unrelated angles.
|
||||
|
||||
---
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
uv run pytest
|
||||
uv run pytest # backend
|
||||
cd frontend && npm run build # frontend typecheck/build
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Non-goals (this prototype)
|
||||
|
||||
No accounts/auth, no cloud storage, no realtime ingest, no NeRF/Gaussian splatting, no mobile
|
||||
UI, no Docker. Single local user, one project at a time.
|
||||
|
||||
@ -73,6 +73,14 @@ class AnnotationIn(BaseModel):
|
||||
video_id: int
|
||||
t_video_s: float
|
||||
bbox: list[float] = Field(min_length=4, max_length=4) # [x0, y0, x1, y1] normalized
|
||||
event_id: int | None = None # CR-2: link to the event this annotation locates (M8)
|
||||
|
||||
|
||||
class EventPatch(BaseModel):
|
||||
event_type: str | None = None
|
||||
source: str | None = None
|
||||
description: str | None = None
|
||||
confidence: float | None = None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@ -191,19 +199,57 @@ def detect_events(payload: DetectIn) -> dict:
|
||||
return {"result": result, "events": [_event_dict(e) for e in db.get_events()]}
|
||||
|
||||
|
||||
@app.patch("/api/events/{event_id}")
|
||||
def patch_event(event_id: int, payload: EventPatch) -> dict:
|
||||
"""Correct an event in place (M8). A user correction defaults ``source`` to ``'user'``."""
|
||||
if db.get_event(event_id) is None:
|
||||
raise HTTPException(status_code=404, detail=f"no event with id={event_id}")
|
||||
source = payload.source
|
||||
if source is None and (payload.event_type is not None or payload.description is not None):
|
||||
source = "user" # any user-driven correction is attributed to the user
|
||||
event = db.update_event(
|
||||
event_id,
|
||||
event_type=payload.event_type,
|
||||
source=source,
|
||||
description=payload.description,
|
||||
confidence=payload.confidence,
|
||||
)
|
||||
return _event_dict(event)
|
||||
|
||||
|
||||
@app.post("/api/annotations")
|
||||
def create_annotation(payload: AnnotationIn) -> dict:
|
||||
x0, y0, x1, y1 = payload.bbox
|
||||
annotation = db.add_annotation(payload.video_id, payload.t_video_s, x0, y0, x1, y1)
|
||||
from festival4d import resolve
|
||||
|
||||
x0, y0, x1, y1 = payload.bbox
|
||||
if db.get_video(payload.video_id) is None:
|
||||
raise HTTPException(status_code=404, detail=f"no video with id={payload.video_id}")
|
||||
annotation = db.add_annotation(
|
||||
payload.video_id, payload.t_video_s, x0, y0, x1, y1, event_id=payload.event_id
|
||||
)
|
||||
|
||||
# M8: cast a ray through the bbox center; triangulate against another view of the same
|
||||
# event, else fall back to the nearest point-cloud point / centroid depth.
|
||||
result = resolve.resolve_annotation(
|
||||
payload.video_id, payload.t_video_s, payload.bbox, event_id=payload.event_id
|
||||
)
|
||||
|
||||
anchor = None
|
||||
if result.point is not None:
|
||||
label = "annotation"
|
||||
color = None
|
||||
if payload.event_id is not None:
|
||||
ev = db.get_event(payload.event_id)
|
||||
if ev is not None:
|
||||
label = ev.description or ev.event_type or label
|
||||
anchor = db.add_anchor(label, result.point[0], result.point[1], result.point[2], color)
|
||||
db.set_annotation_anchor(annotation.id, anchor.id)
|
||||
|
||||
# M8 resolution (triangulate / nearest-point) is integration work; geometry stubs raise
|
||||
# until lane B lands. Store the annotation now; resolve to an anchor later.
|
||||
anchor_id = None
|
||||
point = None
|
||||
note = "annotation stored; 3D resolution is M8/integration work"
|
||||
return {
|
||||
"annotation_id": annotation.id,
|
||||
"anchor_id": anchor_id,
|
||||
"point": point,
|
||||
"note": note,
|
||||
"anchor_id": anchor.id if anchor else None,
|
||||
"point": result.point,
|
||||
"method": result.method,
|
||||
"gap": result.gap,
|
||||
"anchor": _anchor_dict(anchor) if anchor else None,
|
||||
}
|
||||
|
||||
@ -120,6 +120,9 @@ class Annotation(Base):
|
||||
y0: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
x1: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
y1: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
# Nullable link to the event this annotation locates (CR-2, M8). Two annotations sharing an
|
||||
# event_id from different videos triangulate to a 3D anchor. Nullable => additive/back-compat.
|
||||
event_id: Mapped[int | None] = mapped_column(ForeignKey("events.id"), nullable=True)
|
||||
resolved_anchor_id: Mapped[int | None] = mapped_column(
|
||||
ForeignKey("anchors.id"), nullable=True
|
||||
)
|
||||
@ -356,6 +359,39 @@ def clear_events(source: str | None = None) -> int:
|
||||
return int(result.rowcount or 0)
|
||||
|
||||
|
||||
def get_event(event_id: int) -> Event | None:
|
||||
with session_scope() as s:
|
||||
return s.get(Event, event_id)
|
||||
|
||||
|
||||
def update_event(
|
||||
event_id: int,
|
||||
event_type: str | None = None,
|
||||
source: str | None = None,
|
||||
description: str | None = None,
|
||||
confidence: float | None = None,
|
||||
) -> Event:
|
||||
"""Patch an existing event in place (M8 correction). Only provided fields change.
|
||||
|
||||
Returns the updated row; raises ``KeyError`` if the id is unknown.
|
||||
"""
|
||||
with session_scope() as s:
|
||||
event = s.get(Event, event_id)
|
||||
if event is None:
|
||||
raise KeyError(f"no event with id={event_id}")
|
||||
if event_type is not None:
|
||||
event.event_type = event_type
|
||||
if source is not None:
|
||||
event.source = source
|
||||
if description is not None:
|
||||
event.description = description
|
||||
if confidence is not None:
|
||||
event.confidence = confidence
|
||||
s.flush()
|
||||
s.refresh(event)
|
||||
return event
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Annotations
|
||||
# ---------------------------------------------------------------------------
|
||||
@ -366,6 +402,7 @@ def add_annotation(
|
||||
y0: float,
|
||||
x1: float,
|
||||
y1: float,
|
||||
event_id: int | None = None,
|
||||
resolved_anchor_id: int | None = None,
|
||||
) -> Annotation:
|
||||
with session_scope() as s:
|
||||
@ -373,6 +410,7 @@ def add_annotation(
|
||||
video_id=video_id,
|
||||
t_video_s=t_video_s,
|
||||
x0=x0, y0=y0, x1=x1, y1=y1,
|
||||
event_id=event_id,
|
||||
resolved_anchor_id=resolved_anchor_id,
|
||||
)
|
||||
s.add(annotation)
|
||||
|
||||
152
backend/festival4d/resolve.py
Normal file
152
backend/festival4d/resolve.py
Normal file
@ -0,0 +1,152 @@
|
||||
"""Annotation -> 3D point resolution (spec M8, integration phase).
|
||||
|
||||
Given a 2D bounding box drawn on one video at a local video time, recover a 3D scene point:
|
||||
|
||||
1. Cast a ray from that camera through the bbox center, using the pose interpolated at
|
||||
``t_video_s`` and the stored pinhole intrinsics.
|
||||
2. If another annotation for the **same event** exists from a **different video**,
|
||||
triangulate the two rays (reject near-parallel rays or a mutual gap > 0.5 scene units).
|
||||
3. Else take the nearest point-cloud point to the ray (within a 0.3-unit cylinder).
|
||||
4. Else return the ray at depth = distance from the camera to the point-cloud centroid.
|
||||
|
||||
This module is pure orchestration over lane B's frozen geometry primitives
|
||||
(:func:`geometry.ray_from_pixel`, :func:`geometry.triangulate_rays`,
|
||||
:func:`geometry.nearest_point_on_ray`, :func:`geometry.slerp_pose`) — it never reimplements
|
||||
the pose math. It reads the point cloud via :func:`synthetic.read_ply`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
|
||||
import numpy as np
|
||||
|
||||
from festival4d import config, db, geometry
|
||||
from festival4d.synthetic import read_ply
|
||||
|
||||
log = logging.getLogger("festival4d.resolve")
|
||||
|
||||
TRIANGULATION_MAX_GAP = 0.5 # scene units — reject looser two-view intersections (spec M8)
|
||||
NEAREST_RADIUS = 0.3 # scene units — cylinder for the point-cloud fallback (spec M8)
|
||||
PARALLEL_DOT = 0.9995 # |cos(angle)| above this => rays too parallel to triangulate
|
||||
|
||||
|
||||
@dataclass
|
||||
class Resolution:
|
||||
"""Outcome of resolving one annotation to a 3D point."""
|
||||
|
||||
point: list[float] | None
|
||||
method: str # 'triangulated' | 'nearest_point' | 'ray_depth' | 'no_geometry'
|
||||
gap: float | None = None # mutual-approach distance when method == 'triangulated'
|
||||
|
||||
|
||||
def _pose_tuple(p) -> tuple[list[float], list[float], tuple[float, float, float, float]]:
|
||||
return (
|
||||
[p.qw, p.qx, p.qy, p.qz],
|
||||
[p.tx, p.ty, p.tz],
|
||||
(p.fx, p.fy, p.cx, p.cy),
|
||||
)
|
||||
|
||||
|
||||
def _pose_at(poses, t_video_s: float):
|
||||
"""Interpolate a COLMAP pose ``(q, t, (fx,fy,cx,cy))`` at ``t_video_s``.
|
||||
|
||||
``poses`` is ascending by ``t_video_s`` (as returned by :func:`db.get_poses`). Clamps to
|
||||
the endpoints — no extrapolation, matching the frontend ``poseAt`` and spec M2.
|
||||
"""
|
||||
if not poses:
|
||||
return None
|
||||
if t_video_s <= poses[0].t_video_s:
|
||||
return _pose_tuple(poses[0])
|
||||
if t_video_s >= poses[-1].t_video_s:
|
||||
return _pose_tuple(poses[-1])
|
||||
|
||||
lo, hi = 0, len(poses) - 1
|
||||
while hi - lo > 1:
|
||||
mid = (lo + hi) // 2
|
||||
if poses[mid].t_video_s <= t_video_s:
|
||||
lo = mid
|
||||
else:
|
||||
hi = mid
|
||||
a, b = poses[lo], poses[hi]
|
||||
span = b.t_video_s - a.t_video_s
|
||||
alpha = (t_video_s - a.t_video_s) / span if span > 1e-9 else 0.0
|
||||
q, t = geometry.slerp_pose(
|
||||
[a.qw, a.qx, a.qy, a.qz], [a.tx, a.ty, a.tz],
|
||||
[b.qw, b.qx, b.qy, b.qz], [b.tx, b.ty, b.tz],
|
||||
alpha,
|
||||
)
|
||||
# Intrinsics don't interpolate meaningfully across a fixed camera; use the earlier pose's.
|
||||
return list(q), list(t), (a.fx, a.fy, a.cx, a.cy)
|
||||
|
||||
|
||||
def _ray_for_annotation(video_id: int, t_video_s: float, bbox):
|
||||
"""World-space ray (origin, unit direction) through the bbox center, or ``None``.
|
||||
|
||||
Returns ``None`` when the video has no poses (reconstruction absent for that camera).
|
||||
"""
|
||||
video = db.get_video(video_id)
|
||||
if video is None:
|
||||
raise KeyError(f"no video with id={video_id}")
|
||||
pose = _pose_at(db.get_poses(video_id), t_video_s)
|
||||
if pose is None:
|
||||
return None
|
||||
q, t, (fx, fy, cx, cy) = pose
|
||||
x0, y0, x1, y1 = bbox
|
||||
px = (x0 + x1) / 2.0 * video.width
|
||||
py = (y0 + y1) / 2.0 * video.height
|
||||
return geometry.ray_from_pixel(q, t, fx, fy, cx, cy, px, py)
|
||||
|
||||
|
||||
def _load_points() -> np.ndarray | None:
|
||||
if not config.POINTS_PLY.exists():
|
||||
return None
|
||||
try:
|
||||
pts, _colors = read_ply(config.POINTS_PLY)
|
||||
except Exception as exc: # pragma: no cover - defensive
|
||||
log.warning("resolve: failed to read point cloud: %s", exc)
|
||||
return None
|
||||
return np.asarray(pts, dtype=np.float64).reshape(-1, 3)
|
||||
|
||||
|
||||
def resolve_annotation(video_id: int, t_video_s: float, bbox, event_id: int | None = None) -> Resolution:
|
||||
"""Resolve one annotation to a 3D point (see module docstring for the cascade)."""
|
||||
ray = _ray_for_annotation(video_id, t_video_s, bbox)
|
||||
if ray is None:
|
||||
return Resolution(point=None, method="no_geometry")
|
||||
origin, direction = ray
|
||||
|
||||
# 1. Two-view triangulation against another annotation of the same event.
|
||||
if event_id is not None:
|
||||
for other in reversed(db.get_annotations()): # most recent first
|
||||
if other.event_id != event_id or other.video_id == video_id:
|
||||
continue
|
||||
other_ray = _ray_for_annotation(
|
||||
other.video_id, other.t_video_s, [other.x0, other.y0, other.x1, other.y1]
|
||||
)
|
||||
if other_ray is None:
|
||||
continue
|
||||
o2, d2 = other_ray
|
||||
if abs(float(np.dot(direction, d2))) >= PARALLEL_DOT:
|
||||
continue # near-parallel: no reliable intersection
|
||||
point, gap = geometry.triangulate_rays(origin, direction, o2, d2)
|
||||
if gap <= TRIANGULATION_MAX_GAP:
|
||||
return Resolution(
|
||||
point=[float(v) for v in point], method="triangulated", gap=float(gap)
|
||||
)
|
||||
|
||||
# 2/3. Point-cloud fallbacks.
|
||||
pts = _load_points()
|
||||
if pts is not None and len(pts):
|
||||
near = geometry.nearest_point_on_ray(origin, direction, pts, NEAREST_RADIUS)
|
||||
if near is not None:
|
||||
return Resolution(point=[float(v) for v in near], method="nearest_point")
|
||||
centroid = pts.mean(axis=0)
|
||||
depth = float(np.linalg.norm(centroid - origin))
|
||||
pt = origin + direction * depth
|
||||
return Resolution(point=[float(v) for v in pt], method="ray_depth")
|
||||
|
||||
# 4. No point cloud at all: park at unit depth so a usable anchor still comes back.
|
||||
pt = origin + direction * 1.0
|
||||
return Resolution(point=[float(v) for v in pt], method="ray_depth")
|
||||
@ -103,10 +103,26 @@ def test_detect_events_endpoint(client):
|
||||
assert client.post("/api/events/detect", json={}).status_code == 200 # idempotent
|
||||
|
||||
|
||||
def test_annotation_stored(client):
|
||||
def test_annotation_resolves_to_anchor(client):
|
||||
# M8 (integration): a stored annotation now resolves to a 3D anchor. Single-view (no
|
||||
# event_id) falls back to the nearest point-cloud point / centroid-depth ray, so an
|
||||
# anchor + point come back. (Foundation asserted the stub contract anchor_id is None;
|
||||
# superseded by M8 landing — same situation as CR-1.)
|
||||
data = client.post(
|
||||
"/api/annotations",
|
||||
json={"video_id": 1, "t_video_s": 0.5, "bbox": [0.4, 0.4, 0.6, 0.6]},
|
||||
).json()
|
||||
assert data["annotation_id"] > 0
|
||||
assert data["anchor_id"] is None # M8 resolution is integration work
|
||||
assert data["anchor_id"] is not None
|
||||
assert data["point"] is not None and len(data["point"]) == 3
|
||||
assert data["method"] in {"nearest_point", "ray_depth", "triangulated"}
|
||||
assert data["anchor"]["id"] == data["anchor_id"]
|
||||
|
||||
|
||||
def test_event_patch_sets_user_source(client):
|
||||
events = client.get("/api/events").json()
|
||||
eid = events[0]["id"]
|
||||
patched = client.patch(f"/api/events/{eid}", json={"event_type": "pyro"}).json()
|
||||
assert patched["event_type"] == "pyro"
|
||||
assert patched["source"] == "user" # user correction attribution
|
||||
assert client.patch("/api/events/99999", json={"event_type": "x"}).status_code == 404
|
||||
|
||||
127
backend/tests/test_resolve.py
Normal file
127
backend/tests/test_resolve.py
Normal file
@ -0,0 +1,127 @@
|
||||
"""M8 annotation -> 3D resolution tests (integration phase).
|
||||
|
||||
The headline acceptance (spec M8): annotating the *same* stage corner from two videos
|
||||
produces an anchor within 0.2 scene units of the true corner. We build the synthetic fixture
|
||||
without ffmpeg (``run_ffmpeg=False`` populates the DB + point cloud only), forward-project a
|
||||
known corner into each camera to get faithful bbox centers, then resolve and check the error.
|
||||
|
||||
Also covers: the single-view nearest-point fallback, the near-parallel/gap rejection path,
|
||||
and the ``update_event`` correction helper.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from festival4d import config, db, resolve, synthetic
|
||||
from festival4d.geometry import quat_to_mat
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def fixture():
|
||||
# No ffmpeg: this writes videos/poses/anchors/events into the DB and points.ply to WORK_DIR.
|
||||
synthetic.build(run_ffmpeg=False)
|
||||
return synthetic.STAGE_CORNERS
|
||||
|
||||
|
||||
def _project(video, pose, world_pt):
|
||||
"""Forward-project a world point into a video, returning (px, py, z_cam).
|
||||
|
||||
Mirrors the COLMAP pinhole the fixture uses: x_cam = R@X + t; px = fx*x/z + cx (y-down).
|
||||
"""
|
||||
q = [pose.qw, pose.qx, pose.qy, pose.qz]
|
||||
t = np.array([pose.tx, pose.ty, pose.tz])
|
||||
R = quat_to_mat(q)
|
||||
x_cam = R @ np.asarray(world_pt, dtype=float) + t
|
||||
z = x_cam[2]
|
||||
px = pose.fx * x_cam[0] / z + pose.cx
|
||||
py = pose.fy * x_cam[1] / z + pose.cy
|
||||
return px, py, z
|
||||
|
||||
|
||||
def _bbox_center(px, py, W, H, half=0.02):
|
||||
"""A small normalized bbox centered on the projected pixel (clamped to [0,1])."""
|
||||
cx, cy = px / W, py / H
|
||||
return [
|
||||
max(0.0, cx - half), max(0.0, cy - half),
|
||||
min(1.0, cx + half), min(1.0, cy + half),
|
||||
]
|
||||
|
||||
|
||||
def _views_seeing(corner):
|
||||
"""Videos whose first pose projects `corner` in front of the camera and inside the frame."""
|
||||
seen = []
|
||||
for v in db.get_videos():
|
||||
poses = db.get_poses(v.id)
|
||||
if not poses:
|
||||
continue
|
||||
p0 = poses[0]
|
||||
px, py, z = _project(v, p0, corner)
|
||||
if z > 0 and 0 <= px < v.width and 0 <= py < v.height:
|
||||
seen.append((v, p0, px, py))
|
||||
return seen
|
||||
|
||||
|
||||
def test_two_view_triangulation_recovers_corner(fixture):
|
||||
corners = fixture
|
||||
# Pick a corner visible in at least two cameras.
|
||||
chosen = None
|
||||
for label, world in corners.items():
|
||||
views = _views_seeing(world)
|
||||
if len(views) >= 2:
|
||||
chosen = (label, np.asarray(world, dtype=float), views)
|
||||
break
|
||||
assert chosen is not None, "no stage corner is visible in >=2 synthetic cameras"
|
||||
label, world, views = chosen
|
||||
|
||||
event = db.add_event(t_global_s=5.0, event_type="artist_moment", source="ai",
|
||||
description=f"located {label}")
|
||||
|
||||
# Annotate the corner from the first two cameras that see it, then resolve the second one
|
||||
# (which triangulates against the first).
|
||||
(v_a, pose_a, pxa, pya), (v_b, pose_b, pxb, pyb) = views[0], views[1]
|
||||
bbox_a = _bbox_center(pxa, pya, v_a.width, v_a.height)
|
||||
bbox_b = _bbox_center(pxb, pyb, v_b.width, v_b.height)
|
||||
|
||||
db.add_annotation(v_a.id, pose_a.t_video_s, *bbox_a, event_id=event.id)
|
||||
res = resolve.resolve_annotation(v_b.id, pose_b.t_video_s, bbox_b, event_id=event.id)
|
||||
|
||||
assert res.method == "triangulated", f"expected triangulation, got {res.method}"
|
||||
err = float(np.linalg.norm(np.asarray(res.point) - world))
|
||||
assert err < 0.2, f"triangulated {label} off by {err:.3f} units (>0.2)"
|
||||
# Sanity: the mutual-approach gap is tight for exact projections.
|
||||
assert res.gap is not None and res.gap < 0.2
|
||||
|
||||
|
||||
def test_single_view_falls_back_to_point_cloud(fixture):
|
||||
corners = fixture
|
||||
world = np.asarray(corners["Stage FL"], dtype=float)
|
||||
views = _views_seeing(world)
|
||||
assert views, "Stage FL should be visible in at least one camera"
|
||||
v, pose, px, py = views[0]
|
||||
bbox = _bbox_center(px, py, v.width, v.height)
|
||||
|
||||
# No event_id => no other view to triangulate => nearest point-cloud point (corner is in
|
||||
# the cloud) or centroid-depth ray. Either way a point comes back.
|
||||
res = resolve.resolve_annotation(v.id, pose.t_video_s, bbox, event_id=None)
|
||||
assert res.method in {"nearest_point", "ray_depth"}
|
||||
assert res.point is not None and len(res.point) == 3
|
||||
if res.method == "nearest_point":
|
||||
# The exact corner is seeded into the cloud, so the nearest hit should be very close.
|
||||
assert float(np.linalg.norm(np.asarray(res.point) - world)) < resolve.NEAREST_RADIUS
|
||||
|
||||
|
||||
def test_update_event_sets_user_source(fixture):
|
||||
ev = db.add_event(t_global_s=9.0, event_type="candidate", source="audio_auto",
|
||||
description="unlabeled")
|
||||
updated = db.update_event(ev.id, event_type="pyro", source="user")
|
||||
assert updated.event_type == "pyro"
|
||||
assert updated.source == "user"
|
||||
# Unspecified fields are preserved.
|
||||
assert updated.description == "unlabeled"
|
||||
|
||||
|
||||
def test_missing_video_raises(fixture):
|
||||
with pytest.raises(KeyError):
|
||||
resolve.resolve_annotation(9999, 0.0, [0.4, 0.4, 0.6, 0.6], event_id=None)
|
||||
@ -42,7 +42,11 @@
|
||||
#scene3d { position: absolute; inset: 0; width: 100%; height: 100%; display: block; }
|
||||
#scene-controls {
|
||||
position: absolute; top: 0.6rem; left: 0.6rem; display: flex; gap: 0.4rem; z-index: 5;
|
||||
align-items: center; flex-wrap: wrap; max-width: calc(100% - 260px);
|
||||
}
|
||||
#scene-controls .btn { padding: 0.25rem 0.5rem; font-size: 0.78rem; }
|
||||
#key-count { color: var(--accent); font-variant-numeric: tabular-nums; }
|
||||
.ctl-sep { width: 1px; height: 18px; background: var(--border); margin: 0 0.15rem; }
|
||||
#scene-hint {
|
||||
position: absolute; bottom: 0.5rem; left: 0.6rem; color: var(--dim);
|
||||
font-size: 0.72rem; z-index: 5; pointer-events: none;
|
||||
@ -150,6 +154,37 @@
|
||||
.bad { color: var(--bad); }
|
||||
.dim { color: var(--dim); }
|
||||
|
||||
/* M8 annotation correction panel */
|
||||
#annot-panel {
|
||||
position: absolute; top: 0.6rem; right: 0.6rem; width: 232px; z-index: 8;
|
||||
background: rgba(15,20,32,0.94); border: 1px solid var(--border); border-radius: 9px;
|
||||
font-size: 0.78rem; box-shadow: 0 8px 26px rgba(0,0,0,0.5); backdrop-filter: blur(5px);
|
||||
}
|
||||
.ap-head {
|
||||
display: flex; align-items: center; gap: 0.4rem; padding: 0.45rem 0.55rem;
|
||||
border-bottom: 1px solid var(--border);
|
||||
}
|
||||
.ap-dot { width: 9px; height: 9px; border-radius: 3px; flex: 0 0 auto; }
|
||||
.ap-head strong { flex: 1 1 auto; font-size: 0.8rem; }
|
||||
.ap-x {
|
||||
background: none; border: none; color: var(--dim); cursor: pointer; font-size: 0.85rem;
|
||||
padding: 0 0.2rem;
|
||||
}
|
||||
.ap-x:hover { color: var(--text); }
|
||||
.ap-body { padding: 0.5rem 0.55rem 0.6rem; display: flex; flex-direction: column; gap: 0.45rem; }
|
||||
.ap-row { display: flex; align-items: center; justify-content: space-between; gap: 0.4rem; color: var(--dim); }
|
||||
.ap-type { flex: 1 1 auto; margin-left: 0.4rem; background: var(--panel-2); color: var(--text);
|
||||
border: 1px solid var(--border); border-radius: 5px; padding: 0.2rem 0.3rem; text-transform: capitalize; }
|
||||
.ap-desc { color: var(--text); line-height: 1.35; }
|
||||
.ap-meta { text-transform: capitalize; }
|
||||
.ap-annotate { width: 100%; text-align: center; }
|
||||
.ap-hint { line-height: 1.35; min-height: 1.1em; }
|
||||
|
||||
.annot-rect {
|
||||
position: absolute; border: 1.5px solid var(--accent); background: rgba(89,212,153,0.14);
|
||||
pointer-events: none; z-index: 6; border-radius: 2px;
|
||||
}
|
||||
|
||||
/* Loading / error */
|
||||
#loading {
|
||||
position: fixed; inset: 0; display: flex; align-items: center; justify-content: center;
|
||||
@ -171,8 +206,16 @@
|
||||
<canvas id="scene3d"></canvas>
|
||||
<div id="scene-controls">
|
||||
<button class="btn active" id="btn-roam" title="Free roam (Esc)">Free roam</button>
|
||||
<span class="ctl-sep"></span>
|
||||
<button class="btn" id="btn-key" title="Add camera keyframe here (K)">＋ Key <b id="key-count">0</b></button>
|
||||
<button class="btn" id="btn-path" title="Play the keyframed camera path">▶ Path</button>
|
||||
<button class="btn" id="btn-path-export" title="Export path as JSON">⤓</button>
|
||||
<button class="btn" id="btn-path-import" title="Import path JSON">⤒</button>
|
||||
<button class="btn" id="btn-path-clear" title="Clear all keyframes">✕</button>
|
||||
<input type="file" id="path-file" accept="application/json,.json" hidden />
|
||||
</div>
|
||||
<div id="scene-hint">drag to orbit · click a camera or press 1–9 to snap · Esc to detach</div>
|
||||
<div id="scene-hint">drag to orbit · 1–9 snap to camera · K add keyframe · Esc detach</div>
|
||||
<div id="annot-panel"></div>
|
||||
</div>
|
||||
<div id="grid-pane">
|
||||
<div id="video-grid"></div>
|
||||
|
||||
@ -1,6 +1,262 @@
|
||||
// bbox drawing + submission → POST /api/annotations (spec M8).
|
||||
// STUB — this is phase-3 (integration) work; lane C leaves it as a clean seam. Drag a
|
||||
// rectangle on a video's overlay canvas → normalized bbox → POST /api/annotations; the
|
||||
// resolved anchor then appears live in overlays + the 3D scene.
|
||||
// Correction panel + bbox annotation → 3D (spec M8, integration phase).
|
||||
//
|
||||
// Clicking a timeline event opens a panel: it shows the classification, a dropdown to correct
|
||||
// the event_type (PATCH /api/events/{id}, source→'user'), and an "annotate location" mode.
|
||||
// In annotate mode the user drags a rectangle on ANY video's overlay; the normalized bbox is
|
||||
// POSTed to /api/annotations. The backend casts a ray through the bbox center and resolves a
|
||||
// 3D point (triangulate across two views of the same event, else nearest point-cloud point /
|
||||
// centroid-depth ray). The resolved anchor is pushed live into state so overlays + the 3D
|
||||
// scene show it immediately.
|
||||
|
||||
export {};
|
||||
import { state, emit, API_BASE } from "./state.js";
|
||||
import { contentRect } from "./videoGrid.js";
|
||||
import { eventColor } from "./timeline.js";
|
||||
|
||||
const EVENT_TYPES = [
|
||||
"bass_drop", "pyro", "confetti", "crowd_wave",
|
||||
"artist_moment", "light_show", "quiet_moment", "candidate", "other",
|
||||
];
|
||||
|
||||
export class Annotator {
|
||||
constructor() {
|
||||
this.panel = null;
|
||||
this.cells = [];
|
||||
this.currentEvent = null;
|
||||
this.annotating = false;
|
||||
this._rubber = null; // { cell, div, sx, sy }
|
||||
this._cleanup = [];
|
||||
}
|
||||
|
||||
init(panelEl, cells) {
|
||||
this.panel = panelEl;
|
||||
this.cells = cells;
|
||||
this.panel.style.display = "none";
|
||||
}
|
||||
|
||||
/** Open the correction panel for a timeline event. */
|
||||
openForEvent(ev) {
|
||||
this.currentEvent = ev;
|
||||
this.stopAnnotateMode();
|
||||
this._render();
|
||||
this.panel.style.display = "block";
|
||||
}
|
||||
|
||||
close() {
|
||||
this.stopAnnotateMode();
|
||||
this.currentEvent = null;
|
||||
if (this.panel) this.panel.style.display = "none";
|
||||
}
|
||||
|
||||
_render() {
|
||||
const ev = this.currentEvent;
|
||||
if (!ev) return;
|
||||
const options = EVENT_TYPES.map(
|
||||
(t) => `<option value="${t}"${t === ev.event_type ? " selected" : ""}>${t.replace(/_/g, " ")}</option>`
|
||||
).join("");
|
||||
this.panel.innerHTML = `
|
||||
<div class="ap-head">
|
||||
<span class="ap-dot" style="background:${eventColor(ev.event_type)}"></span>
|
||||
<strong>Event @ ${ev.t_global_s.toFixed(1)}s</strong>
|
||||
<button class="ap-x" title="Close">✕</button>
|
||||
</div>
|
||||
<div class="ap-body">
|
||||
<label class="ap-row">type
|
||||
<select class="ap-type">${options}</select>
|
||||
</label>
|
||||
<div class="ap-desc">${ev.description || "<span class='dim'>no description</span>"}</div>
|
||||
<div class="ap-meta dim">${ev.source}${ev.confidence != null ? ` · ${(ev.confidence * 100).toFixed(0)}%` : ""}</div>
|
||||
<button class="ap-annotate btn">📍 Annotate location</button>
|
||||
<div class="ap-hint dim"></div>
|
||||
</div>`;
|
||||
|
||||
this.panel.querySelector(".ap-x").addEventListener("click", () => this.close());
|
||||
this.panel.querySelector(".ap-type").addEventListener("change", (e) =>
|
||||
this._correctType(e.target.value)
|
||||
);
|
||||
this.panel.querySelector(".ap-annotate").addEventListener("click", () =>
|
||||
this.annotating ? this.stopAnnotateMode() : this.startAnnotateMode()
|
||||
);
|
||||
}
|
||||
|
||||
_hint(msg) {
|
||||
const h = this.panel?.querySelector(".ap-hint");
|
||||
if (h) h.innerHTML = msg;
|
||||
}
|
||||
|
||||
async _correctType(newType) {
|
||||
const ev = this.currentEvent;
|
||||
if (!ev) return;
|
||||
try {
|
||||
const resp = await fetch(`${API_BASE}/api/events/${ev.id}`, {
|
||||
method: "PATCH",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ event_type: newType }),
|
||||
});
|
||||
if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
|
||||
const updated = await resp.json();
|
||||
// Reflect into state + timeline marker color.
|
||||
const i = state.events.findIndex((e) => e.id === ev.id);
|
||||
if (i >= 0) state.events[i] = updated;
|
||||
this.currentEvent = updated;
|
||||
emit("events-changed", updated);
|
||||
this._hint(`corrected → <b>${newType.replace(/_/g, " ")}</b> (source: user)`);
|
||||
} catch (err) {
|
||||
this._hint(`<span class="bad">correction failed: ${err.message}</span>`);
|
||||
}
|
||||
}
|
||||
|
||||
startAnnotateMode() {
|
||||
this.annotating = true;
|
||||
const btn = this.panel?.querySelector(".ap-annotate");
|
||||
if (btn) {
|
||||
btn.textContent = "✓ Done annotating";
|
||||
btn.classList.add("active");
|
||||
}
|
||||
this._hint("drag a box on any video over the object. Repeat on a 2nd video to triangulate.");
|
||||
for (const cell of this.cells) {
|
||||
cell.canvas.style.pointerEvents = "auto";
|
||||
cell.canvas.style.cursor = "crosshair";
|
||||
const down = (e) => this._onDown(cell, e);
|
||||
cell.canvas.addEventListener("pointerdown", down);
|
||||
this._cleanup.push(() => cell.canvas.removeEventListener("pointerdown", down));
|
||||
}
|
||||
}
|
||||
|
||||
stopAnnotateMode() {
|
||||
this.annotating = false;
|
||||
const btn = this.panel?.querySelector(".ap-annotate");
|
||||
if (btn) {
|
||||
btn.textContent = "📍 Annotate location";
|
||||
btn.classList.remove("active");
|
||||
}
|
||||
for (const cell of this.cells) {
|
||||
cell.canvas.style.pointerEvents = "none";
|
||||
cell.canvas.style.cursor = "";
|
||||
}
|
||||
this._teardownRubber();
|
||||
for (const fn of this._cleanup) fn();
|
||||
this._cleanup = [];
|
||||
}
|
||||
|
||||
_onDown(cell, e) {
|
||||
e.preventDefault();
|
||||
cell.canvas.setPointerCapture(e.pointerId);
|
||||
const rect = cell.el.getBoundingClientRect();
|
||||
const sx = e.clientX - rect.left;
|
||||
const sy = e.clientY - rect.top;
|
||||
const div = document.createElement("div");
|
||||
div.className = "annot-rect";
|
||||
cell.el.appendChild(div);
|
||||
this._rubber = { cell, div, sx, sy, rect };
|
||||
this._positionRubber(sx, sy);
|
||||
|
||||
const move = (ev) => this._onMove(ev);
|
||||
const up = (ev) => this._onUp(ev, move, up);
|
||||
cell.canvas.addEventListener("pointermove", move);
|
||||
cell.canvas.addEventListener("pointerup", up);
|
||||
cell.canvas.addEventListener("pointercancel", up);
|
||||
}
|
||||
|
||||
_positionRubber(cx, cy) {
|
||||
const r = this._rubber;
|
||||
if (!r) return;
|
||||
const x0 = Math.min(r.sx, cx);
|
||||
const y0 = Math.min(r.sy, cy);
|
||||
r.div.style.left = `${x0}px`;
|
||||
r.div.style.top = `${y0}px`;
|
||||
r.div.style.width = `${Math.abs(cx - r.sx)}px`;
|
||||
r.div.style.height = `${Math.abs(cy - r.sy)}px`;
|
||||
}
|
||||
|
||||
_onMove(e) {
|
||||
const r = this._rubber;
|
||||
if (!r) return;
|
||||
const rect = r.cell.el.getBoundingClientRect();
|
||||
this._positionRubber(e.clientX - rect.left, e.clientY - rect.top);
|
||||
}
|
||||
|
||||
async _onUp(e, move, up) {
|
||||
const r = this._rubber;
|
||||
if (!r) return;
|
||||
const { cell } = r;
|
||||
cell.canvas.removeEventListener("pointermove", move);
|
||||
cell.canvas.removeEventListener("pointerup", up);
|
||||
cell.canvas.removeEventListener("pointercancel", up);
|
||||
|
||||
const rect = cell.el.getBoundingClientRect();
|
||||
const ex = e.clientX - rect.left;
|
||||
const ey = e.clientY - rect.top;
|
||||
const bbox = this._toNormalizedBbox(cell, r.sx, r.sy, ex, ey);
|
||||
this._teardownRubber();
|
||||
if (!bbox) {
|
||||
this._hint('<span class="bad">draw the box over the video content area.</span>');
|
||||
return;
|
||||
}
|
||||
await this._submit(cell, bbox);
|
||||
}
|
||||
|
||||
_teardownRubber() {
|
||||
if (this._rubber?.div?.parentNode) this._rubber.div.parentNode.removeChild(this._rubber.div);
|
||||
this._rubber = null;
|
||||
}
|
||||
|
||||
/** Element-px rectangle → normalized [0,1] bbox in the video's CONTENT space (letterbox-aware). */
|
||||
_toNormalizedBbox(cell, ax, ay, bx, by) {
|
||||
const cr = contentRect(cell.video);
|
||||
const nx = (v) => (v - cr.ox) / cr.cw;
|
||||
const ny = (v) => (v - cr.oy) / cr.ch;
|
||||
let x0 = nx(Math.min(ax, bx));
|
||||
let y0 = ny(Math.min(ay, by));
|
||||
let x1 = nx(Math.max(ax, bx));
|
||||
let y1 = ny(Math.max(ay, by));
|
||||
x0 = Math.max(0, Math.min(1, x0));
|
||||
y0 = Math.max(0, Math.min(1, y0));
|
||||
x1 = Math.max(0, Math.min(1, x1));
|
||||
y1 = Math.max(0, Math.min(1, y1));
|
||||
// A near-zero drag (a click) becomes a tiny box centered on the point.
|
||||
if (x1 - x0 < 1e-4 && y1 - y0 < 1e-4) {
|
||||
const cx = (x0 + x1) / 2;
|
||||
const cy = (y0 + y1) / 2;
|
||||
x0 = Math.max(0, cx - 0.01); x1 = Math.min(1, cx + 0.01);
|
||||
y0 = Math.max(0, cy - 0.01); y1 = Math.min(1, cy + 0.01);
|
||||
}
|
||||
if (x1 <= x0 || y1 <= y0) return null;
|
||||
return [x0, y0, x1, y1];
|
||||
}
|
||||
|
||||
async _submit(cell, bbox) {
|
||||
const ev = this.currentEvent;
|
||||
this._hint("resolving 3D position…");
|
||||
try {
|
||||
const resp = await fetch(`${API_BASE}/api/annotations`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
video_id: cell.id,
|
||||
t_video_s: cell.video.currentTime,
|
||||
bbox,
|
||||
event_id: ev ? ev.id : null,
|
||||
}),
|
||||
});
|
||||
if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
|
||||
const res = await resp.json();
|
||||
if (res.anchor) {
|
||||
state.anchors.push(res.anchor);
|
||||
emit("anchors-changed", res.anchor);
|
||||
const how =
|
||||
res.method === "triangulated"
|
||||
? `triangulated (gap ${res.gap != null ? res.gap.toFixed(2) : "?"})`
|
||||
: res.method === "nearest_point"
|
||||
? "nearest point-cloud point"
|
||||
: "ray at scene depth";
|
||||
this._hint(`anchor placed via <b>${how}</b>. Annotate another view to refine.`);
|
||||
} else {
|
||||
this._hint('<span class="bad">could not resolve a 3D point (no poses?).</span>');
|
||||
}
|
||||
} catch (err) {
|
||||
this._hint(`<span class="bad">annotation failed: ${err.message}</span>`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export const annotator = new Annotator();
|
||||
|
||||
@ -1,6 +1,166 @@
|
||||
// Keyframed camera paths (spec M9).
|
||||
// STUB — this is phase-3 (integration) work; lane C leaves it as a clean seam. "Add
|
||||
// keyframe" captures the current free-roam pose at the current t_global; playback
|
||||
// interpolates position (Catmull-Rom) + orientation (slerp); export/import path as JSON.
|
||||
// Keyframed "god's-eye" camera paths (spec M9, integration phase).
|
||||
//
|
||||
// "Add keyframe" captures the current free-roam camera pose at the current t_global. Keyframes
|
||||
// render as octahedron gizmos with a smooth preview line. On playback (while the transport
|
||||
// plays) the viewer camera follows the path: position is interpolated with a Catmull-Rom curve
|
||||
// through the keyframe centers, orientation with quaternion slerp, both parameterized by
|
||||
// t_global between the first and last keyframe. Paths export/import as JSON.
|
||||
|
||||
export {};
|
||||
import * as THREE from "three";
|
||||
import { state } from "./state.js";
|
||||
import { scene3d } from "./scene3d.js";
|
||||
|
||||
const _qa = new THREE.Quaternion();
|
||||
const _qb = new THREE.Quaternion();
|
||||
const _pos = new THREE.Vector3();
|
||||
|
||||
export class CamPath {
|
||||
constructor() {
|
||||
this.keyframes = []; // [{ t_global, pos:[x,y,z], quat:[x,y,z,w], fov }]
|
||||
this.playing = false;
|
||||
this.group = null; // gizmos + preview line
|
||||
this._curve = null;
|
||||
this.onChange = null; // optional UI callback (count)
|
||||
}
|
||||
|
||||
_ensureGroup() {
|
||||
if (!this.group) {
|
||||
this.group = new THREE.Group();
|
||||
scene3d.addObject(this.group);
|
||||
}
|
||||
}
|
||||
|
||||
/** Capture the current viewer-camera pose at the current master time. */
|
||||
addKeyframe() {
|
||||
const cam = scene3d.camera;
|
||||
if (!cam) return;
|
||||
this.keyframes.push({
|
||||
t_global: state.tGlobal,
|
||||
pos: [cam.position.x, cam.position.y, cam.position.z],
|
||||
quat: [cam.quaternion.x, cam.quaternion.y, cam.quaternion.z, cam.quaternion.w],
|
||||
fov: cam.fov,
|
||||
});
|
||||
this.keyframes.sort((a, b) => a.t_global - b.t_global);
|
||||
this._rebuild();
|
||||
}
|
||||
|
||||
clear() {
|
||||
this.stop();
|
||||
this.keyframes = [];
|
||||
this._rebuild();
|
||||
}
|
||||
|
||||
count() {
|
||||
return this.keyframes.length;
|
||||
}
|
||||
|
||||
_rebuild() {
|
||||
this._ensureGroup();
|
||||
for (const child of [...this.group.children]) {
|
||||
this.group.remove(child);
|
||||
child.geometry?.dispose?.();
|
||||
child.material?.dispose?.();
|
||||
}
|
||||
this._curve = null;
|
||||
|
||||
for (const k of this.keyframes) {
|
||||
const giz = new THREE.Mesh(
|
||||
new THREE.OctahedronGeometry(0.16),
|
||||
new THREE.MeshBasicMaterial({ color: 0xffd23b, wireframe: true })
|
||||
);
|
||||
giz.position.set(k.pos[0], k.pos[1], k.pos[2]);
|
||||
this.group.add(giz);
|
||||
}
|
||||
|
||||
if (this.keyframes.length >= 2) {
|
||||
this._curve = new THREE.CatmullRomCurve3(
|
||||
this.keyframes.map((k) => new THREE.Vector3(k.pos[0], k.pos[1], k.pos[2]))
|
||||
);
|
||||
const pts = this._curve.getPoints(Math.max(20, this.keyframes.length * 12));
|
||||
const line = new THREE.Line(
|
||||
new THREE.BufferGeometry().setFromPoints(pts),
|
||||
new THREE.LineBasicMaterial({ color: 0xffd23b, transparent: true, opacity: 0.6 })
|
||||
);
|
||||
this.group.add(line);
|
||||
}
|
||||
this.onChange?.(this.keyframes.length);
|
||||
}
|
||||
|
||||
/** Begin following the path (requires >= 2 keyframes). Caller should also start the transport. */
|
||||
play() {
|
||||
if (this.keyframes.length < 2) return false;
|
||||
this.playing = true;
|
||||
scene3d.enterPathMode();
|
||||
return true;
|
||||
}
|
||||
|
||||
stop() {
|
||||
if (!this.playing) return;
|
||||
this.playing = false;
|
||||
scene3d.exitPathMode();
|
||||
}
|
||||
|
||||
/** Per-frame: if playing, drive the viewer camera from the path at the current master time. */
|
||||
update(tGlobal) {
|
||||
if (!this.playing) return;
|
||||
// If something else took the camera (snap-to-camera exits path mode), stop driving it.
|
||||
if (!scene3d.pathMode) {
|
||||
this.playing = false;
|
||||
this.onChange?.(this.keyframes.length);
|
||||
return;
|
||||
}
|
||||
if (!this._curve || this.keyframes.length < 2) return;
|
||||
const kf = this.keyframes;
|
||||
const n = kf.length;
|
||||
const t0 = kf[0].t_global;
|
||||
const t1 = kf[n - 1].t_global;
|
||||
|
||||
// Locate the bracketing keyframes by time; clamp outside the path's time span.
|
||||
let i = 0;
|
||||
if (tGlobal <= t0) {
|
||||
i = 0;
|
||||
} else if (tGlobal >= t1) {
|
||||
i = n - 2;
|
||||
} else {
|
||||
while (i < n - 2 && kf[i + 1].t_global <= tGlobal) i++;
|
||||
}
|
||||
const a = kf[i];
|
||||
const b = kf[i + 1];
|
||||
const span = b.t_global - a.t_global;
|
||||
let f = span > 1e-6 ? (tGlobal - a.t_global) / span : 0;
|
||||
f = Math.max(0, Math.min(1, f));
|
||||
|
||||
// Position: Catmull-Rom, parameterized by (segment index + local fraction) / (n - 1).
|
||||
const u = Math.max(0, Math.min(1, (i + f) / (n - 1)));
|
||||
this._curve.getPoint(u, _pos);
|
||||
|
||||
// Orientation: slerp the two bracketing keyframe quaternions.
|
||||
_qa.set(a.quat[0], a.quat[1], a.quat[2], a.quat[3]);
|
||||
_qb.set(b.quat[0], b.quat[1], b.quat[2], b.quat[3]);
|
||||
_qa.slerp(_qb, f);
|
||||
|
||||
const fov = a.fov + (b.fov - a.fov) * f;
|
||||
scene3d.applyCameraPose(_pos, _qa, fov);
|
||||
}
|
||||
|
||||
toJSON() {
|
||||
return JSON.stringify({ version: 1, keyframes: this.keyframes }, null, 2);
|
||||
}
|
||||
|
||||
fromJSON(text) {
|
||||
const data = typeof text === "string" ? JSON.parse(text) : text;
|
||||
if (!data || !Array.isArray(data.keyframes)) throw new Error("invalid camera-path file");
|
||||
this.keyframes = data.keyframes
|
||||
.filter((k) => Array.isArray(k.pos) && Array.isArray(k.quat))
|
||||
.map((k) => ({
|
||||
t_global: +k.t_global || 0,
|
||||
pos: k.pos.slice(0, 3).map(Number),
|
||||
quat: k.quat.slice(0, 4).map(Number),
|
||||
fov: +k.fov || 50,
|
||||
}))
|
||||
.sort((a, b) => a.t_global - b.t_global);
|
||||
this._rebuild();
|
||||
}
|
||||
}
|
||||
|
||||
export const camPath = new CamPath();
|
||||
|
||||
@ -14,6 +14,8 @@ import { drawOverlay, projectWorldToVideoPx } from "./overlays.js";
|
||||
import { poseAt } from "./lib/poseTrack.js";
|
||||
import { scene3d, videoColor } from "./scene3d.js";
|
||||
import { timeline } from "./timeline.js";
|
||||
import { annotator } from "./annotate.js";
|
||||
import { camPath } from "./camPath.js";
|
||||
|
||||
async function getJSON(path) {
|
||||
const resp = await fetch(API_BASE + path);
|
||||
@ -99,13 +101,71 @@ function buildUI() {
|
||||
// Timeline
|
||||
timeline.init(el("timeline"), transport);
|
||||
|
||||
// M8: correction panel + bbox annotation → 3D anchors.
|
||||
annotator.init(el("annot-panel"), cells);
|
||||
on("event-selected", (ev) => annotator.openForEvent(ev));
|
||||
on("anchors-changed", () => scene3d.refreshAnchors());
|
||||
on("events-changed", () => timeline.refresh());
|
||||
|
||||
wireTransportControls();
|
||||
wireCamPath();
|
||||
wireKeyboard();
|
||||
positionVideosWhenReady();
|
||||
|
||||
// Reflect follow/roam state on the free-roam button.
|
||||
// Reflect follow/roam state on the free-roam button; snapping cancels a playing path.
|
||||
on("follow", (id) => {
|
||||
el("btn-roam").classList.toggle("active", id == null);
|
||||
if (id != null) resetPathButton();
|
||||
});
|
||||
}
|
||||
|
||||
function resetPathButton() {
|
||||
const b = el("btn-path");
|
||||
b.textContent = "▶ Path";
|
||||
b.classList.remove("active");
|
||||
}
|
||||
|
||||
function wireCamPath() {
|
||||
camPath.onChange = (n) => {
|
||||
el("key-count").textContent = String(n);
|
||||
if (!camPath.playing) resetPathButton();
|
||||
};
|
||||
el("btn-key").addEventListener("click", () => camPath.addKeyframe());
|
||||
el("btn-path").addEventListener("click", () => {
|
||||
const b = el("btn-path");
|
||||
if (camPath.playing) {
|
||||
camPath.stop();
|
||||
resetPathButton();
|
||||
} else if (camPath.play()) {
|
||||
b.textContent = "⏹ Path";
|
||||
b.classList.add("active");
|
||||
if (!state.playing) transport.toggle(); // advance master time so the path animates
|
||||
}
|
||||
});
|
||||
el("btn-path-export").addEventListener("click", () => {
|
||||
const blob = new Blob([camPath.toJSON()], { type: "application/json" });
|
||||
const a = document.createElement("a");
|
||||
a.href = URL.createObjectURL(blob);
|
||||
a.download = "festival4d-campath.json";
|
||||
a.click();
|
||||
URL.revokeObjectURL(a.href);
|
||||
});
|
||||
const fileInput = el("path-file");
|
||||
el("btn-path-import").addEventListener("click", () => fileInput.click());
|
||||
fileInput.addEventListener("change", async () => {
|
||||
const f = fileInput.files?.[0];
|
||||
if (f) {
|
||||
try {
|
||||
camPath.fromJSON(await f.text());
|
||||
} catch (err) {
|
||||
console.error("[festival4d] camera-path import failed", err);
|
||||
}
|
||||
}
|
||||
fileInput.value = "";
|
||||
});
|
||||
el("btn-path-clear").addEventListener("click", () => {
|
||||
camPath.clear();
|
||||
resetPathButton();
|
||||
});
|
||||
}
|
||||
|
||||
@ -116,7 +176,13 @@ function wireTransportControls() {
|
||||
el("rate-select").addEventListener("change", (e) =>
|
||||
transport.setRate(parseFloat(e.target.value))
|
||||
);
|
||||
el("btn-roam").addEventListener("click", () => scene3d.freeRoam());
|
||||
el("btn-roam").addEventListener("click", () => {
|
||||
if (camPath.playing) {
|
||||
camPath.stop();
|
||||
resetPathButton();
|
||||
}
|
||||
scene3d.freeRoam();
|
||||
});
|
||||
}
|
||||
|
||||
function wireKeyboard() {
|
||||
@ -132,7 +198,14 @@ function wireKeyboard() {
|
||||
e.preventDefault();
|
||||
transport.seekBy(e.shiftKey ? 5 : 1);
|
||||
} else if (e.code === "Escape" || e.code === "Digit0") {
|
||||
if (camPath.playing) {
|
||||
camPath.stop();
|
||||
resetPathButton();
|
||||
}
|
||||
scene3d.freeRoam();
|
||||
} else if (e.code === "KeyK") {
|
||||
e.preventDefault();
|
||||
camPath.addKeyframe();
|
||||
} else if (/^Digit[1-9]$/.test(e.code)) {
|
||||
const n = parseInt(e.code.slice(5), 10) - 1;
|
||||
const v = state.videos[n];
|
||||
@ -157,6 +230,7 @@ function startLoop() {
|
||||
const frame = () => {
|
||||
transport.tick();
|
||||
for (const cell of cells) drawOverlay(cell);
|
||||
camPath.update(state.tGlobal); // M9: drive the viewer camera before the scene renders
|
||||
scene3d.update();
|
||||
timeline.update(state.tGlobal);
|
||||
updateHud();
|
||||
|
||||
@ -26,12 +26,42 @@ const _pos = new THREE.Vector3();
|
||||
const _quat = new THREE.Quaternion();
|
||||
const _scale = new THREE.Vector3();
|
||||
|
||||
/** Build a camera-facing text label sprite (used for 3D anchor labels). */
|
||||
function makeLabelSprite(text) {
|
||||
const font = 48;
|
||||
const pad = 10;
|
||||
const c = document.createElement("canvas");
|
||||
let ctx = c.getContext("2d");
|
||||
ctx.font = `600 ${font}px -apple-system, system-ui, sans-serif`;
|
||||
const w = Math.ceil(ctx.measureText(text).width) + pad * 2;
|
||||
const h = font + pad * 2;
|
||||
c.width = w;
|
||||
c.height = h;
|
||||
ctx = c.getContext("2d");
|
||||
ctx.font = `600 ${font}px -apple-system, system-ui, sans-serif`;
|
||||
ctx.fillStyle = "rgba(6,8,14,0.74)";
|
||||
ctx.fillRect(0, 0, w, h);
|
||||
ctx.fillStyle = "#e6e8ee";
|
||||
ctx.textBaseline = "middle";
|
||||
ctx.fillText(text, pad, h / 2 + 1);
|
||||
const tex = new THREE.CanvasTexture(c);
|
||||
tex.minFilter = THREE.LinearFilter;
|
||||
const sprite = new THREE.Sprite(
|
||||
new THREE.SpriteMaterial({ map: tex, depthTest: false, transparent: true })
|
||||
);
|
||||
const s = 0.006;
|
||||
sprite.scale.set(w * s, h * s, 1);
|
||||
return sprite;
|
||||
}
|
||||
|
||||
export class Scene3D {
|
||||
constructor() {
|
||||
this.videoIndex = {}; // id -> index (for stable colors)
|
||||
this.rigs = {}; // id -> { frustum, marker, pick, path }
|
||||
this._tween = null;
|
||||
this._raf = null;
|
||||
this.anchorGroup = null; // 3D anchor spheres + labels (M6 anchors + M8 resolved)
|
||||
this.pathMode = false; // when true, camPath (M9) drives the camera; we don't touch it
|
||||
}
|
||||
|
||||
init(canvas) {
|
||||
@ -64,12 +94,76 @@ export class Scene3D {
|
||||
const axes = new THREE.AxesHelper(1.5);
|
||||
this.scene.add(axes);
|
||||
|
||||
this.anchorGroup = new THREE.Group();
|
||||
this.scene.add(this.anchorGroup);
|
||||
|
||||
this._buildRigs();
|
||||
this._loadPointCloud();
|
||||
this.refreshAnchors();
|
||||
this._wirePicking();
|
||||
this._wireResize(parent);
|
||||
}
|
||||
|
||||
/** Rebuild the 3D anchor markers from state.anchors (M6 seeded + M8 resolved). */
|
||||
refreshAnchors() {
|
||||
if (!this.anchorGroup) return;
|
||||
for (const child of [...this.anchorGroup.children]) {
|
||||
this.anchorGroup.remove(child);
|
||||
child.geometry?.dispose?.();
|
||||
if (child.material) {
|
||||
child.material.map?.dispose?.();
|
||||
child.material.dispose();
|
||||
}
|
||||
}
|
||||
for (const a of state.anchors) {
|
||||
const color = new THREE.Color(a.color || "#59d499");
|
||||
const sphere = new THREE.Mesh(
|
||||
new THREE.SphereGeometry(0.13, 16, 16),
|
||||
new THREE.MeshBasicMaterial({ color })
|
||||
);
|
||||
sphere.position.set(a.x, a.y, a.z);
|
||||
this.anchorGroup.add(sphere);
|
||||
if (a.label) {
|
||||
const label = makeLabelSprite(a.label);
|
||||
label.position.set(a.x, a.y + 0.3, a.z);
|
||||
this.anchorGroup.add(label);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Expose the scene graph for auxiliary overlays (e.g. camPath gizmos, M9). */
|
||||
addObject(obj) {
|
||||
this.scene?.add(obj);
|
||||
}
|
||||
removeObject(obj) {
|
||||
this.scene?.remove(obj);
|
||||
}
|
||||
|
||||
/** M9: hand camera control to camPath. update() then leaves the camera alone. */
|
||||
enterPathMode() {
|
||||
this.pathMode = true;
|
||||
this.controls.enabled = false;
|
||||
state.followCameraId = null;
|
||||
this._tween = null;
|
||||
emit("follow", null);
|
||||
}
|
||||
exitPathMode() {
|
||||
if (!this.pathMode) return;
|
||||
this.pathMode = false;
|
||||
this.controls.target.copy(STAGE_TARGET);
|
||||
this.controls.enabled = true;
|
||||
this.controls.update();
|
||||
}
|
||||
/** M9: set the camera pose directly (called by camPath each frame while path-playing). */
|
||||
applyCameraPose(pos, quat, fov) {
|
||||
this.camera.position.copy(pos);
|
||||
this.camera.quaternion.copy(quat);
|
||||
if (fov != null && Math.abs(this.camera.fov - fov) > 1e-3) {
|
||||
this.camera.fov = fov;
|
||||
this.camera.updateProjectionMatrix();
|
||||
}
|
||||
}
|
||||
|
||||
_buildRigs() {
|
||||
state.videos.forEach((v, i) => {
|
||||
this.videoIndex[v.id] = i;
|
||||
@ -177,6 +271,7 @@ export class Scene3D {
|
||||
snapTo(videoId) {
|
||||
const pose = this._livePose(videoId);
|
||||
if (!pose) return;
|
||||
if (this.pathMode) this.exitPathMode();
|
||||
state.followCameraId = videoId;
|
||||
this.controls.enabled = false;
|
||||
const targetFov = (2 * Math.atan(state.videos.find((v) => v.id === videoId).height /
|
||||
@ -244,7 +339,9 @@ export class Scene3D {
|
||||
_m4.decompose(rig.group.position, rig.group.quaternion, rig.group.scale);
|
||||
}
|
||||
|
||||
if (state.followCameraId != null) {
|
||||
if (this.pathMode) {
|
||||
// camPath (M9) already set the camera pose this frame; don't fight it.
|
||||
} else if (state.followCameraId != null) {
|
||||
const pose = this._livePose(state.followCameraId);
|
||||
if (pose) {
|
||||
const meta = state.videos.find((v) => v.id === state.followCameraId);
|
||||
|
||||
@ -5,7 +5,7 @@
|
||||
// tooltip; click a marker -> jump the playhead there. Event data is already seeded
|
||||
// synthetically; classification QUALITY is lane D's concern, not this renderer's.
|
||||
|
||||
import { state } from "./state.js";
|
||||
import { state, emit } from "./state.js";
|
||||
|
||||
export const EVENT_COLORS = {
|
||||
bass_drop: "#ff3b6b",
|
||||
@ -58,8 +58,10 @@ export class Timeline {
|
||||
tip.style.display = "none";
|
||||
|
||||
root.append(track, tip);
|
||||
this.root = root;
|
||||
this.track = track;
|
||||
this.fill = fill;
|
||||
this.markers = markers;
|
||||
this.playhead = playhead;
|
||||
this.tip = tip;
|
||||
|
||||
@ -68,6 +70,15 @@ export class Timeline {
|
||||
this._wireScrub(track);
|
||||
}
|
||||
|
||||
/** Rebuild markers + legend after events change (e.g. an M8 type correction). */
|
||||
refresh() {
|
||||
if (!this.markers) return;
|
||||
this.markers.innerHTML = "";
|
||||
this._buildMarkers(this.markers, this.tip);
|
||||
this.root.querySelector(".tl-legend")?.remove();
|
||||
this._buildLegend(this.root);
|
||||
}
|
||||
|
||||
_buildMarkers(container, tip) {
|
||||
for (const ev of state.events) {
|
||||
const m = document.createElement("div");
|
||||
@ -77,6 +88,7 @@ export class Timeline {
|
||||
m.addEventListener("click", (e) => {
|
||||
e.stopPropagation();
|
||||
this.transport.seek(ev.t_global_s);
|
||||
emit("event-selected", ev); // open the M8 correction panel
|
||||
});
|
||||
const show = (e) => {
|
||||
tip.innerHTML =
|
||||
|
||||
@ -31,5 +31,28 @@ Instead append an entry here and work around it locally; the integration agent a
|
||||
- **Workaround in place:** applied the minimal edit directly (test file, not a frozen contract
|
||||
file). The frozen `api.py` app is **unchanged**. Only lane D's owned modules + this obsolete
|
||||
stub assertion were touched. Full suite green (89 passed).
|
||||
- **Decision:** (integration) — informational; the change is a direct, unavoidable consequence
|
||||
of lane D landing. Confirm the assertion matches the intended M3 detect-response shape.
|
||||
- **Decision:** **APPLIED** (integration, 2026-07-16). Confirmed against the frozen api.py
|
||||
docstring: `POST /api/events/detect` returns `{result, events}` (spec M3). The renamed
|
||||
`test_detect_events_endpoint` asserts exactly that shape and is idempotent. No frozen file
|
||||
changed; approval also recorded in DIRECTIVES round 1.
|
||||
|
||||
### CR-2 — integration — 2026-07-16 — M8 needs annotation↔event link + event-type correction
|
||||
- **File:** `db.py` (`Annotation` schema), `api.py` (`AnnotationIn`, events routes)
|
||||
- **Problem:** Spec M8 requires (a) triangulating "another annotation **for the same event**
|
||||
from a different video" — but the frozen `annotations` schema has no event reference, and
|
||||
`AnnotationIn` carries no event id, so annotations can't be grouped per event; and (b) a
|
||||
correction dropdown that "writes `source='user'`" to an event's type — but there is no
|
||||
event-update helper or endpoint (only add/clear).
|
||||
- **Proposed change (additive only — no existing shape altered):**
|
||||
1. `annotations.event_id` — new **nullable** `ForeignKey("events.id")` column. Nullable ⇒
|
||||
existing rows/inserts unaffected; the synthetic fixture creates annotations = none, so no
|
||||
migration. Serves the two-view grouping.
|
||||
2. `AnnotationIn.event_id: int | None = None` — additive optional request field.
|
||||
3. `db.update_event(event_id, event_type=?, source=?, description=?)` helper +
|
||||
`PATCH /api/events/{event_id}` — additive route; existing GET/POST/detect unchanged.
|
||||
- **Workaround in place:** none needed — adjudicated and applied directly in this integration
|
||||
phase (the phase where the ownership matrix ends and frozen contracts may change on an
|
||||
approved CR). All changes are strictly additive; every pre-existing route/shape/column and
|
||||
the timebase + pose contracts are byte-for-byte unchanged. Full suite stays green.
|
||||
- **Decision:** **APPLIED** (integration, 2026-07-16). Additive, backward-compatible, and
|
||||
required for M8 acceptance. Frontend and backend M8 both depend on it.
|
||||
|
||||
66
plan/status/integration.md
Normal file
66
plan/status/integration.md
Normal file
@ -0,0 +1,66 @@
|
||||
# Status — integration
|
||||
|
||||
## Round 1 — 2026-07-16 — STATUS: ready_to_merge
|
||||
|
||||
**Directives acknowledged:** round 1 of plan/DIRECTIVES.md (all lanes merged & verified;
|
||||
integration begins; CR-1 approved; drop debris stash + remove laneA/B worktrees; real-browser
|
||||
sync check; test floor 96).
|
||||
|
||||
**Scope done: M8 + M9 + README + cross-lane wiring + cleanup.**
|
||||
|
||||
### Change requests adjudicated (plan/CHANGE_REQUESTS.md)
|
||||
- **CR-1 (lane D): APPLIED** — confirmed `POST /api/events/detect` returns `{result, events}`;
|
||||
renamed test asserts it.
|
||||
- **CR-2 (integration, filed + APPLIED):** M8 needs an annotation↔event link and event-type
|
||||
correction, neither of which the frozen contract had. Added — strictly additive:
|
||||
- `annotations.event_id` (nullable FK) + `AnnotationIn.event_id` — two-view grouping.
|
||||
- `db.update_event` + `db.get_event` + `PATCH /api/events/{id}` — the correction dropdown.
|
||||
Every pre-existing route/shape/column and the timebase + pose contracts are unchanged.
|
||||
- A second foundation stub assertion (`test_annotation_stored`: `anchor_id is None`) was, like
|
||||
CR-1, only valid pre-M8; renamed to `test_annotation_resolves_to_anchor` for the landed shape.
|
||||
|
||||
### M8 — correction + annotation → 3D
|
||||
- New `backend/festival4d/resolve.py`: ray from bbox center (pose interpolated at t_video via
|
||||
`geometry.slerp_pose`), then triangulate across two same-event views (reject near-parallel /
|
||||
gap > 0.5), else nearest point-cloud point (0.3 cyl), else ray at centroid depth. Pure
|
||||
orchestration over lane B's frozen primitives.
|
||||
- `POST /api/annotations` now resolves + creates+links an anchor (label = event description).
|
||||
- Frontend `annotate.js`: click a timeline marker → correction panel (type dropdown → PATCH,
|
||||
source→user; live marker recolor); **Annotate location** → drag a letterbox-correct box on
|
||||
any video → POST. Resolved anchors pushed live into overlays **and** the 3D scene (new anchor
|
||||
spheres + labels in `scene3d.js`).
|
||||
- **Acceptance (spec M8): PASS.** `tests/test_resolve.py::test_two_view_triangulation_recovers_corner`
|
||||
— two-view stage-corner annotation lands **0.00000 units** from ground truth (< 0.2). Verified
|
||||
live in-browser too: two-view annotation triangulated with gap 0.21 through the full
|
||||
frontend→API→geometry path; correction changed pyro→confetti with source=user and recolored
|
||||
the marker.
|
||||
|
||||
### M9 — keyframed camera paths
|
||||
- New `camPath.js`: **＋ Key**/`K` captures the free-roam pose at the current t_global; gizmos +
|
||||
Catmull-Rom preview line; **▶ Path** follows position (Catmull-Rom) + orientation (slerp) by
|
||||
t_global while the transport plays; ⤓/⤒ export/import JSON. `scene3d` gained
|
||||
`enterPathMode`/`exitPathMode`/`applyCameraPose`; snap-to-camera and Free-roam/`Esc` cancel a
|
||||
playing path.
|
||||
- **Verified live:** 3 keyframes → path mode on → at t=6.0 s the viewer camera sat exactly on
|
||||
keyframe #2 `[-10,6,12]` (Catmull-Rom driven).
|
||||
- Polish present from lanes + this phase: loading state, per-video enable/disable, keyboard
|
||||
(`Space`,`←/→`,`Shift`,`1–9`,`Esc/0`,`K`).
|
||||
|
||||
### Full-system pass
|
||||
- **Backend: 101 passed** (`../.venv/bin/python -m pytest`) — up from the 96 merge-day floor
|
||||
(+5: 4 resolve + 1 event-patch; annotation test renamed).
|
||||
- **Frontend: `npm run build` clean** (18 modules).
|
||||
- End-to-end on the synthetic fixture: `synthetic → ingest → sync → events → serve` + frontend
|
||||
loads with 0 console errors; M4 sync error ~16 ms at t=8 s (deterministic pump; half-frame
|
||||
residual, < 50 ms).
|
||||
- Cleanup: dropped the shared-worktree debris stash; removed `../festifun-laneA` and
|
||||
`../festifun-laneB` worktrees; `frontend/dist/` gitignored; pristine demo DB regenerated.
|
||||
|
||||
**Blockers / notes for coordinator:**
|
||||
- **Real-focused-browser M4 check is the one item I could not fully close from here.** The
|
||||
automated browser pane runs `document.hidden=true`, so rAF is throttled and continuous
|
||||
playback can't be observed (same limitation lane C hit). I verified sync deterministically
|
||||
(pump → ~16 ms) and every interaction via real handlers, but a human should open the app in a
|
||||
normal focused window once to eyeball continuous multi-cam sync. Not code-blocking.
|
||||
|
||||
**Next:** merge `integration` → `main`; tag `v0.1.0`.
|
||||
Loading…
Reference in New Issue
Block a user