Foundation (M0 + M3 + frozen contracts): scaffold, synthetic fixtures, full API

What now works:
- M0 scaffold: pyproject (all spec deps), uv/py3.12 env, `python -m festival4d`
  CLI registering synthetic|ingest|sync|reconstruct|events|serve. Vite hello page.
- Synthetic fixture (synthetic.py): 3 shifted-audio videos (offsets 0/+1370/-842 ms),
  camera-arc poses, stage point cloud -> points.ply, seeded events + anchors,
  ground_truth.json. `python -m festival4d synthetic` populates data/ + DB.
- DB schema exactly per spec §2 (db.py) + CRUD helpers all lanes use.
- M3 API (api.py) full against synthetic data: manifest/poses/pointcloud/anchors/
  events/detect/annotations; Range-capable video serving (206 verified); CORS for any
  localhost origin.
- Frozen geometry contract: geometry.colmap_to_threejs (M5 math) + unit test (3 known
  vectors, random round-trip, scipy oracle); mirrored frontend/src/lib/pose.js with
  identical POSE_TEST_VECTORS. Lane-B stubs: slerp_pose, ray_from_pixel, triangulate_rays,
  nearest_point_on_ray.
- Classifier contract (events_ai.py): MomentClassification model + MomentClassifier
  protocol + Gemini/Claude/Local provider stubs.
- Lane-owned modules stubbed with final signatures (ingest, audio_sync, frames, sfm,
  events_ai); cli/api catch NotImplementedError and degrade gracefully.
- plan/CHANGE_REQUESTS.md created; plan/status/foundation.md updated.

Acceptance: pytest 24 passed; serve endpoints verified via curl + browser (video seek,
manifest fetch cross-origin, pose.js self-test, 0 console errors).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
m3ultra 2026-07-16 00:45:55 +10:00
parent 47bec93bc5
commit cf6a6fb2c8
35 changed files with 5930 additions and 0 deletions

57
README.md Normal file
View File

@ -0,0 +1,57 @@
# Festival 4D
Turn multiple fan-shot smartphone videos of the same concert into a synchronized,
explorable **4D experience**: time-aligned multi-video playback, 3D scene reconstruction
with camera poses, a free-roam "god's eye" viewer, AR-style overlays projected onto each
video, and AI-tagged moments on a shared timeline.
> **Status: foundation phase.** The scaffold, database, frozen contracts (API, pose math,
> DB schema), and a synthetic fixture generator are in place. Feature lanes (media sync,
> reconstruction, viewer, AI events) build on top. The full README with the real-footage
> workflow is written in the integration phase (M9). See
> [`OPUS_BUILD_INSTRUCTIONS.md`](OPUS_BUILD_INSTRUCTIONS.md) for the canonical spec and
> [`plan/`](plan/) for the execution plan.
## Prerequisites
- **Python 3.11+** and [`uv`](https://docs.astral.sh/uv/) (or venv + pip)
- **ffmpeg** / **ffprobe** on your `PATH` (required)
- **COLMAP** (optional — reconstruction degrades gracefully without it)
- A classifier API key (optional — `GEMINI_API_KEY` by default; see `events_ai.py`)
- **Node 18+** for the frontend
## Quickstart (synthetic demo — no footage needed)
```bash
# 1. backend env
uv venv --python 3.12
uv pip install -e ".[dev]"
# 2. generate the synthetic fixture project (fake videos + poses + point cloud + events)
uv run python -m festival4d synthetic
# 3. serve the API (http://127.0.0.1:8000)
uv run python -m festival4d serve
# 4. in another terminal, the frontend
cd frontend
npm install
npm run dev # http://localhost:5173
```
## Backend CLI
```
python -m festival4d synthetic # generate the synthetic fixture (M0)
python -m festival4d ingest # probe videos + extract audio (lane A / M1)
python -m festival4d sync # GCC-PHAT audio alignment (lane A / M1)
python -m festival4d reconstruct # COLMAP SfM + pose export (lane B / M2)
python -m festival4d events # audio candidates + AI classify (lane D / M7)
python -m festival4d serve # FastAPI app (M3)
```
## Tests
```bash
uv run pytest
```

View File

@ -0,0 +1,8 @@
"""Festival 4D — synchronized, explorable 4D concert replay.
See ``OPUS_BUILD_INSTRUCTIONS.md`` for the canonical spec and ``plan/`` for the
parallel execution plan. This package is the backend: CLI, database, synthetic
fixture generator, media/reconstruction/AI pipelines, and the FastAPI app.
"""
__version__ = "0.1.0"

View File

@ -0,0 +1,6 @@
"""Enable ``python -m festival4d <cmd>``."""
from festival4d.cli import main
if __name__ == "__main__":
raise SystemExit(main())

209
backend/festival4d/api.py Normal file
View File

@ -0,0 +1,209 @@
"""FastAPI app (spec M3). FROZEN after foundation — lanes never edit routes or shapes.
Serves the synthetic project fully so lane C can treat this API as finished:
GET /api/manifest -> {videos:[{id,filename,url,duration_s,fps,width,height,
offset_ms,drift_ppm}], t_global_max, has_poses}
GET /api/videos/{id}/poses -> [{frame_idx,t_video_s,q:[w,x,y,z],t:[x,y,z],
intrinsics:{fx,fy,cx,cy},registered}]
GET /api/pointcloud -> points.ply (binary; Range-capable)
GET /api/anchors -> [...] POST /api/anchors
GET /api/events -> [...]
POST /api/events/detect {t_global_s?, window_s?} -> runs M7 detection
POST /api/annotations {video_id, t_video_s, bbox:[x0,y0,x1,y1]} -> {anchor_id?, point?}
Source videos in ``data/raw`` are mounted at ``/media`` via Starlette ``StaticFiles``,
which supports HTTP Range (required for ``<video>`` seeking; verify ``curl -H "Range:
bytes=0-100"`` -> 206). Stubbed lane entrypoints (events detection, M8 annotation
resolution) raise ``NotImplementedError``; we catch it here and degrade to a 200 with a
note so the frozen routes never 500 while a lane is still stubbed.
"""
from __future__ import annotations
import logging
from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import FileResponse
from fastapi.staticfiles import StaticFiles
from pydantic import BaseModel, Field
from festival4d import config, db
log = logging.getLogger("festival4d.api")
app = FastAPI(title="Festival 4D", version="0.1.0")
app.add_middleware(
CORSMiddleware,
allow_origins=config.FRONTEND_ORIGINS,
allow_origin_regex=config.FRONTEND_ORIGIN_REGEX,
allow_methods=["*"],
allow_headers=["*"],
)
# Bind the DB and ensure the media directory exists so the static mount is valid even
# before the synthetic fixture has been generated.
db.init_engine()
db.init_db()
config.RAW_DIR.mkdir(parents=True, exist_ok=True)
# HTTP Range-capable static serving of source videos (spec M3, pitfall #3).
app.mount("/media", StaticFiles(directory=config.RAW_DIR), name="media")
# ---------------------------------------------------------------------------
# Request models
# ---------------------------------------------------------------------------
class AnchorIn(BaseModel):
label: str
x: float
y: float
z: float
color: str | None = None
class DetectIn(BaseModel):
t_global_s: float | None = None
window_s: float | None = None
class AnnotationIn(BaseModel):
video_id: int
t_video_s: float
bbox: list[float] = Field(min_length=4, max_length=4) # [x0, y0, x1, y1] normalized
# ---------------------------------------------------------------------------
# Serialization helpers
# ---------------------------------------------------------------------------
def _video_dict(v) -> dict:
return {
"id": v.id,
"filename": v.filename,
"url": f"/media/{v.filename}",
"duration_s": v.duration_s,
"fps": v.fps,
"width": v.width,
"height": v.height,
"offset_ms": v.offset_ms,
"drift_ppm": v.drift_ppm,
}
def _pose_dict(p) -> dict:
return {
"frame_idx": p.frame_idx,
"t_video_s": p.t_video_s,
"q": [p.qw, p.qx, p.qy, p.qz],
"t": [p.tx, p.ty, p.tz],
"intrinsics": {"fx": p.fx, "fy": p.fy, "cx": p.cx, "cy": p.cy},
"registered": p.registered,
}
def _anchor_dict(a) -> dict:
return {"id": a.id, "label": a.label, "x": a.x, "y": a.y, "z": a.z, "color": a.color}
def _event_dict(e) -> dict:
return {
"id": e.id,
"t_global_s": e.t_global_s,
"duration_s": e.duration_s,
"event_type": e.event_type,
"confidence": e.confidence,
"description": e.description,
"source": e.source,
}
# ---------------------------------------------------------------------------
# Routes
# ---------------------------------------------------------------------------
@app.get("/api/health")
def health() -> dict:
return {"status": "ok"}
@app.get("/api/manifest")
def manifest() -> dict:
videos = db.get_videos()
t_global_max = 0.0
for v in videos:
offset_s = (v.offset_ms or 0.0) / 1000.0
t_global_max = max(t_global_max, offset_s + v.duration_s)
return {
"videos": [_video_dict(v) for v in videos],
"t_global_max": t_global_max,
"has_poses": db.has_poses(),
}
@app.get("/api/videos/{video_id}/poses")
def video_poses(video_id: int) -> list[dict]:
if db.get_video(video_id) is None:
raise HTTPException(status_code=404, detail=f"no video with id={video_id}")
return [_pose_dict(p) for p in db.get_poses(video_id)]
@app.get("/api/pointcloud")
def pointcloud() -> FileResponse:
if not config.POINTS_PLY.exists():
raise HTTPException(status_code=404, detail="no point cloud (run `synthetic` or `reconstruct`)")
return FileResponse(
config.POINTS_PLY,
media_type="application/octet-stream",
filename="points.ply",
)
@app.get("/api/anchors")
def get_anchors() -> list[dict]:
return [_anchor_dict(a) for a in db.get_anchors()]
@app.post("/api/anchors")
def create_anchor(payload: AnchorIn) -> dict:
anchor = db.add_anchor(payload.label, payload.x, payload.y, payload.z, payload.color)
return _anchor_dict(anchor)
@app.get("/api/events")
def get_events() -> list[dict]:
return [_event_dict(e) for e in db.get_events()]
@app.post("/api/events/detect")
def detect_events(payload: DetectIn) -> dict:
from festival4d import events_ai
try:
result = events_ai.run_events(t_global_s=payload.t_global_s, window_s=payload.window_s)
except NotImplementedError as exc:
log.info("events detection not implemented yet: %s", exc)
return {
"result": None,
"note": "moment detection not implemented yet (lane D / M7)",
"events": [_event_dict(e) for e in db.get_events()],
}
return {"result": result, "events": [_event_dict(e) for e in db.get_events()]}
@app.post("/api/annotations")
def create_annotation(payload: AnnotationIn) -> dict:
x0, y0, x1, y1 = payload.bbox
annotation = db.add_annotation(payload.video_id, payload.t_video_s, x0, y0, x1, y1)
# M8 resolution (triangulate / nearest-point) is integration work; geometry stubs raise
# until lane B lands. Store the annotation now; resolve to an anchor later.
anchor_id = None
point = None
note = "annotation stored; 3D resolution is M8/integration work"
return {
"annotation_id": annotation.id,
"anchor_id": anchor_id,
"point": point,
"note": note,
}

View File

@ -0,0 +1,66 @@
"""GCC-PHAT pairwise audio offsets + global graph solve + drift (spec M1, lane A).
STUB lane A fills the bodies; the public signatures below are frozen (``cli.py`` calls
``run_sync``).
Algorithm (spec M1, for lane A's reference):
- **GCC-PHAT**, not raw cross-correlation: whiten the cross-spectrum
``R = X * conj(Y); R /= |R| + eps; r = irfft(R)`` robust to loud/clipped/reverberant
concert audio. numpy/scipy FFTs; librosa only for loading.
- All **pairwise** offsets, each scored by peak-to-second-peak ratio; then a global
least-squares solve over the offset graph, weighted by confidence, rejecting pairs that
violate cycle consistency (``off_ab + off_bc - off_ac > 50 ms``).
- **Drift**: re-run GCC-PHAT on 10 s windows every 30 s along the overlap; fit a line;
store the slope as ``drift_ppm`` (store 0 if ``|drift| < 5 ppm``).
- Disconnected sync-graph components -> ``offset_ms = None`` (never a guess; spec pitfall #5).
- Persist via ``db.update_video_sync`` and export ``config.SYNC_JSON``.
"""
from __future__ import annotations
import logging
import numpy as np
log = logging.getLogger("festival4d.audio_sync")
def gcc_phat(sig: np.ndarray, ref: np.ndarray, sr: int,
max_lag_s: float | None = None) -> tuple[float, float]:
"""GCC-PHAT time delay of ``sig`` relative to ``ref`` (spec M1, lane A).
Returns ``(offset_s, confidence)`` where confidence is the peak-to-second-peak ratio.
"""
raise NotImplementedError("lane A (M1): implement gcc_phat")
def pairwise_offsets(signals: dict[int, np.ndarray], sr: int) -> list[dict]:
"""All pairwise GCC-PHAT offsets with confidences (spec M1, lane A).
``signals`` maps ``video_id -> mono samples``. Returns a list of
``{a, b, offset_s, confidence}`` edges.
"""
raise NotImplementedError("lane A (M1): implement pairwise_offsets")
def solve_global_offsets(edges: list[dict], video_ids: list[int]) -> dict[int, float | None]:
"""Least-squares global solve over the pairwise-offset graph (spec M1, lane A).
Reference node is pinned to 0; disconnected components get ``None`` (spec pitfall #5).
Returns ``video_id -> offset_ms`` (or ``None``).
"""
raise NotImplementedError("lane A (M1): implement solve_global_offsets")
def estimate_drift(sig: np.ndarray, ref: np.ndarray, sr: int) -> float:
"""Estimate clock drift in ppm by fitting window-offsets over the overlap (spec M1)."""
raise NotImplementedError("lane A (M1): implement estimate_drift")
def run_sync() -> dict:
"""Sync every ingested video: pairwise GCC-PHAT, global solve, drift, persist.
Entrypoint for ``python -m festival4d sync``. Writes ``offset_ms``/``drift_ppm``/
``sync_confidence`` via ``db`` and exports ``config.SYNC_JSON``. Returns the solution.
"""
raise NotImplementedError("lane A (M1): implement run_sync")

130
backend/festival4d/cli.py Normal file
View File

@ -0,0 +1,130 @@
"""Command-line entrypoint: ``python -m festival4d <cmd>``.
FROZEN after foundation. Lanes fill in the bodies of the functions this dispatches
to (in ``ingest.py``, ``audio_sync.py``, ``sfm.py``, ``events_ai.py``); they never
edit this file or ``api.py``.
Subcommands: ``synthetic | ingest | sync | reconstruct | events | serve``.
Every subcommand dispatches to a single lane entrypoint. While a lane is still a stub,
its entrypoint raises :class:`NotImplementedError`; we catch that here and print a clear
"not implemented yet" message with a non-zero exit code, rather than a traceback. Once the
lane fills the body, the same dispatch runs it for real no change to this file needed.
"""
from __future__ import annotations
import argparse
import logging
import sys
log = logging.getLogger("festival4d.cli")
def _cmd_synthetic(args: argparse.Namespace) -> int:
from festival4d import synthetic
synthetic.build()
return 0
def _cmd_ingest(args: argparse.Namespace) -> int:
from festival4d import ingest
ingest.run_ingest()
return 0
def _cmd_sync(args: argparse.Namespace) -> int:
from festival4d import audio_sync
audio_sync.run_sync()
return 0
def _cmd_reconstruct(args: argparse.Namespace) -> int:
from festival4d import sfm
sfm.run_reconstruct()
return 0
def _cmd_events(args: argparse.Namespace) -> int:
from festival4d import events_ai
result = events_ai.run_events(t_global_s=args.t_global_s, window_s=args.window_s)
log.info("events: %s", result)
return 0
def _cmd_serve(args: argparse.Namespace) -> int:
import uvicorn
uvicorn.run(
"festival4d.api:app",
host=args.host,
port=args.port,
reload=args.reload,
)
return 0
def build_parser() -> argparse.ArgumentParser:
from festival4d import config
parser = argparse.ArgumentParser(
prog="festival4d",
description="Festival 4D — synchronized, explorable 4D concert replay.",
)
sub = parser.add_subparsers(dest="command", required=True)
p_syn = sub.add_parser("synthetic", help="generate the synthetic fixture project (M0)")
p_syn.set_defaults(func=_cmd_synthetic)
p_ing = sub.add_parser("ingest", help="probe videos + extract audio (lane A / M1)")
p_ing.set_defaults(func=_cmd_ingest)
p_sync = sub.add_parser("sync", help="GCC-PHAT audio alignment (lane A / M1)")
p_sync.set_defaults(func=_cmd_sync)
p_rec = sub.add_parser("reconstruct", help="COLMAP SfM + pose export (lane B / M2)")
p_rec.set_defaults(func=_cmd_reconstruct)
p_evt = sub.add_parser("events", help="audio candidates + AI classification (lane D / M7)")
p_evt.add_argument("--t-global-s", dest="t_global_s", type=float, default=None,
help="detect within a single window centered here (default: whole track)")
p_evt.add_argument("--window-s", dest="window_s", type=float, default=None,
help="window width in seconds when --t-global-s is given")
p_evt.set_defaults(func=_cmd_events)
p_srv = sub.add_parser("serve", help="run the FastAPI app (M3)")
p_srv.add_argument("--host", default=config.API_HOST)
p_srv.add_argument("--port", type=int, default=config.API_PORT)
p_srv.add_argument("--reload", action="store_true", help="uvicorn autoreload (dev)")
p_srv.set_defaults(func=_cmd_serve)
return parser
def main(argv: list[str] | None = None) -> int:
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)-7s %(name)s: %(message)s",
)
parser = build_parser()
args = parser.parse_args(argv)
try:
return args.func(args)
except NotImplementedError as exc:
detail = f" ({exc})" if str(exc) else ""
print(
f"festival4d {args.command}: not implemented yet{detail}.\n"
f"This subcommand is owned by a feature lane that hasn't landed. "
f"Run `python -m festival4d synthetic` for the working demo path.",
file=sys.stderr,
)
return 2
if __name__ == "__main__":
raise SystemExit(main())

View File

@ -0,0 +1,78 @@
"""Paths, constants, and the timebase convention.
Timebase convention (FROZEN CONTRACT do not change without a change request):
``t_global`` is the master timeline in seconds.
For a video *v* with ``offset_ms`` and ``drift_ppm``::
t_video = (t_global - offset_ms / 1000) * (1 + drift_ppm * 1e-6)
The reference video has ``offset_ms == 0`` (and ``drift_ppm == 0``). A positive
``offset_ms`` means the video started recording *later* than the master zero, so at a
given ``t_global`` its local playhead is earlier. Use :func:`t_video_from_global` and
:func:`t_global_from_video` everywhere never re-derive the algebra inline. The
frontend mirrors :func:`t_video_from_global` in ``transport.js``.
"""
from __future__ import annotations
import os
from pathlib import Path
# ---------------------------------------------------------------------------
# Paths. The data root can be overridden with FESTIVAL4D_DATA_DIR (used by tests
# to run against a temporary project without touching the repo's data/).
# ---------------------------------------------------------------------------
REPO_ROOT = Path(__file__).resolve().parents[2]
def _data_dir() -> Path:
env = os.environ.get("FESTIVAL4D_DATA_DIR")
return Path(env).resolve() if env else REPO_ROOT / "data"
DATA_DIR = _data_dir()
RAW_DIR = DATA_DIR / "raw" # user drops source videos here (gitignored)
WORK_DIR = DATA_DIR / "work" # extracted wavs, frames, colmap workspace (gitignored)
AUDIO_DIR = WORK_DIR / "audio" # per-video mono 16 kHz wavs (ingest)
FRAMES_DIR = WORK_DIR / "frames" # sampled JPEGs for SfM
COLMAP_DIR = WORK_DIR / "colmap" # COLMAP workspace
DB_PATH = DATA_DIR / "project.db" # SQLite (gitignored)
POINTS_PLY = WORK_DIR / "points.ply" # reconstructed / synthetic point cloud
SYNC_JSON = WORK_DIR / "sync.json" # exported sync solution (lane A)
GROUND_TRUTH_JSON = WORK_DIR / "ground_truth.json" # synthetic fixture ground truth
# ---------------------------------------------------------------------------
# Constants
# ---------------------------------------------------------------------------
FRONTEND_ORIGINS = [
"http://localhost:5173",
"http://127.0.0.1:5173",
]
# Local-only, single-user, offline tool: allow any localhost origin so a Vite dev server on
# a fallback port (5174, …) still works. Kept alongside the explicit 5173 default.
FRONTEND_ORIGIN_REGEX = r"http://(localhost|127\.0\.0\.1)(:\d+)?"
API_HOST = "127.0.0.1"
API_PORT = 8000
AUDIO_SAMPLE_RATE = 16_000 # ingest extracts mono 16 kHz for sync (M1)
# Synthetic fixture parameters (M0). Ground-truth offsets are deliberately odd,
# non-round numbers so a naive integer-second guesser can't accidentally "pass".
SYNTH_VIDEO_W = 640
SYNTH_VIDEO_H = 360
SYNTH_FPS = 30
SYNTH_DURATION_S = 20.0
SYNTH_AUDIO_SR = 48_000
SYNTH_OFFSETS_MS = (0.0, 1370.0, -842.0) # cam0 is the reference (offset 0)
def t_video_from_global(t_global: float, offset_ms: float, drift_ppm: float = 0.0) -> float:
"""Map master-timeline seconds to a video's local playhead seconds (FROZEN)."""
return (t_global - offset_ms / 1000.0) * (1.0 + drift_ppm * 1e-6)
def t_global_from_video(t_video: float, offset_ms: float, drift_ppm: float = 0.0) -> float:
"""Inverse of :func:`t_video_from_global` (FROZEN)."""
return t_video / (1.0 + drift_ppm * 1e-6) + offset_ms / 1000.0

397
backend/festival4d/db.py Normal file
View File

@ -0,0 +1,397 @@
"""SQLAlchemy engine/session and schema (spec §2) plus small CRUD helpers.
FROZEN CONTRACT after foundation: the schema (table/column names and types) and the
helper signatures are what every lane depends on. Lanes write through these helpers only.
Schema (spec §2)::
videos(id, filename, duration_s, fps, width, height, offset_ms, drift_ppm,
sync_confidence, created_at)
camera_poses(id, video_id FK, frame_idx, t_video_s, qw,qx,qy,qz, tx,ty,tz,
fx,fy,cx,cy, registered) # COLMAP world->camera; registered=False => interpolated
anchors(id, label, x,y,z, color)
events(id, t_global_s, duration_s, event_type, confidence, description, source)
annotations(id, video_id FK, t_video_s, x0,y0,x1,y1, resolved_anchor_id FK NULL)
Engine management: :func:`init_engine` (re)binds the module to a SQLite file. It defaults
to ``config.DB_PATH`` but tests point it at a temp file. Helpers open and commit their own
short-lived sessions and return detached ORM instances (``expire_on_commit=False``), so the
returned objects are safe to read after the session closes.
"""
from __future__ import annotations
from contextlib import contextmanager
from datetime import datetime, timezone
from pathlib import Path
from typing import Iterable, Iterator, Sequence
from sqlalchemy import (
Boolean,
Float,
ForeignKey,
Integer,
String,
create_engine,
delete,
select,
)
from sqlalchemy.engine import Engine
from sqlalchemy.orm import DeclarativeBase, Mapped, Session, mapped_column, sessionmaker
from festival4d import config
# ---------------------------------------------------------------------------
# ORM models
# ---------------------------------------------------------------------------
class Base(DeclarativeBase):
pass
class Video(Base):
__tablename__ = "videos"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
filename: Mapped[str] = mapped_column(String, nullable=False, unique=True)
duration_s: Mapped[float] = mapped_column(Float, nullable=False)
fps: Mapped[float] = mapped_column(Float, nullable=False)
width: Mapped[int] = mapped_column(Integer, nullable=False)
height: Mapped[int] = mapped_column(Integer, nullable=False)
offset_ms: Mapped[float | None] = mapped_column(Float, nullable=True)
drift_ppm: Mapped[float | None] = mapped_column(Float, nullable=True)
sync_confidence: Mapped[float | None] = mapped_column(Float, nullable=True)
created_at: Mapped[str] = mapped_column(String, nullable=False)
class CameraPose(Base):
__tablename__ = "camera_poses"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
video_id: Mapped[int] = mapped_column(ForeignKey("videos.id"), nullable=False, index=True)
frame_idx: Mapped[int] = mapped_column(Integer, nullable=False)
t_video_s: Mapped[float] = mapped_column(Float, nullable=False)
# COLMAP world->camera quaternion [w,x,y,z] + translation
qw: Mapped[float] = mapped_column(Float, nullable=False)
qx: Mapped[float] = mapped_column(Float, nullable=False)
qy: Mapped[float] = mapped_column(Float, nullable=False)
qz: Mapped[float] = mapped_column(Float, nullable=False)
tx: Mapped[float] = mapped_column(Float, nullable=False)
ty: Mapped[float] = mapped_column(Float, nullable=False)
tz: Mapped[float] = mapped_column(Float, nullable=False)
# pinhole intrinsics after undistortion
fx: Mapped[float] = mapped_column(Float, nullable=False)
fy: Mapped[float] = mapped_column(Float, nullable=False)
cx: Mapped[float] = mapped_column(Float, nullable=False)
cy: Mapped[float] = mapped_column(Float, nullable=False)
registered: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
class Anchor(Base):
__tablename__ = "anchors"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
label: Mapped[str] = mapped_column(String, nullable=False)
x: Mapped[float] = mapped_column(Float, nullable=False)
y: Mapped[float] = mapped_column(Float, nullable=False)
z: Mapped[float] = mapped_column(Float, nullable=False)
color: Mapped[str | None] = mapped_column(String, nullable=True)
class Event(Base):
__tablename__ = "events"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
t_global_s: Mapped[float] = mapped_column(Float, nullable=False)
duration_s: Mapped[float | None] = mapped_column(Float, nullable=True)
event_type: Mapped[str] = mapped_column(String, nullable=False)
confidence: Mapped[float | None] = mapped_column(Float, nullable=True)
description: Mapped[str | None] = mapped_column(String, nullable=True)
source: Mapped[str] = mapped_column(String, nullable=False) # 'audio_auto' | 'ai' | 'user'
class Annotation(Base):
__tablename__ = "annotations"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
video_id: Mapped[int] = mapped_column(ForeignKey("videos.id"), nullable=False, index=True)
t_video_s: Mapped[float] = mapped_column(Float, nullable=False)
x0: Mapped[float] = mapped_column(Float, nullable=False)
y0: Mapped[float] = mapped_column(Float, nullable=False)
x1: Mapped[float] = mapped_column(Float, nullable=False)
y1: Mapped[float] = mapped_column(Float, nullable=False)
resolved_anchor_id: Mapped[int | None] = mapped_column(
ForeignKey("anchors.id"), nullable=True
)
# ---------------------------------------------------------------------------
# Engine / session management
# ---------------------------------------------------------------------------
_engine: Engine | None = None
_SessionLocal: sessionmaker[Session] | None = None
_bound_path: Path | None = None
def init_engine(db_path: str | Path | None = None) -> Engine:
"""(Re)bind the module to a SQLite database file and return the engine.
Called with no argument uses ``config.DB_PATH``. Idempotent for the same path.
"""
global _engine, _SessionLocal, _bound_path
path = Path(db_path) if db_path is not None else config.DB_PATH
if _engine is not None and _bound_path == path:
return _engine
path.parent.mkdir(parents=True, exist_ok=True)
if _engine is not None:
_engine.dispose()
_engine = create_engine(
f"sqlite:///{path}",
connect_args={"check_same_thread": False},
future=True,
)
_SessionLocal = sessionmaker(bind=_engine, expire_on_commit=False, future=True)
_bound_path = path
return _engine
def engine() -> Engine:
if _engine is None:
init_engine()
assert _engine is not None
return _engine
@contextmanager
def session_scope() -> Iterator[Session]:
"""Transactional session context; commits on success, rolls back on error."""
if _SessionLocal is None:
init_engine()
assert _SessionLocal is not None
session = _SessionLocal()
try:
yield session
session.commit()
except Exception:
session.rollback()
raise
finally:
session.close()
def init_db() -> None:
"""Create all tables if they do not exist."""
Base.metadata.create_all(engine())
def reset_db() -> None:
"""Drop and recreate all tables (used by the synthetic fixture generator)."""
Base.metadata.drop_all(engine())
Base.metadata.create_all(engine())
def _now_iso() -> str:
return datetime.now(timezone.utc).isoformat()
# ---------------------------------------------------------------------------
# Videos
# ---------------------------------------------------------------------------
def add_video(
filename: str,
duration_s: float,
fps: float,
width: int,
height: int,
offset_ms: float | None = None,
drift_ppm: float | None = None,
sync_confidence: float | None = None,
) -> Video:
"""Insert a video row and return it (used by ingest + synthetic)."""
with session_scope() as s:
video = Video(
filename=filename,
duration_s=duration_s,
fps=fps,
width=width,
height=height,
offset_ms=offset_ms,
drift_ppm=drift_ppm,
sync_confidence=sync_confidence,
created_at=_now_iso(),
)
s.add(video)
s.flush()
s.refresh(video)
return video
def get_videos() -> list[Video]:
with session_scope() as s:
return list(s.scalars(select(Video).order_by(Video.id)))
def get_video(video_id: int) -> Video | None:
with session_scope() as s:
return s.get(Video, video_id)
def update_video_sync(
video_id: int,
offset_ms: float | None,
drift_ppm: float | None,
sync_confidence: float | None,
) -> None:
"""Persist the audio-sync solution for a video (lane A / M1)."""
with session_scope() as s:
video = s.get(Video, video_id)
if video is None:
raise KeyError(f"no video with id={video_id}")
video.offset_ms = offset_ms
video.drift_ppm = drift_ppm
video.sync_confidence = sync_confidence
# ---------------------------------------------------------------------------
# Camera poses
# ---------------------------------------------------------------------------
def set_poses(video_id: int, poses: Iterable[dict]) -> int:
"""Replace all poses for a video with ``poses`` (lane B / M2, and synthetic).
Each dict must have keys: frame_idx, t_video_s, qw, qx, qy, qz, tx, ty, tz,
fx, fy, cx, cy, and optionally ``registered`` (default True). Returns the count.
This is atomic: existing poses are only deleted inside the same transaction that
inserts the new ones, so a caller that raises before returning never corrupts the DB.
"""
rows = list(poses)
with session_scope() as s:
s.execute(delete(CameraPose).where(CameraPose.video_id == video_id))
for p in rows:
s.add(
CameraPose(
video_id=video_id,
frame_idx=int(p["frame_idx"]),
t_video_s=float(p["t_video_s"]),
qw=float(p["qw"]), qx=float(p["qx"]), qy=float(p["qy"]), qz=float(p["qz"]),
tx=float(p["tx"]), ty=float(p["ty"]), tz=float(p["tz"]),
fx=float(p["fx"]), fy=float(p["fy"]), cx=float(p["cx"]), cy=float(p["cy"]),
registered=bool(p.get("registered", True)),
)
)
return len(rows)
def get_poses(video_id: int) -> list[CameraPose]:
with session_scope() as s:
return list(
s.scalars(
select(CameraPose)
.where(CameraPose.video_id == video_id)
.order_by(CameraPose.t_video_s)
)
)
def has_poses() -> bool:
with session_scope() as s:
return s.scalar(select(CameraPose.id).limit(1)) is not None
# ---------------------------------------------------------------------------
# Anchors
# ---------------------------------------------------------------------------
def add_anchor(label: str, x: float, y: float, z: float, color: str | None = None) -> Anchor:
with session_scope() as s:
anchor = Anchor(label=label, x=x, y=y, z=z, color=color)
s.add(anchor)
s.flush()
s.refresh(anchor)
return anchor
def get_anchors() -> list[Anchor]:
with session_scope() as s:
return list(s.scalars(select(Anchor).order_by(Anchor.id)))
# ---------------------------------------------------------------------------
# Events
# ---------------------------------------------------------------------------
def add_event(
t_global_s: float,
event_type: str,
source: str,
duration_s: float | None = None,
confidence: float | None = None,
description: str | None = None,
) -> Event:
with session_scope() as s:
event = Event(
t_global_s=t_global_s,
duration_s=duration_s,
event_type=event_type,
confidence=confidence,
description=description,
source=source,
)
s.add(event)
s.flush()
s.refresh(event)
return event
def get_events() -> list[Event]:
with session_scope() as s:
return list(s.scalars(select(Event).order_by(Event.t_global_s)))
def clear_events(source: str | None = None) -> int:
"""Delete events, optionally only those from a given ``source``. Returns count deleted."""
with session_scope() as s:
stmt = delete(Event)
if source is not None:
stmt = stmt.where(Event.source == source)
result = s.execute(stmt)
return int(result.rowcount or 0)
# ---------------------------------------------------------------------------
# Annotations
# ---------------------------------------------------------------------------
def add_annotation(
video_id: int,
t_video_s: float,
x0: float,
y0: float,
x1: float,
y1: float,
resolved_anchor_id: int | None = None,
) -> Annotation:
with session_scope() as s:
annotation = Annotation(
video_id=video_id,
t_video_s=t_video_s,
x0=x0, y0=y0, x1=x1, y1=y1,
resolved_anchor_id=resolved_anchor_id,
)
s.add(annotation)
s.flush()
s.refresh(annotation)
return annotation
def get_annotations(video_id: int | None = None) -> list[Annotation]:
with session_scope() as s:
stmt = select(Annotation).order_by(Annotation.id)
if video_id is not None:
stmt = stmt.where(Annotation.video_id == video_id)
return list(s.scalars(stmt))
def set_annotation_anchor(annotation_id: int, anchor_id: int) -> None:
with session_scope() as s:
annotation = s.get(Annotation, annotation_id)
if annotation is None:
raise KeyError(f"no annotation with id={annotation_id}")
annotation.resolved_anchor_id = anchor_id

View File

@ -0,0 +1,144 @@
"""Moment detection + pluggable AI classification (spec M7, lane D).
Two stages: cheap audio candidates first, then a vision LLM on candidates only.
This module defines the **FROZEN classifier contract** :class:`MomentClassification`
(Pydantic model) and :class:`MomentClassifier` (Protocol) exactly as spec M7. Lanes and
integration depend on these shapes. Everything else here is a STUB that lane D fills:
candidate detection, the provider classes, provider selection, and the ``run_events``
entrypoint (called by both ``cli.py`` and ``api.py``'s ``POST /api/events/detect``).
Provider selection (lane D): ``FESTIVAL4D_CLASSIFIER=gemini|claude|local`` (default
``gemini``). Inputs prepared once per candidate: a ~3 s MP4 clip around ``t_global`` from
the best-covering video and 6 sampled JPEGs for providers without video input. Common
behavior: catch exceptions per candidate (one failure never aborts the batch); if the
selected provider is unconfigured, run candidate detection only (``event_type='candidate'``)
and log that classification was skipped.
The module imports with only ``pydantic`` present provider SDKs (``google-genai``,
``anthropic``, ``openai``) are imported lazily inside the provider bodies, never at load.
"""
from __future__ import annotations
import logging
import os
from pathlib import Path
from typing import Literal, Protocol, runtime_checkable
from pydantic import BaseModel, Field
log = logging.getLogger("festival4d.events_ai")
# The closed set of moment classes (spec M7). Lane C colors timeline markers by these.
EventType = Literal[
"bass_drop", "pyro", "confetti", "crowd_wave",
"artist_moment", "light_show", "quiet_moment", "other",
]
EVENT_TYPES: tuple[str, ...] = (
"bass_drop", "pyro", "confetti", "crowd_wave",
"artist_moment", "light_show", "quiet_moment", "other",
)
class MomentClassification(BaseModel):
"""Structured classifier output (FROZEN CONTRACT, spec M7)."""
event_type: EventType
confidence: float = Field(ge=0.0, le=1.0) # 0..1
description: str # one sentence, human-readable
@runtime_checkable
class MomentClassifier(Protocol):
"""Interchangeable classifier provider (FROZEN CONTRACT, spec M7)."""
def classify(self, clip_path: Path, frames: list[Path]) -> MomentClassification:
"""Classify one candidate moment from a short clip + sampled frames."""
...
# ---------------------------------------------------------------------------
# Provider stubs (lane D fills bodies; classifiers import their SDK lazily).
# ---------------------------------------------------------------------------
class GeminiClassifier:
"""Default provider — Gemini 2.5 Flash, native video input (``GEMINI_API_KEY``)."""
model = "gemini-2.5-flash"
def __init__(self) -> None:
raise NotImplementedError("lane D (M7): implement GeminiClassifier")
def classify(self, clip_path: Path, frames: list[Path]) -> MomentClassification:
raise NotImplementedError("lane D (M7): implement GeminiClassifier.classify")
class ClaudeClassifier:
"""Optional provider — Claude Opus 4.8, 6 frames as images (``ANTHROPIC_API_KEY``).
Uses ``client.messages.parse(..., thinking={'type': 'adaptive'},
output_format=MomentClassification)``; do **not** pass ``temperature`` (rejected on
Opus 4.8). Model id ``claude-opus-4-8`` (never date-suffixed).
"""
model = "claude-opus-4-8"
def __init__(self) -> None:
raise NotImplementedError("lane D (M7): implement ClaudeClassifier")
def classify(self, clip_path: Path, frames: list[Path]) -> MomentClassification:
raise NotImplementedError("lane D (M7): implement ClaudeClassifier.classify")
class LocalClassifier:
"""Offline/free provider — any OpenAI-compatible vision endpoint (Ollama/LM Studio/
OpenRouter). Config: ``FESTIVAL4D_OPENAI_BASE_URL``, ``FESTIVAL4D_OPENAI_KEY``,
``FESTIVAL4D_OPENAI_MODEL``. Sends the 6 frames as image parts; validates JSON with one
retry on failure.
"""
def __init__(self) -> None:
raise NotImplementedError("lane D (M7): implement LocalClassifier")
def classify(self, clip_path: Path, frames: list[Path]) -> MomentClassification:
raise NotImplementedError("lane D (M7): implement LocalClassifier.classify")
def get_classifier(name: str | None = None) -> MomentClassifier | None:
"""Select a provider by name or ``FESTIVAL4D_CLASSIFIER`` (default ``gemini``).
Returns a ready classifier, or ``None`` if the selected provider is unconfigured (no
key/endpoint) callers then run candidates-only. STUB lane D implements selection.
"""
_ = name or os.environ.get("FESTIVAL4D_CLASSIFIER", "gemini")
raise NotImplementedError("lane D (M7): implement get_classifier")
# ---------------------------------------------------------------------------
# Candidate detection + orchestration (lane D fills bodies).
# ---------------------------------------------------------------------------
def detect_candidates(audio: "object", sr: int) -> list[float]:
"""Audio candidate moments on the reference track (spec M7, lane D).
Compute RMS energy + spectral-flux onset strength (librosa); return the times
(reference-local seconds) of peaks that are local maxima over a 10 s neighborhood and
above the 90th percentile. Verified against the fixture's ground-truth pulse times
(±0.5 s).
"""
raise NotImplementedError("lane D (M7): implement detect_candidates")
def prepare_inputs(t_global_s: float) -> tuple[Path, list[Path]]:
"""Prepare classifier inputs for a candidate: a ~3 s MP4 clip + 6 sampled JPEGs (M7)."""
raise NotImplementedError("lane D (M7): implement prepare_inputs")
def run_events(t_global_s: float | None = None, window_s: float | None = None) -> dict:
"""Detect audio candidates and (if a provider is configured) classify them.
Entrypoint for ``python -m festival4d events`` and ``POST /api/events/detect``. Inserts
events via ``db`` (``source='audio_auto'``; classified ones updated to ``source='ai'``).
Optionally restrict to a single window around ``t_global_s`` of width ``window_s``.
Returns a summary dict. Per-candidate exceptions are isolated (spec M7).
"""
raise NotImplementedError("lane D (M7): implement run_events")

View File

@ -0,0 +1,28 @@
"""Sharpness-aware frame sampling for SfM (spec M2, lane B).
STUB lane B fills the bodies; the public signatures below are frozen (``sfm.py`` calls
these). Sample candidate frames ~2/s per video; within each 0.5 s window keep the sharpest
frame (variance of Laplacian via OpenCV). Write JPEGs named ``{video_id}_{frame_idx}.jpg``
into ``config.FRAMES_DIR`` (one subfolder per video for COLMAP's single-camera-per-folder).
"""
from __future__ import annotations
import logging
from pathlib import Path
log = logging.getLogger("festival4d.frames")
def sharpness(image) -> float:
"""Variance of the Laplacian of a grayscale image (OpenCV) — higher is sharper."""
raise NotImplementedError("lane B (M2): implement sharpness")
def sample_frames(video_path: Path, video_id: int, out_dir: Path,
target_fps: float = 2.0, window_s: float = 0.5) -> list[Path]:
"""Sample the sharpest frame per ``window_s`` window from ``video_path`` (spec M2).
Returns the written JPEG paths (named ``{video_id}_{frame_idx}.jpg``).
"""
raise NotImplementedError("lane B (M2): implement sample_frames")

View File

@ -0,0 +1,217 @@
"""Coordinate conversions, quaternion math, ray casting, triangulation.
The single most important thing in this file is :func:`colmap_to_threejs` the
COLMAP world->camera pose to Three.js camera conversion (spec M5). It is a **FROZEN
CONTRACT**: it ships in foundation with a passing unit test (``test_geometry.py``), and
the JS mirror lives in ``frontend/src/lib/pose.js`` with the *same* embedded test
vectors. Lanes B and C **consume** it; they never reimplement or modify it.
The remaining functions (:func:`slerp_pose`, :func:`ray_from_pixel`,
:func:`triangulate_rays`, :func:`nearest_point_on_ray`) are stubs with frozen
signatures for lane B (spec M2 + M8). Lane B fills the bodies and adds their tests.
Conventions
-----------
- Quaternions are COLMAP order **[w, x, y, z]** (scalar first), unit norm.
- A pose ``(q, t)`` is world->camera: ``x_cam = R(q) @ x_world + t``.
- COLMAP camera axes: +x right, +y down, +z forward (into the scene).
- Three.js cameras look down -z with +y up; hence the ``diag(1, -1, -1)`` flip.
"""
from __future__ import annotations
import numpy as np
from numpy.typing import ArrayLike, NDArray
# Camera-axis flip that takes COLMAP camera-local axes (x right, y down, z forward)
# to Three.js camera-local axes (x right, y up, z backward). Part of the frozen contract.
_FLIP_YZ = np.diag([1.0, -1.0, -1.0])
def quat_to_mat(q: ArrayLike) -> NDArray[np.float64]:
"""Convert a unit quaternion ``[w, x, y, z]`` to a 3x3 rotation matrix.
Hamilton convention, right-handed, active rotation: the returned ``R`` is COLMAP's
world->camera matrix when ``q`` is a COLMAP pose quaternion. FROZEN (used by the
conversion contract).
"""
w, x, y, z = (float(v) for v in np.asarray(q, dtype=np.float64).reshape(4))
n = w * w + x * x + y * y + z * z
if n < 1e-12:
raise ValueError("quaternion has near-zero norm")
s = 2.0 / n
wx, wy, wz = s * w * x, s * w * y, s * w * z
xx, xy, xz = s * x * x, s * x * y, s * x * z
yy, yz, zz = s * y * y, s * y * z, s * z * z
return np.array(
[
[1.0 - (yy + zz), xy - wz, xz + wy],
[xy + wz, 1.0 - (xx + zz), yz - wx],
[xz - wy, yz + wx, 1.0 - (xx + yy)],
],
dtype=np.float64,
)
def mat_to_quat(R: ArrayLike) -> NDArray[np.float64]:
"""Convert a 3x3 rotation matrix to a unit quaternion ``[w, x, y, z]`` (w >= 0).
Inverse of :func:`quat_to_mat`. Used by tests and the pose export path.
"""
m = np.asarray(R, dtype=np.float64).reshape(3, 3)
trace = m[0, 0] + m[1, 1] + m[2, 2]
if trace > 0.0:
s = np.sqrt(trace + 1.0) * 2.0
w = 0.25 * s
x = (m[2, 1] - m[1, 2]) / s
y = (m[0, 2] - m[2, 0]) / s
z = (m[1, 0] - m[0, 1]) / s
elif m[0, 0] > m[1, 1] and m[0, 0] > m[2, 2]:
s = np.sqrt(1.0 + m[0, 0] - m[1, 1] - m[2, 2]) * 2.0
w = (m[2, 1] - m[1, 2]) / s
x = 0.25 * s
y = (m[0, 1] + m[1, 0]) / s
z = (m[0, 2] + m[2, 0]) / s
elif m[1, 1] > m[2, 2]:
s = np.sqrt(1.0 + m[1, 1] - m[0, 0] - m[2, 2]) * 2.0
w = (m[0, 2] - m[2, 0]) / s
x = (m[0, 1] + m[1, 0]) / s
y = 0.25 * s
z = (m[1, 2] + m[2, 1]) / s
else:
s = np.sqrt(1.0 + m[2, 2] - m[0, 0] - m[1, 1]) * 2.0
w = (m[1, 0] - m[0, 1]) / s
x = (m[0, 2] + m[2, 0]) / s
y = (m[1, 2] + m[2, 1]) / s
z = 0.25 * s
q = np.array([w, x, y, z], dtype=np.float64)
q /= np.linalg.norm(q)
if q[0] < 0: # canonical sign: non-negative scalar part
q = -q
return q
def colmap_to_threejs(q: ArrayLike, t: ArrayLike) -> tuple[NDArray[np.float64], NDArray[np.float64]]:
"""Convert a COLMAP world->camera pose to a Three.js camera pose (spec M5).
**FROZEN CONTRACT.** Mirrored in ``frontend/src/lib/pose.js``; do not change the math
without a change request and a synchronized update to both sides + their test vectors.
Parameters
----------
q : array_like, shape (4,)
COLMAP world->camera quaternion ``[w, x, y, z]``.
t : array_like, shape (3,)
COLMAP world->camera translation ``[tx, ty, tz]``.
Returns
-------
position : ndarray, shape (3,)
Camera center in world coordinates, ``C = -R^T t``. Assign to ``camera.position``.
rotation_matrix : ndarray, shape (3, 3)
Three.js camera world rotation ``R_three = R^T @ diag(1, -1, -1)``. Assign via
``camera.setRotationFromMatrix(...)`` (a proper rotation, det = +1).
"""
R = quat_to_mat(q) # world -> cam
t_vec = np.asarray(t, dtype=np.float64).reshape(3)
R_c2w = R.T # cam -> world
position = -R_c2w @ t_vec # camera center in world coords
rotation_matrix = R_c2w @ _FLIP_YZ # flip camera-local y,z for Three.js
return position, rotation_matrix
# ---------------------------------------------------------------------------
# Lane B stubs (spec M2 + M8). Signatures FROZEN; bodies raise NotImplementedError.
# ---------------------------------------------------------------------------
def slerp_pose(
q0: ArrayLike,
t0: ArrayLike,
q1: ArrayLike,
t1: ArrayLike,
alpha: float,
) -> tuple[NDArray[np.float64], NDArray[np.float64]]:
"""Interpolate between two world->camera poses (spec M2 pose interpolation).
Spherically interpolate the rotation (slerp on the shorter arc, handling the
double-cover sign) and linearly interpolate the translation, at fraction
``alpha in [0, 1]`` from pose 0 to pose 1.
Parameters
----------
q0, q1 : array_like, shape (4,)
COLMAP quaternions ``[w, x, y, z]`` at the endpoints.
t0, t1 : array_like, shape (3,)
COLMAP translations at the endpoints.
alpha : float
Interpolation fraction; 0 returns pose 0, 1 returns pose 1.
Returns
-------
(q, t) : the interpolated quaternion ``[w, x, y, z]`` and translation ``[x, y, z]``.
"""
raise NotImplementedError("lane B (M2): implement slerp_pose")
def ray_from_pixel(
q: ArrayLike,
t: ArrayLike,
fx: float,
fy: float,
cx: float,
cy: float,
px: float,
py: float,
) -> tuple[NDArray[np.float64], NDArray[np.float64]]:
"""Unproject a pixel into a world-space ray (spec M8 annotation resolution).
Given the world->camera pose ``(q, t)`` and pinhole intrinsics, build the ray that
passes through image pixel ``(px, py)``.
Returns
-------
origin : ndarray, shape (3,)
Ray origin = camera center in world coords.
direction : ndarray, shape (3,)
Unit ray direction in world coords, pointing into the scene.
"""
raise NotImplementedError("lane B (M8): implement ray_from_pixel")
def triangulate_rays(
origin_a: ArrayLike,
dir_a: ArrayLike,
origin_b: ArrayLike,
dir_b: ArrayLike,
) -> tuple[NDArray[np.float64], float]:
"""Closest point between two world-space rays (spec M8 two-view triangulation).
Returns
-------
point : ndarray, shape (3,)
Midpoint of the shortest segment connecting the two rays.
gap : float
Length of that shortest segment (the mutual-nearest-approach distance). Callers
reject the triangulation when the rays are near-parallel or ``gap`` exceeds the
spec threshold (0.5 scene units).
"""
raise NotImplementedError("lane B (M8): implement triangulate_rays")
def nearest_point_on_ray(
origin: ArrayLike,
direction: ArrayLike,
points: ArrayLike,
radius: float = 0.3,
) -> NDArray[np.float64] | None:
"""Nearest point-cloud point to a ray, within a cylinder (spec M8 single-view fallback).
Among ``points`` (shape ``(N, 3)``) find the one whose perpendicular distance to the
ray is smallest, considering only points within ``radius`` of the ray and in front of
the origin.
Returns
-------
point : ndarray shape (3,) or None
The selected point-cloud point, or ``None`` if none lie within ``radius``.
"""
raise NotImplementedError("lane B (M8): implement nearest_point_on_ray")

View File

@ -0,0 +1,31 @@
"""Video ingest: probe files in ``data/raw`` and extract audio (spec M1, lane A).
STUB lane A fills the bodies; the public signatures below are frozen (``cli.py`` calls
``run_ingest``). For each file in ``config.RAW_DIR``: ffprobe it, insert/update the
``videos`` row via ``db`` helpers, and extract a mono 16 kHz WAV into ``config.AUDIO_DIR``.
"""
from __future__ import annotations
import logging
from pathlib import Path
log = logging.getLogger("festival4d.ingest")
def probe_video(path: Path) -> dict:
"""ffprobe a video; return ``{duration_s, fps, width, height}`` (lane A / M1)."""
raise NotImplementedError("lane A (M1): implement probe_video")
def extract_audio(path: Path, out_wav: Path, sample_rate: int = 16_000) -> Path:
"""Extract a mono ``sample_rate`` WAV from ``path`` via ffmpeg (lane A / M1)."""
raise NotImplementedError("lane A (M1): implement extract_audio")
def run_ingest() -> list[dict]:
"""Ingest every video in ``config.RAW_DIR``: probe, register in DB, extract audio.
Entrypoint for ``python -m festival4d ingest``. Returns a per-video summary list.
"""
raise NotImplementedError("lane A (M1): implement run_ingest")

71
backend/festival4d/sfm.py Normal file
View File

@ -0,0 +1,71 @@
"""COLMAP orchestration, model parsing, scene normalization, pose export (spec M2, lane B).
STUB lane B fills the bodies; the public signatures below are frozen (``cli.py`` calls
``run_reconstruct``).
Pipeline (spec M2, for lane B's reference):
- Sample frames (``frames.sample_frames``), then drive the COLMAP CLI via subprocess:
``feature_extractor`` (OPENCV model, single camera per folder) -> ``sequential_matcher``
within each video + ``exhaustive_matcher`` across -> ``mapper`` -> ``model_converter`` to
a TXT model. Parse ``images.txt`` / ``cameras.txt`` / ``points3D.txt`` by hand (no
pycolmap).
- **Normalize** the scene: centroid -> origin, camera bounding sphere radius -> 10, world-up
-> average camera up. Apply the same similarity transform to poses and points.
- **Interpolate** poses for unregistered frames (slerp rotation, lerp translation, mark
``registered=False``); never extrapolate past the first/last registered frame.
- Export ``config.POINTS_PLY`` (binary little-endian, matching ``synthetic.write_ply``'s
format) and fill ``camera_poses`` via ``db.set_poses``.
- **Failure handling** (spec M2): COLMAP optional at runtime (``shutil.which("colmap")``);
if it registers < 60% of frames or < 2 videos into one model, print a clear diagnostic and
leave existing poses untouched never corrupt the DB. The app degrades to "synced videos,
no 3D".
"""
from __future__ import annotations
import logging
from pathlib import Path
log = logging.getLogger("festival4d.sfm")
def colmap_available() -> bool:
"""Whether the COLMAP binary is on PATH (spec M2 optional-at-runtime rule)."""
import shutil
return shutil.which("colmap") is not None
def parse_images_txt(path: Path) -> list[dict]:
"""Parse a COLMAP ``images.txt`` into per-image pose records (spec M2, lane B)."""
raise NotImplementedError("lane B (M2): implement parse_images_txt")
def parse_cameras_txt(path: Path) -> dict[int, dict]:
"""Parse a COLMAP ``cameras.txt`` into ``camera_id -> intrinsics`` (spec M2, lane B)."""
raise NotImplementedError("lane B (M2): implement parse_cameras_txt")
def parse_points3d_txt(path: Path):
"""Parse a COLMAP ``points3D.txt`` into ``(points Nx3, colors Nx3)`` (spec M2, lane B)."""
raise NotImplementedError("lane B (M2): implement parse_points3d_txt")
def normalize_scene(points, poses):
"""Similarity transform: centroid->origin, camera sphere radius->10, up->+Y (spec M2)."""
raise NotImplementedError("lane B (M2): implement normalize_scene")
def run_colmap(frames_dir: Path, workspace: Path) -> Path:
"""Drive the COLMAP CLI and return the exported TXT-model directory (spec M2, lane B)."""
raise NotImplementedError("lane B (M2): implement run_colmap")
def run_reconstruct() -> dict:
"""Full reconstruction: sample frames, run COLMAP, normalize, interpolate, export.
Entrypoint for ``python -m festival4d reconstruct``. Degrades gracefully when COLMAP is
absent or the reconstruction is too weak (spec M2 failure handling): existing poses are
left untouched. Returns a summary dict.
"""
raise NotImplementedError("lane B (M2): implement run_reconstruct")

View File

@ -0,0 +1,556 @@
"""Synthetic fixture generator (spec M0) — the universal test bed.
``python -m festival4d synthetic`` builds a complete fake project so every later layer is
testable without real footage:
- **3 fake videos** (``cam0/1/2.mp4``, 640x360, 30 fps, 20 s), each a distinct ``testsrc2``
visual but sharing the *same* synthesized music-like audio, shifted by known offsets
(0, +1370, -842 ms). ``cam0`` is the reference (offset 0).
- **Fake poses**: 3 camera trajectories on arcs around a "stage" box at the origin,
looking at the stage, written into ``camera_poses`` (COLMAP world->camera convention).
- **Point cloud**: random points on the stage box + ground plane, written to
``points.ply`` in the exact binary-little-endian format the real COLMAP path produces.
- **Seeded events + anchors** so the timeline and overlays have data immediately.
- **``ground_truth.json``**: offsets, drift, audio pulse (bang) times, events, and stage
corners part of the frozen contract (lanes A and D assert against it).
FROZEN CONTRACT after foundation. The module is factored so the numeric pieces
(audio shifts, poses, PLY round-trip) are unit-testable without invoking ffmpeg.
"""
from __future__ import annotations
import json
import logging
import shutil
import struct
import subprocess
import tempfile
import wave
from pathlib import Path
import numpy as np
from festival4d import config, db
from festival4d.geometry import mat_to_quat
log = logging.getLogger("festival4d.synthetic")
# Audio timeline: the master signal covers global time [G_START, G_END] so that even the
# negatively-offset video (started before global 0) samples a valid window.
G_START = -2.0
G_END = 24.0
BEAT_INTERVAL_S = 0.5 # regular beat clicks
BANG_TIMES_S = (3.0, 10.0, 17.0) # loud transients (M7 candidates); well separated
TONE_FREQS_HZ = (220.0, 330.0)
# Camera intrinsics (pinhole, after undistortion) for the synthetic cameras.
SYNTH_HFOV_DEG = 60.0
# Stage geometry (world coords, +y up, +z toward the cameras/audience).
STAGE_HALF_W = 3.0 # x extent
STAGE_HALF_D = 2.0 # z extent
STAGE_HEIGHT = 1.0 # y top
STAGE_TARGET = (0.0, 0.5, 0.0) # cameras aim here
STAGE_CORNERS = {
"Stage FL": (-STAGE_HALF_W, STAGE_HEIGHT, STAGE_HALF_D),
"Stage FR": (STAGE_HALF_W, STAGE_HEIGHT, STAGE_HALF_D),
"Stage BL": (-STAGE_HALF_W, STAGE_HEIGHT, -STAGE_HALF_D),
"Stage BR": (STAGE_HALF_W, STAGE_HEIGHT, -STAGE_HALF_D),
}
# Seeded timeline events (pre-classified) so lane C can render markers immediately.
SEED_EVENTS = [
(3.0, "bass_drop", 0.94, "Bass drops and the crowd erupts as the beat hits."),
(6.0, "crowd_wave", 0.72, "A wave ripples through the audience."),
(8.0, "quiet_moment", 0.61, "A brief hush before the next build."),
(10.0, "pyro", 0.88, "Pyro columns fire across the front of the stage."),
(13.0, "light_show", 0.90, "Sweeping beams sync to the breakdown."),
(17.0, "confetti", 0.81, "Confetti cannons burst over the crowd."),
(19.0, "artist_moment", 0.77, "The artist steps to the edge for a solo."),
]
# Fonts to try for the "CAM N" label (optional nicety; skipped if none exist).
_FONT_CANDIDATES = (
"/System/Library/Fonts/Supplemental/Arial.ttf",
"/System/Library/Fonts/Supplemental/Helvetica.ttc",
"/Library/Fonts/Arial.ttf",
"/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf",
)
_PLY_DTYPE = np.dtype(
[("x", "<f4"), ("y", "<f4"), ("z", "<f4"),
("red", "u1"), ("green", "u1"), ("blue", "u1")]
)
# ---------------------------------------------------------------------------
# Audio
# ---------------------------------------------------------------------------
def synth_master_audio(sr: int = config.SYNTH_AUDIO_SR) -> tuple[np.ndarray, dict]:
"""Synthesize the master music-like signal over global time [G_START, G_END].
Returns the float32 signal (peak-normalized to ~0.95) and a ground-truth dict with the
sample rate, the window start, and the beat/bang times (global seconds).
"""
n = int(round((G_END - G_START) * sr))
t = G_START + np.arange(n) / sr
sig = np.zeros(n, dtype=np.float64)
# Low tone bed (kept quiet so it never dominates the transients GCC-PHAT keys on).
for f in TONE_FREQS_HZ:
sig += 0.05 * np.sin(2.0 * np.pi * f * t)
# Regular beat clicks: short 2 kHz bursts with a fast exponential decay -> sharp,
# broadband-ish transients that give a clean cross-correlation peak.
beat_times = np.arange(np.ceil(G_START / BEAT_INTERVAL_S) * BEAT_INTERVAL_S,
G_END, BEAT_INTERVAL_S)
for bt in beat_times:
_add_transient(sig, t, sr, center=bt, amp=0.25, freq=2000.0, decay_s=0.03)
# Loud "bangs" (bass drops / pyro): louder, longer, low-frequency thump + noise burst.
rng = np.random.default_rng(7)
for bang in BANG_TIMES_S:
_add_transient(sig, t, sr, center=bang, amp=0.9, freq=60.0, decay_s=0.18)
_add_noise_burst(sig, t, sr, center=bang, amp=0.5, decay_s=0.12, rng=rng)
# Faint noise floor for realism (SNR stays high so sync remains exact).
sig += 0.004 * rng.standard_normal(n)
peak = float(np.max(np.abs(sig)))
if peak > 0:
sig *= 0.95 / peak
ground_truth = {
"sample_rate": sr,
"g_start_s": G_START,
"g_end_s": G_END,
"beat_interval_s": BEAT_INTERVAL_S,
"bang_times_global_s": list(BANG_TIMES_S),
"pulse_times_global_s": list(BANG_TIMES_S), # alias lane D matches candidates against
}
return sig.astype(np.float32), ground_truth
def _add_transient(sig, t, sr, center, amp, freq, decay_s):
"""Add an exponentially decaying sinusoid burst centered at global time ``center``."""
length = int(decay_s * 5 * sr)
start = int(round((center - G_START) * sr))
i0 = max(0, start)
i1 = min(len(sig), start + length)
if i1 <= i0:
return
local = (np.arange(i0, i1) - start) / sr
env = np.exp(-local / decay_s)
sig[i0:i1] += amp * env * np.sin(2.0 * np.pi * freq * local)
def _add_noise_burst(sig, t, sr, center, amp, decay_s, rng):
length = int(decay_s * 5 * sr)
start = int(round((center - G_START) * sr))
i0 = max(0, start)
i1 = min(len(sig), start + length)
if i1 <= i0:
return
local = (np.arange(i0, i1) - start) / sr
env = np.exp(-local / decay_s)
sig[i0:i1] += amp * env * rng.standard_normal(i1 - i0)
def slice_for_offset(master: np.ndarray, sr: int, offset_ms: float,
duration_s: float) -> np.ndarray:
"""Extract a video's audio window from the master (spec M0 known-offset shift).
A video with ``offset_ms`` has local time ``tau`` mapping to global ``offset_ms/1000 +
tau`` (drift 0), so its audio is the master window starting at that global time.
"""
start_global = offset_ms / 1000.0
start = int(round((start_global - G_START) * sr))
length = int(round(duration_s * sr))
if start < 0 or start + length > len(master):
raise ValueError(f"offset {offset_ms} ms window falls outside the master signal")
return master[start:start + length]
def write_wav(path: Path, signal: np.ndarray, sr: int) -> None:
"""Write a mono 16-bit PCM WAV (stdlib only)."""
path.parent.mkdir(parents=True, exist_ok=True)
clipped = np.clip(signal, -1.0, 1.0)
pcm = (clipped * 32767.0).astype("<i2")
with wave.open(str(path), "wb") as w:
w.setnchannels(1)
w.setsampwidth(2)
w.setframerate(sr)
w.writeframes(pcm.tobytes())
def read_wav(path: Path) -> tuple[np.ndarray, int]:
"""Read a mono 16-bit PCM WAV back to float [-1, 1] (used by tests)."""
with wave.open(str(path), "rb") as w:
sr = w.getframerate()
frames = w.readframes(w.getnframes())
pcm = np.frombuffer(frames, dtype="<i2").astype(np.float64) / 32767.0
return pcm, sr
# ---------------------------------------------------------------------------
# Geometry: poses + point cloud
# ---------------------------------------------------------------------------
def intrinsics(width: int, height: int) -> dict:
fx = (width / 2.0) / np.tan(np.radians(SYNTH_HFOV_DEG) / 2.0)
return {"fx": float(fx), "fy": float(fx), "cx": width / 2.0, "cy": height / 2.0}
def look_at_colmap(center, target, world_up=(0.0, 1.0, 0.0)) -> tuple[np.ndarray, np.ndarray]:
"""Build a COLMAP world->camera pose (q=[w,x,y,z], t) for a camera at ``center``
looking at ``target`` with the given world up. Camera axes: +x right, +y down, +z fwd.
"""
C = np.asarray(center, dtype=np.float64)
T = np.asarray(target, dtype=np.float64)
up = np.asarray(world_up, dtype=np.float64)
z_c = T - C
z_c /= np.linalg.norm(z_c)
if abs(np.dot(up, z_c)) > 0.999: # looking near-vertical: pick a safe up
up = np.array([0.0, 0.0, 1.0])
up_c = up - np.dot(up, z_c) * z_c # image up (perp to forward)
up_c /= np.linalg.norm(up_c)
x_c = np.cross(z_c, up_c) # right
y_c = -up_c # down
R_c2w = np.column_stack([x_c, y_c, z_c]) # camera->world
R_w2c = R_c2w.T # world->camera (COLMAP R)
t = -R_w2c @ C
q = mat_to_quat(R_w2c)
return q, t
def camera_track(cam_index: int, duration_s: float, fps: float,
width: int, height: int) -> list[dict]:
"""One camera's trajectory: a gentle arc in front of the stage, ~2 poses/second."""
base_az = np.radians([-35.0, 0.0, 35.0][cam_index])
base_radius = 9.0
base_height = [1.6, 2.4, 2.0][cam_index]
sweep = np.radians(12.0)
intr = intrinsics(width, height)
poses = []
step = 0.5 # seconds -> ~2 poses/second
n = int(round(duration_s / step)) + 1
for k in range(n):
t_video = min(k * step, duration_s)
phase = 2.0 * np.pi * (t_video / duration_s)
az = base_az + sweep * np.sin(phase)
radius = base_radius + 0.5 * np.sin(phase + cam_index)
height_y = base_height + 0.3 * np.sin(phase * 2.0)
center = (radius * np.sin(az), height_y, radius * np.cos(az))
q, t = look_at_colmap(center, STAGE_TARGET)
poses.append({
"frame_idx": int(round(t_video * fps)),
"t_video_s": float(t_video),
"qw": float(q[0]), "qx": float(q[1]), "qy": float(q[2]), "qz": float(q[3]),
"tx": float(t[0]), "ty": float(t[1]), "tz": float(t[2]),
**intr,
"registered": True,
})
return poses
def generate_point_cloud(seed: int = 42) -> tuple[np.ndarray, np.ndarray]:
"""Random points on the stage box + ground plane (with the named corners included).
Returns ``(points Nx3 float32, colors Nx3 uint8)``.
"""
rng = np.random.default_rng(seed)
pts: list[np.ndarray] = []
cols: list[np.ndarray] = []
hw, hd, h = STAGE_HALF_W, STAGE_HALF_D, STAGE_HEIGHT
stage_color = np.array([210, 90, 60]) # warm stage
ground_color = np.array([70, 80, 70]) # dim ground
# Stage box: sample points on all 6 faces.
n_face = 260
faces = [
# (fixed axis, value, u range, v range) -> build points
("x", -hw), ("x", hw), ("z", -hd), ("z", hd), ("y", 0.0), ("y", h),
]
for axis, val in faces:
u = rng.uniform(-hw, hw, n_face)
v = rng.uniform(0.0, h, n_face) if axis in ("x", "z") else rng.uniform(-hd, hd, n_face)
w = rng.uniform(-hd, hd, n_face)
if axis == "x":
p = np.column_stack([np.full(n_face, val), v, w])
elif axis == "z":
p = np.column_stack([u, v, np.full(n_face, val)])
else: # y
p = np.column_stack([u, np.full(n_face, val), w])
pts.append(p)
jitter = rng.integers(-15, 16, size=(n_face, 3))
cols.append(np.clip(stage_color + jitter, 0, 255))
# Ground plane: scattered points on y=0 out to a radius, excluding the stage footprint.
n_ground = 1600
gx = rng.uniform(-12.0, 12.0, n_ground)
gz = rng.uniform(-12.0, 12.0, n_ground)
on_stage = (np.abs(gx) < hw) & (np.abs(gz) < hd)
gx, gz = gx[~on_stage], gz[~on_stage]
gp = np.column_stack([gx, np.zeros_like(gx), gz])
pts.append(gp)
jitter = rng.integers(-12, 13, size=(len(gx), 3))
cols.append(np.clip(ground_color + jitter, 0, 255))
# Include the exact named stage corners so the M8 nearest-point fallback can hit them.
corners = np.array(list(STAGE_CORNERS.values()))
pts.append(corners)
cols.append(np.tile(np.array([255, 230, 120]), (len(corners), 1)))
points = np.vstack(pts).astype(np.float32)
colors = np.vstack(cols).astype(np.uint8)
return points, colors
def write_ply(path: Path, points: np.ndarray, colors: np.ndarray) -> None:
"""Write a binary-little-endian PLY (x,y,z float + red,green,blue uchar). FROZEN format."""
points = np.asarray(points, dtype=np.float32).reshape(-1, 3)
colors = np.asarray(colors, dtype=np.uint8).reshape(-1, 3)
if len(points) != len(colors):
raise ValueError("points and colors length mismatch")
n = len(points)
arr = np.empty(n, dtype=_PLY_DTYPE)
arr["x"], arr["y"], arr["z"] = points[:, 0], points[:, 1], points[:, 2]
arr["red"], arr["green"], arr["blue"] = colors[:, 0], colors[:, 1], colors[:, 2]
assert arr.dtype.itemsize == 15 # no padding — 3*4 + 3*1
header = (
"ply\n"
"format binary_little_endian 1.0\n"
"comment Festival 4D point cloud\n"
f"element vertex {n}\n"
"property float x\n"
"property float y\n"
"property float z\n"
"property uchar red\n"
"property uchar green\n"
"property uchar blue\n"
"end_header\n"
)
path.parent.mkdir(parents=True, exist_ok=True)
with open(path, "wb") as f:
f.write(header.encode("ascii"))
f.write(arr.tobytes())
def read_ply(path: Path) -> tuple[np.ndarray, np.ndarray]:
"""Read a PLY written by :func:`write_ply` (used by tests). Returns (points, colors)."""
with open(path, "rb") as f:
line = f.readline()
if line.strip() != b"ply":
raise ValueError("not a PLY file")
n = 0
while True:
line = f.readline()
if not line:
raise ValueError("unexpected EOF in PLY header")
if line.startswith(b"element vertex"):
n = int(line.split()[-1])
if line.strip() == b"end_header":
break
arr = np.frombuffer(f.read(n * _PLY_DTYPE.itemsize), dtype=_PLY_DTYPE, count=n)
points = np.column_stack([arr["x"], arr["y"], arr["z"]]).astype(np.float32)
colors = np.column_stack([arr["red"], arr["green"], arr["blue"]]).astype(np.uint8)
return points, colors
# ---------------------------------------------------------------------------
# Video rendering (ffmpeg)
# ---------------------------------------------------------------------------
def _find_font() -> str | None:
for candidate in _FONT_CANDIDATES:
if Path(candidate).exists():
return candidate
return None
_DRAWTEXT_AVAILABLE: bool | None = None
def _has_drawtext() -> bool:
"""Whether this ffmpeg build includes the ``drawtext`` filter (needs libfreetype)."""
global _DRAWTEXT_AVAILABLE
if _DRAWTEXT_AVAILABLE is None:
try:
out = subprocess.run(
["ffmpeg", "-hide_banner", "-filters"],
check=True, capture_output=True, text=True,
).stdout
_DRAWTEXT_AVAILABLE = any(
line.split()[1] == "drawtext"
for line in out.splitlines()
if len(line.split()) > 1
)
except Exception:
_DRAWTEXT_AVAILABLE = False
return _DRAWTEXT_AVAILABLE
def render_video(out_path: Path, wav_path: Path, cam_index: int,
width: int, height: int, fps: float, duration_s: float) -> None:
"""Render one synthetic clip: a distinct ``testsrc2`` visual muxed with ``wav_path``."""
if shutil.which("ffmpeg") is None:
raise RuntimeError("ffmpeg not found on PATH — required for the synthetic fixture")
out_path.parent.mkdir(parents=True, exist_ok=True)
hue = cam_index * 47
vf = f"hue=h={hue}"
font = _find_font()
if font and _has_drawtext():
vf += (f",drawtext=fontfile='{font}':text='CAM {cam_index}':"
"x=24:y=24:fontsize=44:fontcolor=white:box=1:boxcolor=black@0.5")
src = f"testsrc2=size={width}x{height}:rate={fps}:duration={duration_s}"
cmd = [
"ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
"-f", "lavfi", "-i", src,
"-i", str(wav_path),
"-vf", vf,
"-c:v", "libx264", "-pix_fmt", "yuv420p", "-preset", "veryfast", "-crf", "28",
"-profile:v", "baseline", "-level", "3.1",
"-movflags", "+faststart",
"-c:a", "aac", "-b:a", "128k",
"-shortest",
str(out_path),
]
subprocess.run(cmd, check=True)
def ffprobe_video(path: Path) -> dict:
"""Probe width/height/fps/duration of a video with ffprobe."""
out = subprocess.run(
["ffprobe", "-v", "error", "-select_streams", "v:0",
"-show_entries", "stream=width,height,avg_frame_rate:format=duration",
"-of", "json", str(path)],
check=True, capture_output=True, text=True,
).stdout
data = json.loads(out)
stream = data["streams"][0]
num, den = stream["avg_frame_rate"].split("/")
fps = float(num) / float(den) if float(den) else 0.0
return {
"width": int(stream["width"]),
"height": int(stream["height"]),
"fps": fps,
"duration_s": float(data["format"]["duration"]),
}
# ---------------------------------------------------------------------------
# Orchestration
# ---------------------------------------------------------------------------
def build(base_dir: Path | str | None = None, duration_s: float | None = None,
run_ffmpeg: bool = True) -> dict:
"""Generate the full synthetic project. Returns a summary dict.
``base_dir`` defaults to ``config.DATA_DIR``. ``duration_s`` defaults to the fixture's
20 s (tests may pass a shorter value). ``run_ffmpeg=False`` skips video rendering (for
fast, hermetic tests of the DB/pose/PLY path).
"""
base = Path(base_dir) if base_dir is not None else config.DATA_DIR
duration = float(duration_s) if duration_s is not None else config.SYNTH_DURATION_S
raw = base / "raw"
work = base / "work"
raw.mkdir(parents=True, exist_ok=True)
work.mkdir(parents=True, exist_ok=True)
sr = config.SYNTH_AUDIO_SR
w, h, fps = config.SYNTH_VIDEO_W, config.SYNTH_VIDEO_H, config.SYNTH_FPS
log.info("synthetic: building project at %s (duration=%.1fs, ffmpeg=%s)",
base, duration, run_ffmpeg)
db.init_engine(base / "project.db")
db.reset_db()
master, audio_gt = synth_master_audio(sr)
videos_gt = []
with tempfile.TemporaryDirectory() as tmp:
tmp = Path(tmp)
for i, offset_ms in enumerate(config.SYNTH_OFFSETS_MS):
filename = f"cam{i}.mp4"
out_path = raw / filename
clip = slice_for_offset(master, sr, offset_ms, duration)
if run_ffmpeg:
wav_path = tmp / f"cam{i}.wav"
write_wav(wav_path, clip, sr)
render_video(out_path, wav_path, i, w, h, fps, duration)
probed = ffprobe_video(out_path)
vw, vh, vfps, vdur = (probed["width"], probed["height"],
probed["fps"], probed["duration_s"])
else:
vw, vh, vfps, vdur = w, h, float(fps), duration
video = db.add_video(
filename=filename, duration_s=vdur, fps=vfps, width=vw, height=vh,
offset_ms=float(offset_ms), drift_ppm=0.0, sync_confidence=1.0,
)
db.set_poses(video.id, camera_track(i, duration, fps, w, h))
videos_gt.append({
"video_id": video.id, "filename": filename,
"offset_ms": float(offset_ms), "drift_ppm": 0.0,
"is_reference": i == 0,
})
log.info("synthetic: cam%d offset=%+.0fms -> %s", i, offset_ms, out_path.name)
# Point cloud
points, colors = generate_point_cloud()
write_ply(work / "points.ply", points, colors)
# Anchors (stage corners)
corner_colors = {"Stage FL": "#ff5533", "Stage FR": "#ffaa33",
"Stage BL": "#33aaff", "Stage BR": "#aa55ff"}
for label, (x, y, z) in STAGE_CORNERS.items():
db.add_anchor(label, x, y, z, corner_colors.get(label))
# Seeded events
for t_global, event_type, conf, desc in SEED_EVENTS:
db.add_event(t_global_s=t_global, event_type=event_type, source="ai",
duration_s=1.0, confidence=conf, description=desc)
# Ground truth JSON
ground_truth = {
"videos": videos_gt,
"reference_filename": "cam0.mp4",
"audio": audio_gt,
"events": [
{"t_global_s": t, "event_type": et, "confidence": c, "description": d}
for (t, et, c, d) in SEED_EVENTS
],
"scene": {
"stage_corners": {k: list(v) for k, v in STAGE_CORNERS.items()},
"stage_target": list(STAGE_TARGET),
"point_count": int(len(points)),
},
}
gt_path = work / "ground_truth.json"
gt_path.write_text(json.dumps(ground_truth, indent=2))
summary = {
"base_dir": str(base),
"videos": [v["filename"] for v in videos_gt],
"points_ply": str(work / "points.ply"),
"point_count": int(len(points)),
"ground_truth": str(gt_path),
"events": len(SEED_EVENTS),
"anchors": len(STAGE_CORNERS),
}
log.info("synthetic: done — %d videos, %d points, %d events, %d anchors",
len(videos_gt), len(points), len(SEED_EVENTS), len(STAGE_CORNERS))
return summary
if __name__ == "__main__": # pragma: no cover
logging.basicConfig(level=logging.INFO)
print(json.dumps(build(), indent=2))

13
backend/tests/conftest.py Normal file
View File

@ -0,0 +1,13 @@
"""Pytest session setup.
Redirect the Festival 4D data directory to a throwaway temp dir for the whole test
session, so tests never touch the repo's real ``data/`` (which may hold the demo fixture).
Set before any ``festival4d.config`` import so the paths resolve to the temp dir.
"""
import os
import tempfile
os.environ.setdefault(
"FESTIVAL4D_DATA_DIR", tempfile.mkdtemp(prefix="festival4d-test-")
)

106
backend/tests/test_api.py Normal file
View File

@ -0,0 +1,106 @@
"""API contract tests (spec M3), against a synthetic fixture built into the test data dir.
Locks the frozen response shapes lane C depends on, and verifies HTTP Range serving (206),
CORS, and graceful degradation of the still-stubbed endpoints.
"""
from __future__ import annotations
import shutil
import pytest
pytestmark = pytest.mark.skipif(
shutil.which("ffmpeg") is None or shutil.which("ffprobe") is None,
reason="ffmpeg/ffprobe required to build the API test fixture",
)
@pytest.fixture(scope="module")
def client():
from fastapi.testclient import TestClient
from festival4d import synthetic
synthetic.build(duration_s=1.0, run_ffmpeg=True) # into the temp data dir (conftest)
from festival4d import api # imported after build so the /media mount + DB are ready
with TestClient(api.app) as c:
yield c
def test_manifest_shape(client):
data = client.get("/api/manifest").json()
assert set(data) == {"videos", "t_global_max", "has_poses"}
assert data["has_poses"] is True
assert len(data["videos"]) == 3
v = data["videos"][0]
assert set(v) == {
"id", "filename", "url", "duration_s", "fps", "width", "height",
"offset_ms", "drift_ppm",
}
offsets = {vv["filename"]: vv["offset_ms"] for vv in data["videos"]}
assert offsets == {"cam0.mp4": 0.0, "cam1.mp4": 1370.0, "cam2.mp4": -842.0}
def test_poses_shape(client):
poses = client.get("/api/videos/1/poses").json()
assert len(poses) > 0
p = poses[0]
assert set(p) == {"frame_idx", "t_video_s", "q", "t", "intrinsics", "registered"}
assert len(p["q"]) == 4 and len(p["t"]) == 3
assert set(p["intrinsics"]) == {"fx", "fy", "cx", "cy"}
def test_missing_video_404(client):
assert client.get("/api/videos/999/poses").status_code == 404
def test_anchors_get_and_post(client):
anchors = client.get("/api/anchors").json()
assert len(anchors) == 4
created = client.post(
"/api/anchors", json={"label": "T", "x": 1, "y": 2, "z": 3, "color": "#fff"}
).json()
assert created["label"] == "T" and created["id"] > 0
def test_events(client):
events = client.get("/api/events").json()
assert len(events) == len(__import__("festival4d.synthetic", fromlist=["x"]).SEED_EVENTS)
assert {"id", "t_global_s", "event_type", "source", "description"} <= set(events[0])
def test_pointcloud_is_ply(client):
resp = client.get("/api/pointcloud")
assert resp.status_code == 200
assert resp.content[:3] == b"ply"
def test_video_range_returns_206(client):
"""Spec pitfall #3: <video> seeking needs HTTP Range -> 206."""
resp = client.get("/media/cam0.mp4", headers={"Range": "bytes=0-100"})
assert resp.status_code == 206
assert resp.headers["content-range"].startswith("bytes 0-100/")
assert len(resp.content) == 101
def test_cors_allows_vite_origin(client):
resp = client.get("/api/manifest", headers={"Origin": "http://localhost:5173"})
assert resp.headers.get("access-control-allow-origin") == "http://localhost:5173"
def test_detect_degrades_gracefully(client):
data = client.post("/api/events/detect", json={}).json()
assert "note" in data and "events" in data # stubbed lane D -> 200 with note
assert client.post("/api/events/detect", json={}).status_code == 200
def test_annotation_stored(client):
data = client.post(
"/api/annotations",
json={"video_id": 1, "t_video_s": 0.5, "bbox": [0.4, 0.4, 0.6, 0.6]},
).json()
assert data["annotation_id"] > 0
assert data["anchor_id"] is None # M8 resolution is integration work

View File

@ -0,0 +1,110 @@
"""Tests for geometry.py.
The ``colmap_to_threejs`` tests below are a **FROZEN CONTRACT** (foundation). The embedded
``POSE_TEST_VECTORS`` are duplicated verbatim in ``frontend/src/lib/pose.js`` so both sides
provably agree if you touch the conversion math, update both files and both vector sets,
via a change request. Lane B owns this file but must not modify these three cases; it adds
tests for slerp_pose / ray_from_pixel / triangulate_rays / nearest_point_on_ray below them.
"""
from __future__ import annotations
import numpy as np
import pytest
from scipy.spatial.transform import Rotation
from festival4d.geometry import (
colmap_to_threejs,
mat_to_quat,
quat_to_mat,
)
# Sqrt(2)/2, spelled out so it matches the JS constant character-for-character.
SQRT1_2 = 0.7071067811865476
# --- FROZEN test vectors (must equal POSE_TEST_VECTORS in frontend/src/lib/pose.js) ----
# Each: COLMAP world->camera (q=[w,x,y,z], t) -> Three.js (position, rotation 3x3 row-major).
POSE_TEST_VECTORS = [
{
"name": "identity",
"q": [1.0, 0.0, 0.0, 0.0],
"t": [0.0, 0.0, -10.0],
"position": [0.0, 0.0, 10.0],
"rotation": [[1.0, 0.0, 0.0], [0.0, -1.0, 0.0], [0.0, 0.0, -1.0]],
},
{
"name": "yaw90", # +90 deg about world Y
"q": [SQRT1_2, 0.0, SQRT1_2, 0.0],
"t": [0.0, 0.0, 10.0],
"position": [10.0, 0.0, 0.0],
"rotation": [[0.0, 0.0, 1.0], [0.0, -1.0, 0.0], [1.0, 0.0, 0.0]],
},
{
"name": "lookat", # camera at +Z looking at origin -> Three.js rotation is identity
"q": [0.0, 1.0, 0.0, 0.0],
"t": [0.0, 0.0, 8.0],
"position": [0.0, 0.0, 8.0],
"rotation": [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]],
},
]
@pytest.mark.parametrize("vec", POSE_TEST_VECTORS, ids=lambda v: v["name"])
def test_colmap_to_threejs_known_vectors(vec):
position, rotation = colmap_to_threejs(vec["q"], vec["t"])
np.testing.assert_allclose(position, vec["position"], atol=1e-9)
np.testing.assert_allclose(rotation, vec["rotation"], atol=1e-9)
@pytest.mark.parametrize("vec", POSE_TEST_VECTORS, ids=lambda v: v["name"])
def test_threejs_rotation_is_proper(vec):
"""R_three must be a proper rotation (orthonormal, det = +1)."""
_, R = colmap_to_threejs(vec["q"], vec["t"])
np.testing.assert_allclose(R @ R.T, np.eye(3), atol=1e-9)
assert abs(np.linalg.det(R) - 1.0) < 1e-9
def _colmap_pose(R_w2c: np.ndarray, center: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
"""Build a COLMAP (q, t) from a world->camera rotation and a world camera center."""
t = -R_w2c @ center
q = mat_to_quat(R_w2c)
return q, t
def test_colmap_to_threejs_roundtrip_random_poses():
"""Forward COLMAP->Three.js then invert it; recover the original (q, t)."""
rng = np.random.default_rng(20260716)
flip = np.diag([1.0, -1.0, -1.0])
for _ in range(50):
R_w2c = Rotation.random(random_state=rng).as_matrix()
center = rng.uniform(-10, 10, size=3)
q, t = _colmap_pose(R_w2c, center)
position, R_three = colmap_to_threejs(q, t)
# position is the camera center
np.testing.assert_allclose(position, center, atol=1e-9)
# invert: R_c2w = R_three @ flip -> R_w2c = R_c2w^T -> t = -R_w2c @ C
R_c2w = R_three @ flip
R_back = R_c2w.T
t_back = -R_back @ position
q_back = mat_to_quat(R_back)
np.testing.assert_allclose(t_back, t, atol=1e-9)
# quaternions are double-cover; compare canonical (w>=0) forms
q_canon = q if q[0] >= 0 else -q
np.testing.assert_allclose(q_back, q_canon, atol=1e-9)
def test_quat_to_mat_matches_scipy_oracle():
"""quat_to_mat agrees with an independent implementation (scipy)."""
rng = np.random.default_rng(11)
for _ in range(50):
# scipy quaternions are scalar-LAST [x,y,z,w]; ours are scalar-FIRST [w,x,y,z]
q_xyzw = rng.normal(size=4)
q_xyzw /= np.linalg.norm(q_xyzw)
q_wxyz = np.array([q_xyzw[3], q_xyzw[0], q_xyzw[1], q_xyzw[2]])
np.testing.assert_allclose(
quat_to_mat(q_wxyz), Rotation.from_quat(q_xyzw).as_matrix(), atol=1e-9
)

View File

@ -0,0 +1,118 @@
"""Fixture tests for synthetic.py (foundation-owned; synthetic.py is a frozen contract).
These lock the synthetic fixture's contract: known audio offsets are recoverable, bang
times stand out for M7 candidate detection, the PLY round-trips, generated poses face the
stage under the frozen conversion, and ``build`` populates the DB + files as ground truth.
"""
from __future__ import annotations
import json
import shutil
import numpy as np
import pytest
from scipy.signal import correlate
from festival4d import config, db, synthetic
from festival4d.geometry import colmap_to_threejs
def _estimate_offset_ms(clip_ref, clip_v, sr):
"""Recover clip_v's offset vs the reference via cross-correlation (see derivation)."""
corr = correlate(clip_v, clip_ref, mode="full")
lag = int(np.argmax(corr)) - (len(clip_ref) - 1)
return -lag / sr * 1000.0
def test_audio_slices_recover_known_offsets():
sr = config.SYNTH_AUDIO_SR
master, _ = synthetic.synth_master_audio(sr)
ref = synthetic.slice_for_offset(master, sr, 0.0, config.SYNTH_DURATION_S)
for offset_ms in config.SYNTH_OFFSETS_MS:
clip = synthetic.slice_for_offset(master, sr, offset_ms, config.SYNTH_DURATION_S)
recovered = _estimate_offset_ms(ref, clip, sr)
assert abs(recovered - offset_ms) < 1.0, (offset_ms, recovered)
def test_bang_times_dominate_energy():
"""Loud bangs (M7 candidates) must clearly exceed the regular-beat energy floor."""
sr = config.SYNTH_AUDIO_SR
master, gt = synthetic.synth_master_audio(sr)
def window_rms(center_s, half=0.05):
i0 = int((center_s - synthetic.G_START - half) * sr)
i1 = int((center_s - synthetic.G_START + half) * sr)
return float(np.sqrt(np.mean(master[i0:i1] ** 2)))
bang_rms = [window_rms(t) for t in gt["bang_times_global_s"]]
# a spread of non-bang beat times for the baseline
beat_rms = [window_rms(t) for t in (1.0, 4.5, 7.0, 11.5, 14.0)]
assert min(bang_rms) > 3.0 * np.median(beat_rms)
def test_ply_roundtrip(tmp_path):
pts, cols = synthetic.generate_point_cloud()
path = tmp_path / "points.ply"
synthetic.write_ply(path, pts, cols)
pts2, cols2 = synthetic.read_ply(path)
assert pts2.shape == pts.shape and cols2.shape == cols.shape
np.testing.assert_allclose(pts2, pts, atol=1e-6)
np.testing.assert_array_equal(cols2, cols)
def test_generated_poses_face_the_stage():
"""Each synthetic pose, run through the frozen conversion, must look at the stage."""
poses = synthetic.camera_track(1, config.SYNTH_DURATION_S, config.SYNTH_FPS,
config.SYNTH_VIDEO_W, config.SYNTH_VIDEO_H)
target = np.array(synthetic.STAGE_TARGET)
for p in poses:
q = [p["qw"], p["qx"], p["qy"], p["qz"]]
t = [p["tx"], p["ty"], p["tz"]]
position, R = colmap_to_threejs(q, t)
look = R @ np.array([0.0, 0.0, -1.0]) # Three.js camera looks down -z
expected = target - position
expected /= np.linalg.norm(expected)
np.testing.assert_allclose(look, expected, atol=1e-9)
def test_build_no_ffmpeg_populates_db_and_files(tmp_path):
summary = synthetic.build(base_dir=tmp_path, duration_s=5.0, run_ffmpeg=False)
videos = db.get_videos()
assert len(videos) == 3
by_name = {v.filename: v for v in videos}
assert by_name["cam0.mp4"].offset_ms == 0.0
assert by_name["cam1.mp4"].offset_ms == 1370.0
assert by_name["cam2.mp4"].offset_ms == -842.0
for v in videos:
assert len(db.get_poses(v.id)) > 0
assert db.get_poses(v.id)[0].registered is True
assert db.has_poses()
assert len(db.get_anchors()) == 4
assert len(db.get_events()) == len(synthetic.SEED_EVENTS)
ply = tmp_path / "work" / "points.ply"
assert ply.exists()
pts, _ = synthetic.read_ply(ply)
assert len(pts) == summary["point_count"] > 1000
gt = json.loads((tmp_path / "work" / "ground_truth.json").read_text())
assert [vv["offset_ms"] for vv in gt["videos"]] == [0.0, 1370.0, -842.0]
assert gt["reference_filename"] == "cam0.mp4"
@pytest.mark.skipif(shutil.which("ffmpeg") is None or shutil.which("ffprobe") is None,
reason="ffmpeg/ffprobe required for the full fixture build")
def test_build_full_ffmpeg_renders_seekable_videos(tmp_path):
synthetic.build(base_dir=tmp_path, duration_s=4.0, run_ffmpeg=True)
raw = tmp_path / "raw"
for name in ("cam0.mp4", "cam1.mp4", "cam2.mp4"):
path = raw / name
assert path.exists() and path.stat().st_size > 10_000
videos = db.get_videos()
assert len(videos) == 3
for v in videos:
assert 3.5 < v.duration_s < 4.6 # ~4 s clips
assert v.width == 640 and v.height == 360

43
frontend/index.html Normal file
View File

@ -0,0 +1,43 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Festival 4D</title>
<style>
:root { color-scheme: dark; }
body {
margin: 0;
font: 15px/1.5 -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
background: #0b0d12;
color: #e6e8ee;
padding: 2rem;
}
h1 { margin: 0 0 0.25rem; font-size: 1.6rem; }
.sub { color: #8a92a6; margin-bottom: 1.5rem; }
.card {
background: #141824;
border: 1px solid #232838;
border-radius: 10px;
padding: 1rem 1.25rem;
margin-bottom: 1rem;
max-width: 720px;
}
.ok { color: #59d499; }
.bad { color: #ff6b6b; }
table { border-collapse: collapse; width: 100%; }
th, td { text-align: left; padding: 0.35rem 0.6rem; border-bottom: 1px solid #232838; }
th { color: #8a92a6; font-weight: 600; }
code { background: #1c2233; padding: 0.1rem 0.35rem; border-radius: 4px; }
.pill { display: inline-block; padding: 0.1rem 0.5rem; border-radius: 999px; font-size: 0.8rem; }
.pill.ok { background: rgba(89,212,153,0.15); }
.pill.bad { background: rgba(255,107,107,0.15); }
</style>
</head>
<body>
<h1>Festival 4D</h1>
<div class="sub">Foundation hello page — proves the API contract, CORS, and the frozen pose helper.</div>
<div id="app"><div class="card">Loading…</div></div>
<script type="module" src="/src/main.js"></script>
</body>
</html>

1152
frontend/package-lock.json generated Normal file

File diff suppressed because it is too large Load Diff

17
frontend/package.json Normal file
View File

@ -0,0 +1,17 @@
{
"name": "festival4d-frontend",
"private": true,
"version": "0.1.0",
"type": "module",
"scripts": {
"dev": "vite",
"build": "vite build",
"preview": "vite preview"
},
"dependencies": {
"three": "^0.170.0"
},
"devDependencies": {
"vite": "^6.0.0"
}
}

6
frontend/src/annotate.js Normal file
View File

@ -0,0 +1,6 @@
// bbox drawing + submission → POST /api/annotations (spec M8).
// STUB — this is phase-3 (integration) work; lane C leaves it as a clean seam. Drag a
// rectangle on a video's overlay canvas → normalized bbox → POST /api/annotations; the
// resolved anchor then appears live in overlays + the 3D scene.
export {};

6
frontend/src/camPath.js Normal file
View File

@ -0,0 +1,6 @@
// Keyframed camera paths (spec M9).
// STUB — this is phase-3 (integration) work; lane C leaves it as a clean seam. "Add
// keyframe" captures the current free-roam pose at the current t_global; playback
// interpolates position (Catmull-Rom) + orientation (slerp); export/import path as JSON.
export {};

143
frontend/src/lib/pose.js Normal file
View File

@ -0,0 +1,143 @@
// COLMAP world->camera pose -> Three.js camera pose (spec M5).
//
// FROZEN CONTRACT. This is the JavaScript mirror of `backend/festival4d/geometry.py`
// (`colmap_to_threejs`, `quat_to_mat`). The math and the POSE_TEST_VECTORS below are
// duplicated verbatim on the Python side (`backend/tests/test_geometry.py`) so both agree.
// Lanes B and C CONSUME this; do not reimplement the conversion elsewhere. Changing it
// requires a change request and a synchronized edit to both files + both vector sets.
//
// Conventions:
// - Quaternions are COLMAP order [w, x, y, z] (scalar first), unit norm.
// - Pose (q, t) is world->camera: x_cam = R(q) * x_world + t.
// - COLMAP camera axes: +x right, +y down, +z forward.
// - Three.js cameras look down -z with +y up; hence the diag(1, -1, -1) flip.
const SQRT1_2 = 0.7071067811865476;
// Camera-axis flip: COLMAP camera-local (x right, y down, z forward) -> Three.js (x right,
// y up, z backward). Same _FLIP_YZ as the Python side.
const FLIP_YZ = [
[1, 0, 0],
[0, -1, 0],
[0, 0, -1],
];
/**
* Convert a unit quaternion [w, x, y, z] to a 3x3 rotation matrix (row-major nested array).
* Hamilton convention, right-handed, active rotation (COLMAP world->camera when q is a
* COLMAP pose quaternion).
* @param {number[]} q - [w, x, y, z]
* @returns {number[][]} 3x3 rotation matrix
*/
export function quatToMat(q) {
const [w, x, y, z] = q;
const n = w * w + x * x + y * y + z * z;
if (n < 1e-12) throw new Error("quaternion has near-zero norm");
const s = 2.0 / n;
const wx = s * w * x, wy = s * w * y, wz = s * w * z;
const xx = s * x * x, xy = s * x * y, xz = s * x * z;
const yy = s * y * y, yz = s * y * z, zz = s * z * z;
return [
[1.0 - (yy + zz), xy - wz, xz + wy],
[xy + wz, 1.0 - (xx + zz), yz - wx],
[xz - wy, yz + wx, 1.0 - (xx + yy)],
];
}
function transpose3(m) {
return [
[m[0][0], m[1][0], m[2][0]],
[m[0][1], m[1][1], m[2][1]],
[m[0][2], m[1][2], m[2][2]],
];
}
function matMul3(a, b) {
const out = [[0, 0, 0], [0, 0, 0], [0, 0, 0]];
for (let i = 0; i < 3; i++)
for (let j = 0; j < 3; j++)
out[i][j] = a[i][0] * b[0][j] + a[i][1] * b[1][j] + a[i][2] * b[2][j];
return out;
}
function matVec3(m, v) {
return [
m[0][0] * v[0] + m[0][1] * v[1] + m[0][2] * v[2],
m[1][0] * v[0] + m[1][1] * v[1] + m[1][2] * v[2],
m[2][0] * v[0] + m[2][1] * v[1] + m[2][2] * v[2],
];
}
/**
* Convert a COLMAP world->camera pose to a Three.js camera pose (spec M5). FROZEN.
* @param {number[]} q - COLMAP world->camera quaternion [w, x, y, z]
* @param {number[]} t - COLMAP world->camera translation [tx, ty, tz]
* @returns {{position: number[], rotation: number[][], matrixWorld: number[]}}
* position - camera center in world coords, C = -R^T t (assign to camera.position)
* rotation - Three.js camera world rotation R_three = R^T * diag(1,-1,-1),
* 3x3 row-major (feed to camera.setRotationFromMatrix via a Matrix4)
* matrixWorld - column-major 16-array [R_three | position], ready for
* THREE.Matrix4().fromArray(...) when driving the camera by matrix.
*/
export function colmapToThreejs(q, t) {
const R = quatToMat(q); // world -> cam
const Rc2w = transpose3(R); // cam -> world
const position = matVec3(Rc2w, [-t[0], -t[1], -t[2]]); // C = -R^T t
const rotation = matMul3(Rc2w, FLIP_YZ); // R_three
// Column-major Matrix4 with rotation in the upper-left 3x3 and translation = position.
const m = rotation;
const p = position;
const matrixWorld = [
m[0][0], m[1][0], m[2][0], 0,
m[0][1], m[1][1], m[2][1], 0,
m[0][2], m[1][2], m[2][2], 0,
p[0], p[1], p[2], 1,
];
return { position, rotation, matrixWorld };
}
// --- FROZEN test vectors (must equal POSE_TEST_VECTORS in backend/tests/test_geometry.py).
export const POSE_TEST_VECTORS = [
{
name: "identity",
q: [1.0, 0.0, 0.0, 0.0],
t: [0.0, 0.0, -10.0],
position: [0.0, 0.0, 10.0],
rotation: [[1.0, 0.0, 0.0], [0.0, -1.0, 0.0], [0.0, 0.0, -1.0]],
},
{
name: "yaw90",
q: [SQRT1_2, 0.0, SQRT1_2, 0.0],
t: [0.0, 0.0, 10.0],
position: [10.0, 0.0, 0.0],
rotation: [[0.0, 0.0, 1.0], [0.0, -1.0, 0.0], [1.0, 0.0, 0.0]],
},
{
name: "lookat",
q: [0.0, 1.0, 0.0, 0.0],
t: [0.0, 0.0, 8.0],
position: [0.0, 0.0, 8.0],
rotation: [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]],
},
];
/**
* Self-check the conversion against POSE_TEST_VECTORS. Returns true on success, throws on
* mismatch. Called by main.js so the frozen contract is verified live in the browser.
* @param {number} [atol=1e-9]
*/
export function selfTest(atol = 1e-9) {
for (const vec of POSE_TEST_VECTORS) {
const { position, rotation } = colmapToThreejs(vec.q, vec.t);
for (let i = 0; i < 3; i++) {
if (Math.abs(position[i] - vec.position[i]) > atol)
throw new Error(`pose.js selfTest[${vec.name}]: position mismatch at ${i}`);
for (let j = 0; j < 3; j++) {
if (Math.abs(rotation[i][j] - vec.rotation[i][j]) > atol)
throw new Error(`pose.js selfTest[${vec.name}]: rotation mismatch at ${i},${j}`);
}
}
}
return true;
}

91
frontend/src/main.js Normal file
View File

@ -0,0 +1,91 @@
// Foundation hello page. Fetches /api/manifest (proving the API + CORS work cross-origin)
// and runs the frozen pose.js self-test in the browser. Lane C (M4M6) replaces this with
// the real app; the modules alongside it are stubs to fill in.
import { selfTest as poseSelfTest } from "./lib/pose.js";
// The FastAPI backend (`python -m festival4d serve`). The Vite dev server runs on :5173;
// this cross-origin fetch exercises the CORS config in api.py.
const API_BASE = "http://localhost:8000";
const app = document.getElementById("app");
function card(html) {
const el = document.createElement("div");
el.className = "card";
el.innerHTML = html;
return el;
}
function pill(ok, text) {
return `<span class="pill ${ok ? "ok" : "bad"}">${text}</span>`;
}
function renderPoseSelfTest() {
try {
poseSelfTest();
return card(
`<strong>Pose contract</strong> ${pill(true, "self-test passed")}<br />` +
`<span class="sub">frontend/src/lib/pose.js matches the frozen COLMAP→Three.js vectors.</span>`
);
} catch (err) {
return card(
`<strong>Pose contract</strong> ${pill(false, "self-test FAILED")}<br />` +
`<code>${err.message}</code>`
);
}
}
function renderManifest(m) {
const rows = m.videos
.map(
(v) => `
<tr>
<td>${v.id}</td>
<td><code>${v.filename}</code></td>
<td>${v.duration_s.toFixed(2)} s</td>
<td>${v.width}×${v.height} @ ${v.fps}</td>
<td>${v.offset_ms === null ? "—" : v.offset_ms + " ms"}</td>
</tr>`
)
.join("");
return card(`
<strong>API manifest</strong> ${pill(true, "fetched (CORS ok)")}<br />
<span class="sub">
t_global_max = ${m.t_global_max.toFixed(2)} s ·
has_poses = ${m.has_poses ? "yes" : "no"} ·
${m.videos.length} videos
</span>
<table>
<thead><tr><th>id</th><th>file</th><th>duration</th><th>frame</th><th>offset</th></tr></thead>
<tbody>${rows}</tbody>
</table>
<div class="sub" style="margin-top:0.75rem">
Videos served with HTTP Range from <code>${API_BASE}/media/</code>
</div>
`);
}
async function main() {
app.innerHTML = "";
app.appendChild(renderPoseSelfTest());
try {
const resp = await fetch(`${API_BASE}/api/manifest`);
if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
const manifest = await resp.json();
app.appendChild(renderManifest(manifest));
console.log("[festival4d] manifest loaded", manifest);
} catch (err) {
app.appendChild(
card(
`<strong>API manifest</strong> ${pill(false, "fetch failed")}<br />` +
`<code>${err.message}</code><br />` +
`<span class="sub">Is the backend running? <code>python -m festival4d serve</code></span>`
)
);
console.error("[festival4d] manifest fetch failed", err);
}
}
main();

6
frontend/src/overlays.js Normal file
View File

@ -0,0 +1,6 @@
// 3D→2D anchor projection onto each video's overlay canvas — the "x-ray" HUD (spec M6).
// STUB — lane C fills this in. For each anchor + visible video: build the video's current
// Three.js camera (pose.js helper), project the anchor, convert NDC→canvas px, skip if
// behind camera (check z_cam < 0). Anchors render regardless of occlusion — that's the feature.
export {};

6
frontend/src/scene3d.js Normal file
View File

@ -0,0 +1,6 @@
// Three.js scene: point cloud (/api/pointcloud via PLYLoader), per-video camera paths +
// current-pose frusta, OrbitControls, snap-to-camera (spec M5).
// STUB — lane C fills this in. Use frontend/src/lib/pose.js (frozen contract) for ALL
// COLMAP→Three.js conversion — do not reimplement the diag(1,-1,-1) math.
export {};

9
frontend/src/state.js Normal file
View File

@ -0,0 +1,9 @@
// Global store: manifest, playhead (t_global), playing, rate, selected camera, audio source.
// STUB — lane C (M4) fills this in. Kept as an empty module so the file exists per the
// spec's repo layout; main.js does not import it yet.
// Example shape lane C will build toward:
// export const state = { manifest: null, tGlobal: 0, playing: false, rate: 1,
// selectedCamera: null, audioSource: null };
export {};

6
frontend/src/timeline.js Normal file
View File

@ -0,0 +1,6 @@
// Scrubber + event markers (spec M4 transport + M7 timeline markers).
// STUB — lane C fills this in. Render GET /api/events as colored markers (color by
// event_type, legend in a corner); hover → description tooltip; click → jump the playhead.
// The event data is already seeded synthetically; classification quality is lane D's job.
export {};

View File

@ -0,0 +1,8 @@
// Master clock + per-video sync (spec M4 — the heart of the app).
// STUB — lane C fills this in. Build and verify the master-clock transport before the 3D
// work. Use the timebase from config.py (mirrored here):
// t_video = (t_global - offset_ms/1000) * (1 + drift_ppm * 1e-6)
// Never trust a single <video> element as the clock; drive t_global from performance.now()
// and continuously correct each video (hard-seek >150ms, nudge playbackRate 20150ms).
export {};

View File

@ -0,0 +1,5 @@
// <video> grid + per-video transparent overlay canvases (spec M4/M6).
// STUB — lane C fills this in. Mind spec pitfall #4: object-fit letterboxing means canvas
// px != video px; compute each video's content rect for the overlay canvas.
export {};

20
plan/CHANGE_REQUESTS.md Normal file
View File

@ -0,0 +1,20 @@
# Change Requests
Lanes that believe a **frozen** file must change (`db.py` schema, `api.py` routes/shapes,
`cli.py`, `config.py`, `synthetic.py`, `pyproject.toml`, the pose contract) do **not** edit it.
Instead append an entry here and work around it locally; the integration agent adjudicates.
**Format** (one entry per request, newest at the bottom):
```
### CR-<n><lane><date><one-line title>
- **File:** <frozen file + symbol>
- **Problem:** why the current contract blocks you
- **Proposed change:** the minimal change you need
- **Workaround in place:** what you did locally so you're not blocked
- **Decision:** (integration fills this: applied | rejected — rationale)
```
---
<!-- entries below -->

31
plan/status/foundation.md Normal file
View File

@ -0,0 +1,31 @@
# Status — foundation
## Round 0 — 2026-07-16 — STATUS: ready_to_merge
**Directives acknowledged:** round 0 of plan/DIRECTIVES.md (run `foundation` only; contracts
correctness beats speed; lanes hard-depend on API shapes, schema, pose math, synthetic fixtures).
**Acceptance checklist** (plan/00-foundation.md + spec M0/M3) — all pass:
- [x] `python -m festival4d synthetic` → 3 fixture videos + populated DB + `points.ply` (3106 pts) + `ground_truth.json`. Evidence: CLI run logs "3 videos, 3106 points, 7 events, 4 anchors".
- [x] `python -m festival4d serve` → every M3 endpoint returns synthetic data; videos seekable. Evidence: curl of manifest/poses/anchors/events/pointcloud; `curl -H "Range: bytes=0-100" /media/cam0.mp4`**206** with `content-range`; browser `<video>` seeked to 15s, `seekable=[0,20]`, readyState 4.
- [x] `pytest` green — **24 passed** (test_geometry, test_synthetic, test_api).
- [x] `npm run dev` serves the hello page; browser fetched `/api/manifest` cross-origin (CORS ok), pose.js self-test passed, **0 console errors**. (Vite fell back to :5174 — an unrelated dev server holds :5173; CORS now allows any localhost port.)
- [x] All lane-owned modules stubbed with frozen signatures; `cli.py` registers all 6 subcommands (dispatch + graceful NotImplementedError → exit 2).
- [x] Frozen geometry contract: `colmap_to_threejs` + `test_geometry.py` (3 known vectors, random round-trip, scipy oracle) + mirrored `frontend/src/lib/pose.js` with identical `POSE_TEST_VECTORS` (node self-test passes).
- [x] Classifier contract: `MomentClassification` (Pydantic) + `MomentClassifier` (runtime Protocol) + Gemini/Claude/Local provider stubs.
- [x] `plan/CHANGE_REQUESTS.md` created.
**Done this round:** full M0 + M3 + all frozen contracts. Env: Python 3.12 venv via uv, all
spec deps installed. DB schema per spec §2 with CRUD helpers. Timebase helpers in `config.py`.
API is Range-capable and CORS-enabled. Frontend scaffolds every layout file (lane-C modules
are stubs; `lib/pose.js` is the live frozen contract).
**Frozen-contract note for lanes:** CORS was widened from a single `:5173` origin to any
`http://localhost|127.0.0.1:<port>` (local single-user tool; robust to Vite port fallback).
Explicit `:5173` still included. If a lane needs a different API shape/schema/pose math,
file a CR in `plan/CHANGE_REQUESTS.md` — do not edit `db.py`/`api.py`/`cli.py`/`config.py`/
`synthetic.py`/`pyproject.toml`.
**Blockers / questions for coordinator:** none.
**Next:** merge `foundation``main`; lanes may start (per directive: C, then A, D, B).

38
pyproject.toml Normal file
View File

@ -0,0 +1,38 @@
[project]
name = "festival4d"
version = "0.1.0"
description = "Turn multiple fan-shot smartphone concert videos into a synchronized, explorable 4D experience."
readme = "README.md"
requires-python = ">=3.11"
dependencies = [
"fastapi>=0.110",
"uvicorn[standard]>=0.29",
"numpy>=1.26",
"scipy>=1.11",
"librosa>=0.10",
"soundfile>=0.12",
"sqlalchemy>=2.0",
"pydantic>=2.6",
"opencv-python-headless>=4.9",
# classifier providers (lane D) — imported lazily, never at module load
"google-genai>=0.3",
"anthropic>=0.40",
"openai>=1.30",
]
[project.optional-dependencies]
dev = ["pytest>=8.0"]
[project.scripts]
festival4d = "festival4d.cli:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["backend/festival4d"]
[tool.pytest.ini_options]
testpaths = ["backend/tests"]
addopts = "-q"

1998
uv.lock generated Normal file

File diff suppressed because it is too large Load Diff