Foundation (M0 + M3 + frozen contracts): scaffold, synthetic fixtures, full API
What now works: - M0 scaffold: pyproject (all spec deps), uv/py3.12 env, `python -m festival4d` CLI registering synthetic|ingest|sync|reconstruct|events|serve. Vite hello page. - Synthetic fixture (synthetic.py): 3 shifted-audio videos (offsets 0/+1370/-842 ms), camera-arc poses, stage point cloud -> points.ply, seeded events + anchors, ground_truth.json. `python -m festival4d synthetic` populates data/ + DB. - DB schema exactly per spec §2 (db.py) + CRUD helpers all lanes use. - M3 API (api.py) full against synthetic data: manifest/poses/pointcloud/anchors/ events/detect/annotations; Range-capable video serving (206 verified); CORS for any localhost origin. - Frozen geometry contract: geometry.colmap_to_threejs (M5 math) + unit test (3 known vectors, random round-trip, scipy oracle); mirrored frontend/src/lib/pose.js with identical POSE_TEST_VECTORS. Lane-B stubs: slerp_pose, ray_from_pixel, triangulate_rays, nearest_point_on_ray. - Classifier contract (events_ai.py): MomentClassification model + MomentClassifier protocol + Gemini/Claude/Local provider stubs. - Lane-owned modules stubbed with final signatures (ingest, audio_sync, frames, sfm, events_ai); cli/api catch NotImplementedError and degrade gracefully. - plan/CHANGE_REQUESTS.md created; plan/status/foundation.md updated. Acceptance: pytest 24 passed; serve endpoints verified via curl + browser (video seek, manifest fetch cross-origin, pose.js self-test, 0 console errors). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
47bec93bc5
commit
cf6a6fb2c8
57
README.md
Normal file
57
README.md
Normal file
@ -0,0 +1,57 @@
|
||||
# Festival 4D
|
||||
|
||||
Turn multiple fan-shot smartphone videos of the same concert into a synchronized,
|
||||
explorable **4D experience**: time-aligned multi-video playback, 3D scene reconstruction
|
||||
with camera poses, a free-roam "god's eye" viewer, AR-style overlays projected onto each
|
||||
video, and AI-tagged moments on a shared timeline.
|
||||
|
||||
> **Status: foundation phase.** The scaffold, database, frozen contracts (API, pose math,
|
||||
> DB schema), and a synthetic fixture generator are in place. Feature lanes (media sync,
|
||||
> reconstruction, viewer, AI events) build on top. The full README with the real-footage
|
||||
> workflow is written in the integration phase (M9). See
|
||||
> [`OPUS_BUILD_INSTRUCTIONS.md`](OPUS_BUILD_INSTRUCTIONS.md) for the canonical spec and
|
||||
> [`plan/`](plan/) for the execution plan.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Python 3.11+** and [`uv`](https://docs.astral.sh/uv/) (or venv + pip)
|
||||
- **ffmpeg** / **ffprobe** on your `PATH` (required)
|
||||
- **COLMAP** (optional — reconstruction degrades gracefully without it)
|
||||
- A classifier API key (optional — `GEMINI_API_KEY` by default; see `events_ai.py`)
|
||||
- **Node 18+** for the frontend
|
||||
|
||||
## Quickstart (synthetic demo — no footage needed)
|
||||
|
||||
```bash
|
||||
# 1. backend env
|
||||
uv venv --python 3.12
|
||||
uv pip install -e ".[dev]"
|
||||
|
||||
# 2. generate the synthetic fixture project (fake videos + poses + point cloud + events)
|
||||
uv run python -m festival4d synthetic
|
||||
|
||||
# 3. serve the API (http://127.0.0.1:8000)
|
||||
uv run python -m festival4d serve
|
||||
|
||||
# 4. in another terminal, the frontend
|
||||
cd frontend
|
||||
npm install
|
||||
npm run dev # http://localhost:5173
|
||||
```
|
||||
|
||||
## Backend CLI
|
||||
|
||||
```
|
||||
python -m festival4d synthetic # generate the synthetic fixture (M0)
|
||||
python -m festival4d ingest # probe videos + extract audio (lane A / M1)
|
||||
python -m festival4d sync # GCC-PHAT audio alignment (lane A / M1)
|
||||
python -m festival4d reconstruct # COLMAP SfM + pose export (lane B / M2)
|
||||
python -m festival4d events # audio candidates + AI classify (lane D / M7)
|
||||
python -m festival4d serve # FastAPI app (M3)
|
||||
```
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
uv run pytest
|
||||
```
|
||||
8
backend/festival4d/__init__.py
Normal file
8
backend/festival4d/__init__.py
Normal file
@ -0,0 +1,8 @@
|
||||
"""Festival 4D — synchronized, explorable 4D concert replay.
|
||||
|
||||
See ``OPUS_BUILD_INSTRUCTIONS.md`` for the canonical spec and ``plan/`` for the
|
||||
parallel execution plan. This package is the backend: CLI, database, synthetic
|
||||
fixture generator, media/reconstruction/AI pipelines, and the FastAPI app.
|
||||
"""
|
||||
|
||||
__version__ = "0.1.0"
|
||||
6
backend/festival4d/__main__.py
Normal file
6
backend/festival4d/__main__.py
Normal file
@ -0,0 +1,6 @@
|
||||
"""Enable ``python -m festival4d <cmd>``."""
|
||||
|
||||
from festival4d.cli import main
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
209
backend/festival4d/api.py
Normal file
209
backend/festival4d/api.py
Normal file
@ -0,0 +1,209 @@
|
||||
"""FastAPI app (spec M3). FROZEN after foundation — lanes never edit routes or shapes.
|
||||
|
||||
Serves the synthetic project fully so lane C can treat this API as finished:
|
||||
|
||||
GET /api/manifest -> {videos:[{id,filename,url,duration_s,fps,width,height,
|
||||
offset_ms,drift_ppm}], t_global_max, has_poses}
|
||||
GET /api/videos/{id}/poses -> [{frame_idx,t_video_s,q:[w,x,y,z],t:[x,y,z],
|
||||
intrinsics:{fx,fy,cx,cy},registered}]
|
||||
GET /api/pointcloud -> points.ply (binary; Range-capable)
|
||||
GET /api/anchors -> [...] POST /api/anchors
|
||||
GET /api/events -> [...]
|
||||
POST /api/events/detect {t_global_s?, window_s?} -> runs M7 detection
|
||||
POST /api/annotations {video_id, t_video_s, bbox:[x0,y0,x1,y1]} -> {anchor_id?, point?}
|
||||
|
||||
Source videos in ``data/raw`` are mounted at ``/media`` via Starlette ``StaticFiles``,
|
||||
which supports HTTP Range (required for ``<video>`` seeking; verify ``curl -H "Range:
|
||||
bytes=0-100"`` -> 206). Stubbed lane entrypoints (events detection, M8 annotation
|
||||
resolution) raise ``NotImplementedError``; we catch it here and degrade to a 200 with a
|
||||
note so the frozen routes never 500 while a lane is still stubbed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from fastapi import FastAPI, HTTPException
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.responses import FileResponse
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from festival4d import config, db
|
||||
|
||||
log = logging.getLogger("festival4d.api")
|
||||
|
||||
app = FastAPI(title="Festival 4D", version="0.1.0")
|
||||
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=config.FRONTEND_ORIGINS,
|
||||
allow_origin_regex=config.FRONTEND_ORIGIN_REGEX,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
# Bind the DB and ensure the media directory exists so the static mount is valid even
|
||||
# before the synthetic fixture has been generated.
|
||||
db.init_engine()
|
||||
db.init_db()
|
||||
config.RAW_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# HTTP Range-capable static serving of source videos (spec M3, pitfall #3).
|
||||
app.mount("/media", StaticFiles(directory=config.RAW_DIR), name="media")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Request models
|
||||
# ---------------------------------------------------------------------------
|
||||
class AnchorIn(BaseModel):
|
||||
label: str
|
||||
x: float
|
||||
y: float
|
||||
z: float
|
||||
color: str | None = None
|
||||
|
||||
|
||||
class DetectIn(BaseModel):
|
||||
t_global_s: float | None = None
|
||||
window_s: float | None = None
|
||||
|
||||
|
||||
class AnnotationIn(BaseModel):
|
||||
video_id: int
|
||||
t_video_s: float
|
||||
bbox: list[float] = Field(min_length=4, max_length=4) # [x0, y0, x1, y1] normalized
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Serialization helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
def _video_dict(v) -> dict:
|
||||
return {
|
||||
"id": v.id,
|
||||
"filename": v.filename,
|
||||
"url": f"/media/{v.filename}",
|
||||
"duration_s": v.duration_s,
|
||||
"fps": v.fps,
|
||||
"width": v.width,
|
||||
"height": v.height,
|
||||
"offset_ms": v.offset_ms,
|
||||
"drift_ppm": v.drift_ppm,
|
||||
}
|
||||
|
||||
|
||||
def _pose_dict(p) -> dict:
|
||||
return {
|
||||
"frame_idx": p.frame_idx,
|
||||
"t_video_s": p.t_video_s,
|
||||
"q": [p.qw, p.qx, p.qy, p.qz],
|
||||
"t": [p.tx, p.ty, p.tz],
|
||||
"intrinsics": {"fx": p.fx, "fy": p.fy, "cx": p.cx, "cy": p.cy},
|
||||
"registered": p.registered,
|
||||
}
|
||||
|
||||
|
||||
def _anchor_dict(a) -> dict:
|
||||
return {"id": a.id, "label": a.label, "x": a.x, "y": a.y, "z": a.z, "color": a.color}
|
||||
|
||||
|
||||
def _event_dict(e) -> dict:
|
||||
return {
|
||||
"id": e.id,
|
||||
"t_global_s": e.t_global_s,
|
||||
"duration_s": e.duration_s,
|
||||
"event_type": e.event_type,
|
||||
"confidence": e.confidence,
|
||||
"description": e.description,
|
||||
"source": e.source,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Routes
|
||||
# ---------------------------------------------------------------------------
|
||||
@app.get("/api/health")
|
||||
def health() -> dict:
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
@app.get("/api/manifest")
|
||||
def manifest() -> dict:
|
||||
videos = db.get_videos()
|
||||
t_global_max = 0.0
|
||||
for v in videos:
|
||||
offset_s = (v.offset_ms or 0.0) / 1000.0
|
||||
t_global_max = max(t_global_max, offset_s + v.duration_s)
|
||||
return {
|
||||
"videos": [_video_dict(v) for v in videos],
|
||||
"t_global_max": t_global_max,
|
||||
"has_poses": db.has_poses(),
|
||||
}
|
||||
|
||||
|
||||
@app.get("/api/videos/{video_id}/poses")
|
||||
def video_poses(video_id: int) -> list[dict]:
|
||||
if db.get_video(video_id) is None:
|
||||
raise HTTPException(status_code=404, detail=f"no video with id={video_id}")
|
||||
return [_pose_dict(p) for p in db.get_poses(video_id)]
|
||||
|
||||
|
||||
@app.get("/api/pointcloud")
|
||||
def pointcloud() -> FileResponse:
|
||||
if not config.POINTS_PLY.exists():
|
||||
raise HTTPException(status_code=404, detail="no point cloud (run `synthetic` or `reconstruct`)")
|
||||
return FileResponse(
|
||||
config.POINTS_PLY,
|
||||
media_type="application/octet-stream",
|
||||
filename="points.ply",
|
||||
)
|
||||
|
||||
|
||||
@app.get("/api/anchors")
|
||||
def get_anchors() -> list[dict]:
|
||||
return [_anchor_dict(a) for a in db.get_anchors()]
|
||||
|
||||
|
||||
@app.post("/api/anchors")
|
||||
def create_anchor(payload: AnchorIn) -> dict:
|
||||
anchor = db.add_anchor(payload.label, payload.x, payload.y, payload.z, payload.color)
|
||||
return _anchor_dict(anchor)
|
||||
|
||||
|
||||
@app.get("/api/events")
|
||||
def get_events() -> list[dict]:
|
||||
return [_event_dict(e) for e in db.get_events()]
|
||||
|
||||
|
||||
@app.post("/api/events/detect")
|
||||
def detect_events(payload: DetectIn) -> dict:
|
||||
from festival4d import events_ai
|
||||
|
||||
try:
|
||||
result = events_ai.run_events(t_global_s=payload.t_global_s, window_s=payload.window_s)
|
||||
except NotImplementedError as exc:
|
||||
log.info("events detection not implemented yet: %s", exc)
|
||||
return {
|
||||
"result": None,
|
||||
"note": "moment detection not implemented yet (lane D / M7)",
|
||||
"events": [_event_dict(e) for e in db.get_events()],
|
||||
}
|
||||
return {"result": result, "events": [_event_dict(e) for e in db.get_events()]}
|
||||
|
||||
|
||||
@app.post("/api/annotations")
|
||||
def create_annotation(payload: AnnotationIn) -> dict:
|
||||
x0, y0, x1, y1 = payload.bbox
|
||||
annotation = db.add_annotation(payload.video_id, payload.t_video_s, x0, y0, x1, y1)
|
||||
|
||||
# M8 resolution (triangulate / nearest-point) is integration work; geometry stubs raise
|
||||
# until lane B lands. Store the annotation now; resolve to an anchor later.
|
||||
anchor_id = None
|
||||
point = None
|
||||
note = "annotation stored; 3D resolution is M8/integration work"
|
||||
return {
|
||||
"annotation_id": annotation.id,
|
||||
"anchor_id": anchor_id,
|
||||
"point": point,
|
||||
"note": note,
|
||||
}
|
||||
66
backend/festival4d/audio_sync.py
Normal file
66
backend/festival4d/audio_sync.py
Normal file
@ -0,0 +1,66 @@
|
||||
"""GCC-PHAT pairwise audio offsets + global graph solve + drift (spec M1, lane A).
|
||||
|
||||
STUB — lane A fills the bodies; the public signatures below are frozen (``cli.py`` calls
|
||||
``run_sync``).
|
||||
|
||||
Algorithm (spec M1, for lane A's reference):
|
||||
- **GCC-PHAT**, not raw cross-correlation: whiten the cross-spectrum
|
||||
``R = X * conj(Y); R /= |R| + eps; r = irfft(R)`` — robust to loud/clipped/reverberant
|
||||
concert audio. numpy/scipy FFTs; librosa only for loading.
|
||||
- All **pairwise** offsets, each scored by peak-to-second-peak ratio; then a global
|
||||
least-squares solve over the offset graph, weighted by confidence, rejecting pairs that
|
||||
violate cycle consistency (``off_ab + off_bc - off_ac > 50 ms``).
|
||||
- **Drift**: re-run GCC-PHAT on 10 s windows every 30 s along the overlap; fit a line;
|
||||
store the slope as ``drift_ppm`` (store 0 if ``|drift| < 5 ppm``).
|
||||
- Disconnected sync-graph components -> ``offset_ms = None`` (never a guess; spec pitfall #5).
|
||||
- Persist via ``db.update_video_sync`` and export ``config.SYNC_JSON``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
import numpy as np
|
||||
|
||||
log = logging.getLogger("festival4d.audio_sync")
|
||||
|
||||
|
||||
def gcc_phat(sig: np.ndarray, ref: np.ndarray, sr: int,
|
||||
max_lag_s: float | None = None) -> tuple[float, float]:
|
||||
"""GCC-PHAT time delay of ``sig`` relative to ``ref`` (spec M1, lane A).
|
||||
|
||||
Returns ``(offset_s, confidence)`` where confidence is the peak-to-second-peak ratio.
|
||||
"""
|
||||
raise NotImplementedError("lane A (M1): implement gcc_phat")
|
||||
|
||||
|
||||
def pairwise_offsets(signals: dict[int, np.ndarray], sr: int) -> list[dict]:
|
||||
"""All pairwise GCC-PHAT offsets with confidences (spec M1, lane A).
|
||||
|
||||
``signals`` maps ``video_id -> mono samples``. Returns a list of
|
||||
``{a, b, offset_s, confidence}`` edges.
|
||||
"""
|
||||
raise NotImplementedError("lane A (M1): implement pairwise_offsets")
|
||||
|
||||
|
||||
def solve_global_offsets(edges: list[dict], video_ids: list[int]) -> dict[int, float | None]:
|
||||
"""Least-squares global solve over the pairwise-offset graph (spec M1, lane A).
|
||||
|
||||
Reference node is pinned to 0; disconnected components get ``None`` (spec pitfall #5).
|
||||
Returns ``video_id -> offset_ms`` (or ``None``).
|
||||
"""
|
||||
raise NotImplementedError("lane A (M1): implement solve_global_offsets")
|
||||
|
||||
|
||||
def estimate_drift(sig: np.ndarray, ref: np.ndarray, sr: int) -> float:
|
||||
"""Estimate clock drift in ppm by fitting window-offsets over the overlap (spec M1)."""
|
||||
raise NotImplementedError("lane A (M1): implement estimate_drift")
|
||||
|
||||
|
||||
def run_sync() -> dict:
|
||||
"""Sync every ingested video: pairwise GCC-PHAT, global solve, drift, persist.
|
||||
|
||||
Entrypoint for ``python -m festival4d sync``. Writes ``offset_ms``/``drift_ppm``/
|
||||
``sync_confidence`` via ``db`` and exports ``config.SYNC_JSON``. Returns the solution.
|
||||
"""
|
||||
raise NotImplementedError("lane A (M1): implement run_sync")
|
||||
130
backend/festival4d/cli.py
Normal file
130
backend/festival4d/cli.py
Normal file
@ -0,0 +1,130 @@
|
||||
"""Command-line entrypoint: ``python -m festival4d <cmd>``.
|
||||
|
||||
FROZEN after foundation. Lanes fill in the bodies of the functions this dispatches
|
||||
to (in ``ingest.py``, ``audio_sync.py``, ``sfm.py``, ``events_ai.py``); they never
|
||||
edit this file or ``api.py``.
|
||||
|
||||
Subcommands: ``synthetic | ingest | sync | reconstruct | events | serve``.
|
||||
|
||||
Every subcommand dispatches to a single lane entrypoint. While a lane is still a stub,
|
||||
its entrypoint raises :class:`NotImplementedError`; we catch that here and print a clear
|
||||
"not implemented yet" message with a non-zero exit code, rather than a traceback. Once the
|
||||
lane fills the body, the same dispatch runs it for real — no change to this file needed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
import sys
|
||||
|
||||
log = logging.getLogger("festival4d.cli")
|
||||
|
||||
|
||||
def _cmd_synthetic(args: argparse.Namespace) -> int:
|
||||
from festival4d import synthetic
|
||||
|
||||
synthetic.build()
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_ingest(args: argparse.Namespace) -> int:
|
||||
from festival4d import ingest
|
||||
|
||||
ingest.run_ingest()
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_sync(args: argparse.Namespace) -> int:
|
||||
from festival4d import audio_sync
|
||||
|
||||
audio_sync.run_sync()
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_reconstruct(args: argparse.Namespace) -> int:
|
||||
from festival4d import sfm
|
||||
|
||||
sfm.run_reconstruct()
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_events(args: argparse.Namespace) -> int:
|
||||
from festival4d import events_ai
|
||||
|
||||
result = events_ai.run_events(t_global_s=args.t_global_s, window_s=args.window_s)
|
||||
log.info("events: %s", result)
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_serve(args: argparse.Namespace) -> int:
|
||||
import uvicorn
|
||||
|
||||
uvicorn.run(
|
||||
"festival4d.api:app",
|
||||
host=args.host,
|
||||
port=args.port,
|
||||
reload=args.reload,
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
from festival4d import config
|
||||
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="festival4d",
|
||||
description="Festival 4D — synchronized, explorable 4D concert replay.",
|
||||
)
|
||||
sub = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
p_syn = sub.add_parser("synthetic", help="generate the synthetic fixture project (M0)")
|
||||
p_syn.set_defaults(func=_cmd_synthetic)
|
||||
|
||||
p_ing = sub.add_parser("ingest", help="probe videos + extract audio (lane A / M1)")
|
||||
p_ing.set_defaults(func=_cmd_ingest)
|
||||
|
||||
p_sync = sub.add_parser("sync", help="GCC-PHAT audio alignment (lane A / M1)")
|
||||
p_sync.set_defaults(func=_cmd_sync)
|
||||
|
||||
p_rec = sub.add_parser("reconstruct", help="COLMAP SfM + pose export (lane B / M2)")
|
||||
p_rec.set_defaults(func=_cmd_reconstruct)
|
||||
|
||||
p_evt = sub.add_parser("events", help="audio candidates + AI classification (lane D / M7)")
|
||||
p_evt.add_argument("--t-global-s", dest="t_global_s", type=float, default=None,
|
||||
help="detect within a single window centered here (default: whole track)")
|
||||
p_evt.add_argument("--window-s", dest="window_s", type=float, default=None,
|
||||
help="window width in seconds when --t-global-s is given")
|
||||
p_evt.set_defaults(func=_cmd_events)
|
||||
|
||||
p_srv = sub.add_parser("serve", help="run the FastAPI app (M3)")
|
||||
p_srv.add_argument("--host", default=config.API_HOST)
|
||||
p_srv.add_argument("--port", type=int, default=config.API_PORT)
|
||||
p_srv.add_argument("--reload", action="store_true", help="uvicorn autoreload (dev)")
|
||||
p_srv.set_defaults(func=_cmd_serve)
|
||||
|
||||
return parser
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s %(levelname)-7s %(name)s: %(message)s",
|
||||
)
|
||||
parser = build_parser()
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
return args.func(args)
|
||||
except NotImplementedError as exc:
|
||||
detail = f" ({exc})" if str(exc) else ""
|
||||
print(
|
||||
f"festival4d {args.command}: not implemented yet{detail}.\n"
|
||||
f"This subcommand is owned by a feature lane that hasn't landed. "
|
||||
f"Run `python -m festival4d synthetic` for the working demo path.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
78
backend/festival4d/config.py
Normal file
78
backend/festival4d/config.py
Normal file
@ -0,0 +1,78 @@
|
||||
"""Paths, constants, and the timebase convention.
|
||||
|
||||
Timebase convention (FROZEN CONTRACT — do not change without a change request):
|
||||
|
||||
``t_global`` is the master timeline in seconds.
|
||||
For a video *v* with ``offset_ms`` and ``drift_ppm``::
|
||||
|
||||
t_video = (t_global - offset_ms / 1000) * (1 + drift_ppm * 1e-6)
|
||||
|
||||
The reference video has ``offset_ms == 0`` (and ``drift_ppm == 0``). A positive
|
||||
``offset_ms`` means the video started recording *later* than the master zero, so at a
|
||||
given ``t_global`` its local playhead is earlier. Use :func:`t_video_from_global` and
|
||||
:func:`t_global_from_video` everywhere — never re-derive the algebra inline. The
|
||||
frontend mirrors :func:`t_video_from_global` in ``transport.js``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Paths. The data root can be overridden with FESTIVAL4D_DATA_DIR (used by tests
|
||||
# to run against a temporary project without touching the repo's data/).
|
||||
# ---------------------------------------------------------------------------
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
|
||||
def _data_dir() -> Path:
|
||||
env = os.environ.get("FESTIVAL4D_DATA_DIR")
|
||||
return Path(env).resolve() if env else REPO_ROOT / "data"
|
||||
|
||||
|
||||
DATA_DIR = _data_dir()
|
||||
RAW_DIR = DATA_DIR / "raw" # user drops source videos here (gitignored)
|
||||
WORK_DIR = DATA_DIR / "work" # extracted wavs, frames, colmap workspace (gitignored)
|
||||
AUDIO_DIR = WORK_DIR / "audio" # per-video mono 16 kHz wavs (ingest)
|
||||
FRAMES_DIR = WORK_DIR / "frames" # sampled JPEGs for SfM
|
||||
COLMAP_DIR = WORK_DIR / "colmap" # COLMAP workspace
|
||||
DB_PATH = DATA_DIR / "project.db" # SQLite (gitignored)
|
||||
|
||||
POINTS_PLY = WORK_DIR / "points.ply" # reconstructed / synthetic point cloud
|
||||
SYNC_JSON = WORK_DIR / "sync.json" # exported sync solution (lane A)
|
||||
GROUND_TRUTH_JSON = WORK_DIR / "ground_truth.json" # synthetic fixture ground truth
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Constants
|
||||
# ---------------------------------------------------------------------------
|
||||
FRONTEND_ORIGINS = [
|
||||
"http://localhost:5173",
|
||||
"http://127.0.0.1:5173",
|
||||
]
|
||||
# Local-only, single-user, offline tool: allow any localhost origin so a Vite dev server on
|
||||
# a fallback port (5174, …) still works. Kept alongside the explicit 5173 default.
|
||||
FRONTEND_ORIGIN_REGEX = r"http://(localhost|127\.0\.0\.1)(:\d+)?"
|
||||
API_HOST = "127.0.0.1"
|
||||
API_PORT = 8000
|
||||
|
||||
AUDIO_SAMPLE_RATE = 16_000 # ingest extracts mono 16 kHz for sync (M1)
|
||||
|
||||
# Synthetic fixture parameters (M0). Ground-truth offsets are deliberately odd,
|
||||
# non-round numbers so a naive integer-second guesser can't accidentally "pass".
|
||||
SYNTH_VIDEO_W = 640
|
||||
SYNTH_VIDEO_H = 360
|
||||
SYNTH_FPS = 30
|
||||
SYNTH_DURATION_S = 20.0
|
||||
SYNTH_AUDIO_SR = 48_000
|
||||
SYNTH_OFFSETS_MS = (0.0, 1370.0, -842.0) # cam0 is the reference (offset 0)
|
||||
|
||||
|
||||
def t_video_from_global(t_global: float, offset_ms: float, drift_ppm: float = 0.0) -> float:
|
||||
"""Map master-timeline seconds to a video's local playhead seconds (FROZEN)."""
|
||||
return (t_global - offset_ms / 1000.0) * (1.0 + drift_ppm * 1e-6)
|
||||
|
||||
|
||||
def t_global_from_video(t_video: float, offset_ms: float, drift_ppm: float = 0.0) -> float:
|
||||
"""Inverse of :func:`t_video_from_global` (FROZEN)."""
|
||||
return t_video / (1.0 + drift_ppm * 1e-6) + offset_ms / 1000.0
|
||||
397
backend/festival4d/db.py
Normal file
397
backend/festival4d/db.py
Normal file
@ -0,0 +1,397 @@
|
||||
"""SQLAlchemy engine/session and schema (spec §2) plus small CRUD helpers.
|
||||
|
||||
FROZEN CONTRACT after foundation: the schema (table/column names and types) and the
|
||||
helper signatures are what every lane depends on. Lanes write through these helpers only.
|
||||
|
||||
Schema (spec §2)::
|
||||
|
||||
videos(id, filename, duration_s, fps, width, height, offset_ms, drift_ppm,
|
||||
sync_confidence, created_at)
|
||||
camera_poses(id, video_id FK, frame_idx, t_video_s, qw,qx,qy,qz, tx,ty,tz,
|
||||
fx,fy,cx,cy, registered) # COLMAP world->camera; registered=False => interpolated
|
||||
anchors(id, label, x,y,z, color)
|
||||
events(id, t_global_s, duration_s, event_type, confidence, description, source)
|
||||
annotations(id, video_id FK, t_video_s, x0,y0,x1,y1, resolved_anchor_id FK NULL)
|
||||
|
||||
Engine management: :func:`init_engine` (re)binds the module to a SQLite file. It defaults
|
||||
to ``config.DB_PATH`` but tests point it at a temp file. Helpers open and commit their own
|
||||
short-lived sessions and return detached ORM instances (``expire_on_commit=False``), so the
|
||||
returned objects are safe to read after the session closes.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from contextlib import contextmanager
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Iterable, Iterator, Sequence
|
||||
|
||||
from sqlalchemy import (
|
||||
Boolean,
|
||||
Float,
|
||||
ForeignKey,
|
||||
Integer,
|
||||
String,
|
||||
create_engine,
|
||||
delete,
|
||||
select,
|
||||
)
|
||||
from sqlalchemy.engine import Engine
|
||||
from sqlalchemy.orm import DeclarativeBase, Mapped, Session, mapped_column, sessionmaker
|
||||
|
||||
from festival4d import config
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ORM models
|
||||
# ---------------------------------------------------------------------------
|
||||
class Base(DeclarativeBase):
|
||||
pass
|
||||
|
||||
|
||||
class Video(Base):
|
||||
__tablename__ = "videos"
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
filename: Mapped[str] = mapped_column(String, nullable=False, unique=True)
|
||||
duration_s: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
fps: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
width: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
height: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
offset_ms: Mapped[float | None] = mapped_column(Float, nullable=True)
|
||||
drift_ppm: Mapped[float | None] = mapped_column(Float, nullable=True)
|
||||
sync_confidence: Mapped[float | None] = mapped_column(Float, nullable=True)
|
||||
created_at: Mapped[str] = mapped_column(String, nullable=False)
|
||||
|
||||
|
||||
class CameraPose(Base):
|
||||
__tablename__ = "camera_poses"
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
video_id: Mapped[int] = mapped_column(ForeignKey("videos.id"), nullable=False, index=True)
|
||||
frame_idx: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
t_video_s: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
# COLMAP world->camera quaternion [w,x,y,z] + translation
|
||||
qw: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
qx: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
qy: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
qz: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
tx: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
ty: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
tz: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
# pinhole intrinsics after undistortion
|
||||
fx: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
fy: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
cx: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
cy: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
registered: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
|
||||
|
||||
|
||||
class Anchor(Base):
|
||||
__tablename__ = "anchors"
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
label: Mapped[str] = mapped_column(String, nullable=False)
|
||||
x: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
y: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
z: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
color: Mapped[str | None] = mapped_column(String, nullable=True)
|
||||
|
||||
|
||||
class Event(Base):
|
||||
__tablename__ = "events"
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
t_global_s: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
duration_s: Mapped[float | None] = mapped_column(Float, nullable=True)
|
||||
event_type: Mapped[str] = mapped_column(String, nullable=False)
|
||||
confidence: Mapped[float | None] = mapped_column(Float, nullable=True)
|
||||
description: Mapped[str | None] = mapped_column(String, nullable=True)
|
||||
source: Mapped[str] = mapped_column(String, nullable=False) # 'audio_auto' | 'ai' | 'user'
|
||||
|
||||
|
||||
class Annotation(Base):
|
||||
__tablename__ = "annotations"
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
video_id: Mapped[int] = mapped_column(ForeignKey("videos.id"), nullable=False, index=True)
|
||||
t_video_s: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
x0: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
y0: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
x1: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
y1: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
resolved_anchor_id: Mapped[int | None] = mapped_column(
|
||||
ForeignKey("anchors.id"), nullable=True
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Engine / session management
|
||||
# ---------------------------------------------------------------------------
|
||||
_engine: Engine | None = None
|
||||
_SessionLocal: sessionmaker[Session] | None = None
|
||||
_bound_path: Path | None = None
|
||||
|
||||
|
||||
def init_engine(db_path: str | Path | None = None) -> Engine:
|
||||
"""(Re)bind the module to a SQLite database file and return the engine.
|
||||
|
||||
Called with no argument uses ``config.DB_PATH``. Idempotent for the same path.
|
||||
"""
|
||||
global _engine, _SessionLocal, _bound_path
|
||||
path = Path(db_path) if db_path is not None else config.DB_PATH
|
||||
if _engine is not None and _bound_path == path:
|
||||
return _engine
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
if _engine is not None:
|
||||
_engine.dispose()
|
||||
_engine = create_engine(
|
||||
f"sqlite:///{path}",
|
||||
connect_args={"check_same_thread": False},
|
||||
future=True,
|
||||
)
|
||||
_SessionLocal = sessionmaker(bind=_engine, expire_on_commit=False, future=True)
|
||||
_bound_path = path
|
||||
return _engine
|
||||
|
||||
|
||||
def engine() -> Engine:
|
||||
if _engine is None:
|
||||
init_engine()
|
||||
assert _engine is not None
|
||||
return _engine
|
||||
|
||||
|
||||
@contextmanager
|
||||
def session_scope() -> Iterator[Session]:
|
||||
"""Transactional session context; commits on success, rolls back on error."""
|
||||
if _SessionLocal is None:
|
||||
init_engine()
|
||||
assert _SessionLocal is not None
|
||||
session = _SessionLocal()
|
||||
try:
|
||||
yield session
|
||||
session.commit()
|
||||
except Exception:
|
||||
session.rollback()
|
||||
raise
|
||||
finally:
|
||||
session.close()
|
||||
|
||||
|
||||
def init_db() -> None:
|
||||
"""Create all tables if they do not exist."""
|
||||
Base.metadata.create_all(engine())
|
||||
|
||||
|
||||
def reset_db() -> None:
|
||||
"""Drop and recreate all tables (used by the synthetic fixture generator)."""
|
||||
Base.metadata.drop_all(engine())
|
||||
Base.metadata.create_all(engine())
|
||||
|
||||
|
||||
def _now_iso() -> str:
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Videos
|
||||
# ---------------------------------------------------------------------------
|
||||
def add_video(
|
||||
filename: str,
|
||||
duration_s: float,
|
||||
fps: float,
|
||||
width: int,
|
||||
height: int,
|
||||
offset_ms: float | None = None,
|
||||
drift_ppm: float | None = None,
|
||||
sync_confidence: float | None = None,
|
||||
) -> Video:
|
||||
"""Insert a video row and return it (used by ingest + synthetic)."""
|
||||
with session_scope() as s:
|
||||
video = Video(
|
||||
filename=filename,
|
||||
duration_s=duration_s,
|
||||
fps=fps,
|
||||
width=width,
|
||||
height=height,
|
||||
offset_ms=offset_ms,
|
||||
drift_ppm=drift_ppm,
|
||||
sync_confidence=sync_confidence,
|
||||
created_at=_now_iso(),
|
||||
)
|
||||
s.add(video)
|
||||
s.flush()
|
||||
s.refresh(video)
|
||||
return video
|
||||
|
||||
|
||||
def get_videos() -> list[Video]:
|
||||
with session_scope() as s:
|
||||
return list(s.scalars(select(Video).order_by(Video.id)))
|
||||
|
||||
|
||||
def get_video(video_id: int) -> Video | None:
|
||||
with session_scope() as s:
|
||||
return s.get(Video, video_id)
|
||||
|
||||
|
||||
def update_video_sync(
|
||||
video_id: int,
|
||||
offset_ms: float | None,
|
||||
drift_ppm: float | None,
|
||||
sync_confidence: float | None,
|
||||
) -> None:
|
||||
"""Persist the audio-sync solution for a video (lane A / M1)."""
|
||||
with session_scope() as s:
|
||||
video = s.get(Video, video_id)
|
||||
if video is None:
|
||||
raise KeyError(f"no video with id={video_id}")
|
||||
video.offset_ms = offset_ms
|
||||
video.drift_ppm = drift_ppm
|
||||
video.sync_confidence = sync_confidence
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Camera poses
|
||||
# ---------------------------------------------------------------------------
|
||||
def set_poses(video_id: int, poses: Iterable[dict]) -> int:
|
||||
"""Replace all poses for a video with ``poses`` (lane B / M2, and synthetic).
|
||||
|
||||
Each dict must have keys: frame_idx, t_video_s, qw, qx, qy, qz, tx, ty, tz,
|
||||
fx, fy, cx, cy, and optionally ``registered`` (default True). Returns the count.
|
||||
|
||||
This is atomic: existing poses are only deleted inside the same transaction that
|
||||
inserts the new ones, so a caller that raises before returning never corrupts the DB.
|
||||
"""
|
||||
rows = list(poses)
|
||||
with session_scope() as s:
|
||||
s.execute(delete(CameraPose).where(CameraPose.video_id == video_id))
|
||||
for p in rows:
|
||||
s.add(
|
||||
CameraPose(
|
||||
video_id=video_id,
|
||||
frame_idx=int(p["frame_idx"]),
|
||||
t_video_s=float(p["t_video_s"]),
|
||||
qw=float(p["qw"]), qx=float(p["qx"]), qy=float(p["qy"]), qz=float(p["qz"]),
|
||||
tx=float(p["tx"]), ty=float(p["ty"]), tz=float(p["tz"]),
|
||||
fx=float(p["fx"]), fy=float(p["fy"]), cx=float(p["cx"]), cy=float(p["cy"]),
|
||||
registered=bool(p.get("registered", True)),
|
||||
)
|
||||
)
|
||||
return len(rows)
|
||||
|
||||
|
||||
def get_poses(video_id: int) -> list[CameraPose]:
|
||||
with session_scope() as s:
|
||||
return list(
|
||||
s.scalars(
|
||||
select(CameraPose)
|
||||
.where(CameraPose.video_id == video_id)
|
||||
.order_by(CameraPose.t_video_s)
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def has_poses() -> bool:
|
||||
with session_scope() as s:
|
||||
return s.scalar(select(CameraPose.id).limit(1)) is not None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Anchors
|
||||
# ---------------------------------------------------------------------------
|
||||
def add_anchor(label: str, x: float, y: float, z: float, color: str | None = None) -> Anchor:
|
||||
with session_scope() as s:
|
||||
anchor = Anchor(label=label, x=x, y=y, z=z, color=color)
|
||||
s.add(anchor)
|
||||
s.flush()
|
||||
s.refresh(anchor)
|
||||
return anchor
|
||||
|
||||
|
||||
def get_anchors() -> list[Anchor]:
|
||||
with session_scope() as s:
|
||||
return list(s.scalars(select(Anchor).order_by(Anchor.id)))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Events
|
||||
# ---------------------------------------------------------------------------
|
||||
def add_event(
|
||||
t_global_s: float,
|
||||
event_type: str,
|
||||
source: str,
|
||||
duration_s: float | None = None,
|
||||
confidence: float | None = None,
|
||||
description: str | None = None,
|
||||
) -> Event:
|
||||
with session_scope() as s:
|
||||
event = Event(
|
||||
t_global_s=t_global_s,
|
||||
duration_s=duration_s,
|
||||
event_type=event_type,
|
||||
confidence=confidence,
|
||||
description=description,
|
||||
source=source,
|
||||
)
|
||||
s.add(event)
|
||||
s.flush()
|
||||
s.refresh(event)
|
||||
return event
|
||||
|
||||
|
||||
def get_events() -> list[Event]:
|
||||
with session_scope() as s:
|
||||
return list(s.scalars(select(Event).order_by(Event.t_global_s)))
|
||||
|
||||
|
||||
def clear_events(source: str | None = None) -> int:
|
||||
"""Delete events, optionally only those from a given ``source``. Returns count deleted."""
|
||||
with session_scope() as s:
|
||||
stmt = delete(Event)
|
||||
if source is not None:
|
||||
stmt = stmt.where(Event.source == source)
|
||||
result = s.execute(stmt)
|
||||
return int(result.rowcount or 0)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Annotations
|
||||
# ---------------------------------------------------------------------------
|
||||
def add_annotation(
|
||||
video_id: int,
|
||||
t_video_s: float,
|
||||
x0: float,
|
||||
y0: float,
|
||||
x1: float,
|
||||
y1: float,
|
||||
resolved_anchor_id: int | None = None,
|
||||
) -> Annotation:
|
||||
with session_scope() as s:
|
||||
annotation = Annotation(
|
||||
video_id=video_id,
|
||||
t_video_s=t_video_s,
|
||||
x0=x0, y0=y0, x1=x1, y1=y1,
|
||||
resolved_anchor_id=resolved_anchor_id,
|
||||
)
|
||||
s.add(annotation)
|
||||
s.flush()
|
||||
s.refresh(annotation)
|
||||
return annotation
|
||||
|
||||
|
||||
def get_annotations(video_id: int | None = None) -> list[Annotation]:
|
||||
with session_scope() as s:
|
||||
stmt = select(Annotation).order_by(Annotation.id)
|
||||
if video_id is not None:
|
||||
stmt = stmt.where(Annotation.video_id == video_id)
|
||||
return list(s.scalars(stmt))
|
||||
|
||||
|
||||
def set_annotation_anchor(annotation_id: int, anchor_id: int) -> None:
|
||||
with session_scope() as s:
|
||||
annotation = s.get(Annotation, annotation_id)
|
||||
if annotation is None:
|
||||
raise KeyError(f"no annotation with id={annotation_id}")
|
||||
annotation.resolved_anchor_id = anchor_id
|
||||
144
backend/festival4d/events_ai.py
Normal file
144
backend/festival4d/events_ai.py
Normal file
@ -0,0 +1,144 @@
|
||||
"""Moment detection + pluggable AI classification (spec M7, lane D).
|
||||
|
||||
Two stages: cheap audio candidates first, then a vision LLM on candidates only.
|
||||
|
||||
This module defines the **FROZEN classifier contract** — :class:`MomentClassification`
|
||||
(Pydantic model) and :class:`MomentClassifier` (Protocol) — exactly as spec M7. Lanes and
|
||||
integration depend on these shapes. Everything else here is a STUB that lane D fills:
|
||||
candidate detection, the provider classes, provider selection, and the ``run_events``
|
||||
entrypoint (called by both ``cli.py`` and ``api.py``'s ``POST /api/events/detect``).
|
||||
|
||||
Provider selection (lane D): ``FESTIVAL4D_CLASSIFIER=gemini|claude|local`` (default
|
||||
``gemini``). Inputs prepared once per candidate: a ~3 s MP4 clip around ``t_global`` from
|
||||
the best-covering video and 6 sampled JPEGs for providers without video input. Common
|
||||
behavior: catch exceptions per candidate (one failure never aborts the batch); if the
|
||||
selected provider is unconfigured, run candidate detection only (``event_type='candidate'``)
|
||||
and log that classification was skipped.
|
||||
|
||||
The module imports with only ``pydantic`` present — provider SDKs (``google-genai``,
|
||||
``anthropic``, ``openai``) are imported lazily inside the provider bodies, never at load.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Literal, Protocol, runtime_checkable
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
log = logging.getLogger("festival4d.events_ai")
|
||||
|
||||
# The closed set of moment classes (spec M7). Lane C colors timeline markers by these.
|
||||
EventType = Literal[
|
||||
"bass_drop", "pyro", "confetti", "crowd_wave",
|
||||
"artist_moment", "light_show", "quiet_moment", "other",
|
||||
]
|
||||
EVENT_TYPES: tuple[str, ...] = (
|
||||
"bass_drop", "pyro", "confetti", "crowd_wave",
|
||||
"artist_moment", "light_show", "quiet_moment", "other",
|
||||
)
|
||||
|
||||
|
||||
class MomentClassification(BaseModel):
|
||||
"""Structured classifier output (FROZEN CONTRACT, spec M7)."""
|
||||
|
||||
event_type: EventType
|
||||
confidence: float = Field(ge=0.0, le=1.0) # 0..1
|
||||
description: str # one sentence, human-readable
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class MomentClassifier(Protocol):
|
||||
"""Interchangeable classifier provider (FROZEN CONTRACT, spec M7)."""
|
||||
|
||||
def classify(self, clip_path: Path, frames: list[Path]) -> MomentClassification:
|
||||
"""Classify one candidate moment from a short clip + sampled frames."""
|
||||
...
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Provider stubs (lane D fills bodies; classifiers import their SDK lazily).
|
||||
# ---------------------------------------------------------------------------
|
||||
class GeminiClassifier:
|
||||
"""Default provider — Gemini 2.5 Flash, native video input (``GEMINI_API_KEY``)."""
|
||||
|
||||
model = "gemini-2.5-flash"
|
||||
|
||||
def __init__(self) -> None:
|
||||
raise NotImplementedError("lane D (M7): implement GeminiClassifier")
|
||||
|
||||
def classify(self, clip_path: Path, frames: list[Path]) -> MomentClassification:
|
||||
raise NotImplementedError("lane D (M7): implement GeminiClassifier.classify")
|
||||
|
||||
|
||||
class ClaudeClassifier:
|
||||
"""Optional provider — Claude Opus 4.8, 6 frames as images (``ANTHROPIC_API_KEY``).
|
||||
|
||||
Uses ``client.messages.parse(..., thinking={'type': 'adaptive'},
|
||||
output_format=MomentClassification)``; do **not** pass ``temperature`` (rejected on
|
||||
Opus 4.8). Model id ``claude-opus-4-8`` (never date-suffixed).
|
||||
"""
|
||||
|
||||
model = "claude-opus-4-8"
|
||||
|
||||
def __init__(self) -> None:
|
||||
raise NotImplementedError("lane D (M7): implement ClaudeClassifier")
|
||||
|
||||
def classify(self, clip_path: Path, frames: list[Path]) -> MomentClassification:
|
||||
raise NotImplementedError("lane D (M7): implement ClaudeClassifier.classify")
|
||||
|
||||
|
||||
class LocalClassifier:
|
||||
"""Offline/free provider — any OpenAI-compatible vision endpoint (Ollama/LM Studio/
|
||||
OpenRouter). Config: ``FESTIVAL4D_OPENAI_BASE_URL``, ``FESTIVAL4D_OPENAI_KEY``,
|
||||
``FESTIVAL4D_OPENAI_MODEL``. Sends the 6 frames as image parts; validates JSON with one
|
||||
retry on failure.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
raise NotImplementedError("lane D (M7): implement LocalClassifier")
|
||||
|
||||
def classify(self, clip_path: Path, frames: list[Path]) -> MomentClassification:
|
||||
raise NotImplementedError("lane D (M7): implement LocalClassifier.classify")
|
||||
|
||||
|
||||
def get_classifier(name: str | None = None) -> MomentClassifier | None:
|
||||
"""Select a provider by name or ``FESTIVAL4D_CLASSIFIER`` (default ``gemini``).
|
||||
|
||||
Returns a ready classifier, or ``None`` if the selected provider is unconfigured (no
|
||||
key/endpoint) — callers then run candidates-only. STUB — lane D implements selection.
|
||||
"""
|
||||
_ = name or os.environ.get("FESTIVAL4D_CLASSIFIER", "gemini")
|
||||
raise NotImplementedError("lane D (M7): implement get_classifier")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Candidate detection + orchestration (lane D fills bodies).
|
||||
# ---------------------------------------------------------------------------
|
||||
def detect_candidates(audio: "object", sr: int) -> list[float]:
|
||||
"""Audio candidate moments on the reference track (spec M7, lane D).
|
||||
|
||||
Compute RMS energy + spectral-flux onset strength (librosa); return the times
|
||||
(reference-local seconds) of peaks that are local maxima over a 10 s neighborhood and
|
||||
above the 90th percentile. Verified against the fixture's ground-truth pulse times
|
||||
(±0.5 s).
|
||||
"""
|
||||
raise NotImplementedError("lane D (M7): implement detect_candidates")
|
||||
|
||||
|
||||
def prepare_inputs(t_global_s: float) -> tuple[Path, list[Path]]:
|
||||
"""Prepare classifier inputs for a candidate: a ~3 s MP4 clip + 6 sampled JPEGs (M7)."""
|
||||
raise NotImplementedError("lane D (M7): implement prepare_inputs")
|
||||
|
||||
|
||||
def run_events(t_global_s: float | None = None, window_s: float | None = None) -> dict:
|
||||
"""Detect audio candidates and (if a provider is configured) classify them.
|
||||
|
||||
Entrypoint for ``python -m festival4d events`` and ``POST /api/events/detect``. Inserts
|
||||
events via ``db`` (``source='audio_auto'``; classified ones updated to ``source='ai'``).
|
||||
Optionally restrict to a single window around ``t_global_s`` of width ``window_s``.
|
||||
Returns a summary dict. Per-candidate exceptions are isolated (spec M7).
|
||||
"""
|
||||
raise NotImplementedError("lane D (M7): implement run_events")
|
||||
28
backend/festival4d/frames.py
Normal file
28
backend/festival4d/frames.py
Normal file
@ -0,0 +1,28 @@
|
||||
"""Sharpness-aware frame sampling for SfM (spec M2, lane B).
|
||||
|
||||
STUB — lane B fills the bodies; the public signatures below are frozen (``sfm.py`` calls
|
||||
these). Sample candidate frames ~2/s per video; within each 0.5 s window keep the sharpest
|
||||
frame (variance of Laplacian via OpenCV). Write JPEGs named ``{video_id}_{frame_idx}.jpg``
|
||||
into ``config.FRAMES_DIR`` (one subfolder per video for COLMAP's single-camera-per-folder).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
log = logging.getLogger("festival4d.frames")
|
||||
|
||||
|
||||
def sharpness(image) -> float:
|
||||
"""Variance of the Laplacian of a grayscale image (OpenCV) — higher is sharper."""
|
||||
raise NotImplementedError("lane B (M2): implement sharpness")
|
||||
|
||||
|
||||
def sample_frames(video_path: Path, video_id: int, out_dir: Path,
|
||||
target_fps: float = 2.0, window_s: float = 0.5) -> list[Path]:
|
||||
"""Sample the sharpest frame per ``window_s`` window from ``video_path`` (spec M2).
|
||||
|
||||
Returns the written JPEG paths (named ``{video_id}_{frame_idx}.jpg``).
|
||||
"""
|
||||
raise NotImplementedError("lane B (M2): implement sample_frames")
|
||||
217
backend/festival4d/geometry.py
Normal file
217
backend/festival4d/geometry.py
Normal file
@ -0,0 +1,217 @@
|
||||
"""Coordinate conversions, quaternion math, ray casting, triangulation.
|
||||
|
||||
The single most important thing in this file is :func:`colmap_to_threejs` — the
|
||||
COLMAP world->camera pose to Three.js camera conversion (spec M5). It is a **FROZEN
|
||||
CONTRACT**: it ships in foundation with a passing unit test (``test_geometry.py``), and
|
||||
the JS mirror lives in ``frontend/src/lib/pose.js`` with the *same* embedded test
|
||||
vectors. Lanes B and C **consume** it; they never reimplement or modify it.
|
||||
|
||||
The remaining functions (:func:`slerp_pose`, :func:`ray_from_pixel`,
|
||||
:func:`triangulate_rays`, :func:`nearest_point_on_ray`) are stubs with frozen
|
||||
signatures for lane B (spec M2 + M8). Lane B fills the bodies and adds their tests.
|
||||
|
||||
Conventions
|
||||
-----------
|
||||
- Quaternions are COLMAP order **[w, x, y, z]** (scalar first), unit norm.
|
||||
- A pose ``(q, t)`` is world->camera: ``x_cam = R(q) @ x_world + t``.
|
||||
- COLMAP camera axes: +x right, +y down, +z forward (into the scene).
|
||||
- Three.js cameras look down -z with +y up; hence the ``diag(1, -1, -1)`` flip.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
from numpy.typing import ArrayLike, NDArray
|
||||
|
||||
# Camera-axis flip that takes COLMAP camera-local axes (x right, y down, z forward)
|
||||
# to Three.js camera-local axes (x right, y up, z backward). Part of the frozen contract.
|
||||
_FLIP_YZ = np.diag([1.0, -1.0, -1.0])
|
||||
|
||||
|
||||
def quat_to_mat(q: ArrayLike) -> NDArray[np.float64]:
|
||||
"""Convert a unit quaternion ``[w, x, y, z]`` to a 3x3 rotation matrix.
|
||||
|
||||
Hamilton convention, right-handed, active rotation: the returned ``R`` is COLMAP's
|
||||
world->camera matrix when ``q`` is a COLMAP pose quaternion. FROZEN (used by the
|
||||
conversion contract).
|
||||
"""
|
||||
w, x, y, z = (float(v) for v in np.asarray(q, dtype=np.float64).reshape(4))
|
||||
n = w * w + x * x + y * y + z * z
|
||||
if n < 1e-12:
|
||||
raise ValueError("quaternion has near-zero norm")
|
||||
s = 2.0 / n
|
||||
wx, wy, wz = s * w * x, s * w * y, s * w * z
|
||||
xx, xy, xz = s * x * x, s * x * y, s * x * z
|
||||
yy, yz, zz = s * y * y, s * y * z, s * z * z
|
||||
return np.array(
|
||||
[
|
||||
[1.0 - (yy + zz), xy - wz, xz + wy],
|
||||
[xy + wz, 1.0 - (xx + zz), yz - wx],
|
||||
[xz - wy, yz + wx, 1.0 - (xx + yy)],
|
||||
],
|
||||
dtype=np.float64,
|
||||
)
|
||||
|
||||
|
||||
def mat_to_quat(R: ArrayLike) -> NDArray[np.float64]:
|
||||
"""Convert a 3x3 rotation matrix to a unit quaternion ``[w, x, y, z]`` (w >= 0).
|
||||
|
||||
Inverse of :func:`quat_to_mat`. Used by tests and the pose export path.
|
||||
"""
|
||||
m = np.asarray(R, dtype=np.float64).reshape(3, 3)
|
||||
trace = m[0, 0] + m[1, 1] + m[2, 2]
|
||||
if trace > 0.0:
|
||||
s = np.sqrt(trace + 1.0) * 2.0
|
||||
w = 0.25 * s
|
||||
x = (m[2, 1] - m[1, 2]) / s
|
||||
y = (m[0, 2] - m[2, 0]) / s
|
||||
z = (m[1, 0] - m[0, 1]) / s
|
||||
elif m[0, 0] > m[1, 1] and m[0, 0] > m[2, 2]:
|
||||
s = np.sqrt(1.0 + m[0, 0] - m[1, 1] - m[2, 2]) * 2.0
|
||||
w = (m[2, 1] - m[1, 2]) / s
|
||||
x = 0.25 * s
|
||||
y = (m[0, 1] + m[1, 0]) / s
|
||||
z = (m[0, 2] + m[2, 0]) / s
|
||||
elif m[1, 1] > m[2, 2]:
|
||||
s = np.sqrt(1.0 + m[1, 1] - m[0, 0] - m[2, 2]) * 2.0
|
||||
w = (m[0, 2] - m[2, 0]) / s
|
||||
x = (m[0, 1] + m[1, 0]) / s
|
||||
y = 0.25 * s
|
||||
z = (m[1, 2] + m[2, 1]) / s
|
||||
else:
|
||||
s = np.sqrt(1.0 + m[2, 2] - m[0, 0] - m[1, 1]) * 2.0
|
||||
w = (m[1, 0] - m[0, 1]) / s
|
||||
x = (m[0, 2] + m[2, 0]) / s
|
||||
y = (m[1, 2] + m[2, 1]) / s
|
||||
z = 0.25 * s
|
||||
q = np.array([w, x, y, z], dtype=np.float64)
|
||||
q /= np.linalg.norm(q)
|
||||
if q[0] < 0: # canonical sign: non-negative scalar part
|
||||
q = -q
|
||||
return q
|
||||
|
||||
|
||||
def colmap_to_threejs(q: ArrayLike, t: ArrayLike) -> tuple[NDArray[np.float64], NDArray[np.float64]]:
|
||||
"""Convert a COLMAP world->camera pose to a Three.js camera pose (spec M5).
|
||||
|
||||
**FROZEN CONTRACT.** Mirrored in ``frontend/src/lib/pose.js``; do not change the math
|
||||
without a change request and a synchronized update to both sides + their test vectors.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
q : array_like, shape (4,)
|
||||
COLMAP world->camera quaternion ``[w, x, y, z]``.
|
||||
t : array_like, shape (3,)
|
||||
COLMAP world->camera translation ``[tx, ty, tz]``.
|
||||
|
||||
Returns
|
||||
-------
|
||||
position : ndarray, shape (3,)
|
||||
Camera center in world coordinates, ``C = -R^T t``. Assign to ``camera.position``.
|
||||
rotation_matrix : ndarray, shape (3, 3)
|
||||
Three.js camera world rotation ``R_three = R^T @ diag(1, -1, -1)``. Assign via
|
||||
``camera.setRotationFromMatrix(...)`` (a proper rotation, det = +1).
|
||||
"""
|
||||
R = quat_to_mat(q) # world -> cam
|
||||
t_vec = np.asarray(t, dtype=np.float64).reshape(3)
|
||||
R_c2w = R.T # cam -> world
|
||||
position = -R_c2w @ t_vec # camera center in world coords
|
||||
rotation_matrix = R_c2w @ _FLIP_YZ # flip camera-local y,z for Three.js
|
||||
return position, rotation_matrix
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Lane B stubs (spec M2 + M8). Signatures FROZEN; bodies raise NotImplementedError.
|
||||
# ---------------------------------------------------------------------------
|
||||
def slerp_pose(
|
||||
q0: ArrayLike,
|
||||
t0: ArrayLike,
|
||||
q1: ArrayLike,
|
||||
t1: ArrayLike,
|
||||
alpha: float,
|
||||
) -> tuple[NDArray[np.float64], NDArray[np.float64]]:
|
||||
"""Interpolate between two world->camera poses (spec M2 pose interpolation).
|
||||
|
||||
Spherically interpolate the rotation (slerp on the shorter arc, handling the
|
||||
double-cover sign) and linearly interpolate the translation, at fraction
|
||||
``alpha in [0, 1]`` from pose 0 to pose 1.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
q0, q1 : array_like, shape (4,)
|
||||
COLMAP quaternions ``[w, x, y, z]`` at the endpoints.
|
||||
t0, t1 : array_like, shape (3,)
|
||||
COLMAP translations at the endpoints.
|
||||
alpha : float
|
||||
Interpolation fraction; 0 returns pose 0, 1 returns pose 1.
|
||||
|
||||
Returns
|
||||
-------
|
||||
(q, t) : the interpolated quaternion ``[w, x, y, z]`` and translation ``[x, y, z]``.
|
||||
"""
|
||||
raise NotImplementedError("lane B (M2): implement slerp_pose")
|
||||
|
||||
|
||||
def ray_from_pixel(
|
||||
q: ArrayLike,
|
||||
t: ArrayLike,
|
||||
fx: float,
|
||||
fy: float,
|
||||
cx: float,
|
||||
cy: float,
|
||||
px: float,
|
||||
py: float,
|
||||
) -> tuple[NDArray[np.float64], NDArray[np.float64]]:
|
||||
"""Unproject a pixel into a world-space ray (spec M8 annotation resolution).
|
||||
|
||||
Given the world->camera pose ``(q, t)`` and pinhole intrinsics, build the ray that
|
||||
passes through image pixel ``(px, py)``.
|
||||
|
||||
Returns
|
||||
-------
|
||||
origin : ndarray, shape (3,)
|
||||
Ray origin = camera center in world coords.
|
||||
direction : ndarray, shape (3,)
|
||||
Unit ray direction in world coords, pointing into the scene.
|
||||
"""
|
||||
raise NotImplementedError("lane B (M8): implement ray_from_pixel")
|
||||
|
||||
|
||||
def triangulate_rays(
|
||||
origin_a: ArrayLike,
|
||||
dir_a: ArrayLike,
|
||||
origin_b: ArrayLike,
|
||||
dir_b: ArrayLike,
|
||||
) -> tuple[NDArray[np.float64], float]:
|
||||
"""Closest point between two world-space rays (spec M8 two-view triangulation).
|
||||
|
||||
Returns
|
||||
-------
|
||||
point : ndarray, shape (3,)
|
||||
Midpoint of the shortest segment connecting the two rays.
|
||||
gap : float
|
||||
Length of that shortest segment (the mutual-nearest-approach distance). Callers
|
||||
reject the triangulation when the rays are near-parallel or ``gap`` exceeds the
|
||||
spec threshold (0.5 scene units).
|
||||
"""
|
||||
raise NotImplementedError("lane B (M8): implement triangulate_rays")
|
||||
|
||||
|
||||
def nearest_point_on_ray(
|
||||
origin: ArrayLike,
|
||||
direction: ArrayLike,
|
||||
points: ArrayLike,
|
||||
radius: float = 0.3,
|
||||
) -> NDArray[np.float64] | None:
|
||||
"""Nearest point-cloud point to a ray, within a cylinder (spec M8 single-view fallback).
|
||||
|
||||
Among ``points`` (shape ``(N, 3)``) find the one whose perpendicular distance to the
|
||||
ray is smallest, considering only points within ``radius`` of the ray and in front of
|
||||
the origin.
|
||||
|
||||
Returns
|
||||
-------
|
||||
point : ndarray shape (3,) or None
|
||||
The selected point-cloud point, or ``None`` if none lie within ``radius``.
|
||||
"""
|
||||
raise NotImplementedError("lane B (M8): implement nearest_point_on_ray")
|
||||
31
backend/festival4d/ingest.py
Normal file
31
backend/festival4d/ingest.py
Normal file
@ -0,0 +1,31 @@
|
||||
"""Video ingest: probe files in ``data/raw`` and extract audio (spec M1, lane A).
|
||||
|
||||
STUB — lane A fills the bodies; the public signatures below are frozen (``cli.py`` calls
|
||||
``run_ingest``). For each file in ``config.RAW_DIR``: ffprobe it, insert/update the
|
||||
``videos`` row via ``db`` helpers, and extract a mono 16 kHz WAV into ``config.AUDIO_DIR``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
log = logging.getLogger("festival4d.ingest")
|
||||
|
||||
|
||||
def probe_video(path: Path) -> dict:
|
||||
"""ffprobe a video; return ``{duration_s, fps, width, height}`` (lane A / M1)."""
|
||||
raise NotImplementedError("lane A (M1): implement probe_video")
|
||||
|
||||
|
||||
def extract_audio(path: Path, out_wav: Path, sample_rate: int = 16_000) -> Path:
|
||||
"""Extract a mono ``sample_rate`` WAV from ``path`` via ffmpeg (lane A / M1)."""
|
||||
raise NotImplementedError("lane A (M1): implement extract_audio")
|
||||
|
||||
|
||||
def run_ingest() -> list[dict]:
|
||||
"""Ingest every video in ``config.RAW_DIR``: probe, register in DB, extract audio.
|
||||
|
||||
Entrypoint for ``python -m festival4d ingest``. Returns a per-video summary list.
|
||||
"""
|
||||
raise NotImplementedError("lane A (M1): implement run_ingest")
|
||||
71
backend/festival4d/sfm.py
Normal file
71
backend/festival4d/sfm.py
Normal file
@ -0,0 +1,71 @@
|
||||
"""COLMAP orchestration, model parsing, scene normalization, pose export (spec M2, lane B).
|
||||
|
||||
STUB — lane B fills the bodies; the public signatures below are frozen (``cli.py`` calls
|
||||
``run_reconstruct``).
|
||||
|
||||
Pipeline (spec M2, for lane B's reference):
|
||||
- Sample frames (``frames.sample_frames``), then drive the COLMAP CLI via subprocess:
|
||||
``feature_extractor`` (OPENCV model, single camera per folder) -> ``sequential_matcher``
|
||||
within each video + ``exhaustive_matcher`` across -> ``mapper`` -> ``model_converter`` to
|
||||
a TXT model. Parse ``images.txt`` / ``cameras.txt`` / ``points3D.txt`` by hand (no
|
||||
pycolmap).
|
||||
- **Normalize** the scene: centroid -> origin, camera bounding sphere radius -> 10, world-up
|
||||
-> average camera up. Apply the same similarity transform to poses and points.
|
||||
- **Interpolate** poses for unregistered frames (slerp rotation, lerp translation, mark
|
||||
``registered=False``); never extrapolate past the first/last registered frame.
|
||||
- Export ``config.POINTS_PLY`` (binary little-endian, matching ``synthetic.write_ply``'s
|
||||
format) and fill ``camera_poses`` via ``db.set_poses``.
|
||||
- **Failure handling** (spec M2): COLMAP optional at runtime (``shutil.which("colmap")``);
|
||||
if it registers < 60% of frames or < 2 videos into one model, print a clear diagnostic and
|
||||
leave existing poses untouched — never corrupt the DB. The app degrades to "synced videos,
|
||||
no 3D".
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
log = logging.getLogger("festival4d.sfm")
|
||||
|
||||
|
||||
def colmap_available() -> bool:
|
||||
"""Whether the COLMAP binary is on PATH (spec M2 optional-at-runtime rule)."""
|
||||
import shutil
|
||||
|
||||
return shutil.which("colmap") is not None
|
||||
|
||||
|
||||
def parse_images_txt(path: Path) -> list[dict]:
|
||||
"""Parse a COLMAP ``images.txt`` into per-image pose records (spec M2, lane B)."""
|
||||
raise NotImplementedError("lane B (M2): implement parse_images_txt")
|
||||
|
||||
|
||||
def parse_cameras_txt(path: Path) -> dict[int, dict]:
|
||||
"""Parse a COLMAP ``cameras.txt`` into ``camera_id -> intrinsics`` (spec M2, lane B)."""
|
||||
raise NotImplementedError("lane B (M2): implement parse_cameras_txt")
|
||||
|
||||
|
||||
def parse_points3d_txt(path: Path):
|
||||
"""Parse a COLMAP ``points3D.txt`` into ``(points Nx3, colors Nx3)`` (spec M2, lane B)."""
|
||||
raise NotImplementedError("lane B (M2): implement parse_points3d_txt")
|
||||
|
||||
|
||||
def normalize_scene(points, poses):
|
||||
"""Similarity transform: centroid->origin, camera sphere radius->10, up->+Y (spec M2)."""
|
||||
raise NotImplementedError("lane B (M2): implement normalize_scene")
|
||||
|
||||
|
||||
def run_colmap(frames_dir: Path, workspace: Path) -> Path:
|
||||
"""Drive the COLMAP CLI and return the exported TXT-model directory (spec M2, lane B)."""
|
||||
raise NotImplementedError("lane B (M2): implement run_colmap")
|
||||
|
||||
|
||||
def run_reconstruct() -> dict:
|
||||
"""Full reconstruction: sample frames, run COLMAP, normalize, interpolate, export.
|
||||
|
||||
Entrypoint for ``python -m festival4d reconstruct``. Degrades gracefully when COLMAP is
|
||||
absent or the reconstruction is too weak (spec M2 failure handling): existing poses are
|
||||
left untouched. Returns a summary dict.
|
||||
"""
|
||||
raise NotImplementedError("lane B (M2): implement run_reconstruct")
|
||||
556
backend/festival4d/synthetic.py
Normal file
556
backend/festival4d/synthetic.py
Normal file
@ -0,0 +1,556 @@
|
||||
"""Synthetic fixture generator (spec M0) — the universal test bed.
|
||||
|
||||
``python -m festival4d synthetic`` builds a complete fake project so every later layer is
|
||||
testable without real footage:
|
||||
|
||||
- **3 fake videos** (``cam0/1/2.mp4``, 640x360, 30 fps, 20 s), each a distinct ``testsrc2``
|
||||
visual but sharing the *same* synthesized music-like audio, shifted by known offsets
|
||||
(0, +1370, -842 ms). ``cam0`` is the reference (offset 0).
|
||||
- **Fake poses**: 3 camera trajectories on arcs around a "stage" box at the origin,
|
||||
looking at the stage, written into ``camera_poses`` (COLMAP world->camera convention).
|
||||
- **Point cloud**: random points on the stage box + ground plane, written to
|
||||
``points.ply`` in the exact binary-little-endian format the real COLMAP path produces.
|
||||
- **Seeded events + anchors** so the timeline and overlays have data immediately.
|
||||
- **``ground_truth.json``**: offsets, drift, audio pulse (bang) times, events, and stage
|
||||
corners — part of the frozen contract (lanes A and D assert against it).
|
||||
|
||||
FROZEN CONTRACT after foundation. The module is factored so the numeric pieces
|
||||
(audio shifts, poses, PLY round-trip) are unit-testable without invoking ffmpeg.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import shutil
|
||||
import struct
|
||||
import subprocess
|
||||
import tempfile
|
||||
import wave
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
||||
from festival4d import config, db
|
||||
from festival4d.geometry import mat_to_quat
|
||||
|
||||
log = logging.getLogger("festival4d.synthetic")
|
||||
|
||||
# Audio timeline: the master signal covers global time [G_START, G_END] so that even the
|
||||
# negatively-offset video (started before global 0) samples a valid window.
|
||||
G_START = -2.0
|
||||
G_END = 24.0
|
||||
BEAT_INTERVAL_S = 0.5 # regular beat clicks
|
||||
BANG_TIMES_S = (3.0, 10.0, 17.0) # loud transients (M7 candidates); well separated
|
||||
TONE_FREQS_HZ = (220.0, 330.0)
|
||||
|
||||
# Camera intrinsics (pinhole, after undistortion) for the synthetic cameras.
|
||||
SYNTH_HFOV_DEG = 60.0
|
||||
|
||||
# Stage geometry (world coords, +y up, +z toward the cameras/audience).
|
||||
STAGE_HALF_W = 3.0 # x extent
|
||||
STAGE_HALF_D = 2.0 # z extent
|
||||
STAGE_HEIGHT = 1.0 # y top
|
||||
STAGE_TARGET = (0.0, 0.5, 0.0) # cameras aim here
|
||||
|
||||
STAGE_CORNERS = {
|
||||
"Stage FL": (-STAGE_HALF_W, STAGE_HEIGHT, STAGE_HALF_D),
|
||||
"Stage FR": (STAGE_HALF_W, STAGE_HEIGHT, STAGE_HALF_D),
|
||||
"Stage BL": (-STAGE_HALF_W, STAGE_HEIGHT, -STAGE_HALF_D),
|
||||
"Stage BR": (STAGE_HALF_W, STAGE_HEIGHT, -STAGE_HALF_D),
|
||||
}
|
||||
|
||||
# Seeded timeline events (pre-classified) so lane C can render markers immediately.
|
||||
SEED_EVENTS = [
|
||||
(3.0, "bass_drop", 0.94, "Bass drops and the crowd erupts as the beat hits."),
|
||||
(6.0, "crowd_wave", 0.72, "A wave ripples through the audience."),
|
||||
(8.0, "quiet_moment", 0.61, "A brief hush before the next build."),
|
||||
(10.0, "pyro", 0.88, "Pyro columns fire across the front of the stage."),
|
||||
(13.0, "light_show", 0.90, "Sweeping beams sync to the breakdown."),
|
||||
(17.0, "confetti", 0.81, "Confetti cannons burst over the crowd."),
|
||||
(19.0, "artist_moment", 0.77, "The artist steps to the edge for a solo."),
|
||||
]
|
||||
|
||||
# Fonts to try for the "CAM N" label (optional nicety; skipped if none exist).
|
||||
_FONT_CANDIDATES = (
|
||||
"/System/Library/Fonts/Supplemental/Arial.ttf",
|
||||
"/System/Library/Fonts/Supplemental/Helvetica.ttc",
|
||||
"/Library/Fonts/Arial.ttf",
|
||||
"/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf",
|
||||
)
|
||||
|
||||
_PLY_DTYPE = np.dtype(
|
||||
[("x", "<f4"), ("y", "<f4"), ("z", "<f4"),
|
||||
("red", "u1"), ("green", "u1"), ("blue", "u1")]
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Audio
|
||||
# ---------------------------------------------------------------------------
|
||||
def synth_master_audio(sr: int = config.SYNTH_AUDIO_SR) -> tuple[np.ndarray, dict]:
|
||||
"""Synthesize the master music-like signal over global time [G_START, G_END].
|
||||
|
||||
Returns the float32 signal (peak-normalized to ~0.95) and a ground-truth dict with the
|
||||
sample rate, the window start, and the beat/bang times (global seconds).
|
||||
"""
|
||||
n = int(round((G_END - G_START) * sr))
|
||||
t = G_START + np.arange(n) / sr
|
||||
sig = np.zeros(n, dtype=np.float64)
|
||||
|
||||
# Low tone bed (kept quiet so it never dominates the transients GCC-PHAT keys on).
|
||||
for f in TONE_FREQS_HZ:
|
||||
sig += 0.05 * np.sin(2.0 * np.pi * f * t)
|
||||
|
||||
# Regular beat clicks: short 2 kHz bursts with a fast exponential decay -> sharp,
|
||||
# broadband-ish transients that give a clean cross-correlation peak.
|
||||
beat_times = np.arange(np.ceil(G_START / BEAT_INTERVAL_S) * BEAT_INTERVAL_S,
|
||||
G_END, BEAT_INTERVAL_S)
|
||||
for bt in beat_times:
|
||||
_add_transient(sig, t, sr, center=bt, amp=0.25, freq=2000.0, decay_s=0.03)
|
||||
|
||||
# Loud "bangs" (bass drops / pyro): louder, longer, low-frequency thump + noise burst.
|
||||
rng = np.random.default_rng(7)
|
||||
for bang in BANG_TIMES_S:
|
||||
_add_transient(sig, t, sr, center=bang, amp=0.9, freq=60.0, decay_s=0.18)
|
||||
_add_noise_burst(sig, t, sr, center=bang, amp=0.5, decay_s=0.12, rng=rng)
|
||||
|
||||
# Faint noise floor for realism (SNR stays high so sync remains exact).
|
||||
sig += 0.004 * rng.standard_normal(n)
|
||||
|
||||
peak = float(np.max(np.abs(sig)))
|
||||
if peak > 0:
|
||||
sig *= 0.95 / peak
|
||||
|
||||
ground_truth = {
|
||||
"sample_rate": sr,
|
||||
"g_start_s": G_START,
|
||||
"g_end_s": G_END,
|
||||
"beat_interval_s": BEAT_INTERVAL_S,
|
||||
"bang_times_global_s": list(BANG_TIMES_S),
|
||||
"pulse_times_global_s": list(BANG_TIMES_S), # alias lane D matches candidates against
|
||||
}
|
||||
return sig.astype(np.float32), ground_truth
|
||||
|
||||
|
||||
def _add_transient(sig, t, sr, center, amp, freq, decay_s):
|
||||
"""Add an exponentially decaying sinusoid burst centered at global time ``center``."""
|
||||
length = int(decay_s * 5 * sr)
|
||||
start = int(round((center - G_START) * sr))
|
||||
i0 = max(0, start)
|
||||
i1 = min(len(sig), start + length)
|
||||
if i1 <= i0:
|
||||
return
|
||||
local = (np.arange(i0, i1) - start) / sr
|
||||
env = np.exp(-local / decay_s)
|
||||
sig[i0:i1] += amp * env * np.sin(2.0 * np.pi * freq * local)
|
||||
|
||||
|
||||
def _add_noise_burst(sig, t, sr, center, amp, decay_s, rng):
|
||||
length = int(decay_s * 5 * sr)
|
||||
start = int(round((center - G_START) * sr))
|
||||
i0 = max(0, start)
|
||||
i1 = min(len(sig), start + length)
|
||||
if i1 <= i0:
|
||||
return
|
||||
local = (np.arange(i0, i1) - start) / sr
|
||||
env = np.exp(-local / decay_s)
|
||||
sig[i0:i1] += amp * env * rng.standard_normal(i1 - i0)
|
||||
|
||||
|
||||
def slice_for_offset(master: np.ndarray, sr: int, offset_ms: float,
|
||||
duration_s: float) -> np.ndarray:
|
||||
"""Extract a video's audio window from the master (spec M0 known-offset shift).
|
||||
|
||||
A video with ``offset_ms`` has local time ``tau`` mapping to global ``offset_ms/1000 +
|
||||
tau`` (drift 0), so its audio is the master window starting at that global time.
|
||||
"""
|
||||
start_global = offset_ms / 1000.0
|
||||
start = int(round((start_global - G_START) * sr))
|
||||
length = int(round(duration_s * sr))
|
||||
if start < 0 or start + length > len(master):
|
||||
raise ValueError(f"offset {offset_ms} ms window falls outside the master signal")
|
||||
return master[start:start + length]
|
||||
|
||||
|
||||
def write_wav(path: Path, signal: np.ndarray, sr: int) -> None:
|
||||
"""Write a mono 16-bit PCM WAV (stdlib only)."""
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
clipped = np.clip(signal, -1.0, 1.0)
|
||||
pcm = (clipped * 32767.0).astype("<i2")
|
||||
with wave.open(str(path), "wb") as w:
|
||||
w.setnchannels(1)
|
||||
w.setsampwidth(2)
|
||||
w.setframerate(sr)
|
||||
w.writeframes(pcm.tobytes())
|
||||
|
||||
|
||||
def read_wav(path: Path) -> tuple[np.ndarray, int]:
|
||||
"""Read a mono 16-bit PCM WAV back to float [-1, 1] (used by tests)."""
|
||||
with wave.open(str(path), "rb") as w:
|
||||
sr = w.getframerate()
|
||||
frames = w.readframes(w.getnframes())
|
||||
pcm = np.frombuffer(frames, dtype="<i2").astype(np.float64) / 32767.0
|
||||
return pcm, sr
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Geometry: poses + point cloud
|
||||
# ---------------------------------------------------------------------------
|
||||
def intrinsics(width: int, height: int) -> dict:
|
||||
fx = (width / 2.0) / np.tan(np.radians(SYNTH_HFOV_DEG) / 2.0)
|
||||
return {"fx": float(fx), "fy": float(fx), "cx": width / 2.0, "cy": height / 2.0}
|
||||
|
||||
|
||||
def look_at_colmap(center, target, world_up=(0.0, 1.0, 0.0)) -> tuple[np.ndarray, np.ndarray]:
|
||||
"""Build a COLMAP world->camera pose (q=[w,x,y,z], t) for a camera at ``center``
|
||||
looking at ``target`` with the given world up. Camera axes: +x right, +y down, +z fwd.
|
||||
"""
|
||||
C = np.asarray(center, dtype=np.float64)
|
||||
T = np.asarray(target, dtype=np.float64)
|
||||
up = np.asarray(world_up, dtype=np.float64)
|
||||
|
||||
z_c = T - C
|
||||
z_c /= np.linalg.norm(z_c)
|
||||
if abs(np.dot(up, z_c)) > 0.999: # looking near-vertical: pick a safe up
|
||||
up = np.array([0.0, 0.0, 1.0])
|
||||
up_c = up - np.dot(up, z_c) * z_c # image up (perp to forward)
|
||||
up_c /= np.linalg.norm(up_c)
|
||||
x_c = np.cross(z_c, up_c) # right
|
||||
y_c = -up_c # down
|
||||
|
||||
R_c2w = np.column_stack([x_c, y_c, z_c]) # camera->world
|
||||
R_w2c = R_c2w.T # world->camera (COLMAP R)
|
||||
t = -R_w2c @ C
|
||||
q = mat_to_quat(R_w2c)
|
||||
return q, t
|
||||
|
||||
|
||||
def camera_track(cam_index: int, duration_s: float, fps: float,
|
||||
width: int, height: int) -> list[dict]:
|
||||
"""One camera's trajectory: a gentle arc in front of the stage, ~2 poses/second."""
|
||||
base_az = np.radians([-35.0, 0.0, 35.0][cam_index])
|
||||
base_radius = 9.0
|
||||
base_height = [1.6, 2.4, 2.0][cam_index]
|
||||
sweep = np.radians(12.0)
|
||||
intr = intrinsics(width, height)
|
||||
|
||||
poses = []
|
||||
step = 0.5 # seconds -> ~2 poses/second
|
||||
n = int(round(duration_s / step)) + 1
|
||||
for k in range(n):
|
||||
t_video = min(k * step, duration_s)
|
||||
phase = 2.0 * np.pi * (t_video / duration_s)
|
||||
az = base_az + sweep * np.sin(phase)
|
||||
radius = base_radius + 0.5 * np.sin(phase + cam_index)
|
||||
height_y = base_height + 0.3 * np.sin(phase * 2.0)
|
||||
center = (radius * np.sin(az), height_y, radius * np.cos(az))
|
||||
q, t = look_at_colmap(center, STAGE_TARGET)
|
||||
poses.append({
|
||||
"frame_idx": int(round(t_video * fps)),
|
||||
"t_video_s": float(t_video),
|
||||
"qw": float(q[0]), "qx": float(q[1]), "qy": float(q[2]), "qz": float(q[3]),
|
||||
"tx": float(t[0]), "ty": float(t[1]), "tz": float(t[2]),
|
||||
**intr,
|
||||
"registered": True,
|
||||
})
|
||||
return poses
|
||||
|
||||
|
||||
def generate_point_cloud(seed: int = 42) -> tuple[np.ndarray, np.ndarray]:
|
||||
"""Random points on the stage box + ground plane (with the named corners included).
|
||||
|
||||
Returns ``(points Nx3 float32, colors Nx3 uint8)``.
|
||||
"""
|
||||
rng = np.random.default_rng(seed)
|
||||
pts: list[np.ndarray] = []
|
||||
cols: list[np.ndarray] = []
|
||||
|
||||
hw, hd, h = STAGE_HALF_W, STAGE_HALF_D, STAGE_HEIGHT
|
||||
stage_color = np.array([210, 90, 60]) # warm stage
|
||||
ground_color = np.array([70, 80, 70]) # dim ground
|
||||
|
||||
# Stage box: sample points on all 6 faces.
|
||||
n_face = 260
|
||||
faces = [
|
||||
# (fixed axis, value, u range, v range) -> build points
|
||||
("x", -hw), ("x", hw), ("z", -hd), ("z", hd), ("y", 0.0), ("y", h),
|
||||
]
|
||||
for axis, val in faces:
|
||||
u = rng.uniform(-hw, hw, n_face)
|
||||
v = rng.uniform(0.0, h, n_face) if axis in ("x", "z") else rng.uniform(-hd, hd, n_face)
|
||||
w = rng.uniform(-hd, hd, n_face)
|
||||
if axis == "x":
|
||||
p = np.column_stack([np.full(n_face, val), v, w])
|
||||
elif axis == "z":
|
||||
p = np.column_stack([u, v, np.full(n_face, val)])
|
||||
else: # y
|
||||
p = np.column_stack([u, np.full(n_face, val), w])
|
||||
pts.append(p)
|
||||
jitter = rng.integers(-15, 16, size=(n_face, 3))
|
||||
cols.append(np.clip(stage_color + jitter, 0, 255))
|
||||
|
||||
# Ground plane: scattered points on y=0 out to a radius, excluding the stage footprint.
|
||||
n_ground = 1600
|
||||
gx = rng.uniform(-12.0, 12.0, n_ground)
|
||||
gz = rng.uniform(-12.0, 12.0, n_ground)
|
||||
on_stage = (np.abs(gx) < hw) & (np.abs(gz) < hd)
|
||||
gx, gz = gx[~on_stage], gz[~on_stage]
|
||||
gp = np.column_stack([gx, np.zeros_like(gx), gz])
|
||||
pts.append(gp)
|
||||
jitter = rng.integers(-12, 13, size=(len(gx), 3))
|
||||
cols.append(np.clip(ground_color + jitter, 0, 255))
|
||||
|
||||
# Include the exact named stage corners so the M8 nearest-point fallback can hit them.
|
||||
corners = np.array(list(STAGE_CORNERS.values()))
|
||||
pts.append(corners)
|
||||
cols.append(np.tile(np.array([255, 230, 120]), (len(corners), 1)))
|
||||
|
||||
points = np.vstack(pts).astype(np.float32)
|
||||
colors = np.vstack(cols).astype(np.uint8)
|
||||
return points, colors
|
||||
|
||||
|
||||
def write_ply(path: Path, points: np.ndarray, colors: np.ndarray) -> None:
|
||||
"""Write a binary-little-endian PLY (x,y,z float + red,green,blue uchar). FROZEN format."""
|
||||
points = np.asarray(points, dtype=np.float32).reshape(-1, 3)
|
||||
colors = np.asarray(colors, dtype=np.uint8).reshape(-1, 3)
|
||||
if len(points) != len(colors):
|
||||
raise ValueError("points and colors length mismatch")
|
||||
n = len(points)
|
||||
arr = np.empty(n, dtype=_PLY_DTYPE)
|
||||
arr["x"], arr["y"], arr["z"] = points[:, 0], points[:, 1], points[:, 2]
|
||||
arr["red"], arr["green"], arr["blue"] = colors[:, 0], colors[:, 1], colors[:, 2]
|
||||
assert arr.dtype.itemsize == 15 # no padding — 3*4 + 3*1
|
||||
|
||||
header = (
|
||||
"ply\n"
|
||||
"format binary_little_endian 1.0\n"
|
||||
"comment Festival 4D point cloud\n"
|
||||
f"element vertex {n}\n"
|
||||
"property float x\n"
|
||||
"property float y\n"
|
||||
"property float z\n"
|
||||
"property uchar red\n"
|
||||
"property uchar green\n"
|
||||
"property uchar blue\n"
|
||||
"end_header\n"
|
||||
)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(path, "wb") as f:
|
||||
f.write(header.encode("ascii"))
|
||||
f.write(arr.tobytes())
|
||||
|
||||
|
||||
def read_ply(path: Path) -> tuple[np.ndarray, np.ndarray]:
|
||||
"""Read a PLY written by :func:`write_ply` (used by tests). Returns (points, colors)."""
|
||||
with open(path, "rb") as f:
|
||||
line = f.readline()
|
||||
if line.strip() != b"ply":
|
||||
raise ValueError("not a PLY file")
|
||||
n = 0
|
||||
while True:
|
||||
line = f.readline()
|
||||
if not line:
|
||||
raise ValueError("unexpected EOF in PLY header")
|
||||
if line.startswith(b"element vertex"):
|
||||
n = int(line.split()[-1])
|
||||
if line.strip() == b"end_header":
|
||||
break
|
||||
arr = np.frombuffer(f.read(n * _PLY_DTYPE.itemsize), dtype=_PLY_DTYPE, count=n)
|
||||
points = np.column_stack([arr["x"], arr["y"], arr["z"]]).astype(np.float32)
|
||||
colors = np.column_stack([arr["red"], arr["green"], arr["blue"]]).astype(np.uint8)
|
||||
return points, colors
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Video rendering (ffmpeg)
|
||||
# ---------------------------------------------------------------------------
|
||||
def _find_font() -> str | None:
|
||||
for candidate in _FONT_CANDIDATES:
|
||||
if Path(candidate).exists():
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
_DRAWTEXT_AVAILABLE: bool | None = None
|
||||
|
||||
|
||||
def _has_drawtext() -> bool:
|
||||
"""Whether this ffmpeg build includes the ``drawtext`` filter (needs libfreetype)."""
|
||||
global _DRAWTEXT_AVAILABLE
|
||||
if _DRAWTEXT_AVAILABLE is None:
|
||||
try:
|
||||
out = subprocess.run(
|
||||
["ffmpeg", "-hide_banner", "-filters"],
|
||||
check=True, capture_output=True, text=True,
|
||||
).stdout
|
||||
_DRAWTEXT_AVAILABLE = any(
|
||||
line.split()[1] == "drawtext"
|
||||
for line in out.splitlines()
|
||||
if len(line.split()) > 1
|
||||
)
|
||||
except Exception:
|
||||
_DRAWTEXT_AVAILABLE = False
|
||||
return _DRAWTEXT_AVAILABLE
|
||||
|
||||
|
||||
def render_video(out_path: Path, wav_path: Path, cam_index: int,
|
||||
width: int, height: int, fps: float, duration_s: float) -> None:
|
||||
"""Render one synthetic clip: a distinct ``testsrc2`` visual muxed with ``wav_path``."""
|
||||
if shutil.which("ffmpeg") is None:
|
||||
raise RuntimeError("ffmpeg not found on PATH — required for the synthetic fixture")
|
||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
hue = cam_index * 47
|
||||
vf = f"hue=h={hue}"
|
||||
font = _find_font()
|
||||
if font and _has_drawtext():
|
||||
vf += (f",drawtext=fontfile='{font}':text='CAM {cam_index}':"
|
||||
"x=24:y=24:fontsize=44:fontcolor=white:box=1:boxcolor=black@0.5")
|
||||
|
||||
src = f"testsrc2=size={width}x{height}:rate={fps}:duration={duration_s}"
|
||||
cmd = [
|
||||
"ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
|
||||
"-f", "lavfi", "-i", src,
|
||||
"-i", str(wav_path),
|
||||
"-vf", vf,
|
||||
"-c:v", "libx264", "-pix_fmt", "yuv420p", "-preset", "veryfast", "-crf", "28",
|
||||
"-profile:v", "baseline", "-level", "3.1",
|
||||
"-movflags", "+faststart",
|
||||
"-c:a", "aac", "-b:a", "128k",
|
||||
"-shortest",
|
||||
str(out_path),
|
||||
]
|
||||
subprocess.run(cmd, check=True)
|
||||
|
||||
|
||||
def ffprobe_video(path: Path) -> dict:
|
||||
"""Probe width/height/fps/duration of a video with ffprobe."""
|
||||
out = subprocess.run(
|
||||
["ffprobe", "-v", "error", "-select_streams", "v:0",
|
||||
"-show_entries", "stream=width,height,avg_frame_rate:format=duration",
|
||||
"-of", "json", str(path)],
|
||||
check=True, capture_output=True, text=True,
|
||||
).stdout
|
||||
data = json.loads(out)
|
||||
stream = data["streams"][0]
|
||||
num, den = stream["avg_frame_rate"].split("/")
|
||||
fps = float(num) / float(den) if float(den) else 0.0
|
||||
return {
|
||||
"width": int(stream["width"]),
|
||||
"height": int(stream["height"]),
|
||||
"fps": fps,
|
||||
"duration_s": float(data["format"]["duration"]),
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Orchestration
|
||||
# ---------------------------------------------------------------------------
|
||||
def build(base_dir: Path | str | None = None, duration_s: float | None = None,
|
||||
run_ffmpeg: bool = True) -> dict:
|
||||
"""Generate the full synthetic project. Returns a summary dict.
|
||||
|
||||
``base_dir`` defaults to ``config.DATA_DIR``. ``duration_s`` defaults to the fixture's
|
||||
20 s (tests may pass a shorter value). ``run_ffmpeg=False`` skips video rendering (for
|
||||
fast, hermetic tests of the DB/pose/PLY path).
|
||||
"""
|
||||
base = Path(base_dir) if base_dir is not None else config.DATA_DIR
|
||||
duration = float(duration_s) if duration_s is not None else config.SYNTH_DURATION_S
|
||||
raw = base / "raw"
|
||||
work = base / "work"
|
||||
raw.mkdir(parents=True, exist_ok=True)
|
||||
work.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
sr = config.SYNTH_AUDIO_SR
|
||||
w, h, fps = config.SYNTH_VIDEO_W, config.SYNTH_VIDEO_H, config.SYNTH_FPS
|
||||
|
||||
log.info("synthetic: building project at %s (duration=%.1fs, ffmpeg=%s)",
|
||||
base, duration, run_ffmpeg)
|
||||
|
||||
db.init_engine(base / "project.db")
|
||||
db.reset_db()
|
||||
|
||||
master, audio_gt = synth_master_audio(sr)
|
||||
|
||||
videos_gt = []
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
tmp = Path(tmp)
|
||||
for i, offset_ms in enumerate(config.SYNTH_OFFSETS_MS):
|
||||
filename = f"cam{i}.mp4"
|
||||
out_path = raw / filename
|
||||
clip = slice_for_offset(master, sr, offset_ms, duration)
|
||||
|
||||
if run_ffmpeg:
|
||||
wav_path = tmp / f"cam{i}.wav"
|
||||
write_wav(wav_path, clip, sr)
|
||||
render_video(out_path, wav_path, i, w, h, fps, duration)
|
||||
probed = ffprobe_video(out_path)
|
||||
vw, vh, vfps, vdur = (probed["width"], probed["height"],
|
||||
probed["fps"], probed["duration_s"])
|
||||
else:
|
||||
vw, vh, vfps, vdur = w, h, float(fps), duration
|
||||
|
||||
video = db.add_video(
|
||||
filename=filename, duration_s=vdur, fps=vfps, width=vw, height=vh,
|
||||
offset_ms=float(offset_ms), drift_ppm=0.0, sync_confidence=1.0,
|
||||
)
|
||||
db.set_poses(video.id, camera_track(i, duration, fps, w, h))
|
||||
videos_gt.append({
|
||||
"video_id": video.id, "filename": filename,
|
||||
"offset_ms": float(offset_ms), "drift_ppm": 0.0,
|
||||
"is_reference": i == 0,
|
||||
})
|
||||
log.info("synthetic: cam%d offset=%+.0fms -> %s", i, offset_ms, out_path.name)
|
||||
|
||||
# Point cloud
|
||||
points, colors = generate_point_cloud()
|
||||
write_ply(work / "points.ply", points, colors)
|
||||
|
||||
# Anchors (stage corners)
|
||||
corner_colors = {"Stage FL": "#ff5533", "Stage FR": "#ffaa33",
|
||||
"Stage BL": "#33aaff", "Stage BR": "#aa55ff"}
|
||||
for label, (x, y, z) in STAGE_CORNERS.items():
|
||||
db.add_anchor(label, x, y, z, corner_colors.get(label))
|
||||
|
||||
# Seeded events
|
||||
for t_global, event_type, conf, desc in SEED_EVENTS:
|
||||
db.add_event(t_global_s=t_global, event_type=event_type, source="ai",
|
||||
duration_s=1.0, confidence=conf, description=desc)
|
||||
|
||||
# Ground truth JSON
|
||||
ground_truth = {
|
||||
"videos": videos_gt,
|
||||
"reference_filename": "cam0.mp4",
|
||||
"audio": audio_gt,
|
||||
"events": [
|
||||
{"t_global_s": t, "event_type": et, "confidence": c, "description": d}
|
||||
for (t, et, c, d) in SEED_EVENTS
|
||||
],
|
||||
"scene": {
|
||||
"stage_corners": {k: list(v) for k, v in STAGE_CORNERS.items()},
|
||||
"stage_target": list(STAGE_TARGET),
|
||||
"point_count": int(len(points)),
|
||||
},
|
||||
}
|
||||
gt_path = work / "ground_truth.json"
|
||||
gt_path.write_text(json.dumps(ground_truth, indent=2))
|
||||
|
||||
summary = {
|
||||
"base_dir": str(base),
|
||||
"videos": [v["filename"] for v in videos_gt],
|
||||
"points_ply": str(work / "points.ply"),
|
||||
"point_count": int(len(points)),
|
||||
"ground_truth": str(gt_path),
|
||||
"events": len(SEED_EVENTS),
|
||||
"anchors": len(STAGE_CORNERS),
|
||||
}
|
||||
log.info("synthetic: done — %d videos, %d points, %d events, %d anchors",
|
||||
len(videos_gt), len(points), len(SEED_EVENTS), len(STAGE_CORNERS))
|
||||
return summary
|
||||
|
||||
|
||||
if __name__ == "__main__": # pragma: no cover
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
print(json.dumps(build(), indent=2))
|
||||
13
backend/tests/conftest.py
Normal file
13
backend/tests/conftest.py
Normal file
@ -0,0 +1,13 @@
|
||||
"""Pytest session setup.
|
||||
|
||||
Redirect the Festival 4D data directory to a throwaway temp dir for the whole test
|
||||
session, so tests never touch the repo's real ``data/`` (which may hold the demo fixture).
|
||||
Set before any ``festival4d.config`` import so the paths resolve to the temp dir.
|
||||
"""
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
|
||||
os.environ.setdefault(
|
||||
"FESTIVAL4D_DATA_DIR", tempfile.mkdtemp(prefix="festival4d-test-")
|
||||
)
|
||||
106
backend/tests/test_api.py
Normal file
106
backend/tests/test_api.py
Normal file
@ -0,0 +1,106 @@
|
||||
"""API contract tests (spec M3), against a synthetic fixture built into the test data dir.
|
||||
|
||||
Locks the frozen response shapes lane C depends on, and verifies HTTP Range serving (206),
|
||||
CORS, and graceful degradation of the still-stubbed endpoints.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import shutil
|
||||
|
||||
import pytest
|
||||
|
||||
pytestmark = pytest.mark.skipif(
|
||||
shutil.which("ffmpeg") is None or shutil.which("ffprobe") is None,
|
||||
reason="ffmpeg/ffprobe required to build the API test fixture",
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def client():
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from festival4d import synthetic
|
||||
|
||||
synthetic.build(duration_s=1.0, run_ffmpeg=True) # into the temp data dir (conftest)
|
||||
|
||||
from festival4d import api # imported after build so the /media mount + DB are ready
|
||||
|
||||
with TestClient(api.app) as c:
|
||||
yield c
|
||||
|
||||
|
||||
def test_manifest_shape(client):
|
||||
data = client.get("/api/manifest").json()
|
||||
assert set(data) == {"videos", "t_global_max", "has_poses"}
|
||||
assert data["has_poses"] is True
|
||||
assert len(data["videos"]) == 3
|
||||
v = data["videos"][0]
|
||||
assert set(v) == {
|
||||
"id", "filename", "url", "duration_s", "fps", "width", "height",
|
||||
"offset_ms", "drift_ppm",
|
||||
}
|
||||
offsets = {vv["filename"]: vv["offset_ms"] for vv in data["videos"]}
|
||||
assert offsets == {"cam0.mp4": 0.0, "cam1.mp4": 1370.0, "cam2.mp4": -842.0}
|
||||
|
||||
|
||||
def test_poses_shape(client):
|
||||
poses = client.get("/api/videos/1/poses").json()
|
||||
assert len(poses) > 0
|
||||
p = poses[0]
|
||||
assert set(p) == {"frame_idx", "t_video_s", "q", "t", "intrinsics", "registered"}
|
||||
assert len(p["q"]) == 4 and len(p["t"]) == 3
|
||||
assert set(p["intrinsics"]) == {"fx", "fy", "cx", "cy"}
|
||||
|
||||
|
||||
def test_missing_video_404(client):
|
||||
assert client.get("/api/videos/999/poses").status_code == 404
|
||||
|
||||
|
||||
def test_anchors_get_and_post(client):
|
||||
anchors = client.get("/api/anchors").json()
|
||||
assert len(anchors) == 4
|
||||
created = client.post(
|
||||
"/api/anchors", json={"label": "T", "x": 1, "y": 2, "z": 3, "color": "#fff"}
|
||||
).json()
|
||||
assert created["label"] == "T" and created["id"] > 0
|
||||
|
||||
|
||||
def test_events(client):
|
||||
events = client.get("/api/events").json()
|
||||
assert len(events) == len(__import__("festival4d.synthetic", fromlist=["x"]).SEED_EVENTS)
|
||||
assert {"id", "t_global_s", "event_type", "source", "description"} <= set(events[0])
|
||||
|
||||
|
||||
def test_pointcloud_is_ply(client):
|
||||
resp = client.get("/api/pointcloud")
|
||||
assert resp.status_code == 200
|
||||
assert resp.content[:3] == b"ply"
|
||||
|
||||
|
||||
def test_video_range_returns_206(client):
|
||||
"""Spec pitfall #3: <video> seeking needs HTTP Range -> 206."""
|
||||
resp = client.get("/media/cam0.mp4", headers={"Range": "bytes=0-100"})
|
||||
assert resp.status_code == 206
|
||||
assert resp.headers["content-range"].startswith("bytes 0-100/")
|
||||
assert len(resp.content) == 101
|
||||
|
||||
|
||||
def test_cors_allows_vite_origin(client):
|
||||
resp = client.get("/api/manifest", headers={"Origin": "http://localhost:5173"})
|
||||
assert resp.headers.get("access-control-allow-origin") == "http://localhost:5173"
|
||||
|
||||
|
||||
def test_detect_degrades_gracefully(client):
|
||||
data = client.post("/api/events/detect", json={}).json()
|
||||
assert "note" in data and "events" in data # stubbed lane D -> 200 with note
|
||||
assert client.post("/api/events/detect", json={}).status_code == 200
|
||||
|
||||
|
||||
def test_annotation_stored(client):
|
||||
data = client.post(
|
||||
"/api/annotations",
|
||||
json={"video_id": 1, "t_video_s": 0.5, "bbox": [0.4, 0.4, 0.6, 0.6]},
|
||||
).json()
|
||||
assert data["annotation_id"] > 0
|
||||
assert data["anchor_id"] is None # M8 resolution is integration work
|
||||
110
backend/tests/test_geometry.py
Normal file
110
backend/tests/test_geometry.py
Normal file
@ -0,0 +1,110 @@
|
||||
"""Tests for geometry.py.
|
||||
|
||||
The ``colmap_to_threejs`` tests below are a **FROZEN CONTRACT** (foundation). The embedded
|
||||
``POSE_TEST_VECTORS`` are duplicated verbatim in ``frontend/src/lib/pose.js`` so both sides
|
||||
provably agree — if you touch the conversion math, update both files and both vector sets,
|
||||
via a change request. Lane B owns this file but must not modify these three cases; it adds
|
||||
tests for slerp_pose / ray_from_pixel / triangulate_rays / nearest_point_on_ray below them.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
from scipy.spatial.transform import Rotation
|
||||
|
||||
from festival4d.geometry import (
|
||||
colmap_to_threejs,
|
||||
mat_to_quat,
|
||||
quat_to_mat,
|
||||
)
|
||||
|
||||
# Sqrt(2)/2, spelled out so it matches the JS constant character-for-character.
|
||||
SQRT1_2 = 0.7071067811865476
|
||||
|
||||
# --- FROZEN test vectors (must equal POSE_TEST_VECTORS in frontend/src/lib/pose.js) ----
|
||||
# Each: COLMAP world->camera (q=[w,x,y,z], t) -> Three.js (position, rotation 3x3 row-major).
|
||||
POSE_TEST_VECTORS = [
|
||||
{
|
||||
"name": "identity",
|
||||
"q": [1.0, 0.0, 0.0, 0.0],
|
||||
"t": [0.0, 0.0, -10.0],
|
||||
"position": [0.0, 0.0, 10.0],
|
||||
"rotation": [[1.0, 0.0, 0.0], [0.0, -1.0, 0.0], [0.0, 0.0, -1.0]],
|
||||
},
|
||||
{
|
||||
"name": "yaw90", # +90 deg about world Y
|
||||
"q": [SQRT1_2, 0.0, SQRT1_2, 0.0],
|
||||
"t": [0.0, 0.0, 10.0],
|
||||
"position": [10.0, 0.0, 0.0],
|
||||
"rotation": [[0.0, 0.0, 1.0], [0.0, -1.0, 0.0], [1.0, 0.0, 0.0]],
|
||||
},
|
||||
{
|
||||
"name": "lookat", # camera at +Z looking at origin -> Three.js rotation is identity
|
||||
"q": [0.0, 1.0, 0.0, 0.0],
|
||||
"t": [0.0, 0.0, 8.0],
|
||||
"position": [0.0, 0.0, 8.0],
|
||||
"rotation": [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]],
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("vec", POSE_TEST_VECTORS, ids=lambda v: v["name"])
|
||||
def test_colmap_to_threejs_known_vectors(vec):
|
||||
position, rotation = colmap_to_threejs(vec["q"], vec["t"])
|
||||
np.testing.assert_allclose(position, vec["position"], atol=1e-9)
|
||||
np.testing.assert_allclose(rotation, vec["rotation"], atol=1e-9)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("vec", POSE_TEST_VECTORS, ids=lambda v: v["name"])
|
||||
def test_threejs_rotation_is_proper(vec):
|
||||
"""R_three must be a proper rotation (orthonormal, det = +1)."""
|
||||
_, R = colmap_to_threejs(vec["q"], vec["t"])
|
||||
np.testing.assert_allclose(R @ R.T, np.eye(3), atol=1e-9)
|
||||
assert abs(np.linalg.det(R) - 1.0) < 1e-9
|
||||
|
||||
|
||||
def _colmap_pose(R_w2c: np.ndarray, center: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
|
||||
"""Build a COLMAP (q, t) from a world->camera rotation and a world camera center."""
|
||||
t = -R_w2c @ center
|
||||
q = mat_to_quat(R_w2c)
|
||||
return q, t
|
||||
|
||||
|
||||
def test_colmap_to_threejs_roundtrip_random_poses():
|
||||
"""Forward COLMAP->Three.js then invert it; recover the original (q, t)."""
|
||||
rng = np.random.default_rng(20260716)
|
||||
flip = np.diag([1.0, -1.0, -1.0])
|
||||
for _ in range(50):
|
||||
R_w2c = Rotation.random(random_state=rng).as_matrix()
|
||||
center = rng.uniform(-10, 10, size=3)
|
||||
q, t = _colmap_pose(R_w2c, center)
|
||||
|
||||
position, R_three = colmap_to_threejs(q, t)
|
||||
|
||||
# position is the camera center
|
||||
np.testing.assert_allclose(position, center, atol=1e-9)
|
||||
|
||||
# invert: R_c2w = R_three @ flip -> R_w2c = R_c2w^T -> t = -R_w2c @ C
|
||||
R_c2w = R_three @ flip
|
||||
R_back = R_c2w.T
|
||||
t_back = -R_back @ position
|
||||
q_back = mat_to_quat(R_back)
|
||||
|
||||
np.testing.assert_allclose(t_back, t, atol=1e-9)
|
||||
# quaternions are double-cover; compare canonical (w>=0) forms
|
||||
q_canon = q if q[0] >= 0 else -q
|
||||
np.testing.assert_allclose(q_back, q_canon, atol=1e-9)
|
||||
|
||||
|
||||
def test_quat_to_mat_matches_scipy_oracle():
|
||||
"""quat_to_mat agrees with an independent implementation (scipy)."""
|
||||
rng = np.random.default_rng(11)
|
||||
for _ in range(50):
|
||||
# scipy quaternions are scalar-LAST [x,y,z,w]; ours are scalar-FIRST [w,x,y,z]
|
||||
q_xyzw = rng.normal(size=4)
|
||||
q_xyzw /= np.linalg.norm(q_xyzw)
|
||||
q_wxyz = np.array([q_xyzw[3], q_xyzw[0], q_xyzw[1], q_xyzw[2]])
|
||||
np.testing.assert_allclose(
|
||||
quat_to_mat(q_wxyz), Rotation.from_quat(q_xyzw).as_matrix(), atol=1e-9
|
||||
)
|
||||
118
backend/tests/test_synthetic.py
Normal file
118
backend/tests/test_synthetic.py
Normal file
@ -0,0 +1,118 @@
|
||||
"""Fixture tests for synthetic.py (foundation-owned; synthetic.py is a frozen contract).
|
||||
|
||||
These lock the synthetic fixture's contract: known audio offsets are recoverable, bang
|
||||
times stand out for M7 candidate detection, the PLY round-trips, generated poses face the
|
||||
stage under the frozen conversion, and ``build`` populates the DB + files as ground truth.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shutil
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
from scipy.signal import correlate
|
||||
|
||||
from festival4d import config, db, synthetic
|
||||
from festival4d.geometry import colmap_to_threejs
|
||||
|
||||
|
||||
def _estimate_offset_ms(clip_ref, clip_v, sr):
|
||||
"""Recover clip_v's offset vs the reference via cross-correlation (see derivation)."""
|
||||
corr = correlate(clip_v, clip_ref, mode="full")
|
||||
lag = int(np.argmax(corr)) - (len(clip_ref) - 1)
|
||||
return -lag / sr * 1000.0
|
||||
|
||||
|
||||
def test_audio_slices_recover_known_offsets():
|
||||
sr = config.SYNTH_AUDIO_SR
|
||||
master, _ = synthetic.synth_master_audio(sr)
|
||||
ref = synthetic.slice_for_offset(master, sr, 0.0, config.SYNTH_DURATION_S)
|
||||
for offset_ms in config.SYNTH_OFFSETS_MS:
|
||||
clip = synthetic.slice_for_offset(master, sr, offset_ms, config.SYNTH_DURATION_S)
|
||||
recovered = _estimate_offset_ms(ref, clip, sr)
|
||||
assert abs(recovered - offset_ms) < 1.0, (offset_ms, recovered)
|
||||
|
||||
|
||||
def test_bang_times_dominate_energy():
|
||||
"""Loud bangs (M7 candidates) must clearly exceed the regular-beat energy floor."""
|
||||
sr = config.SYNTH_AUDIO_SR
|
||||
master, gt = synthetic.synth_master_audio(sr)
|
||||
|
||||
def window_rms(center_s, half=0.05):
|
||||
i0 = int((center_s - synthetic.G_START - half) * sr)
|
||||
i1 = int((center_s - synthetic.G_START + half) * sr)
|
||||
return float(np.sqrt(np.mean(master[i0:i1] ** 2)))
|
||||
|
||||
bang_rms = [window_rms(t) for t in gt["bang_times_global_s"]]
|
||||
# a spread of non-bang beat times for the baseline
|
||||
beat_rms = [window_rms(t) for t in (1.0, 4.5, 7.0, 11.5, 14.0)]
|
||||
assert min(bang_rms) > 3.0 * np.median(beat_rms)
|
||||
|
||||
|
||||
def test_ply_roundtrip(tmp_path):
|
||||
pts, cols = synthetic.generate_point_cloud()
|
||||
path = tmp_path / "points.ply"
|
||||
synthetic.write_ply(path, pts, cols)
|
||||
pts2, cols2 = synthetic.read_ply(path)
|
||||
assert pts2.shape == pts.shape and cols2.shape == cols.shape
|
||||
np.testing.assert_allclose(pts2, pts, atol=1e-6)
|
||||
np.testing.assert_array_equal(cols2, cols)
|
||||
|
||||
|
||||
def test_generated_poses_face_the_stage():
|
||||
"""Each synthetic pose, run through the frozen conversion, must look at the stage."""
|
||||
poses = synthetic.camera_track(1, config.SYNTH_DURATION_S, config.SYNTH_FPS,
|
||||
config.SYNTH_VIDEO_W, config.SYNTH_VIDEO_H)
|
||||
target = np.array(synthetic.STAGE_TARGET)
|
||||
for p in poses:
|
||||
q = [p["qw"], p["qx"], p["qy"], p["qz"]]
|
||||
t = [p["tx"], p["ty"], p["tz"]]
|
||||
position, R = colmap_to_threejs(q, t)
|
||||
look = R @ np.array([0.0, 0.0, -1.0]) # Three.js camera looks down -z
|
||||
expected = target - position
|
||||
expected /= np.linalg.norm(expected)
|
||||
np.testing.assert_allclose(look, expected, atol=1e-9)
|
||||
|
||||
|
||||
def test_build_no_ffmpeg_populates_db_and_files(tmp_path):
|
||||
summary = synthetic.build(base_dir=tmp_path, duration_s=5.0, run_ffmpeg=False)
|
||||
|
||||
videos = db.get_videos()
|
||||
assert len(videos) == 3
|
||||
by_name = {v.filename: v for v in videos}
|
||||
assert by_name["cam0.mp4"].offset_ms == 0.0
|
||||
assert by_name["cam1.mp4"].offset_ms == 1370.0
|
||||
assert by_name["cam2.mp4"].offset_ms == -842.0
|
||||
for v in videos:
|
||||
assert len(db.get_poses(v.id)) > 0
|
||||
assert db.get_poses(v.id)[0].registered is True
|
||||
|
||||
assert db.has_poses()
|
||||
assert len(db.get_anchors()) == 4
|
||||
assert len(db.get_events()) == len(synthetic.SEED_EVENTS)
|
||||
|
||||
ply = tmp_path / "work" / "points.ply"
|
||||
assert ply.exists()
|
||||
pts, _ = synthetic.read_ply(ply)
|
||||
assert len(pts) == summary["point_count"] > 1000
|
||||
|
||||
gt = json.loads((tmp_path / "work" / "ground_truth.json").read_text())
|
||||
assert [vv["offset_ms"] for vv in gt["videos"]] == [0.0, 1370.0, -842.0]
|
||||
assert gt["reference_filename"] == "cam0.mp4"
|
||||
|
||||
|
||||
@pytest.mark.skipif(shutil.which("ffmpeg") is None or shutil.which("ffprobe") is None,
|
||||
reason="ffmpeg/ffprobe required for the full fixture build")
|
||||
def test_build_full_ffmpeg_renders_seekable_videos(tmp_path):
|
||||
synthetic.build(base_dir=tmp_path, duration_s=4.0, run_ffmpeg=True)
|
||||
raw = tmp_path / "raw"
|
||||
for name in ("cam0.mp4", "cam1.mp4", "cam2.mp4"):
|
||||
path = raw / name
|
||||
assert path.exists() and path.stat().st_size > 10_000
|
||||
videos = db.get_videos()
|
||||
assert len(videos) == 3
|
||||
for v in videos:
|
||||
assert 3.5 < v.duration_s < 4.6 # ~4 s clips
|
||||
assert v.width == 640 and v.height == 360
|
||||
43
frontend/index.html
Normal file
43
frontend/index.html
Normal file
@ -0,0 +1,43 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>Festival 4D</title>
|
||||
<style>
|
||||
:root { color-scheme: dark; }
|
||||
body {
|
||||
margin: 0;
|
||||
font: 15px/1.5 -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
|
||||
background: #0b0d12;
|
||||
color: #e6e8ee;
|
||||
padding: 2rem;
|
||||
}
|
||||
h1 { margin: 0 0 0.25rem; font-size: 1.6rem; }
|
||||
.sub { color: #8a92a6; margin-bottom: 1.5rem; }
|
||||
.card {
|
||||
background: #141824;
|
||||
border: 1px solid #232838;
|
||||
border-radius: 10px;
|
||||
padding: 1rem 1.25rem;
|
||||
margin-bottom: 1rem;
|
||||
max-width: 720px;
|
||||
}
|
||||
.ok { color: #59d499; }
|
||||
.bad { color: #ff6b6b; }
|
||||
table { border-collapse: collapse; width: 100%; }
|
||||
th, td { text-align: left; padding: 0.35rem 0.6rem; border-bottom: 1px solid #232838; }
|
||||
th { color: #8a92a6; font-weight: 600; }
|
||||
code { background: #1c2233; padding: 0.1rem 0.35rem; border-radius: 4px; }
|
||||
.pill { display: inline-block; padding: 0.1rem 0.5rem; border-radius: 999px; font-size: 0.8rem; }
|
||||
.pill.ok { background: rgba(89,212,153,0.15); }
|
||||
.pill.bad { background: rgba(255,107,107,0.15); }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Festival 4D</h1>
|
||||
<div class="sub">Foundation hello page — proves the API contract, CORS, and the frozen pose helper.</div>
|
||||
<div id="app"><div class="card">Loading…</div></div>
|
||||
<script type="module" src="/src/main.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
1152
frontend/package-lock.json
generated
Normal file
1152
frontend/package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load Diff
17
frontend/package.json
Normal file
17
frontend/package.json
Normal file
@ -0,0 +1,17 @@
|
||||
{
|
||||
"name": "festival4d-frontend",
|
||||
"private": true,
|
||||
"version": "0.1.0",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
"build": "vite build",
|
||||
"preview": "vite preview"
|
||||
},
|
||||
"dependencies": {
|
||||
"three": "^0.170.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"vite": "^6.0.0"
|
||||
}
|
||||
}
|
||||
6
frontend/src/annotate.js
Normal file
6
frontend/src/annotate.js
Normal file
@ -0,0 +1,6 @@
|
||||
// bbox drawing + submission → POST /api/annotations (spec M8).
|
||||
// STUB — this is phase-3 (integration) work; lane C leaves it as a clean seam. Drag a
|
||||
// rectangle on a video's overlay canvas → normalized bbox → POST /api/annotations; the
|
||||
// resolved anchor then appears live in overlays + the 3D scene.
|
||||
|
||||
export {};
|
||||
6
frontend/src/camPath.js
Normal file
6
frontend/src/camPath.js
Normal file
@ -0,0 +1,6 @@
|
||||
// Keyframed camera paths (spec M9).
|
||||
// STUB — this is phase-3 (integration) work; lane C leaves it as a clean seam. "Add
|
||||
// keyframe" captures the current free-roam pose at the current t_global; playback
|
||||
// interpolates position (Catmull-Rom) + orientation (slerp); export/import path as JSON.
|
||||
|
||||
export {};
|
||||
143
frontend/src/lib/pose.js
Normal file
143
frontend/src/lib/pose.js
Normal file
@ -0,0 +1,143 @@
|
||||
// COLMAP world->camera pose -> Three.js camera pose (spec M5).
|
||||
//
|
||||
// FROZEN CONTRACT. This is the JavaScript mirror of `backend/festival4d/geometry.py`
|
||||
// (`colmap_to_threejs`, `quat_to_mat`). The math and the POSE_TEST_VECTORS below are
|
||||
// duplicated verbatim on the Python side (`backend/tests/test_geometry.py`) so both agree.
|
||||
// Lanes B and C CONSUME this; do not reimplement the conversion elsewhere. Changing it
|
||||
// requires a change request and a synchronized edit to both files + both vector sets.
|
||||
//
|
||||
// Conventions:
|
||||
// - Quaternions are COLMAP order [w, x, y, z] (scalar first), unit norm.
|
||||
// - Pose (q, t) is world->camera: x_cam = R(q) * x_world + t.
|
||||
// - COLMAP camera axes: +x right, +y down, +z forward.
|
||||
// - Three.js cameras look down -z with +y up; hence the diag(1, -1, -1) flip.
|
||||
|
||||
const SQRT1_2 = 0.7071067811865476;
|
||||
|
||||
// Camera-axis flip: COLMAP camera-local (x right, y down, z forward) -> Three.js (x right,
|
||||
// y up, z backward). Same _FLIP_YZ as the Python side.
|
||||
const FLIP_YZ = [
|
||||
[1, 0, 0],
|
||||
[0, -1, 0],
|
||||
[0, 0, -1],
|
||||
];
|
||||
|
||||
/**
|
||||
* Convert a unit quaternion [w, x, y, z] to a 3x3 rotation matrix (row-major nested array).
|
||||
* Hamilton convention, right-handed, active rotation (COLMAP world->camera when q is a
|
||||
* COLMAP pose quaternion).
|
||||
* @param {number[]} q - [w, x, y, z]
|
||||
* @returns {number[][]} 3x3 rotation matrix
|
||||
*/
|
||||
export function quatToMat(q) {
|
||||
const [w, x, y, z] = q;
|
||||
const n = w * w + x * x + y * y + z * z;
|
||||
if (n < 1e-12) throw new Error("quaternion has near-zero norm");
|
||||
const s = 2.0 / n;
|
||||
const wx = s * w * x, wy = s * w * y, wz = s * w * z;
|
||||
const xx = s * x * x, xy = s * x * y, xz = s * x * z;
|
||||
const yy = s * y * y, yz = s * y * z, zz = s * z * z;
|
||||
return [
|
||||
[1.0 - (yy + zz), xy - wz, xz + wy],
|
||||
[xy + wz, 1.0 - (xx + zz), yz - wx],
|
||||
[xz - wy, yz + wx, 1.0 - (xx + yy)],
|
||||
];
|
||||
}
|
||||
|
||||
function transpose3(m) {
|
||||
return [
|
||||
[m[0][0], m[1][0], m[2][0]],
|
||||
[m[0][1], m[1][1], m[2][1]],
|
||||
[m[0][2], m[1][2], m[2][2]],
|
||||
];
|
||||
}
|
||||
|
||||
function matMul3(a, b) {
|
||||
const out = [[0, 0, 0], [0, 0, 0], [0, 0, 0]];
|
||||
for (let i = 0; i < 3; i++)
|
||||
for (let j = 0; j < 3; j++)
|
||||
out[i][j] = a[i][0] * b[0][j] + a[i][1] * b[1][j] + a[i][2] * b[2][j];
|
||||
return out;
|
||||
}
|
||||
|
||||
function matVec3(m, v) {
|
||||
return [
|
||||
m[0][0] * v[0] + m[0][1] * v[1] + m[0][2] * v[2],
|
||||
m[1][0] * v[0] + m[1][1] * v[1] + m[1][2] * v[2],
|
||||
m[2][0] * v[0] + m[2][1] * v[1] + m[2][2] * v[2],
|
||||
];
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert a COLMAP world->camera pose to a Three.js camera pose (spec M5). FROZEN.
|
||||
* @param {number[]} q - COLMAP world->camera quaternion [w, x, y, z]
|
||||
* @param {number[]} t - COLMAP world->camera translation [tx, ty, tz]
|
||||
* @returns {{position: number[], rotation: number[][], matrixWorld: number[]}}
|
||||
* position - camera center in world coords, C = -R^T t (assign to camera.position)
|
||||
* rotation - Three.js camera world rotation R_three = R^T * diag(1,-1,-1),
|
||||
* 3x3 row-major (feed to camera.setRotationFromMatrix via a Matrix4)
|
||||
* matrixWorld - column-major 16-array [R_three | position], ready for
|
||||
* THREE.Matrix4().fromArray(...) when driving the camera by matrix.
|
||||
*/
|
||||
export function colmapToThreejs(q, t) {
|
||||
const R = quatToMat(q); // world -> cam
|
||||
const Rc2w = transpose3(R); // cam -> world
|
||||
const position = matVec3(Rc2w, [-t[0], -t[1], -t[2]]); // C = -R^T t
|
||||
const rotation = matMul3(Rc2w, FLIP_YZ); // R_three
|
||||
|
||||
// Column-major Matrix4 with rotation in the upper-left 3x3 and translation = position.
|
||||
const m = rotation;
|
||||
const p = position;
|
||||
const matrixWorld = [
|
||||
m[0][0], m[1][0], m[2][0], 0,
|
||||
m[0][1], m[1][1], m[2][1], 0,
|
||||
m[0][2], m[1][2], m[2][2], 0,
|
||||
p[0], p[1], p[2], 1,
|
||||
];
|
||||
return { position, rotation, matrixWorld };
|
||||
}
|
||||
|
||||
// --- FROZEN test vectors (must equal POSE_TEST_VECTORS in backend/tests/test_geometry.py).
|
||||
export const POSE_TEST_VECTORS = [
|
||||
{
|
||||
name: "identity",
|
||||
q: [1.0, 0.0, 0.0, 0.0],
|
||||
t: [0.0, 0.0, -10.0],
|
||||
position: [0.0, 0.0, 10.0],
|
||||
rotation: [[1.0, 0.0, 0.0], [0.0, -1.0, 0.0], [0.0, 0.0, -1.0]],
|
||||
},
|
||||
{
|
||||
name: "yaw90",
|
||||
q: [SQRT1_2, 0.0, SQRT1_2, 0.0],
|
||||
t: [0.0, 0.0, 10.0],
|
||||
position: [10.0, 0.0, 0.0],
|
||||
rotation: [[0.0, 0.0, 1.0], [0.0, -1.0, 0.0], [1.0, 0.0, 0.0]],
|
||||
},
|
||||
{
|
||||
name: "lookat",
|
||||
q: [0.0, 1.0, 0.0, 0.0],
|
||||
t: [0.0, 0.0, 8.0],
|
||||
position: [0.0, 0.0, 8.0],
|
||||
rotation: [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]],
|
||||
},
|
||||
];
|
||||
|
||||
/**
|
||||
* Self-check the conversion against POSE_TEST_VECTORS. Returns true on success, throws on
|
||||
* mismatch. Called by main.js so the frozen contract is verified live in the browser.
|
||||
* @param {number} [atol=1e-9]
|
||||
*/
|
||||
export function selfTest(atol = 1e-9) {
|
||||
for (const vec of POSE_TEST_VECTORS) {
|
||||
const { position, rotation } = colmapToThreejs(vec.q, vec.t);
|
||||
for (let i = 0; i < 3; i++) {
|
||||
if (Math.abs(position[i] - vec.position[i]) > atol)
|
||||
throw new Error(`pose.js selfTest[${vec.name}]: position mismatch at ${i}`);
|
||||
for (let j = 0; j < 3; j++) {
|
||||
if (Math.abs(rotation[i][j] - vec.rotation[i][j]) > atol)
|
||||
throw new Error(`pose.js selfTest[${vec.name}]: rotation mismatch at ${i},${j}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
91
frontend/src/main.js
Normal file
91
frontend/src/main.js
Normal file
@ -0,0 +1,91 @@
|
||||
// Foundation hello page. Fetches /api/manifest (proving the API + CORS work cross-origin)
|
||||
// and runs the frozen pose.js self-test in the browser. Lane C (M4–M6) replaces this with
|
||||
// the real app; the modules alongside it are stubs to fill in.
|
||||
|
||||
import { selfTest as poseSelfTest } from "./lib/pose.js";
|
||||
|
||||
// The FastAPI backend (`python -m festival4d serve`). The Vite dev server runs on :5173;
|
||||
// this cross-origin fetch exercises the CORS config in api.py.
|
||||
const API_BASE = "http://localhost:8000";
|
||||
|
||||
const app = document.getElementById("app");
|
||||
|
||||
function card(html) {
|
||||
const el = document.createElement("div");
|
||||
el.className = "card";
|
||||
el.innerHTML = html;
|
||||
return el;
|
||||
}
|
||||
|
||||
function pill(ok, text) {
|
||||
return `<span class="pill ${ok ? "ok" : "bad"}">${text}</span>`;
|
||||
}
|
||||
|
||||
function renderPoseSelfTest() {
|
||||
try {
|
||||
poseSelfTest();
|
||||
return card(
|
||||
`<strong>Pose contract</strong> ${pill(true, "self-test passed")}<br />` +
|
||||
`<span class="sub">frontend/src/lib/pose.js matches the frozen COLMAP→Three.js vectors.</span>`
|
||||
);
|
||||
} catch (err) {
|
||||
return card(
|
||||
`<strong>Pose contract</strong> ${pill(false, "self-test FAILED")}<br />` +
|
||||
`<code>${err.message}</code>`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function renderManifest(m) {
|
||||
const rows = m.videos
|
||||
.map(
|
||||
(v) => `
|
||||
<tr>
|
||||
<td>${v.id}</td>
|
||||
<td><code>${v.filename}</code></td>
|
||||
<td>${v.duration_s.toFixed(2)} s</td>
|
||||
<td>${v.width}×${v.height} @ ${v.fps}</td>
|
||||
<td>${v.offset_ms === null ? "—" : v.offset_ms + " ms"}</td>
|
||||
</tr>`
|
||||
)
|
||||
.join("");
|
||||
return card(`
|
||||
<strong>API manifest</strong> ${pill(true, "fetched (CORS ok)")}<br />
|
||||
<span class="sub">
|
||||
t_global_max = ${m.t_global_max.toFixed(2)} s ·
|
||||
has_poses = ${m.has_poses ? "yes" : "no"} ·
|
||||
${m.videos.length} videos
|
||||
</span>
|
||||
<table>
|
||||
<thead><tr><th>id</th><th>file</th><th>duration</th><th>frame</th><th>offset</th></tr></thead>
|
||||
<tbody>${rows}</tbody>
|
||||
</table>
|
||||
<div class="sub" style="margin-top:0.75rem">
|
||||
Videos served with HTTP Range from <code>${API_BASE}/media/…</code>
|
||||
</div>
|
||||
`);
|
||||
}
|
||||
|
||||
async function main() {
|
||||
app.innerHTML = "";
|
||||
app.appendChild(renderPoseSelfTest());
|
||||
|
||||
try {
|
||||
const resp = await fetch(`${API_BASE}/api/manifest`);
|
||||
if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
|
||||
const manifest = await resp.json();
|
||||
app.appendChild(renderManifest(manifest));
|
||||
console.log("[festival4d] manifest loaded", manifest);
|
||||
} catch (err) {
|
||||
app.appendChild(
|
||||
card(
|
||||
`<strong>API manifest</strong> ${pill(false, "fetch failed")}<br />` +
|
||||
`<code>${err.message}</code><br />` +
|
||||
`<span class="sub">Is the backend running? <code>python -m festival4d serve</code></span>`
|
||||
)
|
||||
);
|
||||
console.error("[festival4d] manifest fetch failed", err);
|
||||
}
|
||||
}
|
||||
|
||||
main();
|
||||
6
frontend/src/overlays.js
Normal file
6
frontend/src/overlays.js
Normal file
@ -0,0 +1,6 @@
|
||||
// 3D→2D anchor projection onto each video's overlay canvas — the "x-ray" HUD (spec M6).
|
||||
// STUB — lane C fills this in. For each anchor + visible video: build the video's current
|
||||
// Three.js camera (pose.js helper), project the anchor, convert NDC→canvas px, skip if
|
||||
// behind camera (check z_cam < 0). Anchors render regardless of occlusion — that's the feature.
|
||||
|
||||
export {};
|
||||
6
frontend/src/scene3d.js
Normal file
6
frontend/src/scene3d.js
Normal file
@ -0,0 +1,6 @@
|
||||
// Three.js scene: point cloud (/api/pointcloud via PLYLoader), per-video camera paths +
|
||||
// current-pose frusta, OrbitControls, snap-to-camera (spec M5).
|
||||
// STUB — lane C fills this in. Use frontend/src/lib/pose.js (frozen contract) for ALL
|
||||
// COLMAP→Three.js conversion — do not reimplement the diag(1,-1,-1) math.
|
||||
|
||||
export {};
|
||||
9
frontend/src/state.js
Normal file
9
frontend/src/state.js
Normal file
@ -0,0 +1,9 @@
|
||||
// Global store: manifest, playhead (t_global), playing, rate, selected camera, audio source.
|
||||
// STUB — lane C (M4) fills this in. Kept as an empty module so the file exists per the
|
||||
// spec's repo layout; main.js does not import it yet.
|
||||
|
||||
// Example shape lane C will build toward:
|
||||
// export const state = { manifest: null, tGlobal: 0, playing: false, rate: 1,
|
||||
// selectedCamera: null, audioSource: null };
|
||||
|
||||
export {};
|
||||
6
frontend/src/timeline.js
Normal file
6
frontend/src/timeline.js
Normal file
@ -0,0 +1,6 @@
|
||||
// Scrubber + event markers (spec M4 transport + M7 timeline markers).
|
||||
// STUB — lane C fills this in. Render GET /api/events as colored markers (color by
|
||||
// event_type, legend in a corner); hover → description tooltip; click → jump the playhead.
|
||||
// The event data is already seeded synthetically; classification quality is lane D's job.
|
||||
|
||||
export {};
|
||||
8
frontend/src/transport.js
Normal file
8
frontend/src/transport.js
Normal file
@ -0,0 +1,8 @@
|
||||
// Master clock + per-video sync (spec M4 — the heart of the app).
|
||||
// STUB — lane C fills this in. Build and verify the master-clock transport before the 3D
|
||||
// work. Use the timebase from config.py (mirrored here):
|
||||
// t_video = (t_global - offset_ms/1000) * (1 + drift_ppm * 1e-6)
|
||||
// Never trust a single <video> element as the clock; drive t_global from performance.now()
|
||||
// and continuously correct each video (hard-seek >150ms, nudge playbackRate 20–150ms).
|
||||
|
||||
export {};
|
||||
5
frontend/src/videoGrid.js
Normal file
5
frontend/src/videoGrid.js
Normal file
@ -0,0 +1,5 @@
|
||||
// <video> grid + per-video transparent overlay canvases (spec M4/M6).
|
||||
// STUB — lane C fills this in. Mind spec pitfall #4: object-fit letterboxing means canvas
|
||||
// px != video px; compute each video's content rect for the overlay canvas.
|
||||
|
||||
export {};
|
||||
20
plan/CHANGE_REQUESTS.md
Normal file
20
plan/CHANGE_REQUESTS.md
Normal file
@ -0,0 +1,20 @@
|
||||
# Change Requests
|
||||
|
||||
Lanes that believe a **frozen** file must change (`db.py` schema, `api.py` routes/shapes,
|
||||
`cli.py`, `config.py`, `synthetic.py`, `pyproject.toml`, the pose contract) do **not** edit it.
|
||||
Instead append an entry here and work around it locally; the integration agent adjudicates.
|
||||
|
||||
**Format** (one entry per request, newest at the bottom):
|
||||
|
||||
```
|
||||
### CR-<n> — <lane> — <date> — <one-line title>
|
||||
- **File:** <frozen file + symbol>
|
||||
- **Problem:** why the current contract blocks you
|
||||
- **Proposed change:** the minimal change you need
|
||||
- **Workaround in place:** what you did locally so you're not blocked
|
||||
- **Decision:** (integration fills this: applied | rejected — rationale)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
<!-- entries below -->
|
||||
31
plan/status/foundation.md
Normal file
31
plan/status/foundation.md
Normal file
@ -0,0 +1,31 @@
|
||||
# Status — foundation
|
||||
|
||||
## Round 0 — 2026-07-16 — STATUS: ready_to_merge
|
||||
|
||||
**Directives acknowledged:** round 0 of plan/DIRECTIVES.md (run `foundation` only; contracts
|
||||
correctness beats speed; lanes hard-depend on API shapes, schema, pose math, synthetic fixtures).
|
||||
|
||||
**Acceptance checklist** (plan/00-foundation.md + spec M0/M3) — all pass:
|
||||
- [x] `python -m festival4d synthetic` → 3 fixture videos + populated DB + `points.ply` (3106 pts) + `ground_truth.json`. Evidence: CLI run logs "3 videos, 3106 points, 7 events, 4 anchors".
|
||||
- [x] `python -m festival4d serve` → every M3 endpoint returns synthetic data; videos seekable. Evidence: curl of manifest/poses/anchors/events/pointcloud; `curl -H "Range: bytes=0-100" /media/cam0.mp4` → **206** with `content-range`; browser `<video>` seeked to 15s, `seekable=[0,20]`, readyState 4.
|
||||
- [x] `pytest` green — **24 passed** (test_geometry, test_synthetic, test_api).
|
||||
- [x] `npm run dev` serves the hello page; browser fetched `/api/manifest` cross-origin (CORS ok), pose.js self-test passed, **0 console errors**. (Vite fell back to :5174 — an unrelated dev server holds :5173; CORS now allows any localhost port.)
|
||||
- [x] All lane-owned modules stubbed with frozen signatures; `cli.py` registers all 6 subcommands (dispatch + graceful NotImplementedError → exit 2).
|
||||
- [x] Frozen geometry contract: `colmap_to_threejs` + `test_geometry.py` (3 known vectors, random round-trip, scipy oracle) + mirrored `frontend/src/lib/pose.js` with identical `POSE_TEST_VECTORS` (node self-test passes).
|
||||
- [x] Classifier contract: `MomentClassification` (Pydantic) + `MomentClassifier` (runtime Protocol) + Gemini/Claude/Local provider stubs.
|
||||
- [x] `plan/CHANGE_REQUESTS.md` created.
|
||||
|
||||
**Done this round:** full M0 + M3 + all frozen contracts. Env: Python 3.12 venv via uv, all
|
||||
spec deps installed. DB schema per spec §2 with CRUD helpers. Timebase helpers in `config.py`.
|
||||
API is Range-capable and CORS-enabled. Frontend scaffolds every layout file (lane-C modules
|
||||
are stubs; `lib/pose.js` is the live frozen contract).
|
||||
|
||||
**Frozen-contract note for lanes:** CORS was widened from a single `:5173` origin to any
|
||||
`http://localhost|127.0.0.1:<port>` (local single-user tool; robust to Vite port fallback).
|
||||
Explicit `:5173` still included. If a lane needs a different API shape/schema/pose math,
|
||||
file a CR in `plan/CHANGE_REQUESTS.md` — do not edit `db.py`/`api.py`/`cli.py`/`config.py`/
|
||||
`synthetic.py`/`pyproject.toml`.
|
||||
|
||||
**Blockers / questions for coordinator:** none.
|
||||
|
||||
**Next:** merge `foundation` → `main`; lanes may start (per directive: C, then A, D, B).
|
||||
38
pyproject.toml
Normal file
38
pyproject.toml
Normal file
@ -0,0 +1,38 @@
|
||||
[project]
|
||||
name = "festival4d"
|
||||
version = "0.1.0"
|
||||
description = "Turn multiple fan-shot smartphone concert videos into a synchronized, explorable 4D experience."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
"fastapi>=0.110",
|
||||
"uvicorn[standard]>=0.29",
|
||||
"numpy>=1.26",
|
||||
"scipy>=1.11",
|
||||
"librosa>=0.10",
|
||||
"soundfile>=0.12",
|
||||
"sqlalchemy>=2.0",
|
||||
"pydantic>=2.6",
|
||||
"opencv-python-headless>=4.9",
|
||||
# classifier providers (lane D) — imported lazily, never at module load
|
||||
"google-genai>=0.3",
|
||||
"anthropic>=0.40",
|
||||
"openai>=1.30",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = ["pytest>=8.0"]
|
||||
|
||||
[project.scripts]
|
||||
festival4d = "festival4d.cli:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["backend/festival4d"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["backend/tests"]
|
||||
addopts = "-q"
|
||||
Loading…
Reference in New Issue
Block a user