Compare commits
No commits in common. "main" and "lane/d-events" have entirely different histories.
main
...
lane/d-eve
@ -1,61 +0,0 @@
|
||||
{
|
||||
"version": "0.0.1",
|
||||
"configurations": [
|
||||
{
|
||||
"name": "festifun-api",
|
||||
"runtimeExecutable": "uv",
|
||||
"runtimeArgs": [
|
||||
"run",
|
||||
"python",
|
||||
"-m",
|
||||
"festival4d",
|
||||
"serve"
|
||||
],
|
||||
"port": 8000,
|
||||
"autoPort": false
|
||||
},
|
||||
{
|
||||
"name": "festifun-web",
|
||||
"runtimeExecutable": "npm",
|
||||
"runtimeArgs": [
|
||||
"run",
|
||||
"dev",
|
||||
"--prefix",
|
||||
"frontend"
|
||||
],
|
||||
"port": 5173,
|
||||
"autoPort": false
|
||||
},
|
||||
{
|
||||
"name": "festifun-web-alt",
|
||||
"runtimeExecutable": "npm",
|
||||
"runtimeArgs": [
|
||||
"run",
|
||||
"dev",
|
||||
"--prefix",
|
||||
"frontend",
|
||||
"--",
|
||||
"--port",
|
||||
"5177",
|
||||
"--strictPort"
|
||||
],
|
||||
"port": 5177,
|
||||
"autoPort": false
|
||||
},
|
||||
{
|
||||
"name": "festifun-web-dist",
|
||||
"runtimeExecutable": "uv",
|
||||
"runtimeArgs": [
|
||||
"run",
|
||||
"python",
|
||||
"-m",
|
||||
"http.server",
|
||||
"5174",
|
||||
"--directory",
|
||||
"/Users/m3ultra/Documents/festifun/frontend/dist"
|
||||
],
|
||||
"port": 5174,
|
||||
"autoPort": false
|
||||
}
|
||||
]
|
||||
}
|
||||
4
.gitignore
vendored
4
.gitignore
vendored
@ -5,7 +5,3 @@ node_modules/
|
||||
__pycache__/
|
||||
.venv/
|
||||
.DS_Store
|
||||
frontend/dist/
|
||||
.env
|
||||
data/capsule/
|
||||
.claude/worktrees/
|
||||
|
||||
234
README.md
234
README.md
@ -3,243 +3,55 @@
|
||||
Turn multiple fan-shot smartphone videos of the same concert into a synchronized,
|
||||
explorable **4D experience**: time-aligned multi-video playback, 3D scene reconstruction
|
||||
with camera poses, a free-roam "god's eye" viewer, AR-style overlays projected onto each
|
||||
video, AI-tagged moments on a shared timeline, click-to-place 3D annotations, and keyframed
|
||||
cinematic fly-throughs.
|
||||
video, and AI-tagged moments on a shared timeline.
|
||||
|
||||
Everything works **from pixels and audio alone** — no depth sensors, no IMU logs. It runs
|
||||
fully offline and local; the only optional cloud piece is moment classification.
|
||||
|
||||
---
|
||||
|
||||
## What you can do
|
||||
|
||||
- **Scrub a shared timeline** and watch every camera stay locked to the same instant, aligned
|
||||
by their audio (no manual frame-matching).
|
||||
- **Fly through the reconstructed scene** in 3D — orbit freely, snap to any real camera, or
|
||||
play a keyframed cinematic path.
|
||||
- **See an "x-ray" HUD**: 3D anchor points (stage corners, tagged objects) projected onto every
|
||||
video, tracking even when occluded by crowd or scaffolding.
|
||||
- **Jump between AI-tagged moments** (bass drops, pyro, crowd waves…) on the timeline, and
|
||||
correct any misclassification.
|
||||
- **Click to locate things in 3D**: draw a box around an object in one or two videos and the
|
||||
app triangulates its 3D position, dropping an anchor that appears everywhere at once.
|
||||
- **Hear the scene in 3D** 🎧: toggle spatial audio and every camera becomes a positional
|
||||
sound source at its reconstructed pose — fly toward a camera and you hear *that* spot in
|
||||
the crowd get closer.
|
||||
|
||||
---
|
||||
> **Status: foundation phase.** The scaffold, database, frozen contracts (API, pose math,
|
||||
> DB schema), and a synthetic fixture generator are in place. Feature lanes (media sync,
|
||||
> reconstruction, viewer, AI events) build on top. The full README with the real-footage
|
||||
> workflow is written in the integration phase (M9). See
|
||||
> [`OPUS_BUILD_INSTRUCTIONS.md`](OPUS_BUILD_INSTRUCTIONS.md) for the canonical spec and
|
||||
> [`plan/`](plan/) for the execution plan.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
| Tool | Required? | Notes |
|
||||
|---|---|---|
|
||||
| **Python 3.11+** + [`uv`](https://docs.astral.sh/uv/) (or venv+pip) | yes | backend |
|
||||
| **ffmpeg** / **ffprobe** on `PATH` | yes | audio extraction, frame sampling, clip cutting |
|
||||
| **Node 18+** | yes | frontend (Vite + Three.js) |
|
||||
| **COLMAP** on `PATH` | optional | 3D reconstruction. Without it the app still runs — you get synced videos + timeline, just no 3D scene. `brew install colmap` on macOS. |
|
||||
| A classifier API key | optional | AI moment labels. Default **Gemini flash** (`GEMINI_API_KEY`; model `gemini-3.1-flash-lite`, override with `FESTIVAL4D_GEMINI_MODEL`); also Claude or any OpenAI-compatible/local/OpenRouter endpoint via `FESTIVAL4D_CLASSIFIER`. Without a key you still get audio-detected candidate moments, just unlabeled. |
|
||||
|
||||
---
|
||||
- **Python 3.11+** and [`uv`](https://docs.astral.sh/uv/) (or venv + pip)
|
||||
- **ffmpeg** / **ffprobe** on your `PATH` (required)
|
||||
- **COLMAP** (optional — reconstruction degrades gracefully without it)
|
||||
- A classifier API key (optional — `GEMINI_API_KEY` by default; see `events_ai.py`)
|
||||
- **Node 18+** for the frontend
|
||||
|
||||
## Quickstart (synthetic demo — no footage needed)
|
||||
|
||||
The synthetic fixture generates three fake camera videos of the same fake stage, with known
|
||||
audio offsets and known 3D geometry — so you can see the whole app working end-to-end before
|
||||
you have any real footage.
|
||||
|
||||
```bash
|
||||
# 1. backend env
|
||||
uv venv --python 3.12
|
||||
uv pip install -e ".[dev]"
|
||||
|
||||
# 2. generate the synthetic project (fake videos + poses + point cloud + seeded events)
|
||||
# 2. generate the synthetic fixture project (fake videos + poses + point cloud + events)
|
||||
uv run python -m festival4d synthetic
|
||||
|
||||
# 3. serve the API (http://127.0.0.1:8000)
|
||||
uv run python -m festival4d serve
|
||||
|
||||
# 4. in another terminal, run the frontend
|
||||
# 4. in another terminal, the frontend
|
||||
cd frontend
|
||||
npm install
|
||||
npm run dev # opens http://localhost:5173 (or the next free port)
|
||||
npm run dev # http://localhost:5173
|
||||
```
|
||||
|
||||
Open the printed URL. You should see the 3D scene with three camera frusta around a stage box,
|
||||
the three synced video players, stage-corner anchors projected onto each video, and colored
|
||||
moment markers on the timeline.
|
||||
|
||||
---
|
||||
|
||||
## Capture: record straight from webcams & phones
|
||||
|
||||
Don't have footage yet? Record it with whatever cameras you have — USB webcams, the laptop cam,
|
||||
and any phone with a browser. **No app install.** Cameras *record* clips that feed the normal
|
||||
pipeline (this isn't live streaming — see `docs/ideas.md` for that).
|
||||
|
||||
This works because of two properties of the design: browser recordings (`.webm`, or `.mp4` on
|
||||
iOS) are already ingestible formats, and **sync is audio-based** — so the devices need **no clock
|
||||
sync and don't even have to start together**. They only have to record the same moment and hear
|
||||
the same sound.
|
||||
|
||||
```bash
|
||||
# capture is OPT-IN (it accepts uploads) — enable it explicitly:
|
||||
FESTIVAL4D_CAPTURE=1 uv run python -m festival4d serve
|
||||
```
|
||||
|
||||
Then open **`/capture`** on each device (a 📹 Capture link also appears in the viewer): name the
|
||||
device, pick a camera + mic, hit record, hit stop — the clip uploads into `data/raw/`. The page
|
||||
shows a live "who's shooting what" monitor of every connected camera. When you're done:
|
||||
|
||||
```bash
|
||||
uv run python -m festival4d ingest && uv run python -m festival4d sync # then reconstruct / events
|
||||
```
|
||||
|
||||
### Getting a phone connected (the HTTPS bit)
|
||||
|
||||
Browsers **only grant camera access on a secure origin**. A phone opening `http://<lan-ip>:8000`
|
||||
gets *no camera*, silently. The easiest fix if you use Tailscale — it issues a real cert, so
|
||||
there's nothing to trust manually:
|
||||
|
||||
```bash
|
||||
tailscale serve --bg 8000 # prints https://<machine>.<tailnet>.ts.net
|
||||
```
|
||||
|
||||
Join the phone to the tailnet and open `https://<machine>.<tailnet>.ts.net/capture`. (Without
|
||||
Tailscale: any HTTPS reverse proxy works; `localhost` is also treated as secure, which is why the
|
||||
laptop's own cameras work with no setup.)
|
||||
|
||||
### Capture tips
|
||||
|
||||
- **Every clip needs audio**, and all cameras must hear the same sound — that's the only thing
|
||||
aligning them. Testing indoors? Play music.
|
||||
- **USB bandwidth is the real limit** on multiple webcams: two 1080p cams on one controller often
|
||||
fails. Drop to 720p or use separate ports/hubs.
|
||||
- Follow the *Shooting tips* below for reconstruction-friendly angles.
|
||||
- Capture routes stay **unmounted** unless `FESTIVAL4D_CAPTURE=1` — a public deployment must
|
||||
never expose an open upload endpoint.
|
||||
|
||||
## Real-footage workflow
|
||||
|
||||
1. **Shoot / gather** 2–4 videos of the same performance from different positions (see
|
||||
*Shooting tips* below) and drop the files into `data/raw/`.
|
||||
2. **Run the pipeline:**
|
||||
|
||||
```bash
|
||||
uv run python -m festival4d ingest # probe videos, extract 16 kHz mono audio
|
||||
uv run python -m festival4d sync # GCC-PHAT audio alignment (offsets + drift)
|
||||
uv run python -m festival4d reconstruct # COLMAP structure-from-motion → camera poses + point cloud
|
||||
uv run python -m festival4d events # detect audio moments + AI-classify them
|
||||
uv run python -m festival4d serve # serve it
|
||||
```
|
||||
|
||||
Each step is independent and re-runnable. `sync` alone already gives you locked multi-video
|
||||
playback; `reconstruct` adds the 3D scene and overlays; `events` adds the tagged timeline.
|
||||
3. **Explore** in the browser (`cd frontend && npm run dev`).
|
||||
|
||||
If COLMAP can't reconstruct your footage (common with dark, motion-blurred, or low-overlap
|
||||
clips), the pipeline says so and leaves you with the synced-video experience — it never
|
||||
corrupts existing data.
|
||||
|
||||
To enable AI moment labels, set a key before `events`:
|
||||
|
||||
```bash
|
||||
export GEMINI_API_KEY=... # default provider (Gemini flash, native video)
|
||||
# or: export FESTIVAL4D_CLASSIFIER=claude ANTHROPIC_API_KEY=...
|
||||
# or: export FESTIVAL4D_CLASSIFIER=local FESTIVAL4D_OPENAI_BASE_URL=... FESTIVAL4D_OPENAI_MODEL=...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Using the viewer
|
||||
|
||||
| Action | How |
|
||||
|---|---|
|
||||
| Play / pause | `Space` or the ⏵ button |
|
||||
| Nudge time ±1 s (±5 s) | `←` / `→` (hold `Shift`) |
|
||||
| Scrub | drag the timeline |
|
||||
| Pick the audio you hear | click a video (one soundtrack at a time) |
|
||||
| Spatial audio | 🎧 **3D audio** (needs poses): every camera plays from its position in the scene; the mix follows the viewer camera as you fly |
|
||||
| Enable/disable a video | ◉ on the video |
|
||||
| Orbit the 3D scene | drag in the 3D pane |
|
||||
| Snap the 3D view to a camera | click its frustum, its ⛶ button, or press `1`–`9` |
|
||||
| Free roam again | `Esc` / `0` / **Free roam** |
|
||||
| Correct a moment / annotate | click a timeline marker → panel |
|
||||
| Place a 3D anchor | in the panel, **Annotate location**, then drag a box on a video (repeat on a 2nd video to triangulate) |
|
||||
| Add a camera keyframe | **＋ Key** or `K` (captures the current free-roam view at the current time) |
|
||||
| Play the keyframed path | **▶ Path** · export/import with ⤓ / ⤒ |
|
||||
|
||||
The top-right **sync error** panel shows each video's alignment error in ms while playing —
|
||||
a dev aid; it should stay green (<50 ms) once footage is well-synced.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
## Backend CLI
|
||||
|
||||
```
|
||||
data/raw/ ──ingest──▶ data/work/audio ──sync────▶ videos.offset_ms / drift_ppm ┐
|
||||
│
|
||||
──frames──▶ data/work/frames ──reconstruct──▶ camera_poses + points.ply
|
||||
│
|
||||
reference audio ──events──▶ events (audio candidates → AI labels)
|
||||
│
|
||||
SQLite (data/project.db) ◀─────────┘
|
||||
│
|
||||
FastAPI (backend/festival4d/api.py)
|
||||
│ /api/manifest, /poses, /pointcloud,
|
||||
│ /events, /anchors, /annotations
|
||||
▼
|
||||
Vite + Three.js SPA (frontend/src)
|
||||
transport (master clock) · videoGrid + overlays · scene3d · timeline
|
||||
annotate (M8) · camPath (M9)
|
||||
python -m festival4d synthetic # generate the synthetic fixture (M0)
|
||||
python -m festival4d ingest # probe videos + extract audio (lane A / M1)
|
||||
python -m festival4d sync # GCC-PHAT audio alignment (lane A / M1)
|
||||
python -m festival4d reconstruct # COLMAP SfM + pose export (lane B / M2)
|
||||
python -m festival4d events # audio candidates + AI classify (lane D / M7)
|
||||
python -m festival4d serve # FastAPI app (M3)
|
||||
```
|
||||
|
||||
- **Backend** — Python: `ingest`/`audio_sync` (media + GCC-PHAT sync), `frames`/`sfm`/`geometry`
|
||||
(COLMAP orchestration + pose math + triangulation), `events_ai` (moment detection + pluggable
|
||||
classifiers), `resolve` (annotation→3D), `api` (FastAPI, Range-capable video serving), `db`
|
||||
(SQLAlchemy/SQLite), `synthetic` (the test fixture).
|
||||
- **Frontend** — vanilla JS + Three.js. A single master clock (`transport.js`) derives each
|
||||
video's local time from the shared timeline and continuously corrects playback. When possible
|
||||
the clock is **audio**: the selected soundtrack (`GET /api/audio/{id}`, extracted+cached
|
||||
server-side) plays through WebAudio and `t_global` derives from `AudioContext.currentTime` —
|
||||
so every video is muted picture-only and can be seeked/rate-trimmed with zero audible
|
||||
artifacts (falls back to a `performance.now()` clock + `<video>` audio if the track can't
|
||||
load). `lib/pose.js` is the one COLMAP→Three.js pose conversion, mirrored from `geometry.py`
|
||||
and locked by a shared test.
|
||||
- **Timebase contract:** `t_video = (t_global − offset_ms/1000) · (1 + drift_ppm·1e−6)`; the
|
||||
reference video has offset 0.
|
||||
|
||||
Full design + milestone history: [`OPUS_BUILD_INSTRUCTIONS.md`](OPUS_BUILD_INSTRUCTIONS.md);
|
||||
the parallel build plan and per-lane status live in [`plan/`](plan/).
|
||||
|
||||
---
|
||||
|
||||
## Shooting tips (for good reconstructions)
|
||||
|
||||
Concert footage is genuinely hard for structure-from-motion. To give COLMAP a chance:
|
||||
|
||||
- **Spread the cameras out** but keep **overlapping views** — each pair of cameras should see
|
||||
some of the same stage/structure. No overlap → they can't be related in 3D.
|
||||
- **Keep some static structure in frame** (stage edges, truss, speaker stacks). A frame that's
|
||||
all moving crowd and lights has nothing stable to triangulate.
|
||||
- **Avoid pure zoom** — physically moving parallax reconstructs far better than zooming.
|
||||
- **Brighter, sharper is better** — motion blur and near-dark frames are the main failure cause.
|
||||
- **2–4 phones is the sweet spot** for a first reconstruction; more is fine but slower.
|
||||
|
||||
Audio sync is far more forgiving: any clips that share audible sound (the same music/claps)
|
||||
will align, even across otherwise-unrelated angles.
|
||||
|
||||
---
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
uv run pytest # backend
|
||||
cd frontend && npm run build # frontend typecheck/build
|
||||
uv run pytest
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Non-goals (this prototype)
|
||||
|
||||
No accounts/auth, no cloud storage, no realtime ingest, no NeRF/Gaussian splatting, no mobile
|
||||
UI, no Docker. Single local user, one project at a time.
|
||||
|
||||
@ -12,28 +12,6 @@ Serves the synthetic project fully so lane C can treat this API as finished:
|
||||
POST /api/events/detect {t_global_s?, window_s?} -> runs M7 detection
|
||||
POST /api/annotations {video_id, t_video_s, bbox:[x0,y0,x1,y1]} -> {anchor_id?, point?}
|
||||
|
||||
Phase 5 (foundation2; shapes frozen in plan/20-phase5.md contracts):
|
||||
|
||||
GET /api/beats -> beats.json body (contract #1); 404 when not yet analyzed
|
||||
POST /api/director {top_n?, lead_s?} -> camPath JSON (contract #2); while lane E
|
||||
is stubbed, degrades to a VALID empty path + "note"
|
||||
GET /api/paths -> [{id, name, created_at}]
|
||||
GET /api/paths/{id} -> {id, name, created_at, json: <parsed camPath JSON>}
|
||||
POST /api/paths {name, json: <camPath JSON as text>} (validated -> 422)
|
||||
DELETE /api/paths/{id}
|
||||
PATCH /api/anchors/{id} {label?, color?} -> updated anchor dict
|
||||
|
||||
Phase 6 (foundation3 / M18; shapes frozen in plan/30-phase6.md contract #2):
|
||||
|
||||
GET /api/tracks -> [{id, marker_key, label, color, points:[{t_global_s,x,y,z,
|
||||
quality,views}]}] (points ordered by t_global_s)
|
||||
PATCH /api/tracks/{id} {label?, color?} -> updated track dict (404 on missing)
|
||||
DELETE /api/tracks/{id} -> {deleted: id} (404 on missing)
|
||||
POST /api/tracks/solve -> {result, tracks[, note]} — runs lane H's detect->solve
|
||||
(tracker_solve.run_solve); while lane H is stubbed it
|
||||
degrades to a VALID empty result + "note" (house pattern).
|
||||
manifest gains "has_tracks": bool.
|
||||
|
||||
Source videos in ``data/raw`` are mounted at ``/media`` via Starlette ``StaticFiles``,
|
||||
which supports HTTP Range (required for ``<video>`` seeking; verify ``curl -H "Range:
|
||||
bytes=0-100"`` -> 206). Stubbed lane entrypoints (events detection, M8 annotation
|
||||
@ -74,14 +52,6 @@ config.RAW_DIR.mkdir(parents=True, exist_ok=True)
|
||||
# HTTP Range-capable static serving of source videos (spec M3, pitfall #3).
|
||||
app.mount("/media", StaticFiles(directory=config.RAW_DIR), name="media")
|
||||
|
||||
# Live capture (webcams / phones -> data/raw). OPT-IN: these routes accept file uploads, so they
|
||||
# stay unmounted unless FESTIVAL4D_CAPTURE=1 — a public deployment must never expose them.
|
||||
from festival4d import capture # noqa: E402 (import after app/config are ready)
|
||||
|
||||
if capture.capture_enabled():
|
||||
app.include_router(capture.router)
|
||||
log.info("capture routes ENABLED (/capture, /api/capture/*) — FESTIVAL4D_CAPTURE is set")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Request models
|
||||
@ -103,35 +73,6 @@ class AnnotationIn(BaseModel):
|
||||
video_id: int
|
||||
t_video_s: float
|
||||
bbox: list[float] = Field(min_length=4, max_length=4) # [x0, y0, x1, y1] normalized
|
||||
event_id: int | None = None # CR-2: link to the event this annotation locates (M8)
|
||||
|
||||
|
||||
class EventPatch(BaseModel):
|
||||
event_type: str | None = None
|
||||
source: str | None = None
|
||||
description: str | None = None
|
||||
confidence: float | None = None
|
||||
|
||||
|
||||
class AnchorPatch(BaseModel):
|
||||
label: str | None = None
|
||||
color: str | None = None
|
||||
|
||||
|
||||
class DirectorIn(BaseModel):
|
||||
top_n: int = 8
|
||||
lead_s: float = 2.0
|
||||
|
||||
|
||||
class PathIn(BaseModel):
|
||||
# Wire key is "json" (contract #3); aliased because `json` shadows a BaseModel attr.
|
||||
name: str
|
||||
path_json: str = Field(alias="json")
|
||||
|
||||
|
||||
class TrackPatch(BaseModel):
|
||||
label: str | None = None
|
||||
color: str | None = None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@ -178,24 +119,6 @@ def _event_dict(e) -> dict:
|
||||
}
|
||||
|
||||
|
||||
def _track_dict(t) -> dict:
|
||||
"""A track with its points inlined (phase 6 contract #2), ordered by t_global_s."""
|
||||
return {
|
||||
"id": t.id,
|
||||
"marker_key": t.marker_key,
|
||||
"label": t.label,
|
||||
"color": t.color,
|
||||
"points": [
|
||||
{
|
||||
"t_global_s": p.t_global_s,
|
||||
"x": p.x, "y": p.y, "z": p.z,
|
||||
"quality": p.quality, "views": p.views,
|
||||
}
|
||||
for p in db.get_track_points(t.id)
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Routes
|
||||
# ---------------------------------------------------------------------------
|
||||
@ -214,14 +137,7 @@ def manifest() -> dict:
|
||||
return {
|
||||
"videos": [_video_dict(v) for v in videos],
|
||||
"t_global_max": t_global_max,
|
||||
"has_capture": capture.capture_enabled(),
|
||||
"has_poses": db.has_poses(),
|
||||
"has_splat": config.SPLAT_PLY.exists(),
|
||||
"has_tracks": db.has_tracks(), # phase 6 (M18): any solved friend track ⇒ frontend loads them
|
||||
|
||||
# Always False from the live server; the capsule (M17) bakes a manifest with true,
|
||||
# which makes the frontend hide all write UI (phase-5 contract #5).
|
||||
"capsule": False,
|
||||
}
|
||||
|
||||
|
||||
@ -243,51 +159,6 @@ def pointcloud() -> FileResponse:
|
||||
)
|
||||
|
||||
|
||||
@app.get("/api/splat")
|
||||
def splat() -> FileResponse:
|
||||
"""Optional 3DGS splat of the scene (see docs/modelbeast-crossover.md).
|
||||
|
||||
Trained externally from the same COLMAP reconstruction (e.g. MODELBEAST's
|
||||
brush_train) and dropped at ``data/work/splat.ply``. The viewer prefers it
|
||||
over the sparse point cloud when present.
|
||||
"""
|
||||
if not config.SPLAT_PLY.exists():
|
||||
raise HTTPException(status_code=404, detail="no splat (train one — see docs/modelbeast-crossover.md)")
|
||||
return FileResponse(
|
||||
config.SPLAT_PLY,
|
||||
media_type="application/octet-stream",
|
||||
filename="splat.ply",
|
||||
)
|
||||
|
||||
|
||||
@app.get("/api/audio/{video_id}")
|
||||
def video_audio(video_id: int) -> FileResponse:
|
||||
"""Listening-quality audio track of a video (stereo AAC), for the WebAudio master clock.
|
||||
|
||||
Extracted lazily from the source video via ffmpeg on first request and cached in
|
||||
``data/work/audio_hq/`` (re-extracted if the source file is newer). The frontend
|
||||
decodes this whole file with ``decodeAudioData`` and derives ``t_global`` from
|
||||
``AudioContext.currentTime`` — see ``frontend/src/transport.js``.
|
||||
"""
|
||||
from festival4d import ingest
|
||||
import subprocess
|
||||
|
||||
v = db.get_video(video_id)
|
||||
if v is None:
|
||||
raise HTTPException(status_code=404, detail=f"no video with id={video_id}")
|
||||
src = config.RAW_DIR / v.filename
|
||||
if not src.exists():
|
||||
raise HTTPException(status_code=404, detail=f"source file missing: {v.filename}")
|
||||
out = config.AUDIO_HQ_DIR / f"{video_id}.m4a"
|
||||
if not out.exists() or out.stat().st_mtime < src.stat().st_mtime:
|
||||
try:
|
||||
ingest.extract_audio_hq(src, out)
|
||||
except (subprocess.CalledProcessError, RuntimeError) as exc:
|
||||
# No audio stream / no ffmpeg — the frontend falls back to <video> audio.
|
||||
raise HTTPException(status_code=404, detail=f"no extractable audio: {exc}") from exc
|
||||
return FileResponse(out, media_type="audio/mp4", filename=out.name)
|
||||
|
||||
|
||||
@app.get("/api/anchors")
|
||||
def get_anchors() -> list[dict]:
|
||||
return [_anchor_dict(a) for a in db.get_anchors()]
|
||||
@ -320,233 +191,19 @@ def detect_events(payload: DetectIn) -> dict:
|
||||
return {"result": result, "events": [_event_dict(e) for e in db.get_events()]}
|
||||
|
||||
|
||||
@app.patch("/api/events/{event_id}")
|
||||
def patch_event(event_id: int, payload: EventPatch) -> dict:
|
||||
"""Correct an event in place (M8). A user correction defaults ``source`` to ``'user'``."""
|
||||
if db.get_event(event_id) is None:
|
||||
raise HTTPException(status_code=404, detail=f"no event with id={event_id}")
|
||||
source = payload.source
|
||||
if source is None and (payload.event_type is not None or payload.description is not None):
|
||||
source = "user" # any user-driven correction is attributed to the user
|
||||
event = db.update_event(
|
||||
event_id,
|
||||
event_type=payload.event_type,
|
||||
source=source,
|
||||
description=payload.description,
|
||||
confidence=payload.confidence,
|
||||
)
|
||||
return _event_dict(event)
|
||||
|
||||
|
||||
@app.post("/api/annotations")
|
||||
def create_annotation(payload: AnnotationIn) -> dict:
|
||||
from festival4d import resolve
|
||||
|
||||
x0, y0, x1, y1 = payload.bbox
|
||||
if db.get_video(payload.video_id) is None:
|
||||
raise HTTPException(status_code=404, detail=f"no video with id={payload.video_id}")
|
||||
annotation = db.add_annotation(
|
||||
payload.video_id, payload.t_video_s, x0, y0, x1, y1, event_id=payload.event_id
|
||||
)
|
||||
|
||||
# M8: cast a ray through the bbox center; triangulate against another view of the same
|
||||
# event, else fall back to the nearest point-cloud point / centroid depth.
|
||||
result = resolve.resolve_annotation(
|
||||
payload.video_id, payload.t_video_s, payload.bbox, event_id=payload.event_id
|
||||
)
|
||||
|
||||
anchor = None
|
||||
superseded = False
|
||||
if result.point is not None:
|
||||
px, py, pz = result.point
|
||||
label = "annotation"
|
||||
if payload.event_id is not None:
|
||||
ev = db.get_event(payload.event_id)
|
||||
if ev is not None:
|
||||
label = ev.description or ev.event_type or label
|
||||
|
||||
# One anchor per event: the first annotation creates it (a single-view fallback); a later
|
||||
# view that TRIANGULATES supersedes the anchor in place rather than dropping a sibling. A
|
||||
# later fallback links to the existing anchor without moving it (don't regress a good
|
||||
# triangulation). Annotations with no event stay independent (each makes its own anchor).
|
||||
existing_id = _existing_event_anchor_id(payload.event_id) if payload.event_id else None
|
||||
if existing_id is None:
|
||||
anchor = db.add_anchor(label, px, py, pz, None)
|
||||
elif result.method == "triangulated":
|
||||
anchor = db.update_anchor(existing_id, px, py, pz, label=label)
|
||||
superseded = True
|
||||
else:
|
||||
anchor = db.get_anchor(existing_id)
|
||||
if anchor is not None:
|
||||
db.set_annotation_anchor(annotation.id, anchor.id)
|
||||
annotation = db.add_annotation(payload.video_id, payload.t_video_s, x0, y0, x1, y1)
|
||||
|
||||
# M8 resolution (triangulate / nearest-point) is integration work; geometry stubs raise
|
||||
# until lane B lands. Store the annotation now; resolve to an anchor later.
|
||||
anchor_id = None
|
||||
point = None
|
||||
note = "annotation stored; 3D resolution is M8/integration work"
|
||||
return {
|
||||
"annotation_id": annotation.id,
|
||||
"anchor_id": anchor.id if anchor else None,
|
||||
"point": result.point,
|
||||
"method": result.method,
|
||||
"gap": result.gap,
|
||||
"superseded": superseded,
|
||||
"anchor": _anchor_dict(anchor) if anchor else None,
|
||||
"anchor_id": anchor_id,
|
||||
"point": point,
|
||||
"note": note,
|
||||
}
|
||||
|
||||
|
||||
def _existing_event_anchor_id(event_id: int) -> int | None:
|
||||
"""The anchor resolved from a prior annotation of this event (most recent), or None."""
|
||||
for ann in reversed(db.get_annotations(event_id=event_id)):
|
||||
if ann.resolved_anchor_id is not None:
|
||||
return ann.resolved_anchor_id
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 5 (foundation2). Shapes are frozen in plan/20-phase5.md "Contracts".
|
||||
# ---------------------------------------------------------------------------
|
||||
@app.get("/api/beats")
|
||||
def get_beats() -> dict:
|
||||
"""The beats.json analysis (contract #1). 404 until `python -m festival4d features` runs."""
|
||||
import json
|
||||
|
||||
if not config.BEATS_JSON.exists():
|
||||
raise HTTPException(status_code=404, detail="no beat analysis (run `python -m festival4d features`)")
|
||||
return json.loads(config.BEATS_JSON.read_text())
|
||||
|
||||
|
||||
@app.post("/api/director")
|
||||
def run_director(payload: DirectorIn) -> dict:
|
||||
"""Auto-director (M11) -> camPath JSON. Degrades to a valid EMPTY path while stubbed,
|
||||
so the response always round-trips through camPath.fromJSON (contract #2)."""
|
||||
from festival4d import director
|
||||
|
||||
try:
|
||||
return director.generate_path(top_n=payload.top_n, lead_s=payload.lead_s)
|
||||
except NotImplementedError as exc:
|
||||
log.info("auto-director not implemented yet: %s", exc)
|
||||
return {"version": 1, "keyframes": [], "note": "auto-director not implemented yet (lane E / M11)"}
|
||||
|
||||
|
||||
def _validate_campath(text: str) -> None:
|
||||
"""422 unless ``text`` is the frozen camPath JSON (contract #2). Lane G owns hardening."""
|
||||
import json
|
||||
|
||||
try:
|
||||
data = json.loads(text)
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=422, detail="path json is not valid JSON")
|
||||
ok = (
|
||||
isinstance(data, dict)
|
||||
and data.get("version") == 1
|
||||
and isinstance(data.get("keyframes"), list)
|
||||
and all(
|
||||
isinstance(k, dict)
|
||||
and isinstance(k.get("t_global"), (int, float))
|
||||
and isinstance(k.get("pos"), list) and len(k["pos"]) == 3
|
||||
and isinstance(k.get("quat"), list) and len(k["quat"]) == 4
|
||||
and isinstance(k.get("fov"), (int, float))
|
||||
for k in data.get("keyframes", [])
|
||||
)
|
||||
)
|
||||
if not ok:
|
||||
raise HTTPException(status_code=422, detail="not a camPath JSON (see plan/20-phase5.md contract #2)")
|
||||
|
||||
|
||||
def _path_dict(p, with_json: bool = False) -> dict:
|
||||
import json
|
||||
|
||||
d = {"id": p.id, "name": p.name, "created_at": p.created_at}
|
||||
if with_json:
|
||||
d["json"] = json.loads(p.path_json)
|
||||
return d
|
||||
|
||||
|
||||
@app.get("/api/paths")
|
||||
def get_paths() -> list[dict]:
|
||||
return [_path_dict(p) for p in db.get_paths()]
|
||||
|
||||
|
||||
@app.get("/api/paths/{path_id}")
|
||||
def get_path(path_id: int) -> dict:
|
||||
p = db.get_path(path_id)
|
||||
if p is None:
|
||||
raise HTTPException(status_code=404, detail=f"no path with id={path_id}")
|
||||
return _path_dict(p, with_json=True)
|
||||
|
||||
|
||||
@app.post("/api/paths")
|
||||
def create_path(payload: PathIn) -> dict:
|
||||
_validate_campath(payload.path_json)
|
||||
return _path_dict(db.add_path(payload.name, payload.path_json), with_json=True)
|
||||
|
||||
|
||||
@app.delete("/api/paths/{path_id}")
|
||||
def delete_path(path_id: int) -> dict:
|
||||
if not db.delete_path(path_id):
|
||||
raise HTTPException(status_code=404, detail=f"no path with id={path_id}")
|
||||
return {"deleted": path_id}
|
||||
|
||||
|
||||
@app.patch("/api/anchors/{anchor_id}")
|
||||
def patch_anchor(anchor_id: int, payload: AnchorPatch) -> dict:
|
||||
"""Rename / recolor an anchor in place (M13, contract #4)."""
|
||||
try:
|
||||
anchor = db.patch_anchor(anchor_id, label=payload.label, color=payload.color)
|
||||
except KeyError:
|
||||
raise HTTPException(status_code=404, detail=f"no anchor with id={anchor_id}")
|
||||
return _anchor_dict(anchor)
|
||||
|
||||
|
||||
@app.delete("/api/anchors/{anchor_id}")
|
||||
def delete_anchor(anchor_id: int) -> dict:
|
||||
"""Remove an anchor (M8 / Phase 4). Unlinks any annotations that resolved to it."""
|
||||
if not db.delete_anchor(anchor_id):
|
||||
raise HTTPException(status_code=404, detail=f"no anchor with id={anchor_id}")
|
||||
return {"deleted": anchor_id}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 6 (foundation3 / M18). Friend tracks. Shapes frozen in plan/30-phase6.md contract #2.
|
||||
# ---------------------------------------------------------------------------
|
||||
@app.get("/api/tracks")
|
||||
def get_tracks() -> list[dict]:
|
||||
"""All friend tracks with their points inlined (contract #2)."""
|
||||
return [_track_dict(t) for t in db.get_tracks()]
|
||||
|
||||
|
||||
@app.patch("/api/tracks/{track_id}")
|
||||
def patch_track(track_id: int, payload: TrackPatch) -> dict:
|
||||
"""Rename / recolor a track in place (M21 friends panel)."""
|
||||
try:
|
||||
track = db.patch_track(track_id, label=payload.label, color=payload.color)
|
||||
except KeyError:
|
||||
raise HTTPException(status_code=404, detail=f"no track with id={track_id}")
|
||||
return _track_dict(track)
|
||||
|
||||
|
||||
@app.delete("/api/tracks/{track_id}")
|
||||
def delete_track(track_id: int) -> dict:
|
||||
"""Delete a track and its points (M21)."""
|
||||
if not db.delete_track(track_id):
|
||||
raise HTTPException(status_code=404, detail=f"no track with id={track_id}")
|
||||
return {"deleted": track_id}
|
||||
|
||||
|
||||
@app.post("/api/tracks/solve")
|
||||
def solve_tracks() -> dict:
|
||||
"""Run marker detection + track solving (M19+M20) and return the resulting tracks.
|
||||
|
||||
Dispatches into lane H's ``tracker_solve.run_solve`` (which itself consumes
|
||||
``tracker_detect``'s detections). While lane H is stubbed both raise
|
||||
``NotImplementedError``; we catch it and degrade to a VALID empty result + a note, so the
|
||||
frozen route never 500s — the house pattern (mirrors ``POST /api/events/detect``)."""
|
||||
from festival4d import tracker_solve
|
||||
|
||||
try:
|
||||
result = tracker_solve.run_solve()
|
||||
except NotImplementedError as exc:
|
||||
log.info("track solving not implemented yet: %s", exc)
|
||||
return {
|
||||
"result": None,
|
||||
"note": "friend-track solving not implemented yet (lane H / M19-M20)",
|
||||
"tracks": [_track_dict(t) for t in db.get_tracks()],
|
||||
}
|
||||
return {"result": result, "tracks": [_track_dict(t) for t in db.get_tracks()]}
|
||||
|
||||
@ -1,187 +0,0 @@
|
||||
"""Audio features: beats & onsets (spec M10, lane E).
|
||||
|
||||
Contract (frozen at foundation2 merge — phase-5 contract #1):
|
||||
|
||||
:func:`run_features` loads the **reference** video's ingest WAV (16 kHz mono from
|
||||
``config.AUDIO_DIR``; the reference timeline *is* ``t_global``), runs beat/tempo tracking
|
||||
and onset-strength detection (librosa is already a dependency), and writes
|
||||
``config.BEATS_JSON``::
|
||||
|
||||
{"tempo_bpm": float | null,
|
||||
"beats_s": [float, ...],
|
||||
"onsets": [{"t_global_s": float, "strength": float 0..1}, ...],
|
||||
"generated_by": "festival4d.audio_features"}
|
||||
|
||||
All times are ``t_global`` seconds. Quiet / short / beatless audio must produce a
|
||||
*valid, empty* result (``tempo_bpm: null``, empty lists) — never a crash. Returns a
|
||||
small summary dict for the CLI log. ``GET /api/beats`` serves the written file.
|
||||
|
||||
Implementation notes
|
||||
--------------------
|
||||
- Analysis runs on the reference's local timeline and maps to ``t_global`` via the frozen
|
||||
:func:`config.t_global_from_video` (identity for a true reference: offset 0, drift 0;
|
||||
correct either way if we ever fall back to a non-zero-offset "reference").
|
||||
- ``HOP_LENGTH = 160`` (10 ms at 16 kHz): librosa's default 512-sample hop is a 32 ms
|
||||
grid at this rate, which quantizes beats past the ±50 ms acceptance bound. 10 ms frames
|
||||
recover the synthetic 120 BPM pulse grid to ~30 ms / ±0.001 BPM (see test_audio_features).
|
||||
- Reported ``tempo_bpm`` is the median inter-beat interval of the tracked beats (more
|
||||
accurate than the tracker's coarse tempo prior) and ``null`` when < 2 beats are found.
|
||||
- Missing prerequisites (no videos ingested / WAV absent) degrade to a summary dict with a
|
||||
``note`` and write nothing — mirroring ``events_ai.run_events`` house style; degenerate
|
||||
*content* (silence, too short, beatless) writes the valid-empty JSON above.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import wave
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
||||
from festival4d import config, db
|
||||
|
||||
log = logging.getLogger("festival4d.audio_features")
|
||||
|
||||
GENERATED_BY = "festival4d.audio_features"
|
||||
HOP_LENGTH = 160 # analysis hop in samples: 10 ms at the 16 kHz ingest rate
|
||||
MIN_DURATION_S = 1.0 # anything shorter is "no usable audio" -> valid-empty result
|
||||
SILENCE_PEAK = 1e-4 # |peak| below this is treated as silence -> valid-empty result
|
||||
|
||||
|
||||
def _empty_result() -> dict:
|
||||
"""The valid-empty beats.json body (contract #1) for silent/short/beatless audio."""
|
||||
return {
|
||||
"tempo_bpm": None,
|
||||
"beats_s": [],
|
||||
"onsets": [],
|
||||
"generated_by": GENERATED_BY,
|
||||
}
|
||||
|
||||
|
||||
def _select_reference(videos: list) -> "object | None":
|
||||
"""The reference video (spec §2: offset 0). Fall back to the smallest |offset|, else first."""
|
||||
if not videos:
|
||||
return None
|
||||
exact = [v for v in videos if v.offset_ms == 0]
|
||||
if exact:
|
||||
return exact[0]
|
||||
with_offset = [v for v in videos if v.offset_ms is not None]
|
||||
if with_offset:
|
||||
return min(with_offset, key=lambda v: abs(v.offset_ms))
|
||||
return videos[0]
|
||||
|
||||
|
||||
def _read_wav_mono(path: Path) -> tuple[np.ndarray, int]:
|
||||
"""Read a 16-bit PCM WAV (the ingest format) as float64 mono in [-1, 1]."""
|
||||
with wave.open(str(path), "rb") as w:
|
||||
sr = w.getframerate()
|
||||
channels = w.getnchannels()
|
||||
frames = w.readframes(w.getnframes())
|
||||
pcm = np.frombuffer(frames, dtype="<i2").astype(np.float64) / 32767.0
|
||||
if channels > 1:
|
||||
pcm = pcm.reshape(-1, channels).mean(axis=1)
|
||||
return pcm, sr
|
||||
|
||||
|
||||
def analyze(y: np.ndarray, sr: int) -> dict:
|
||||
"""Beat/tempo + onset-strength analysis of a mono signal on its *local* timeline.
|
||||
|
||||
Returns the contract #1 dict with times in local (signal) seconds — the caller maps
|
||||
them to ``t_global``. Degenerate input (short/silent/beatless) returns the valid-empty
|
||||
shape; this function never raises on quiet or short audio.
|
||||
"""
|
||||
import librosa # deferred: keeps `festival4d` import light for the API/CLI
|
||||
|
||||
y = np.asarray(y, dtype=np.float32).reshape(-1)
|
||||
if len(y) < MIN_DURATION_S * sr or float(np.max(np.abs(y), initial=0.0)) < SILENCE_PEAK:
|
||||
return _empty_result()
|
||||
|
||||
try:
|
||||
onset_env = librosa.onset.onset_strength(y=y, sr=sr, hop_length=HOP_LENGTH)
|
||||
_tempo, beat_frames = librosa.beat.beat_track(
|
||||
onset_envelope=onset_env, sr=sr, hop_length=HOP_LENGTH
|
||||
)
|
||||
beat_times = librosa.frames_to_time(beat_frames, sr=sr, hop_length=HOP_LENGTH)
|
||||
|
||||
onset_frames = librosa.onset.onset_detect(
|
||||
onset_envelope=onset_env, sr=sr, hop_length=HOP_LENGTH
|
||||
)
|
||||
onset_times = librosa.frames_to_time(onset_frames, sr=sr, hop_length=HOP_LENGTH)
|
||||
except Exception as exc: # librosa internals on weird signals: degrade, never crash
|
||||
log.warning("audio_features: analysis failed (%s) — returning empty result", exc)
|
||||
return _empty_result()
|
||||
|
||||
beats = [float(t) for t in np.atleast_1d(beat_times)]
|
||||
if len(beats) >= 2:
|
||||
tempo_bpm: float | None = float(60.0 / np.median(np.diff(beats)))
|
||||
elif beats:
|
||||
tempo_bpm = float(np.atleast_1d(_tempo)[0]) or None
|
||||
else:
|
||||
tempo_bpm = None
|
||||
|
||||
env_max = float(onset_env.max()) if len(onset_env) else 0.0
|
||||
onsets = []
|
||||
if env_max > 0:
|
||||
for frame, t in zip(np.atleast_1d(onset_frames), np.atleast_1d(onset_times)):
|
||||
strength = float(onset_env[int(frame)]) / env_max
|
||||
onsets.append({"t_global_s": float(t), "strength": min(1.0, max(0.0, strength))})
|
||||
|
||||
return {
|
||||
"tempo_bpm": tempo_bpm,
|
||||
"beats_s": beats,
|
||||
"onsets": onsets,
|
||||
"generated_by": GENERATED_BY,
|
||||
}
|
||||
|
||||
|
||||
def run_features() -> dict:
|
||||
"""Analyze the reference soundtrack and write ``data/work/beats.json`` (see module doc)."""
|
||||
videos = db.get_videos()
|
||||
reference = _select_reference(videos)
|
||||
if reference is None:
|
||||
log.warning("audio_features: no videos in the project — run ingest first")
|
||||
return {"status": "no_videos", "note": "no videos in the project (run ingest first)"}
|
||||
|
||||
wav_path = config.AUDIO_DIR / f"{reference.id}.wav" # ingest.audio_wav_path convention
|
||||
if not wav_path.exists():
|
||||
log.warning("audio_features: reference WAV missing at %s — run ingest first", wav_path)
|
||||
return {
|
||||
"status": "no_audio",
|
||||
"note": f"reference audio not found ({wav_path.name}) — run `python -m festival4d ingest`",
|
||||
}
|
||||
|
||||
y, sr = _read_wav_mono(wav_path)
|
||||
result = analyze(y, sr)
|
||||
|
||||
# Map local (reference) times onto the master timeline via the frozen timebase helpers.
|
||||
# For a true reference (offset 0, drift 0) this is the identity.
|
||||
offset_ms = reference.offset_ms or 0.0
|
||||
drift_ppm = reference.drift_ppm or 0.0
|
||||
if offset_ms != 0.0 or drift_ppm != 0.0:
|
||||
result["beats_s"] = [
|
||||
float(config.t_global_from_video(t, offset_ms, drift_ppm)) for t in result["beats_s"]
|
||||
]
|
||||
for onset in result["onsets"]:
|
||||
onset["t_global_s"] = float(
|
||||
config.t_global_from_video(onset["t_global_s"], offset_ms, drift_ppm)
|
||||
)
|
||||
|
||||
config.BEATS_JSON.parent.mkdir(parents=True, exist_ok=True)
|
||||
config.BEATS_JSON.write_text(json.dumps(result, indent=2))
|
||||
log.info(
|
||||
"audio_features: %s beats (tempo %s BPM), %s onsets -> %s",
|
||||
len(result["beats_s"]),
|
||||
f"{result['tempo_bpm']:.1f}" if result["tempo_bpm"] else "null",
|
||||
len(result["onsets"]),
|
||||
config.BEATS_JSON,
|
||||
)
|
||||
return {
|
||||
"status": "ok",
|
||||
"reference_video_id": reference.id,
|
||||
"tempo_bpm": result["tempo_bpm"],
|
||||
"beats": len(result["beats_s"]),
|
||||
"onsets": len(result["onsets"]),
|
||||
"beats_json": str(config.BEATS_JSON),
|
||||
}
|
||||
@ -1,349 +1,60 @@
|
||||
"""GCC-PHAT pairwise audio offsets + global graph solve + drift (spec M1, lane A).
|
||||
|
||||
Algorithm (spec M1):
|
||||
STUB — lane A fills the bodies; the public signatures below are frozen (``cli.py`` calls
|
||||
``run_sync``).
|
||||
|
||||
Algorithm (spec M1, for lane A's reference):
|
||||
- **GCC-PHAT**, not raw cross-correlation: whiten the cross-spectrum
|
||||
``R = X·conj(Y); R /= |R| + eps; r = irfft(R)`` — robust to loud/clipped/reverberant
|
||||
concert audio. numpy FFTs; librosa only for loading (in :func:`run_sync`).
|
||||
``R = X * conj(Y); R /= |R| + eps; r = irfft(R)`` — robust to loud/clipped/reverberant
|
||||
concert audio. numpy/scipy FFTs; librosa only for loading.
|
||||
- All **pairwise** offsets, each scored by peak-to-second-peak ratio; then a global
|
||||
least-squares solve over the offset graph, weighted by confidence, rejecting pairs that
|
||||
violate cycle consistency (``|off_ab + off_bc − off_ac| > 50 ms``, generalized here as a
|
||||
residual test against the fitted global solution).
|
||||
- **Drift**: re-run GCC-PHAT on 10 s windows every 30 s along the overlap; fit a line to
|
||||
``delay(t)``; store the slope (dimensionless rate) as ``drift_ppm`` (store 0 if < 5 ppm).
|
||||
- Disconnected sync-graph components → ``offset_ms = None`` (never a guess; spec pitfall #5).
|
||||
violate cycle consistency (``off_ab + off_bc - off_ac > 50 ms``).
|
||||
- **Drift**: re-run GCC-PHAT on 10 s windows every 30 s along the overlap; fit a line;
|
||||
store the slope as ``drift_ppm`` (store 0 if ``|drift| < 5 ppm``).
|
||||
- Disconnected sync-graph components -> ``offset_ms = None`` (never a guess; spec pitfall #5).
|
||||
- Persist via ``db.update_video_sync`` and export ``config.SYNC_JSON``.
|
||||
|
||||
Sign conventions (kept consistent with ``config``'s timebase):
|
||||
|
||||
A video *v*'s audio sample at local time ``τ`` equals the master signal at global time
|
||||
``offset_s + τ`` (drift 0). So for two videos ``a``/``b``:
|
||||
``offset_a − offset_b = −gcc_phat(a, b)`` seconds, where :func:`gcc_phat` returns the
|
||||
delay of its first argument relative to its second. Under drift ``d`` (= ``drift_ppm·1e−6``),
|
||||
the windowed delay of ``v`` vs the reference is ``d·t − offset_s`` — a line whose slope is
|
||||
the drift, recovered by :func:`estimate_drift`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
|
||||
import numpy as np
|
||||
|
||||
from festival4d import config, db
|
||||
from festival4d.ingest import audio_wav_path
|
||||
|
||||
log = logging.getLogger("festival4d.audio_sync")
|
||||
|
||||
_EPS = 1e-10
|
||||
# Edges weaker than this peak-to-second-peak ratio carry no reliable alignment
|
||||
# (flat/ambiguous correlation) and are dropped before the global solve.
|
||||
_MIN_EDGE_CONFIDENCE = 1.15
|
||||
# A pairwise offset whose residual against the fitted global solution exceeds this is a
|
||||
# cycle-consistency violation (spec M1); it is rejected and the component is re-solved.
|
||||
_CYCLE_TOLERANCE_MS = 50.0
|
||||
# Drift smaller than this magnitude is stored as exactly 0 (spec M1).
|
||||
_DRIFT_DEADBAND_PPM = 5.0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Core correlation
|
||||
# ---------------------------------------------------------------------------
|
||||
def _peak_ratio(env: np.ndarray, peak: int, sr: int, guard_s: float = 0.004) -> float:
|
||||
"""Peak-to-second-peak ratio of a correlation envelope (the spec's edge score).
|
||||
|
||||
The second peak is the largest value outside a small guard band around the main peak,
|
||||
so the main lobe's own shoulders don't count as competition.
|
||||
"""
|
||||
main = float(env[peak])
|
||||
if main <= _EPS:
|
||||
return 0.0
|
||||
guard = max(1, int(round(guard_s * sr)))
|
||||
masked = env.copy()
|
||||
masked[max(0, peak - guard):min(len(env), peak + guard + 1)] = 0.0
|
||||
second = float(masked.max()) if masked.size else 0.0
|
||||
if second <= _EPS:
|
||||
return 1.0e6
|
||||
return main / second
|
||||
|
||||
|
||||
def _whitened_cc(SIG: np.ndarray, REF: np.ndarray, n: int) -> np.ndarray:
|
||||
"""PHAT-whitened cross-correlation from precomputed rFFTs (lag 0 at index 0)."""
|
||||
R = SIG * np.conj(REF)
|
||||
R /= np.abs(R) + _EPS # PHAT weighting: flatten magnitude, keep phase
|
||||
return np.fft.irfft(R, n) # lag 0 at index 0; negative lags wrap to the tail
|
||||
|
||||
|
||||
def _peak_offset(cc: np.ndarray, sr: int, max_lag: int) -> tuple[float, float]:
|
||||
"""Peak of a whitened correlation → ``(offset_s, confidence)`` over ±``max_lag`` samples."""
|
||||
# Re-center to contiguous lags [−max_lag … +max_lag].
|
||||
cc = np.concatenate((cc[-max_lag:], cc[:max_lag + 1]))
|
||||
env = np.abs(cc)
|
||||
peak = int(np.argmax(env))
|
||||
# Parabolic sub-sample refinement of the peak location (3-point fit around the max),
|
||||
# so precision isn't capped at one sample (62.5 µs at 16 kHz) and drift slopes stay clean.
|
||||
delta = 0.0
|
||||
if 0 < peak < len(env) - 1:
|
||||
a, b, c = env[peak - 1], env[peak], env[peak + 1]
|
||||
denom = a - 2.0 * b + c
|
||||
if abs(denom) > _EPS:
|
||||
delta = float(np.clip(0.5 * (a - c) / denom, -0.5, 0.5))
|
||||
offset_s = ((peak - max_lag) + delta) / sr
|
||||
return offset_s, _peak_ratio(env, peak, sr)
|
||||
|
||||
|
||||
def gcc_phat(sig: np.ndarray, ref: np.ndarray, sr: int,
|
||||
max_lag_s: float | None = None) -> tuple[float, float]:
|
||||
"""GCC-PHAT time delay of ``sig`` relative to ``ref`` (spec M1, lane A).
|
||||
|
||||
Returns ``(offset_s, confidence)`` where ``offset_s`` is positive when ``sig`` *lags*
|
||||
``ref`` (i.e. ``sig[n] ≈ ref[n − offset_s·sr]``), and confidence is the
|
||||
peak-to-second-peak ratio.
|
||||
Returns ``(offset_s, confidence)`` where confidence is the peak-to-second-peak ratio.
|
||||
"""
|
||||
sig = np.asarray(sig, dtype=np.float64)
|
||||
ref = np.asarray(ref, dtype=np.float64)
|
||||
# Remove DC so the whitened correlation keys on structure, not a bias term.
|
||||
sig = sig - sig.mean()
|
||||
ref = ref - ref.mean()
|
||||
|
||||
n = len(sig) + len(ref) # linear (zero-padded) correlation → no circular wrap
|
||||
cc = _whitened_cc(np.fft.rfft(sig, n), np.fft.rfft(ref, n), n)
|
||||
max_lag = n // 2 if max_lag_s is None else int(round(max_lag_s * sr))
|
||||
max_lag = max(1, min(max_lag, n // 2))
|
||||
return _peak_offset(cc, sr, max_lag)
|
||||
raise NotImplementedError("lane A (M1): implement gcc_phat")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pairwise graph
|
||||
# ---------------------------------------------------------------------------
|
||||
def pairwise_offsets(signals: dict[int, np.ndarray], sr: int) -> list[dict]:
|
||||
"""All pairwise GCC-PHAT offsets with confidences (spec M1, lane A).
|
||||
|
||||
``signals`` maps ``video_id -> mono samples``. Returns a list of
|
||||
``{a, b, offset_s, confidence}`` edges, where ``offset_s`` is the *relative start
|
||||
offset* ``offset_a − offset_b`` in seconds (see the module's sign conventions).
|
||||
|
||||
Each signal's rFFT is computed **once** at a shared padded length and reused across
|
||||
every pair — O(K) FFTs + O(K²) cheap spectrum products, instead of the O(K²) full
|
||||
FFTs a per-pair :func:`gcc_phat` would cost. The shared length (2·max_len ≥ any
|
||||
pair's la+lb) keeps the correlation linear (no circular wrap), and each pair's lag
|
||||
search stays capped at ±(la+lb)/2 — the same window the per-pair transform used.
|
||||
``{a, b, offset_s, confidence}`` edges.
|
||||
"""
|
||||
ids = sorted(signals)
|
||||
if len(ids) < 2:
|
||||
return []
|
||||
lengths = {i: len(signals[i]) for i in ids}
|
||||
n = 2 * max(lengths.values())
|
||||
spectra: dict[int, np.ndarray] = {}
|
||||
for i in ids:
|
||||
s = np.asarray(signals[i], dtype=np.float64)
|
||||
s = s - s.mean() # same DC removal as gcc_phat
|
||||
spectra[i] = np.fft.rfft(s, n)
|
||||
edges: list[dict] = []
|
||||
for i in range(len(ids)):
|
||||
for j in range(i + 1, len(ids)):
|
||||
a, b = ids[i], ids[j]
|
||||
cc = _whitened_cc(spectra[a], spectra[b], n)
|
||||
max_lag = max(1, min(n // 2, (lengths[a] + lengths[b]) // 2))
|
||||
delay_s, conf = _peak_offset(cc, sr, max_lag)
|
||||
edges.append({"a": a, "b": b, "offset_s": -delay_s, "confidence": conf})
|
||||
return edges
|
||||
|
||||
|
||||
def _components(ids: list[int], edges: list[dict]) -> dict[int, list[int]]:
|
||||
"""Connected components of the offset graph (union-find over usable edges)."""
|
||||
parent = {i: i for i in ids}
|
||||
|
||||
def find(x: int) -> int:
|
||||
while parent[x] != x:
|
||||
parent[x] = parent[parent[x]]
|
||||
x = parent[x]
|
||||
return x
|
||||
|
||||
for e in edges:
|
||||
parent[find(e["a"])] = find(e["b"])
|
||||
comps: dict[int, list[int]] = {}
|
||||
for i in ids:
|
||||
comps.setdefault(find(i), []).append(i)
|
||||
return comps
|
||||
|
||||
|
||||
def _connected(nodes: list[int], root: int, edges: list[dict]) -> bool:
|
||||
"""Whether every node is reachable from ``root`` over ``edges``."""
|
||||
adj: dict[int, list[int]] = {n: [] for n in nodes}
|
||||
for e in edges:
|
||||
adj[e["a"]].append(e["b"])
|
||||
adj[e["b"]].append(e["a"])
|
||||
seen = {root}
|
||||
stack = [root]
|
||||
while stack:
|
||||
for y in adj[stack.pop()]:
|
||||
if y not in seen:
|
||||
seen.add(y)
|
||||
stack.append(y)
|
||||
return len(seen) == len(nodes)
|
||||
|
||||
|
||||
def _weighted_lstsq(nodes: list[int], ref: int, edges: list[dict]) -> dict[int, float]:
|
||||
"""Confidence-weighted least-squares offsets (seconds) with ``ref`` pinned to 0.
|
||||
|
||||
Each edge contributes the constraint ``offset_a − offset_b = offset_s``, weighted by
|
||||
``sqrt(confidence)``. Solved for all nodes except the pinned reference.
|
||||
"""
|
||||
unknown = [n for n in nodes if n != ref]
|
||||
col = {n: k for k, n in enumerate(unknown)}
|
||||
A = np.zeros((len(edges), len(unknown)))
|
||||
b = np.zeros(len(edges))
|
||||
w = np.zeros(len(edges))
|
||||
for r, e in enumerate(edges):
|
||||
if e["a"] in col:
|
||||
A[r, col[e["a"]]] += 1.0
|
||||
if e["b"] in col:
|
||||
A[r, col[e["b"]]] -= 1.0
|
||||
b[r] = e["offset_s"] # ref terms are 0, so they drop out of A
|
||||
w[r] = np.sqrt(max(e["confidence"], _EPS))
|
||||
x, *_ = np.linalg.lstsq(A * w[:, None], b * w, rcond=None)
|
||||
sol = {ref: 0.0}
|
||||
for n in unknown:
|
||||
sol[n] = float(x[col[n]])
|
||||
return sol
|
||||
|
||||
|
||||
def _solve_component(nodes: list[int], edges: list[dict]) -> dict[int, float]:
|
||||
"""Solve one connected component, rejecting cycle-inconsistent edges iteratively.
|
||||
|
||||
The reference (smallest ``video_id``) is pinned to 0. While the graph is over-determined
|
||||
(more edges than a spanning tree needs) and the worst edge residual exceeds
|
||||
``_CYCLE_TOLERANCE_MS``, drop that edge (unless doing so would disconnect a node) and
|
||||
re-solve. Returns offsets in seconds.
|
||||
"""
|
||||
nodes = sorted(nodes)
|
||||
ref = min(nodes)
|
||||
edges = list(edges)
|
||||
sol = _weighted_lstsq(nodes, ref, edges)
|
||||
while len(edges) > len(nodes) - 1:
|
||||
worst, worst_res = None, 0.0
|
||||
for e in edges:
|
||||
res = abs((sol[e["a"]] - sol[e["b"]]) - e["offset_s"])
|
||||
if res > worst_res:
|
||||
worst, worst_res = e, res
|
||||
if worst is None or worst_res * 1000.0 <= _CYCLE_TOLERANCE_MS:
|
||||
break
|
||||
trial = [e for e in edges if e is not worst]
|
||||
if not _connected(nodes, ref, trial):
|
||||
break # can't drop it without orphaning a node
|
||||
log.info("audio_sync: rejecting inconsistent edge %d-%d (residual %.1f ms)",
|
||||
worst["a"], worst["b"], worst_res * 1000.0)
|
||||
edges = trial
|
||||
sol = _weighted_lstsq(nodes, ref, edges)
|
||||
return sol
|
||||
raise NotImplementedError("lane A (M1): implement pairwise_offsets")
|
||||
|
||||
|
||||
def solve_global_offsets(edges: list[dict], video_ids: list[int]) -> dict[int, float | None]:
|
||||
"""Least-squares global solve over the pairwise-offset graph (spec M1, lane A).
|
||||
|
||||
The reference component (largest; ties broken by smallest ``video_id``) is solved with
|
||||
its smallest-id node pinned to 0. Every video outside it — including any with no usable
|
||||
edges — gets ``None`` (spec pitfall #5). Returns ``video_id -> offset_ms`` (or ``None``).
|
||||
Reference node is pinned to 0; disconnected components get ``None`` (spec pitfall #5).
|
||||
Returns ``video_id -> offset_ms`` (or ``None``).
|
||||
"""
|
||||
ids = sorted(set(video_ids))
|
||||
usable = [
|
||||
e for e in edges
|
||||
if e["a"] in ids and e["b"] in ids
|
||||
and e["confidence"] >= _MIN_EDGE_CONFIDENCE
|
||||
and np.isfinite(e["offset_s"])
|
||||
]
|
||||
result: dict[int, float | None] = {i: None for i in ids}
|
||||
if not ids:
|
||||
return result
|
||||
|
||||
comps = _components(ids, usable)
|
||||
# Reference component: the one that aligns the most cameras. A lone reference (no
|
||||
# overlap with anyone) is still "aligned" at 0; everyone else stays None.
|
||||
ref_comp = max(comps.values(), key=lambda c: (len(c), -min(c)))
|
||||
if len(ref_comp) == 1:
|
||||
result[ref_comp[0]] = 0.0
|
||||
return result
|
||||
|
||||
ref_set = set(ref_comp)
|
||||
comp_edges = [e for e in usable if e["a"] in ref_set and e["b"] in ref_set]
|
||||
for i, off_s in _solve_component(ref_comp, comp_edges).items():
|
||||
result[i] = off_s * 1000.0
|
||||
return result
|
||||
raise NotImplementedError("lane A (M1): implement solve_global_offsets")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Drift
|
||||
# ---------------------------------------------------------------------------
|
||||
def estimate_drift(sig: np.ndarray, ref: np.ndarray, sr: int,
|
||||
window_s: float = 10.0, hop_s: float = 30.0) -> float:
|
||||
"""Estimate clock drift in ppm by fitting window-offsets over the overlap (spec M1).
|
||||
|
||||
Slides a ``window_s`` window every ``hop_s`` over the common span, measures each
|
||||
window's GCC-PHAT delay, and fits a line ``delay ≈ slope·t + b``. ``slope`` is the
|
||||
dimensionless drift rate; ``drift_ppm = slope·1e6``. Returns 0 when there are too few
|
||||
windows to fit or the estimate is within the ±5 ppm deadband.
|
||||
"""
|
||||
sig = np.asarray(sig, dtype=np.float64)
|
||||
ref = np.asarray(ref, dtype=np.float64)
|
||||
n = min(len(sig), len(ref))
|
||||
win = int(round(window_s * sr))
|
||||
hop = int(round(hop_s * sr))
|
||||
if win < sr or n < win or hop < 1: # need a window of at least ~1 s that fits
|
||||
return 0.0
|
||||
|
||||
centers: list[float] = []
|
||||
delays: list[float] = []
|
||||
start = 0
|
||||
while start + win <= n:
|
||||
delay_s, conf = gcc_phat(sig[start:start + win], ref[start:start + win], sr,
|
||||
max_lag_s=window_s / 2.0)
|
||||
if conf >= _MIN_EDGE_CONFIDENCE:
|
||||
centers.append((start + win / 2.0) / sr)
|
||||
delays.append(delay_s)
|
||||
start += hop
|
||||
|
||||
if len(centers) < 2:
|
||||
return 0.0
|
||||
slope = float(np.polyfit(np.array(centers), np.array(delays), 1)[0])
|
||||
drift_ppm = slope * 1.0e6
|
||||
return 0.0 if abs(drift_ppm) < _DRIFT_DEADBAND_PPM else drift_ppm
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Orchestration
|
||||
# ---------------------------------------------------------------------------
|
||||
def _reference_id(offsets_ms: dict[int, float | None]) -> int | None:
|
||||
"""The reference video: smallest id among those with a solved (non-None) offset."""
|
||||
solved = [i for i, o in offsets_ms.items() if o is not None]
|
||||
return min(solved) if solved else None
|
||||
|
||||
|
||||
def _confidence_by_video(edges: list[dict], offsets_ms: dict[int, float | None],
|
||||
ref_id: int | None) -> dict[int, float | None]:
|
||||
"""Per-video sync confidence in [0, 1): best incident edge, mapped ``1 − 1/ratio``.
|
||||
|
||||
The reference is 1.0; unsolved videos are None. A raw peak ratio of 1 (ambiguous) maps
|
||||
to 0, 2 to 0.5, large ratios approach 1 — a bounded, honest confidence.
|
||||
"""
|
||||
best: dict[int, float] = {}
|
||||
for e in edges:
|
||||
if offsets_ms.get(e["a"]) is None or offsets_ms.get(e["b"]) is None:
|
||||
continue
|
||||
r = e["confidence"]
|
||||
best[e["a"]] = max(best.get(e["a"], 0.0), r)
|
||||
best[e["b"]] = max(best.get(e["b"], 0.0), r)
|
||||
out: dict[int, float | None] = {}
|
||||
for vid, off in offsets_ms.items():
|
||||
if off is None:
|
||||
out[vid] = None
|
||||
elif vid == ref_id:
|
||||
out[vid] = 1.0
|
||||
else:
|
||||
ratio = best.get(vid)
|
||||
out[vid] = None if ratio is None else float(max(0.0, 1.0 - 1.0 / ratio))
|
||||
return out
|
||||
def estimate_drift(sig: np.ndarray, ref: np.ndarray, sr: int) -> float:
|
||||
"""Estimate clock drift in ppm by fitting window-offsets over the overlap (spec M1)."""
|
||||
raise NotImplementedError("lane A (M1): implement estimate_drift")
|
||||
|
||||
|
||||
def run_sync() -> dict:
|
||||
@ -352,87 +63,4 @@ def run_sync() -> dict:
|
||||
Entrypoint for ``python -m festival4d sync``. Writes ``offset_ms``/``drift_ppm``/
|
||||
``sync_confidence`` via ``db`` and exports ``config.SYNC_JSON``. Returns the solution.
|
||||
"""
|
||||
import librosa # spec: librosa only for loading; import lazily (heavy module)
|
||||
|
||||
db.init_engine()
|
||||
db.init_db()
|
||||
videos = db.get_videos()
|
||||
if not videos:
|
||||
raise RuntimeError("no videos in DB — run `python -m festival4d ingest` first")
|
||||
|
||||
sr = config.AUDIO_SAMPLE_RATE
|
||||
signals: dict[int, np.ndarray] = {}
|
||||
missing: list[str] = []
|
||||
for v in videos:
|
||||
wav = audio_wav_path(v.id)
|
||||
if not wav.exists():
|
||||
missing.append(v.filename)
|
||||
continue
|
||||
y, _ = librosa.load(str(wav), sr=sr, mono=True)
|
||||
signals[v.id] = y.astype(np.float64)
|
||||
if missing:
|
||||
log.warning("audio_sync: no extracted audio for %s — run ingest; "
|
||||
"they will be left unsynced (offset=None)", ", ".join(missing))
|
||||
|
||||
edges = pairwise_offsets(signals, sr) if len(signals) >= 2 else []
|
||||
offsets_ms = solve_global_offsets(edges, list(signals.keys()))
|
||||
# Videos with no audio at all are unsynced too.
|
||||
for v in videos:
|
||||
offsets_ms.setdefault(v.id, None)
|
||||
|
||||
ref_id = _reference_id(offsets_ms)
|
||||
ref_sig = signals.get(ref_id) if ref_id is not None else None
|
||||
confidence = _confidence_by_video(edges, offsets_ms, ref_id)
|
||||
|
||||
solution = {
|
||||
"reference_video_id": ref_id,
|
||||
"reference_filename": next((v.filename for v in videos if v.id == ref_id), None),
|
||||
"sample_rate": sr,
|
||||
"videos": [],
|
||||
"edges": [
|
||||
{"a": e["a"], "b": e["b"],
|
||||
"offset_ms": round(e["offset_s"] * 1000.0, 3),
|
||||
"confidence": round(float(e["confidence"]), 4)}
|
||||
for e in edges
|
||||
],
|
||||
"components": sorted(
|
||||
(sorted(c) for c in _components(sorted(signals), [
|
||||
e for e in edges if e["confidence"] >= _MIN_EDGE_CONFIDENCE
|
||||
]).values()),
|
||||
key=lambda c: (-len(c), c),
|
||||
),
|
||||
"generated_by": "festival4d.audio_sync",
|
||||
}
|
||||
|
||||
for v in videos:
|
||||
off = offsets_ms.get(v.id)
|
||||
if off is None:
|
||||
drift = None
|
||||
elif v.id == ref_id:
|
||||
drift = 0.0
|
||||
elif ref_sig is not None and v.id in signals:
|
||||
drift = estimate_drift(signals[v.id], ref_sig, sr)
|
||||
else:
|
||||
drift = 0.0
|
||||
conf = confidence.get(v.id)
|
||||
db.update_video_sync(v.id, off, drift, conf)
|
||||
solution["videos"].append({
|
||||
"video_id": v.id, "filename": v.filename,
|
||||
"offset_ms": None if off is None else round(off, 3),
|
||||
"drift_ppm": None if drift is None else round(drift, 4),
|
||||
"sync_confidence": None if conf is None else round(conf, 4),
|
||||
"connected": off is not None,
|
||||
})
|
||||
log.info("audio_sync: %s offset=%s drift=%s conf=%s",
|
||||
v.filename,
|
||||
"None" if off is None else f"{off:+.1f}ms",
|
||||
"None" if drift is None else f"{drift:+.2f}ppm",
|
||||
"None" if conf is None else f"{conf:.2f}")
|
||||
|
||||
config.SYNC_JSON.parent.mkdir(parents=True, exist_ok=True)
|
||||
config.SYNC_JSON.write_text(json.dumps(solution, indent=2))
|
||||
log.info("audio_sync: wrote %s (%d synced, %d unsynced)",
|
||||
config.SYNC_JSON,
|
||||
sum(1 for vv in solution["videos"] if vv["connected"]),
|
||||
sum(1 for vv in solution["videos"] if not vv["connected"]))
|
||||
return solution
|
||||
raise NotImplementedError("lane A (M1): implement run_sync")
|
||||
|
||||
@ -1,317 +0,0 @@
|
||||
"""Memory capsule: zero-backend shareable export (spec M17, lane G).
|
||||
|
||||
Contract: :func:`build_capsule` writes a self-contained bundle to ``out_dir`` (default
|
||||
``config.CAPSULE_DIR``):
|
||||
|
||||
- copy ``frontend/dist`` (error clearly if missing — tell the user to ``npm run build``);
|
||||
- copy media from ``config.RAW_DIR``;
|
||||
- bake every read-only API response to files at the *same relative paths*
|
||||
(``api/manifest``, ``api/events``, ``api/anchors``, ``api/tracks``, ``api/beats``,
|
||||
``api/videos/{id}/poses``, ``api/pointcloud``, ``api/splat``, ``api/audio/{id}`` —
|
||||
extracting audio via ``ingest.extract_audio_hq`` if not yet cached);
|
||||
- inject ``<script>window.__F4D_API_BASE__=""</script>`` into the copied ``index.html``;
|
||||
- set ``"capsule": true`` in the baked manifest (the frontend hides all write UI);
|
||||
- ship a stdlib ``serve.py`` in the bundle root that serves with HTTP Range support
|
||||
(video seeking needs 206s; plain ``http.server`` can't — pitfall #4).
|
||||
|
||||
Skip what doesn't exist (no splat -> no ``api/splat``) — degrade, don't block.
|
||||
Returns a summary dict (bundle path, file count, bytes) for the CLI log.
|
||||
|
||||
Layout mirror (why paths look the way they do): the frontend resolves every URL as
|
||||
``API_BASE + path`` with API_BASE == "" in a capsule, so requests hit the bundle root:
|
||||
``fetch("/api/manifest")`` (no extension — bake the file extension-less; serve.py maps
|
||||
extension-less ``api/*`` files to a JSON content type), ``video.src = "/media/<filename>"``
|
||||
(manifest ``url`` fields are ``/media/{filename}`` — media files are copied under
|
||||
``media/``), PLY loader hits ``/api/pointcloud``, the WebAudio clock fetches
|
||||
``/api/audio/{id}``. Saved camera paths (M16) are read-only in a capsule, so
|
||||
``api/paths`` + ``api/paths/{id}`` are baked too — the load dropdown keeps working.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
from festival4d import config, db
|
||||
|
||||
log = logging.getLogger("festival4d.capsule")
|
||||
|
||||
# Injected verbatim (phase-5 contract #5): a runtime override that beats the build-time
|
||||
# VITE_API_BASE, turning the same dist bundle zero-backend. Must land in index.html before
|
||||
# the app module reads it (modules are deferred, but we inject ahead of the first <script>
|
||||
# anyway so execution order is obvious).
|
||||
_API_BASE_SNIPPET = '<script>window.__F4D_API_BASE__=""</script>'
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# serve.py shipped in the bundle root (stdlib-only, Range-capable — pitfall #4)
|
||||
# ---------------------------------------------------------------------------
|
||||
SERVE_PY = r'''#!/usr/bin/env python3
|
||||
"""Serve this Festival 4D memory capsule locally.
|
||||
|
||||
Plain `python -m http.server` cannot answer HTTP Range requests, and <video>
|
||||
seeking needs 206 partial responses — so the capsule ships this tiny stdlib
|
||||
server instead. No dependencies beyond Python 3.8+.
|
||||
|
||||
python serve.py [--port 8080] [--host 127.0.0.1]
|
||||
|
||||
Then open http://127.0.0.1:8080/ in a browser.
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
from functools import partial
|
||||
from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
ROOT = os.path.dirname(os.path.abspath(__file__))
|
||||
|
||||
_RANGE = re.compile(r"bytes=(\d*)-(\d*)$")
|
||||
|
||||
|
||||
class _Slice:
|
||||
"""File wrapper that reads at most `length` bytes (the body of a 206)."""
|
||||
|
||||
def __init__(self, f, length):
|
||||
self._f = f
|
||||
self._left = length
|
||||
|
||||
def read(self, n=-1):
|
||||
if self._left <= 0:
|
||||
return b""
|
||||
if n < 0 or n > self._left:
|
||||
n = self._left
|
||||
data = self._f.read(n)
|
||||
self._left -= len(data)
|
||||
return data
|
||||
|
||||
def close(self):
|
||||
self._f.close()
|
||||
|
||||
|
||||
class RangeHandler(SimpleHTTPRequestHandler):
|
||||
protocol_version = "HTTP/1.1"
|
||||
|
||||
def guess_type(self, path):
|
||||
# Baked API responses are extension-less files at the live API's relative
|
||||
# paths; give them the content types the live server would have used.
|
||||
url = self.path.split("?", 1)[0].split("#", 1)[0].lstrip("/")
|
||||
if "." not in url.rsplit("/", 1)[-1]:
|
||||
if url.startswith("api/audio/"):
|
||||
return "audio/mp4"
|
||||
if url in ("api/pointcloud", "api/splat"):
|
||||
return "application/octet-stream"
|
||||
if url.startswith("api/"):
|
||||
return "application/json"
|
||||
return super().guess_type(path)
|
||||
|
||||
def send_head(self):
|
||||
path = self.translate_path(self.path)
|
||||
if os.path.isdir(path):
|
||||
# Directory -> index.html (the SPA entry), stock behavior.
|
||||
return super().send_head()
|
||||
try:
|
||||
f = open(path, "rb")
|
||||
except OSError:
|
||||
self.send_error(404, "File not found")
|
||||
return None
|
||||
try:
|
||||
size = os.fstat(f.fileno()).st_size
|
||||
start, end, status = 0, size - 1, 200
|
||||
m = _RANGE.match(self.headers.get("Range", "").strip())
|
||||
if m and (m.group(1) or m.group(2)):
|
||||
if m.group(1):
|
||||
start = int(m.group(1))
|
||||
if m.group(2):
|
||||
end = min(int(m.group(2)), size - 1)
|
||||
else: # suffix form: bytes=-N (the final N bytes)
|
||||
start = max(size - int(m.group(2)), 0)
|
||||
if start >= size:
|
||||
self.send_response(416)
|
||||
self.send_header("Content-Range", "bytes */%d" % size)
|
||||
self.send_header("Content-Length", "0")
|
||||
self.end_headers()
|
||||
f.close()
|
||||
return None
|
||||
status = 206
|
||||
self.send_response(status)
|
||||
self.send_header("Content-Type", self.guess_type(path))
|
||||
self.send_header("Accept-Ranges", "bytes")
|
||||
self.send_header("Content-Length", str(end - start + 1))
|
||||
if status == 206:
|
||||
self.send_header("Content-Range", "bytes %d-%d/%d" % (start, end, size))
|
||||
self.end_headers()
|
||||
f.seek(start)
|
||||
return _Slice(f, end - start + 1)
|
||||
except Exception:
|
||||
f.close()
|
||||
raise
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description="Serve this Festival 4D capsule (Range-capable).")
|
||||
ap.add_argument("--host", default="127.0.0.1")
|
||||
ap.add_argument("--port", type=int, default=8080)
|
||||
args = ap.parse_args()
|
||||
handler = partial(RangeHandler, directory=ROOT)
|
||||
httpd = ThreadingHTTPServer((args.host, args.port), handler)
|
||||
print("Festival 4D capsule -> http://%s:%d/" % (args.host, args.port))
|
||||
try:
|
||||
httpd.serve_forever()
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
'''
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
def _bake_json(path: Path, obj) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(obj, indent=2))
|
||||
|
||||
|
||||
def _inject_api_base(index_html: Path) -> None:
|
||||
"""Insert the runtime API-base override into the copied index.html."""
|
||||
html = index_html.read_text()
|
||||
if "__F4D_API_BASE__" in html:
|
||||
return # already injected (rebuild over an old bundle copy)
|
||||
i = html.find("<script")
|
||||
if i == -1:
|
||||
i = html.find("</head>")
|
||||
if i != -1:
|
||||
html = html[:i] + _API_BASE_SNIPPET + "\n " + html[i:]
|
||||
else: # no <script> and no </head> — degenerate page; prepend
|
||||
html = _API_BASE_SNIPPET + "\n" + html
|
||||
index_html.write_text(html)
|
||||
|
||||
|
||||
def _prepare_out_dir(out: Path) -> None:
|
||||
"""Refuse to clobber a directory that isn't (or wasn't) a capsule."""
|
||||
if not out.exists():
|
||||
return
|
||||
contents = list(out.iterdir())
|
||||
is_previous_capsule = (out / "serve.py").exists() or (out / "index.html").exists()
|
||||
if contents and not is_previous_capsule:
|
||||
raise RuntimeError(
|
||||
f"output dir {out} exists and does not look like a previous capsule — "
|
||||
"refusing to overwrite; pass an empty/new --out"
|
||||
)
|
||||
shutil.rmtree(out)
|
||||
|
||||
|
||||
def _summary(out: Path) -> dict:
|
||||
files = [p for p in out.rglob("*") if p.is_file()]
|
||||
return {
|
||||
"bundle": str(out),
|
||||
"files": len(files),
|
||||
"bytes": sum(p.stat().st_size for p in files),
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Entry point
|
||||
# ---------------------------------------------------------------------------
|
||||
def build_capsule(out_dir: str | Path | None = None,
|
||||
dist_dir: str | Path | None = None) -> dict:
|
||||
"""Build the static shareable bundle (see module doc).
|
||||
|
||||
``dist_dir`` overrides the frontend build location (default
|
||||
``<repo>/frontend/dist``) — used by tests to point at a fabricated dist.
|
||||
"""
|
||||
# The API module is the single source of truth for response shapes — its route
|
||||
# functions are plain callables returning the exact dicts the live server sends,
|
||||
# so baking through them can never drift from the frozen contract.
|
||||
from festival4d import api, ingest
|
||||
|
||||
dist = Path(dist_dir) if dist_dir is not None else config.REPO_ROOT / "frontend" / "dist"
|
||||
if not (dist / "index.html").exists():
|
||||
raise RuntimeError(
|
||||
f"frontend build not found at {dist} — run `cd frontend && npm run build` first"
|
||||
)
|
||||
|
||||
out = Path(out_dir) if out_dir is not None else config.CAPSULE_DIR
|
||||
_prepare_out_dir(out)
|
||||
|
||||
db.init_engine()
|
||||
db.init_db()
|
||||
videos = db.get_videos()
|
||||
|
||||
# 1. Frontend bundle + runtime API-base override.
|
||||
shutil.copytree(dist, out)
|
||||
_inject_api_base(out / "index.html")
|
||||
|
||||
# 2. Media (only the files the manifest references — RAW_DIR may hold strays).
|
||||
media_dir = out / "media"
|
||||
media_dir.mkdir(parents=True, exist_ok=True)
|
||||
for v in videos:
|
||||
src = config.RAW_DIR / v.filename
|
||||
if src.exists():
|
||||
shutil.copy2(src, media_dir / v.filename)
|
||||
else:
|
||||
log.warning("capsule: source video missing, skipped: %s", src)
|
||||
|
||||
# 3. Baked read-only API responses, at the SAME relative paths the live API serves
|
||||
# (extension-less on purpose — the frontend fetches e.g. `${API_BASE}/api/manifest`).
|
||||
api_dir = out / "api"
|
||||
manifest = api.manifest()
|
||||
manifest["capsule"] = True # frontend hides all write UI (contract #5)
|
||||
manifest["has_capture"] = False # capture routes never exist in a static bundle
|
||||
_bake_json(api_dir / "manifest", manifest)
|
||||
_bake_json(api_dir / "events", api.get_events())
|
||||
_bake_json(api_dir / "anchors", api.get_anchors())
|
||||
# Friend tracks (M18/M20) — read-only in a capsule so ribbons + follow still work zero-backend.
|
||||
_bake_json(api_dir / "tracks", api.get_tracks())
|
||||
for v in videos:
|
||||
_bake_json(api_dir / "videos" / str(v.id) / "poses", api.video_poses(v.id))
|
||||
|
||||
# Saved camera paths (M16) — read-only in a capsule, so the load dropdown still works.
|
||||
# A filesystem can't hold both a FILE `api/paths` and per-id files under `api/paths/`,
|
||||
# so the listing is baked as the directory's index.html: GET /api/paths gets the stock
|
||||
# 301 -> /api/paths/ -> index.html (fetch follows redirects), and serve.py's URL-based
|
||||
# content-type override still labels it application/json.
|
||||
_bake_json(api_dir / "paths" / "index.html", api.get_paths())
|
||||
for p in db.get_paths():
|
||||
_bake_json(api_dir / "paths" / str(p.id), api.get_path(p.id))
|
||||
|
||||
# Optional artifacts: bake when present, silently omit when not (the frontend already
|
||||
# handles the 404s serve.py will answer for the missing files — degrade, don't block).
|
||||
if config.BEATS_JSON.exists():
|
||||
_bake_json(api_dir / "beats", json.loads(config.BEATS_JSON.read_text()))
|
||||
if config.POINTS_PLY.exists():
|
||||
api_dir.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(config.POINTS_PLY, api_dir / "pointcloud")
|
||||
if config.SPLAT_PLY.exists():
|
||||
api_dir.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(config.SPLAT_PLY, api_dir / "splat")
|
||||
|
||||
# 4. Soundtracks for the WebAudio master clock: reuse the live server's lazy cache,
|
||||
# extracting on demand exactly like GET /api/audio/{id} would have.
|
||||
import subprocess
|
||||
|
||||
for v in videos:
|
||||
src = config.RAW_DIR / v.filename
|
||||
cached = config.AUDIO_HQ_DIR / f"{v.id}.m4a"
|
||||
if src.exists() and (not cached.exists() or cached.stat().st_mtime < src.stat().st_mtime):
|
||||
try:
|
||||
ingest.extract_audio_hq(src, cached)
|
||||
except (subprocess.CalledProcessError, RuntimeError, OSError) as exc:
|
||||
log.warning("capsule: no extractable audio for video %s: %s", v.id, exc)
|
||||
if cached.exists():
|
||||
audio_out = api_dir / "audio" / str(v.id)
|
||||
audio_out.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(cached, audio_out)
|
||||
|
||||
# 5. The Range-capable server (pitfall #4).
|
||||
serve_py = out / "serve.py"
|
||||
serve_py.write_text(SERVE_PY)
|
||||
serve_py.chmod(serve_py.stat().st_mode | 0o111)
|
||||
|
||||
summary = _summary(out)
|
||||
log.info("capsule: baked %(files)d files (%(bytes)d bytes) at %(bundle)s", summary)
|
||||
return summary
|
||||
@ -1,166 +0,0 @@
|
||||
"""Live capture — USB webcams, laptop cams, and phones -> ``data/raw`` (Phase 5).
|
||||
|
||||
**Records, it doesn't stream.** Each device records a clip *with audio* and uploads it; the normal
|
||||
offline pipeline (``ingest -> sync -> reconstruct -> events``) takes it from there. This fits the
|
||||
app's design almost for free:
|
||||
|
||||
- Browser ``MediaRecorder`` emits WebM (Safari/iOS: MP4) — both are already in ingest's
|
||||
``_VIDEO_EXTS``, so no format work is needed.
|
||||
- Sync is **audio-based** (GCC-PHAT), so capture devices need **no clock sync and no
|
||||
synchronized start** — they only have to record the same moment and hear the same sound.
|
||||
Every recording MUST carry an audio track or ``sync`` has nothing to align.
|
||||
|
||||
A lightweight device registry powers the "who's shooting what" monitor: each device posts a
|
||||
heartbeat with its status and a small JPEG snapshot (~1 fps), so the capture page can show every
|
||||
camera without a WebRTC stack.
|
||||
|
||||
**Disabled by default.** These routes accept file uploads; they must never be exposed on a public
|
||||
deployment. Set ``FESTIVAL4D_CAPTURE=1`` to mount them (see :func:`capture_enabled`).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import APIRouter, Form, HTTPException, UploadFile
|
||||
from fastapi.responses import HTMLResponse
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from festival4d import config
|
||||
|
||||
log = logging.getLogger("festival4d.capture")
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
STATIC_DIR = Path(__file__).resolve().parent / "static"
|
||||
|
||||
# Containers a browser MediaRecorder can produce, intersected with what ingest accepts.
|
||||
ALLOWED_EXTS = {".webm", ".mp4", ".mov", ".mkv"}
|
||||
MAX_UPLOAD_BYTES = int(os.environ.get("FESTIVAL4D_MAX_UPLOAD_MB", "2048")) * 1024 * 1024
|
||||
MAX_SNAPSHOT_CHARS = 300_000 # data-URL cap for a heartbeat thumbnail
|
||||
DEVICE_STALE_S = 15.0 # drop devices that stop heartbeating
|
||||
|
||||
|
||||
def capture_enabled() -> bool:
|
||||
"""Capture routes are opt-in — they accept uploads, so never on by default."""
|
||||
return os.environ.get("FESTIVAL4D_CAPTURE", "").strip().lower() in {"1", "true", "yes", "on"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Device registry (in-memory, ephemeral — it only powers the live monitor)
|
||||
# ---------------------------------------------------------------------------
|
||||
_devices: dict[str, dict] = {}
|
||||
|
||||
|
||||
class Heartbeat(BaseModel):
|
||||
device_id: str = Field(min_length=1, max_length=64)
|
||||
name: str = Field(default="camera", max_length=64)
|
||||
status: str = Field(default="idle", max_length=16) # idle | recording
|
||||
elapsed_s: float = 0.0
|
||||
snapshot: str | None = None # data:image/jpeg;base64,...
|
||||
|
||||
|
||||
def _prune() -> None:
|
||||
now = time.monotonic()
|
||||
for did in [d for d, v in _devices.items() if now - v["_seen"] > DEVICE_STALE_S]:
|
||||
_devices.pop(did, None)
|
||||
|
||||
|
||||
def _slug(name: str) -> str:
|
||||
"""Filesystem-safe slug from a user-supplied device name."""
|
||||
s = re.sub(r"[^a-zA-Z0-9]+", "-", name).strip("-").lower()
|
||||
return (s or "cam")[:40]
|
||||
|
||||
|
||||
def _safe_raw_path(device_name: str, filename: str) -> Path:
|
||||
"""Build a sanitized destination inside RAW_DIR. Never trusts the client filename."""
|
||||
ext = Path(filename or "").suffix.lower()
|
||||
if ext not in ALLOWED_EXTS:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"unsupported container '{ext or '?'}' (allowed: {sorted(ALLOWED_EXTS)})",
|
||||
)
|
||||
stamp = datetime.now().strftime("%Y%m%d-%H%M%S")
|
||||
dest = (config.RAW_DIR / f"{_slug(device_name)}-{stamp}{ext}").resolve()
|
||||
# Defence in depth: the slug can't traverse, but verify containment anyway.
|
||||
if not str(dest).startswith(str(config.RAW_DIR.resolve()) + os.sep):
|
||||
raise HTTPException(status_code=400, detail="invalid destination path")
|
||||
return dest
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Routes
|
||||
# ---------------------------------------------------------------------------
|
||||
@router.get("/capture", response_class=HTMLResponse)
|
||||
def capture_page() -> str:
|
||||
"""The standalone capture page (open this on a phone over HTTPS — see README)."""
|
||||
page = STATIC_DIR / "capture.html"
|
||||
if not page.is_file():
|
||||
raise HTTPException(status_code=500, detail="capture.html missing from the install")
|
||||
return page.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
@router.post("/api/capture/upload")
|
||||
async def upload(file: UploadFile, device_name: str = Form("camera")) -> dict:
|
||||
"""Stream an uploaded clip into ``data/raw`` under a sanitized name."""
|
||||
dest = _safe_raw_path(device_name, file.filename or "")
|
||||
config.RAW_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
written = 0
|
||||
try:
|
||||
with open(dest, "wb") as out:
|
||||
while chunk := await file.read(1 << 20): # 1 MiB
|
||||
written += len(chunk)
|
||||
if written > MAX_UPLOAD_BYTES:
|
||||
raise HTTPException(
|
||||
status_code=413,
|
||||
detail=f"upload exceeds {MAX_UPLOAD_BYTES // (1024*1024)} MB",
|
||||
)
|
||||
out.write(chunk)
|
||||
except Exception:
|
||||
dest.unlink(missing_ok=True) # never leave a partial file for ingest to trip on
|
||||
raise
|
||||
|
||||
if written == 0:
|
||||
dest.unlink(missing_ok=True)
|
||||
raise HTTPException(status_code=400, detail="empty upload")
|
||||
|
||||
log.info("capture: saved %s (%.1f MB) from %r", dest.name, written / 1e6, device_name)
|
||||
return {"filename": dest.name, "bytes": written, "next": "run `python -m festival4d ingest`"}
|
||||
|
||||
|
||||
@router.post("/api/capture/heartbeat")
|
||||
def heartbeat(hb: Heartbeat) -> dict:
|
||||
"""Register/refresh a capture device and its latest thumbnail (powers the monitor)."""
|
||||
snap = hb.snapshot
|
||||
if snap and len(snap) > MAX_SNAPSHOT_CHARS:
|
||||
snap = None # oversized thumbnail: keep the device, drop the image
|
||||
entry = _devices.setdefault(hb.device_id, {})
|
||||
entry.update(
|
||||
{
|
||||
"device_id": hb.device_id,
|
||||
"name": hb.name,
|
||||
"status": hb.status,
|
||||
"elapsed_s": hb.elapsed_s,
|
||||
"_seen": time.monotonic(),
|
||||
}
|
||||
)
|
||||
if snap:
|
||||
entry["snapshot"] = snap
|
||||
_prune()
|
||||
return {"ok": True, "devices": len(_devices)}
|
||||
|
||||
|
||||
@router.get("/api/capture/devices")
|
||||
def devices() -> list[dict]:
|
||||
"""Live capture devices (stale ones drop off after ~15 s)."""
|
||||
_prune()
|
||||
return [
|
||||
{k: v for k, v in d.items() if not k.startswith("_")}
|
||||
for d in sorted(_devices.values(), key=lambda d: d["name"])
|
||||
]
|
||||
@ -4,8 +4,7 @@ FROZEN after foundation. Lanes fill in the bodies of the functions this dispatch
|
||||
to (in ``ingest.py``, ``audio_sync.py``, ``sfm.py``, ``events_ai.py``); they never
|
||||
edit this file or ``api.py``.
|
||||
|
||||
Subcommands: ``synthetic | ingest | sync | reconstruct | events | features | direct |
|
||||
track | capsule | serve``.
|
||||
Subcommands: ``synthetic | ingest | sync | reconstruct | events | serve``.
|
||||
|
||||
Every subcommand dispatches to a single lane entrypoint. While a lane is still a stub,
|
||||
its entrypoint raises :class:`NotImplementedError`; we catch that here and print a clear
|
||||
@ -58,49 +57,6 @@ def _cmd_events(args: argparse.Namespace) -> int:
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_features(args: argparse.Namespace) -> int:
|
||||
from festival4d import audio_features
|
||||
|
||||
result = audio_features.run_features()
|
||||
log.info("features: %s", result)
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_direct(args: argparse.Namespace) -> int:
|
||||
import json
|
||||
|
||||
from festival4d import director
|
||||
|
||||
path = director.generate_path(top_n=args.top_n, lead_s=args.lead_s)
|
||||
print(json.dumps(path, indent=2))
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_track(args: argparse.Namespace) -> int:
|
||||
"""Detect wearable markers and solve them into 3D friend tracks (lane H / M19-M20).
|
||||
|
||||
Runs detect -> solve by default; ``--detect-only`` / ``--solve-only`` run one stage.
|
||||
Both stages dispatch into lane-H stubs, so while unimplemented this prints the
|
||||
"not implemented yet" message and exits 2 (same as every other stubbed subcommand)."""
|
||||
from festival4d import tracker_detect, tracker_solve
|
||||
|
||||
if not args.solve_only:
|
||||
result = tracker_detect.run_detect()
|
||||
log.info("track detect: %s", result)
|
||||
if not args.detect_only:
|
||||
result = tracker_solve.run_solve()
|
||||
log.info("track solve: %s", result)
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_capsule(args: argparse.Namespace) -> int:
|
||||
from festival4d import capsule
|
||||
|
||||
result = capsule.build_capsule(out_dir=args.out)
|
||||
log.info("capsule: %s", result)
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_serve(args: argparse.Namespace) -> int:
|
||||
import uvicorn
|
||||
|
||||
@ -141,28 +97,6 @@ def build_parser() -> argparse.ArgumentParser:
|
||||
help="window width in seconds when --t-global-s is given")
|
||||
p_evt.set_defaults(func=_cmd_events)
|
||||
|
||||
p_feat = sub.add_parser("features", help="beat/onset analysis of the reference audio (lane E / M10)")
|
||||
p_feat.set_defaults(func=_cmd_features)
|
||||
|
||||
p_dir = sub.add_parser("direct", help="auto-director: events -> camPath JSON on stdout (lane E / M11)")
|
||||
p_dir.add_argument("--top-n", dest="top_n", type=int, default=8,
|
||||
help="number of top-confidence events to cover (default 8)")
|
||||
p_dir.add_argument("--lead-s", dest="lead_s", type=float, default=2.0,
|
||||
help="seconds of lead-in before each event (default 2.0)")
|
||||
p_dir.set_defaults(func=_cmd_direct)
|
||||
|
||||
p_trk = sub.add_parser("track", help="detect markers + solve friend tracks (lane H / M19-M20)")
|
||||
trk_mode = p_trk.add_mutually_exclusive_group()
|
||||
trk_mode.add_argument("--detect-only", dest="detect_only", action="store_true",
|
||||
help="run marker detection only (write detections.json)")
|
||||
trk_mode.add_argument("--solve-only", dest="solve_only", action="store_true",
|
||||
help="run track solving only (from an existing detections.json)")
|
||||
p_trk.set_defaults(func=_cmd_track, detect_only=False, solve_only=False)
|
||||
|
||||
p_cap = sub.add_parser("capsule", help="bake a zero-backend shareable bundle (lane G / M17)")
|
||||
p_cap.add_argument("--out", default=None, help="output dir (default data/capsule/)")
|
||||
p_cap.set_defaults(func=_cmd_capsule)
|
||||
|
||||
p_srv = sub.add_parser("serve", help="run the FastAPI app (M3)")
|
||||
p_srv.add_argument("--host", default=config.API_HOST)
|
||||
p_srv.add_argument("--port", type=int, default=config.API_PORT)
|
||||
|
||||
@ -35,19 +35,13 @@ DATA_DIR = _data_dir()
|
||||
RAW_DIR = DATA_DIR / "raw" # user drops source videos here (gitignored)
|
||||
WORK_DIR = DATA_DIR / "work" # extracted wavs, frames, colmap workspace (gitignored)
|
||||
AUDIO_DIR = WORK_DIR / "audio" # per-video mono 16 kHz wavs (ingest)
|
||||
AUDIO_HQ_DIR = WORK_DIR / "audio_hq" # per-video listening-quality AAC (lazy, /api/audio/{id})
|
||||
FRAMES_DIR = WORK_DIR / "frames" # sampled JPEGs for SfM
|
||||
COLMAP_DIR = WORK_DIR / "colmap" # COLMAP workspace
|
||||
DB_PATH = DATA_DIR / "project.db" # SQLite (gitignored)
|
||||
|
||||
BEATS_JSON = WORK_DIR / "beats.json" # beat/onset analysis (lane E / M10, /api/beats)
|
||||
CAPSULE_DIR = DATA_DIR / "capsule" # default output of `python -m festival4d capsule` (M17)
|
||||
POINTS_PLY = WORK_DIR / "points.ply" # reconstructed / synthetic point cloud
|
||||
SPLAT_PLY = WORK_DIR / "splat.ply" # optional 3DGS splat (trained externally, e.g. via MODELBEAST — see docs/modelbeast-crossover.md); viewer prefers it over the point cloud
|
||||
SYNC_JSON = WORK_DIR / "sync.json" # exported sync solution (lane A)
|
||||
GROUND_TRUTH_JSON = WORK_DIR / "ground_truth.json" # synthetic fixture ground truth
|
||||
DETECTIONS_JSON = WORK_DIR / "detections.json" # marker detections (phase 6 / M19, contract #3)
|
||||
TRACK_TRUTH_JSON = WORK_DIR / "track_truth.json" # synthetic marker ground truth (phase 6 / M18)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Constants
|
||||
|
||||
@ -12,9 +12,6 @@ Schema (spec §2)::
|
||||
anchors(id, label, x,y,z, color)
|
||||
events(id, t_global_s, duration_s, event_type, confidence, description, source)
|
||||
annotations(id, video_id FK, t_video_s, x0,y0,x1,y1, resolved_anchor_id FK NULL)
|
||||
paths(id, name, created_at, path_json) # phase 5 (M16): saved camPath JSON, as text
|
||||
tracks(id, marker_key, label, color, created_at) # phase 6 (M18/M20): friend tracks
|
||||
track_points(id, track_id FK, t_global_s, x,y,z, quality, views) # ordered by t_global_s
|
||||
|
||||
Engine management: :func:`init_engine` (re)binds the module to a SQLite file. It defaults
|
||||
to ``config.DB_PATH`` but tests point it at a temp file. Helpers open and commit their own
|
||||
@ -38,7 +35,6 @@ from sqlalchemy import (
|
||||
create_engine,
|
||||
delete,
|
||||
select,
|
||||
update,
|
||||
)
|
||||
from sqlalchemy.engine import Engine
|
||||
from sqlalchemy.orm import DeclarativeBase, Mapped, Session, mapped_column, sessionmaker
|
||||
@ -124,64 +120,11 @@ class Annotation(Base):
|
||||
y0: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
x1: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
y1: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
# Nullable link to the event this annotation locates (CR-2, M8). Two annotations sharing an
|
||||
# event_id from different videos triangulate to a 3D anchor. Nullable => additive/back-compat.
|
||||
event_id: Mapped[int | None] = mapped_column(ForeignKey("events.id"), nullable=True)
|
||||
resolved_anchor_id: Mapped[int | None] = mapped_column(
|
||||
ForeignKey("anchors.id"), nullable=True
|
||||
)
|
||||
|
||||
|
||||
class SavedPath(Base):
|
||||
"""A saved keyframed camera path (phase 5 contract #3). ``path_json`` is the frozen
|
||||
camPath JSON (``{"version": 1, "keyframes": [...]}``) stored verbatim as text —
|
||||
the DB never interprets it beyond POST-time validation in the API layer."""
|
||||
|
||||
__tablename__ = "paths"
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
name: Mapped[str] = mapped_column(String, nullable=False)
|
||||
created_at: Mapped[str] = mapped_column(String, nullable=False)
|
||||
path_json: Mapped[str] = mapped_column(String, nullable=False)
|
||||
|
||||
|
||||
class Track(Base):
|
||||
"""A friend track (phase 6 contract #2): a wearable marker's identity + metadata.
|
||||
|
||||
``marker_key`` is the detector-emitted identity — ``"hue:<opencv_hue>"`` for a color
|
||||
marker or ``"code:<id>"`` for a blink badge (see ``tracker_detect``). The per-timestep
|
||||
3D path lives in :class:`TrackPoint`. ``label`` / ``color`` are user-editable.
|
||||
"""
|
||||
|
||||
__tablename__ = "tracks"
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
marker_key: Mapped[str] = mapped_column(String, nullable=False)
|
||||
label: Mapped[str | None] = mapped_column(String, nullable=True)
|
||||
color: Mapped[str | None] = mapped_column(String, nullable=True)
|
||||
created_at: Mapped[str] = mapped_column(String, nullable=False)
|
||||
|
||||
|
||||
class TrackPoint(Base):
|
||||
"""One solved position of a track at a master-timeline instant (phase 6 contract #2).
|
||||
|
||||
Coordinates are **Three.js scene space**. ``views`` = number of cameras used to solve the
|
||||
point (1 ⇒ single-view ground-plane fallback, contract #5); ``quality`` is 0..1.
|
||||
Points are always read ordered by ``t_global_s``.
|
||||
"""
|
||||
|
||||
__tablename__ = "track_points"
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
track_id: Mapped[int] = mapped_column(ForeignKey("tracks.id"), nullable=False, index=True)
|
||||
t_global_s: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
x: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
y: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
z: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
quality: Mapped[float | None] = mapped_column(Float, nullable=True)
|
||||
views: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Engine / session management
|
||||
# ---------------------------------------------------------------------------
|
||||
@ -372,75 +315,6 @@ def get_anchors() -> list[Anchor]:
|
||||
return list(s.scalars(select(Anchor).order_by(Anchor.id)))
|
||||
|
||||
|
||||
def get_anchor(anchor_id: int) -> Anchor | None:
|
||||
with session_scope() as s:
|
||||
return s.get(Anchor, anchor_id)
|
||||
|
||||
|
||||
def update_anchor(
|
||||
anchor_id: int,
|
||||
x: float,
|
||||
y: float,
|
||||
z: float,
|
||||
label: str | None = None,
|
||||
color: str | None = None,
|
||||
) -> Anchor:
|
||||
"""Move an anchor in place (M8 supersede). Position always updates; label/color only if given.
|
||||
|
||||
Raises ``KeyError`` if the id is unknown.
|
||||
"""
|
||||
with session_scope() as s:
|
||||
anchor = s.get(Anchor, anchor_id)
|
||||
if anchor is None:
|
||||
raise KeyError(f"no anchor with id={anchor_id}")
|
||||
anchor.x, anchor.y, anchor.z = x, y, z
|
||||
if label is not None:
|
||||
anchor.label = label
|
||||
if color is not None:
|
||||
anchor.color = color
|
||||
s.flush()
|
||||
s.refresh(anchor)
|
||||
return anchor
|
||||
|
||||
|
||||
def patch_anchor(anchor_id: int, label: str | None = None, color: str | None = None) -> Anchor:
|
||||
"""Rename / recolor an anchor without moving it (phase 5 contract #4, M13).
|
||||
|
||||
Only provided fields change. Raises ``KeyError`` if the id is unknown.
|
||||
"""
|
||||
with session_scope() as s:
|
||||
anchor = s.get(Anchor, anchor_id)
|
||||
if anchor is None:
|
||||
raise KeyError(f"no anchor with id={anchor_id}")
|
||||
if label is not None:
|
||||
anchor.label = label
|
||||
if color is not None:
|
||||
anchor.color = color
|
||||
s.flush()
|
||||
s.refresh(anchor)
|
||||
return anchor
|
||||
|
||||
|
||||
def delete_anchor(anchor_id: int) -> bool:
|
||||
"""Delete an anchor, first unlinking any annotations that resolved to it.
|
||||
|
||||
Returns ``True`` if a row was deleted, ``False`` if the id was unknown. SQLite doesn't
|
||||
enforce the FK by default, so we NULL ``annotations.resolved_anchor_id`` explicitly to keep
|
||||
the link consistent.
|
||||
"""
|
||||
with session_scope() as s:
|
||||
anchor = s.get(Anchor, anchor_id)
|
||||
if anchor is None:
|
||||
return False
|
||||
s.execute(
|
||||
update(Annotation)
|
||||
.where(Annotation.resolved_anchor_id == anchor_id)
|
||||
.values(resolved_anchor_id=None)
|
||||
)
|
||||
s.delete(anchor)
|
||||
return True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Events
|
||||
# ---------------------------------------------------------------------------
|
||||
@ -482,172 +356,6 @@ def clear_events(source: str | None = None) -> int:
|
||||
return int(result.rowcount or 0)
|
||||
|
||||
|
||||
def get_event(event_id: int) -> Event | None:
|
||||
with session_scope() as s:
|
||||
return s.get(Event, event_id)
|
||||
|
||||
|
||||
def update_event(
|
||||
event_id: int,
|
||||
event_type: str | None = None,
|
||||
source: str | None = None,
|
||||
description: str | None = None,
|
||||
confidence: float | None = None,
|
||||
) -> Event:
|
||||
"""Patch an existing event in place (M8 correction). Only provided fields change.
|
||||
|
||||
Returns the updated row; raises ``KeyError`` if the id is unknown.
|
||||
"""
|
||||
with session_scope() as s:
|
||||
event = s.get(Event, event_id)
|
||||
if event is None:
|
||||
raise KeyError(f"no event with id={event_id}")
|
||||
if event_type is not None:
|
||||
event.event_type = event_type
|
||||
if source is not None:
|
||||
event.source = source
|
||||
if description is not None:
|
||||
event.description = description
|
||||
if confidence is not None:
|
||||
event.confidence = confidence
|
||||
s.flush()
|
||||
s.refresh(event)
|
||||
return event
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Saved camera paths (phase 5 / M16)
|
||||
# ---------------------------------------------------------------------------
|
||||
def add_path(name: str, path_json: str) -> SavedPath:
|
||||
with session_scope() as s:
|
||||
path = SavedPath(name=name, path_json=path_json, created_at=_now_iso())
|
||||
s.add(path)
|
||||
s.flush()
|
||||
s.refresh(path)
|
||||
return path
|
||||
|
||||
|
||||
def get_paths() -> list[SavedPath]:
|
||||
with session_scope() as s:
|
||||
return list(s.scalars(select(SavedPath).order_by(SavedPath.id)))
|
||||
|
||||
|
||||
def get_path(path_id: int) -> SavedPath | None:
|
||||
with session_scope() as s:
|
||||
return s.get(SavedPath, path_id)
|
||||
|
||||
|
||||
def delete_path(path_id: int) -> bool:
|
||||
"""Delete a saved path. Returns ``True`` if a row was deleted, ``False`` if unknown."""
|
||||
with session_scope() as s:
|
||||
path = s.get(SavedPath, path_id)
|
||||
if path is None:
|
||||
return False
|
||||
s.delete(path)
|
||||
return True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Friend tracks (phase 6 / M18 contract #2). Written by the solver (lane H / M20),
|
||||
# read by the API + frontend. TrackPoints are always returned ordered by t_global_s.
|
||||
# ---------------------------------------------------------------------------
|
||||
def add_track(marker_key: str, label: str | None = None, color: str | None = None) -> Track:
|
||||
"""Insert a track row (no points yet) and return it."""
|
||||
with session_scope() as s:
|
||||
track = Track(marker_key=marker_key, label=label, color=color, created_at=_now_iso())
|
||||
s.add(track)
|
||||
s.flush()
|
||||
s.refresh(track)
|
||||
return track
|
||||
|
||||
|
||||
def get_tracks() -> list[Track]:
|
||||
"""All tracks, ordered by id."""
|
||||
with session_scope() as s:
|
||||
return list(s.scalars(select(Track).order_by(Track.id)))
|
||||
|
||||
|
||||
def get_track(track_id: int) -> Track | None:
|
||||
with session_scope() as s:
|
||||
return s.get(Track, track_id)
|
||||
|
||||
|
||||
def get_track_points(track_id: int) -> list[TrackPoint]:
|
||||
"""A track's points, ordered by ``t_global_s`` (contract #2)."""
|
||||
with session_scope() as s:
|
||||
return list(
|
||||
s.scalars(
|
||||
select(TrackPoint)
|
||||
.where(TrackPoint.track_id == track_id)
|
||||
.order_by(TrackPoint.t_global_s)
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def set_track_points(track_id: int, points: Iterable[dict]) -> int:
|
||||
"""Replace all points for a track with ``points`` (solver output, M20). Returns the count.
|
||||
|
||||
Each dict needs ``t_global_s, x, y, z`` and optionally ``quality`` (default None) and
|
||||
``views`` (default None). Atomic: the delete + inserts share one transaction, so a caller
|
||||
that raises mid-way never leaves a half-written track. Raises ``KeyError`` if the track
|
||||
is unknown.
|
||||
"""
|
||||
rows = list(points)
|
||||
with session_scope() as s:
|
||||
if s.get(Track, track_id) is None:
|
||||
raise KeyError(f"no track with id={track_id}")
|
||||
s.execute(delete(TrackPoint).where(TrackPoint.track_id == track_id))
|
||||
for p in rows:
|
||||
s.add(
|
||||
TrackPoint(
|
||||
track_id=track_id,
|
||||
t_global_s=float(p["t_global_s"]),
|
||||
x=float(p["x"]), y=float(p["y"]), z=float(p["z"]),
|
||||
quality=None if p.get("quality") is None else float(p["quality"]),
|
||||
views=None if p.get("views") is None else int(p["views"]),
|
||||
)
|
||||
)
|
||||
return len(rows)
|
||||
|
||||
|
||||
def patch_track(track_id: int, label: str | None = None, color: str | None = None) -> Track:
|
||||
"""Rename / recolor a track (M21 panel). Only provided fields change.
|
||||
|
||||
Raises ``KeyError`` if the id is unknown.
|
||||
"""
|
||||
with session_scope() as s:
|
||||
track = s.get(Track, track_id)
|
||||
if track is None:
|
||||
raise KeyError(f"no track with id={track_id}")
|
||||
if label is not None:
|
||||
track.label = label
|
||||
if color is not None:
|
||||
track.color = color
|
||||
s.flush()
|
||||
s.refresh(track)
|
||||
return track
|
||||
|
||||
|
||||
def delete_track(track_id: int) -> bool:
|
||||
"""Delete a track and its points. Returns ``True`` if a row was deleted, else ``False``.
|
||||
|
||||
SQLite doesn't cascade by default, so the points are removed explicitly.
|
||||
"""
|
||||
with session_scope() as s:
|
||||
track = s.get(Track, track_id)
|
||||
if track is None:
|
||||
return False
|
||||
s.execute(delete(TrackPoint).where(TrackPoint.track_id == track_id))
|
||||
s.delete(track)
|
||||
return True
|
||||
|
||||
|
||||
def has_tracks() -> bool:
|
||||
"""Whether any track exists (drives ``manifest.has_tracks``)."""
|
||||
with session_scope() as s:
|
||||
return s.scalar(select(Track.id).limit(1)) is not None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Annotations
|
||||
# ---------------------------------------------------------------------------
|
||||
@ -658,7 +366,6 @@ def add_annotation(
|
||||
y0: float,
|
||||
x1: float,
|
||||
y1: float,
|
||||
event_id: int | None = None,
|
||||
resolved_anchor_id: int | None = None,
|
||||
) -> Annotation:
|
||||
with session_scope() as s:
|
||||
@ -666,7 +373,6 @@ def add_annotation(
|
||||
video_id=video_id,
|
||||
t_video_s=t_video_s,
|
||||
x0=x0, y0=y0, x1=x1, y1=y1,
|
||||
event_id=event_id,
|
||||
resolved_anchor_id=resolved_anchor_id,
|
||||
)
|
||||
s.add(annotation)
|
||||
@ -675,15 +381,11 @@ def add_annotation(
|
||||
return annotation
|
||||
|
||||
|
||||
def get_annotations(
|
||||
video_id: int | None = None, event_id: int | None = None
|
||||
) -> list[Annotation]:
|
||||
def get_annotations(video_id: int | None = None) -> list[Annotation]:
|
||||
with session_scope() as s:
|
||||
stmt = select(Annotation).order_by(Annotation.id)
|
||||
if video_id is not None:
|
||||
stmt = stmt.where(Annotation.video_id == video_id)
|
||||
if event_id is not None:
|
||||
stmt = stmt.where(Annotation.event_id == event_id)
|
||||
return list(s.scalars(stmt))
|
||||
|
||||
|
||||
|
||||
@ -1,152 +0,0 @@
|
||||
"""Auto-director: events -> keyframed camera path (spec M11, lane E).
|
||||
|
||||
Contract (frozen at foundation2 merge — phase-5 contract #2): :func:`generate_path`
|
||||
returns a **camPath-compatible** dict::
|
||||
|
||||
{"version": 1,
|
||||
"keyframes": [{"t_global": float, "pos": [x, y, z], "quat": [x, y, z, w], "fov": float}]}
|
||||
|
||||
Poses are **Three.js scene space**, quaternion order **[x, y, z, w]** — convert from the
|
||||
COLMAP rows via :func:`festival4d.geometry.colmap_to_threejs`, never inline (pitfall #1).
|
||||
|
||||
Deterministic v1 rules — no ML: take the top-N events by confidence (ties -> earlier);
|
||||
for each, choose the registered camera with a pose nearest the event time; keyframes at
|
||||
``t_event - lead_s`` and ``t_event + duration``; when ``config.BEATS_JSON`` exists, snap
|
||||
keyframe times to the nearest beat (no beats file -> no snapping — degrade, don't block);
|
||||
FOV from that camera's intrinsics (``2 * atan(height / (2 * fy))`` in degrees, the same
|
||||
formula as ``scene3d.snapTo``).
|
||||
|
||||
Degradation (house style): no events -> valid empty path; no registered poses -> valid
|
||||
empty path with a clear ``note`` (the response always round-trips through
|
||||
``camPath.fromJSON``); no beats.json -> same path, just unsnapped. Never a crash.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
|
||||
from festival4d import config, db
|
||||
from festival4d.geometry import colmap_to_threejs, mat_to_quat
|
||||
|
||||
log = logging.getLogger("festival4d.director")
|
||||
|
||||
DEFAULT_EVENT_DURATION_S = 1.0 # used when an event has no duration_s
|
||||
|
||||
|
||||
def _empty(note: str | None = None) -> dict:
|
||||
path: dict = {"version": 1, "keyframes": []}
|
||||
if note:
|
||||
path["note"] = note
|
||||
log.info("director: %s -> empty path", note)
|
||||
return path
|
||||
|
||||
|
||||
def _load_beats() -> list[float]:
|
||||
"""The beat grid from beats.json, if valid; otherwise [] (no snapping — pitfall #5)."""
|
||||
if not config.BEATS_JSON.exists():
|
||||
return []
|
||||
try:
|
||||
beats = json.loads(config.BEATS_JSON.read_text()).get("beats_s", [])
|
||||
return sorted(float(b) for b in beats)
|
||||
except (ValueError, TypeError, AttributeError) as exc:
|
||||
log.warning("director: unreadable beats.json (%s) — skipping beat snap", exc)
|
||||
return []
|
||||
|
||||
|
||||
def _snap_to_beat(t: float, beats: list[float]) -> float:
|
||||
"""Nearest beat to t (beats sorted ascending); t unchanged when the grid is empty."""
|
||||
if not beats:
|
||||
return t
|
||||
return min(beats, key=lambda b: abs(b - t))
|
||||
|
||||
|
||||
def _nearest_pose(poses: list, t_video: float):
|
||||
"""The registered pose nearest a local time, and its |dt|. Poses are t_video_s-sorted."""
|
||||
best = min(poses, key=lambda p: abs(p.t_video_s - t_video))
|
||||
return best, abs(best.t_video_s - t_video)
|
||||
|
||||
|
||||
def _keyframe(pose, video, t_global: float) -> dict:
|
||||
"""Build one frozen-shape keyframe from a COLMAP pose row (contract #2).
|
||||
|
||||
The COLMAP [w,x,y,z] world->camera pose goes through the frozen
|
||||
``geometry.colmap_to_threejs`` (never inline — pitfall #1); the resulting Three.js
|
||||
world rotation matrix becomes the camPath quaternion, reordered to Three.js [x,y,z,w].
|
||||
"""
|
||||
position, rotation = colmap_to_threejs(
|
||||
[pose.qw, pose.qx, pose.qy, pose.qz], [pose.tx, pose.ty, pose.tz]
|
||||
)
|
||||
qw, qx, qy, qz = mat_to_quat(rotation) # [w,x,y,z] of the *Three.js* world rotation
|
||||
fov_deg = math.degrees(2.0 * math.atan(video.height / (2.0 * pose.fy))) # scene3d.snapTo
|
||||
return {
|
||||
"t_global": float(t_global),
|
||||
"pos": [float(v) for v in position],
|
||||
"quat": [float(qx), float(qy), float(qz), float(qw)], # Three.js order [x,y,z,w]
|
||||
"fov": float(fov_deg),
|
||||
}
|
||||
|
||||
|
||||
def generate_path(top_n: int = 8, lead_s: float = 2.0) -> dict:
|
||||
"""Build a camPath JSON dict from the detected events (see module doc)."""
|
||||
events = db.get_events()
|
||||
if not events:
|
||||
return _empty("no events to direct (run `python -m festival4d events`)")
|
||||
|
||||
videos = db.get_videos()
|
||||
# Registered poses per video (interpolated rows are excluded: the director only cuts
|
||||
# to cameras that were actually solved at that moment).
|
||||
tracks: dict[int, list] = {}
|
||||
for video in videos:
|
||||
registered = [p for p in db.get_poses(video.id) if p.registered]
|
||||
if registered:
|
||||
tracks[video.id] = registered
|
||||
if not tracks:
|
||||
return _empty("no registered camera poses (run `python -m festival4d reconstruct`)")
|
||||
video_by_id = {v.id: v for v in videos}
|
||||
|
||||
# Top-N events by confidence, ties -> earlier; then shoot them in chronological order.
|
||||
ranked = sorted(events, key=lambda e: (-(e.confidence or 0.0), e.t_global_s, e.id))
|
||||
chosen = sorted(ranked[: max(0, int(top_n))], key=lambda e: (e.t_global_s, e.id))
|
||||
|
||||
beats = _load_beats()
|
||||
|
||||
keyframes: list[dict] = []
|
||||
for event in chosen:
|
||||
# The registered camera whose pose track comes closest to the event time.
|
||||
best_id, best_dt = None, None
|
||||
for vid, poses in sorted(tracks.items()): # sorted -> deterministic tie-break
|
||||
video = video_by_id[vid]
|
||||
tv_event = config.t_video_from_global(
|
||||
event.t_global_s, video.offset_ms or 0.0, video.drift_ppm or 0.0
|
||||
)
|
||||
_, dt = _nearest_pose(poses, tv_event)
|
||||
if best_dt is None or dt < best_dt:
|
||||
best_id, best_dt = vid, dt
|
||||
video = video_by_id[best_id]
|
||||
poses = tracks[best_id]
|
||||
|
||||
duration = event.duration_s if event.duration_s is not None else DEFAULT_EVENT_DURATION_S
|
||||
for t_key in (event.t_global_s - lead_s, event.t_global_s + duration):
|
||||
t_key = _snap_to_beat(max(0.0, t_key), beats)
|
||||
tv_key = config.t_video_from_global(
|
||||
t_key, video.offset_ms or 0.0, video.drift_ppm or 0.0
|
||||
)
|
||||
pose, _ = _nearest_pose(poses, tv_key)
|
||||
keyframes.append(_keyframe(pose, video, t_key))
|
||||
|
||||
# Chronological, and collapse keyframes that landed on the same instant (adjacent events
|
||||
# snapping to a shared beat) — camPath needs a monotone timeline.
|
||||
keyframes.sort(key=lambda k: k["t_global"])
|
||||
deduped: list[dict] = []
|
||||
for kf in keyframes:
|
||||
if deduped and abs(kf["t_global"] - deduped[-1]["t_global"]) < 1e-6:
|
||||
continue
|
||||
deduped.append(kf)
|
||||
|
||||
log.info(
|
||||
"director: %d event(s) -> %d keyframe(s)%s",
|
||||
len(chosen), len(deduped), " (beat-snapped)" if beats else "",
|
||||
)
|
||||
return {"version": 1, "keyframes": deduped}
|
||||
@ -102,18 +102,13 @@ _CLASSIFY_PROMPT = (
|
||||
# only pydantic present; construction reads the key/endpoint from the environment.
|
||||
# ---------------------------------------------------------------------------
|
||||
class GeminiClassifier:
|
||||
"""Default provider — Gemini flash tier, native video input (``GEMINI_API_KEY``).
|
||||
"""Default provider — Gemini 2.5 Flash, native video input (``GEMINI_API_KEY``)."""
|
||||
|
||||
Model is overridable via ``FESTIVAL4D_GEMINI_MODEL``. Default was ``gemini-2.5-flash``
|
||||
until Google retired it for new API keys (404 "no longer available to new users",
|
||||
field-found 2026-07-16); ``gemini-3.1-flash-lite`` is the cheapest current model verified
|
||||
to accept inline video + JSON-schema structured output.
|
||||
"""
|
||||
model = "gemini-2.5-flash"
|
||||
|
||||
def __init__(self) -> None:
|
||||
from google import genai
|
||||
|
||||
self.model = os.environ.get("FESTIVAL4D_GEMINI_MODEL", "gemini-3.1-flash-lite")
|
||||
self._genai = genai
|
||||
self._client = genai.Client() # reads GEMINI_API_KEY from the environment
|
||||
|
||||
|
||||
@ -11,78 +11,18 @@ from __future__ import annotations
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
||||
log = logging.getLogger("festival4d.frames")
|
||||
|
||||
|
||||
def sharpness(image) -> float:
|
||||
"""Variance of the Laplacian of an image (OpenCV) — higher is sharper.
|
||||
|
||||
Accepts a BGR or grayscale ``ndarray`` (as returned by ``cv2.VideoCapture.read``).
|
||||
The variance of the Laplacian is the standard focus/blur measure: a sharp frame has
|
||||
strong high-frequency edges (high variance), a blurred one is smooth (low variance).
|
||||
"""
|
||||
import cv2
|
||||
|
||||
img = np.asarray(image)
|
||||
if img.ndim == 3:
|
||||
img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
|
||||
return float(cv2.Laplacian(img, cv2.CV_64F).var())
|
||||
"""Variance of the Laplacian of a grayscale image (OpenCV) — higher is sharper."""
|
||||
raise NotImplementedError("lane B (M2): implement sharpness")
|
||||
|
||||
|
||||
def sample_frames(video_path: Path, video_id: int, out_dir: Path,
|
||||
target_fps: float = 2.0, window_s: float = 0.5) -> list[Path]:
|
||||
"""Sample the sharpest frame per ``window_s`` window from ``video_path`` (spec M2).
|
||||
|
||||
Decodes every frame, buckets them into non-overlapping ``window_s``-second windows, and
|
||||
keeps the single sharpest frame (variance of Laplacian) in each window. This yields
|
||||
~``1 / window_s`` frames per second (≈ ``target_fps`` at the defaults), biased toward the
|
||||
in-focus frames COLMAP needs. Writes JPEGs named ``{video_id}_{frame_idx}.jpg`` (the
|
||||
original decoded ``frame_idx``, so ``t_video = frame_idx / fps``) into ``out_dir``.
|
||||
|
||||
Returns the written JPEG paths, ordered by frame index.
|
||||
Returns the written JPEG paths (named ``{video_id}_{frame_idx}.jpg``).
|
||||
"""
|
||||
import cv2
|
||||
|
||||
out_dir = Path(out_dir)
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
cap = cv2.VideoCapture(str(video_path))
|
||||
if not cap.isOpened():
|
||||
raise RuntimeError(f"could not open video for frame sampling: {video_path}")
|
||||
try:
|
||||
fps = float(cap.get(cv2.CAP_PROP_FPS))
|
||||
if not np.isfinite(fps) or fps <= 0.0:
|
||||
log.warning("frames: %s reported fps=%s; falling back to 30.0", video_path, fps)
|
||||
fps = 30.0
|
||||
bucket_s = float(window_s) if window_s and window_s > 0 else 0.5
|
||||
|
||||
# window_idx -> (sharpness, frame_idx, frame_bgr). Only the current best per window
|
||||
# is retained, so memory stays ~O(number of windows), not O(frames).
|
||||
best: dict[int, tuple[float, int, np.ndarray]] = {}
|
||||
frame_idx = 0
|
||||
while True:
|
||||
ok, frame = cap.read()
|
||||
if not ok:
|
||||
break
|
||||
t = frame_idx / fps
|
||||
win = int(t / bucket_s)
|
||||
s = sharpness(frame)
|
||||
cur = best.get(win)
|
||||
if cur is None or s > cur[0]:
|
||||
best[win] = (s, frame_idx, frame)
|
||||
frame_idx += 1
|
||||
finally:
|
||||
cap.release()
|
||||
|
||||
written: list[Path] = []
|
||||
for win in sorted(best):
|
||||
_, fidx, frame = best[win]
|
||||
path = out_dir / f"{video_id}_{fidx}.jpg"
|
||||
if not cv2.imwrite(str(path), frame):
|
||||
raise RuntimeError(f"failed to write frame JPEG: {path}")
|
||||
written.append(path)
|
||||
log.info("frames: %s -> %d frames sampled into %s", Path(video_path).name,
|
||||
len(written), out_dir)
|
||||
return written
|
||||
raise NotImplementedError("lane B (M2): implement sample_frames")
|
||||
|
||||
@ -121,8 +121,7 @@ def colmap_to_threejs(q: ArrayLike, t: ArrayLike) -> tuple[NDArray[np.float64],
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Lane B (spec M2 + M8): pose interpolation, pixel ray casting, two-view triangulation,
|
||||
# and the single-view nearest-point fallback. Signatures FROZEN; bodies implemented below.
|
||||
# Lane B stubs (spec M2 + M8). Signatures FROZEN; bodies raise NotImplementedError.
|
||||
# ---------------------------------------------------------------------------
|
||||
def slerp_pose(
|
||||
q0: ArrayLike,
|
||||
@ -150,39 +149,7 @@ def slerp_pose(
|
||||
-------
|
||||
(q, t) : the interpolated quaternion ``[w, x, y, z]`` and translation ``[x, y, z]``.
|
||||
"""
|
||||
a = float(alpha)
|
||||
q0 = np.asarray(q0, dtype=np.float64).reshape(4)
|
||||
q1 = np.asarray(q1, dtype=np.float64).reshape(4)
|
||||
n0 = np.linalg.norm(q0)
|
||||
n1 = np.linalg.norm(q1)
|
||||
if n0 < 1e-12 or n1 < 1e-12:
|
||||
raise ValueError("slerp endpoint quaternion has near-zero norm")
|
||||
q0 = q0 / n0
|
||||
q1 = q1 / n1
|
||||
|
||||
# Double cover: pick the sign of q1 that lies on the same hemisphere as q0, so slerp
|
||||
# takes the shorter arc (a rotation and its negation are the same orientation).
|
||||
dot = float(np.dot(q0, q1))
|
||||
if dot < 0.0:
|
||||
q1 = -q1
|
||||
dot = -dot
|
||||
dot = min(1.0, max(-1.0, dot))
|
||||
|
||||
if dot > 0.9995:
|
||||
# Endpoints almost coincide: nlerp is numerically safe and visually identical.
|
||||
q_interp = q0 + a * (q1 - q0)
|
||||
else:
|
||||
theta_0 = np.arccos(dot)
|
||||
sin_0 = np.sin(theta_0)
|
||||
s0 = np.sin((1.0 - a) * theta_0) / sin_0
|
||||
s1 = np.sin(a * theta_0) / sin_0
|
||||
q_interp = s0 * q0 + s1 * q1
|
||||
q_interp = q_interp / np.linalg.norm(q_interp)
|
||||
|
||||
t0 = np.asarray(t0, dtype=np.float64).reshape(3)
|
||||
t1 = np.asarray(t1, dtype=np.float64).reshape(3)
|
||||
t_interp = (1.0 - a) * t0 + a * t1
|
||||
return q_interp, t_interp
|
||||
raise NotImplementedError("lane B (M2): implement slerp_pose")
|
||||
|
||||
|
||||
def ray_from_pixel(
|
||||
@ -207,20 +174,7 @@ def ray_from_pixel(
|
||||
direction : ndarray, shape (3,)
|
||||
Unit ray direction in world coords, pointing into the scene.
|
||||
"""
|
||||
R = quat_to_mat(q) # world -> cam
|
||||
t_vec = np.asarray(t, dtype=np.float64).reshape(3)
|
||||
origin = -R.T @ t_vec # camera center in world coords
|
||||
|
||||
# Pinhole back-projection. A world point X projects with x_cam = R X + t and
|
||||
# px = fx * x_cam.x / x_cam.z + cx, py = fy * x_cam.y / x_cam.z + cy
|
||||
# (COLMAP camera axes: +x right, +y down, +z forward). So the camera-space direction
|
||||
# through pixel (px, py) is [(px-cx)/fx, (py-cy)/fy, 1], pointing forward into the scene.
|
||||
d_cam = np.array([(px - cx) / fx, (py - cy) / fy, 1.0], dtype=np.float64)
|
||||
d_world = R.T @ d_cam # rotate direction cam -> world
|
||||
norm = np.linalg.norm(d_world)
|
||||
if norm < 1e-12:
|
||||
raise ValueError("degenerate ray direction")
|
||||
return origin, d_world / norm
|
||||
raise NotImplementedError("lane B (M8): implement ray_from_pixel")
|
||||
|
||||
|
||||
def triangulate_rays(
|
||||
@ -240,37 +194,7 @@ def triangulate_rays(
|
||||
reject the triangulation when the rays are near-parallel or ``gap`` exceeds the
|
||||
spec threshold (0.5 scene units).
|
||||
"""
|
||||
oa = np.asarray(origin_a, dtype=np.float64).reshape(3)
|
||||
ob = np.asarray(origin_b, dtype=np.float64).reshape(3)
|
||||
da = np.asarray(dir_a, dtype=np.float64).reshape(3)
|
||||
db = np.asarray(dir_b, dtype=np.float64).reshape(3)
|
||||
na, nb = np.linalg.norm(da), np.linalg.norm(db)
|
||||
if na < 1e-12 or nb < 1e-12:
|
||||
raise ValueError("triangulate_rays: zero-length direction")
|
||||
da = da / na
|
||||
db = db / nb
|
||||
|
||||
# Shortest segment between two lines P(s)=oa+s*da, Q(u)=ob+u*db. Minimize |P-Q|^2.
|
||||
# With unit directions: b = da.db, denom = 1 - b^2 (0 when parallel).
|
||||
w0 = oa - ob
|
||||
b = float(np.dot(da, db))
|
||||
d = float(np.dot(da, w0))
|
||||
e = float(np.dot(db, w0))
|
||||
denom = 1.0 - b * b
|
||||
if denom < 1e-9:
|
||||
# Near-parallel: no unique closest pair. Anchor on oa, take the closest point on
|
||||
# line b to it; gap is the line-to-line perpendicular distance. Callers reject
|
||||
# near-parallel rays up front, so this branch just stays numerically safe.
|
||||
s = 0.0
|
||||
u = e
|
||||
else:
|
||||
s = (b * e - d) / denom
|
||||
u = (e - b * d) / denom
|
||||
pa = oa + s * da
|
||||
pb = ob + u * db
|
||||
point = 0.5 * (pa + pb)
|
||||
gap = float(np.linalg.norm(pa - pb))
|
||||
return point, gap
|
||||
raise NotImplementedError("lane B (M8): implement triangulate_rays")
|
||||
|
||||
|
||||
def nearest_point_on_ray(
|
||||
@ -290,26 +214,4 @@ def nearest_point_on_ray(
|
||||
point : ndarray shape (3,) or None
|
||||
The selected point-cloud point, or ``None`` if none lie within ``radius``.
|
||||
"""
|
||||
o = np.asarray(origin, dtype=np.float64).reshape(3)
|
||||
d = np.asarray(direction, dtype=np.float64).reshape(3)
|
||||
nd = np.linalg.norm(d)
|
||||
if nd < 1e-12:
|
||||
raise ValueError("nearest_point_on_ray: zero-length direction")
|
||||
d = d / nd
|
||||
|
||||
pts = np.asarray(points, dtype=np.float64).reshape(-1, 3)
|
||||
if len(pts) == 0:
|
||||
return None
|
||||
|
||||
v = pts - o # origin -> each point
|
||||
proj = v @ d # signed distance along the ray
|
||||
perp = v - np.outer(proj, d) # component perpendicular to the ray
|
||||
perp_dist = np.linalg.norm(perp, axis=1)
|
||||
|
||||
# In front of the origin and inside the cylinder of the given radius.
|
||||
mask = (proj > 0.0) & (perp_dist <= radius)
|
||||
if not np.any(mask):
|
||||
return None
|
||||
idx_in = np.where(mask)[0]
|
||||
best = idx_in[np.argmin(perp_dist[idx_in])]
|
||||
return pts[best].copy()
|
||||
raise NotImplementedError("lane B (M8): implement nearest_point_on_ray")
|
||||
|
||||
@ -1,123 +1,26 @@
|
||||
"""Video ingest: probe files in ``data/raw`` and extract audio (spec M1, lane A).
|
||||
|
||||
For each file in ``config.RAW_DIR``: ffprobe it, insert/update the ``videos`` row via
|
||||
``db`` helpers, and extract a mono 16 kHz WAV into ``config.AUDIO_DIR`` (one per video,
|
||||
keyed by ``video_id``) — that WAV is exactly what ``audio_sync.run_sync`` loads.
|
||||
|
||||
``run_ingest`` is idempotent: re-running after the synthetic fixture (which already
|
||||
registered the videos) reuses the existing rows and just re-extracts audio, so the
|
||||
demo path ``synthetic -> ingest -> sync`` works without unique-constraint collisions.
|
||||
STUB — lane A fills the bodies; the public signatures below are frozen (``cli.py`` calls
|
||||
``run_ingest``). For each file in ``config.RAW_DIR``: ffprobe it, insert/update the
|
||||
``videos`` row via ``db`` helpers, and extract a mono 16 kHz WAV into ``config.AUDIO_DIR``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
from festival4d import config, db
|
||||
|
||||
log = logging.getLogger("festival4d.ingest")
|
||||
|
||||
# Container extensions we treat as ingestible video (audio is pulled from any of them).
|
||||
_VIDEO_EXTS = {".mp4", ".mov", ".m4v", ".mkv", ".avi", ".webm"}
|
||||
|
||||
|
||||
def _require(tool: str) -> None:
|
||||
if shutil.which(tool) is None:
|
||||
raise RuntimeError(
|
||||
f"{tool} not found on PATH — required for ingest (macOS: `brew install ffmpeg`)"
|
||||
)
|
||||
|
||||
|
||||
def audio_wav_path(video_id: int) -> Path:
|
||||
"""Canonical path of a video's extracted mono WAV.
|
||||
|
||||
Shared convention between ingest (writer) and :mod:`festival4d.audio_sync` (reader),
|
||||
so neither side has to guess filenames.
|
||||
"""
|
||||
return config.AUDIO_DIR / f"{video_id}.wav"
|
||||
|
||||
|
||||
def _parse_fps(rate: str | None) -> float | None:
|
||||
"""Parse an ffprobe frame-rate string (``"30/1"`` or ``"29.97"``) to fps."""
|
||||
if not rate or rate in ("0/0", "N/A"):
|
||||
return None
|
||||
if "/" in rate:
|
||||
num, den = rate.split("/", 1)
|
||||
den = float(den)
|
||||
return float(num) / den if den else None
|
||||
return float(rate)
|
||||
|
||||
|
||||
def probe_video(path: Path) -> dict:
|
||||
"""ffprobe a video; return ``{duration_s, fps, width, height}`` (lane A / M1)."""
|
||||
_require("ffprobe")
|
||||
out = subprocess.run(
|
||||
["ffprobe", "-v", "error", "-select_streams", "v:0",
|
||||
"-show_entries",
|
||||
"stream=width,height,avg_frame_rate,r_frame_rate:format=duration",
|
||||
"-of", "json", str(path)],
|
||||
check=True, capture_output=True, text=True,
|
||||
).stdout
|
||||
data = json.loads(out)
|
||||
streams = data.get("streams") or []
|
||||
if not streams:
|
||||
raise ValueError(f"{path.name}: no video stream found")
|
||||
stream = streams[0]
|
||||
# avg_frame_rate is the true average; r_frame_rate is a fallback for odd containers.
|
||||
fps = _parse_fps(stream.get("avg_frame_rate")) or _parse_fps(stream.get("r_frame_rate")) or 0.0
|
||||
duration = float(data.get("format", {}).get("duration") or 0.0)
|
||||
return {
|
||||
"duration_s": duration,
|
||||
"fps": float(fps),
|
||||
"width": int(stream["width"]),
|
||||
"height": int(stream["height"]),
|
||||
}
|
||||
raise NotImplementedError("lane A (M1): implement probe_video")
|
||||
|
||||
|
||||
def extract_audio(path: Path, out_wav: Path, sample_rate: int = 16_000) -> Path:
|
||||
"""Extract a mono ``sample_rate`` 16-bit PCM WAV from ``path`` via ffmpeg (lane A / M1)."""
|
||||
_require("ffmpeg")
|
||||
out_wav.parent.mkdir(parents=True, exist_ok=True)
|
||||
subprocess.run(
|
||||
["ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
|
||||
"-i", str(path),
|
||||
"-vn", # drop video
|
||||
"-ac", "1", # mono
|
||||
"-ar", str(sample_rate), # resample
|
||||
"-c:a", "pcm_s16le", # uncompressed PCM (lossless for sync)
|
||||
str(out_wav)],
|
||||
check=True,
|
||||
)
|
||||
return out_wav
|
||||
|
||||
|
||||
def extract_audio_hq(path: Path, out_m4a: Path) -> Path:
|
||||
"""Extract a listening-quality stereo AAC track (48 kHz, 192 kbps) from ``path``.
|
||||
|
||||
Feeds the frontend's WebAudio master clock (``GET /api/audio/{id}``). AAC-in-m4a is
|
||||
re-encoded (not stream-copied) so any source codec — including webm/Opus from browser
|
||||
capture — decodes in every browser's ``decodeAudioData``. Written atomically (tmp +
|
||||
rename) so a request racing the first extraction never reads a half-written file.
|
||||
"""
|
||||
_require("ffmpeg")
|
||||
out_m4a.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = out_m4a.with_suffix(".tmp.m4a")
|
||||
subprocess.run(
|
||||
["ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
|
||||
"-i", str(path),
|
||||
"-vn",
|
||||
"-ac", "2",
|
||||
"-ar", "48000",
|
||||
"-c:a", "aac", "-b:a", "192k",
|
||||
str(tmp)],
|
||||
check=True,
|
||||
)
|
||||
tmp.replace(out_m4a)
|
||||
return out_m4a
|
||||
"""Extract a mono ``sample_rate`` WAV from ``path`` via ffmpeg (lane A / M1)."""
|
||||
raise NotImplementedError("lane A (M1): implement extract_audio")
|
||||
|
||||
|
||||
def run_ingest() -> list[dict]:
|
||||
@ -125,39 +28,4 @@ def run_ingest() -> list[dict]:
|
||||
|
||||
Entrypoint for ``python -m festival4d ingest``. Returns a per-video summary list.
|
||||
"""
|
||||
db.init_engine()
|
||||
db.init_db()
|
||||
|
||||
files = sorted(
|
||||
p for p in config.RAW_DIR.glob("*") if p.suffix.lower() in _VIDEO_EXTS
|
||||
)
|
||||
if not files:
|
||||
log.warning("ingest: no video files in %s (drop clips there first)", config.RAW_DIR)
|
||||
return []
|
||||
|
||||
existing = {v.filename: v for v in db.get_videos()}
|
||||
summary: list[dict] = []
|
||||
for path in files:
|
||||
info = probe_video(path)
|
||||
video = existing.get(path.name)
|
||||
reused = video is not None
|
||||
if video is None:
|
||||
video = db.add_video(
|
||||
filename=path.name,
|
||||
duration_s=info["duration_s"], fps=info["fps"],
|
||||
width=info["width"], height=info["height"],
|
||||
)
|
||||
wav = audio_wav_path(video.id)
|
||||
extract_audio(path, wav)
|
||||
summary.append({
|
||||
"video_id": video.id, "filename": path.name,
|
||||
"duration_s": info["duration_s"], "fps": info["fps"],
|
||||
"width": info["width"], "height": info["height"],
|
||||
"audio_wav": str(wav), "reused_db_row": reused,
|
||||
})
|
||||
log.info("ingest: %s -> video_id=%d (%.1fs, %.2f fps, %dx%d)%s",
|
||||
path.name, video.id, info["duration_s"], info["fps"],
|
||||
info["width"], info["height"], " [reused row]" if reused else "")
|
||||
|
||||
log.info("ingest: %d video(s) ready; audio in %s", len(summary), config.AUDIO_DIR)
|
||||
return summary
|
||||
raise NotImplementedError("lane A (M1): implement run_ingest")
|
||||
|
||||
@ -1,152 +0,0 @@
|
||||
"""Annotation -> 3D point resolution (spec M8, integration phase).
|
||||
|
||||
Given a 2D bounding box drawn on one video at a local video time, recover a 3D scene point:
|
||||
|
||||
1. Cast a ray from that camera through the bbox center, using the pose interpolated at
|
||||
``t_video_s`` and the stored pinhole intrinsics.
|
||||
2. If another annotation for the **same event** exists from a **different video**,
|
||||
triangulate the two rays (reject near-parallel rays or a mutual gap > 0.5 scene units).
|
||||
3. Else take the nearest point-cloud point to the ray (within a 0.3-unit cylinder).
|
||||
4. Else return the ray at depth = distance from the camera to the point-cloud centroid.
|
||||
|
||||
This module is pure orchestration over lane B's frozen geometry primitives
|
||||
(:func:`geometry.ray_from_pixel`, :func:`geometry.triangulate_rays`,
|
||||
:func:`geometry.nearest_point_on_ray`, :func:`geometry.slerp_pose`) — it never reimplements
|
||||
the pose math. It reads the point cloud via :func:`synthetic.read_ply`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
|
||||
import numpy as np
|
||||
|
||||
from festival4d import config, db, geometry
|
||||
from festival4d.synthetic import read_ply
|
||||
|
||||
log = logging.getLogger("festival4d.resolve")
|
||||
|
||||
TRIANGULATION_MAX_GAP = 0.5 # scene units — reject looser two-view intersections (spec M8)
|
||||
NEAREST_RADIUS = 0.3 # scene units — cylinder for the point-cloud fallback (spec M8)
|
||||
PARALLEL_DOT = 0.9995 # |cos(angle)| above this => rays too parallel to triangulate
|
||||
|
||||
|
||||
@dataclass
|
||||
class Resolution:
|
||||
"""Outcome of resolving one annotation to a 3D point."""
|
||||
|
||||
point: list[float] | None
|
||||
method: str # 'triangulated' | 'nearest_point' | 'ray_depth' | 'no_geometry'
|
||||
gap: float | None = None # mutual-approach distance when method == 'triangulated'
|
||||
|
||||
|
||||
def _pose_tuple(p) -> tuple[list[float], list[float], tuple[float, float, float, float]]:
|
||||
return (
|
||||
[p.qw, p.qx, p.qy, p.qz],
|
||||
[p.tx, p.ty, p.tz],
|
||||
(p.fx, p.fy, p.cx, p.cy),
|
||||
)
|
||||
|
||||
|
||||
def _pose_at(poses, t_video_s: float):
|
||||
"""Interpolate a COLMAP pose ``(q, t, (fx,fy,cx,cy))`` at ``t_video_s``.
|
||||
|
||||
``poses`` is ascending by ``t_video_s`` (as returned by :func:`db.get_poses`). Clamps to
|
||||
the endpoints — no extrapolation, matching the frontend ``poseAt`` and spec M2.
|
||||
"""
|
||||
if not poses:
|
||||
return None
|
||||
if t_video_s <= poses[0].t_video_s:
|
||||
return _pose_tuple(poses[0])
|
||||
if t_video_s >= poses[-1].t_video_s:
|
||||
return _pose_tuple(poses[-1])
|
||||
|
||||
lo, hi = 0, len(poses) - 1
|
||||
while hi - lo > 1:
|
||||
mid = (lo + hi) // 2
|
||||
if poses[mid].t_video_s <= t_video_s:
|
||||
lo = mid
|
||||
else:
|
||||
hi = mid
|
||||
a, b = poses[lo], poses[hi]
|
||||
span = b.t_video_s - a.t_video_s
|
||||
alpha = (t_video_s - a.t_video_s) / span if span > 1e-9 else 0.0
|
||||
q, t = geometry.slerp_pose(
|
||||
[a.qw, a.qx, a.qy, a.qz], [a.tx, a.ty, a.tz],
|
||||
[b.qw, b.qx, b.qy, b.qz], [b.tx, b.ty, b.tz],
|
||||
alpha,
|
||||
)
|
||||
# Intrinsics don't interpolate meaningfully across a fixed camera; use the earlier pose's.
|
||||
return list(q), list(t), (a.fx, a.fy, a.cx, a.cy)
|
||||
|
||||
|
||||
def _ray_for_annotation(video_id: int, t_video_s: float, bbox):
|
||||
"""World-space ray (origin, unit direction) through the bbox center, or ``None``.
|
||||
|
||||
Returns ``None`` when the video has no poses (reconstruction absent for that camera).
|
||||
"""
|
||||
video = db.get_video(video_id)
|
||||
if video is None:
|
||||
raise KeyError(f"no video with id={video_id}")
|
||||
pose = _pose_at(db.get_poses(video_id), t_video_s)
|
||||
if pose is None:
|
||||
return None
|
||||
q, t, (fx, fy, cx, cy) = pose
|
||||
x0, y0, x1, y1 = bbox
|
||||
px = (x0 + x1) / 2.0 * video.width
|
||||
py = (y0 + y1) / 2.0 * video.height
|
||||
return geometry.ray_from_pixel(q, t, fx, fy, cx, cy, px, py)
|
||||
|
||||
|
||||
def _load_points() -> np.ndarray | None:
|
||||
if not config.POINTS_PLY.exists():
|
||||
return None
|
||||
try:
|
||||
pts, _colors = read_ply(config.POINTS_PLY)
|
||||
except Exception as exc: # pragma: no cover - defensive
|
||||
log.warning("resolve: failed to read point cloud: %s", exc)
|
||||
return None
|
||||
return np.asarray(pts, dtype=np.float64).reshape(-1, 3)
|
||||
|
||||
|
||||
def resolve_annotation(video_id: int, t_video_s: float, bbox, event_id: int | None = None) -> Resolution:
|
||||
"""Resolve one annotation to a 3D point (see module docstring for the cascade)."""
|
||||
ray = _ray_for_annotation(video_id, t_video_s, bbox)
|
||||
if ray is None:
|
||||
return Resolution(point=None, method="no_geometry")
|
||||
origin, direction = ray
|
||||
|
||||
# 1. Two-view triangulation against another annotation of the same event.
|
||||
if event_id is not None:
|
||||
for other in reversed(db.get_annotations()): # most recent first
|
||||
if other.event_id != event_id or other.video_id == video_id:
|
||||
continue
|
||||
other_ray = _ray_for_annotation(
|
||||
other.video_id, other.t_video_s, [other.x0, other.y0, other.x1, other.y1]
|
||||
)
|
||||
if other_ray is None:
|
||||
continue
|
||||
o2, d2 = other_ray
|
||||
if abs(float(np.dot(direction, d2))) >= PARALLEL_DOT:
|
||||
continue # near-parallel: no reliable intersection
|
||||
point, gap = geometry.triangulate_rays(origin, direction, o2, d2)
|
||||
if gap <= TRIANGULATION_MAX_GAP:
|
||||
return Resolution(
|
||||
point=[float(v) for v in point], method="triangulated", gap=float(gap)
|
||||
)
|
||||
|
||||
# 2/3. Point-cloud fallbacks.
|
||||
pts = _load_points()
|
||||
if pts is not None and len(pts):
|
||||
near = geometry.nearest_point_on_ray(origin, direction, pts, NEAREST_RADIUS)
|
||||
if near is not None:
|
||||
return Resolution(point=[float(v) for v in near], method="nearest_point")
|
||||
centroid = pts.mean(axis=0)
|
||||
depth = float(np.linalg.norm(centroid - origin))
|
||||
pt = origin + direction * depth
|
||||
return Resolution(point=[float(v) for v in pt], method="ray_depth")
|
||||
|
||||
# 4. No point cloud at all: park at unit depth so a usable anchor still comes back.
|
||||
pt = origin + direction * 1.0
|
||||
return Resolution(point=[float(v) for v in pt], method="ray_depth")
|
||||
@ -1,61 +1,33 @@
|
||||
"""COLMAP orchestration, model parsing, scene normalization, pose export (spec M2, lane B).
|
||||
|
||||
Pipeline (spec M2):
|
||||
STUB — lane B fills the bodies; the public signatures below are frozen (``cli.py`` calls
|
||||
``run_reconstruct``).
|
||||
|
||||
Pipeline (spec M2, for lane B's reference):
|
||||
- Sample frames (``frames.sample_frames``), then drive the COLMAP CLI via subprocess:
|
||||
``feature_extractor`` (OPENCV model, single camera per folder) -> ``exhaustive_matcher``
|
||||
(small frame counts; exhaustive covers within- and cross-video pairs) -> ``mapper`` ->
|
||||
``image_undistorter`` + ``model_converter`` to a PINHOLE TXT model. Parse ``images.txt`` /
|
||||
``cameras.txt`` / ``points3D.txt`` by hand (no pycolmap).
|
||||
``feature_extractor`` (OPENCV model, single camera per folder) -> ``sequential_matcher``
|
||||
within each video + ``exhaustive_matcher`` across -> ``mapper`` -> ``model_converter`` to
|
||||
a TXT model. Parse ``images.txt`` / ``cameras.txt`` / ``points3D.txt`` by hand (no
|
||||
pycolmap).
|
||||
- **Normalize** the scene: centroid -> origin, camera bounding sphere radius -> 10, world-up
|
||||
-> average camera up. The same similarity transform is applied to poses and points.
|
||||
-> average camera up. Apply the same similarity transform to poses and points.
|
||||
- **Interpolate** poses for unregistered frames (slerp rotation, lerp translation, mark
|
||||
``registered=False``); never extrapolate past the first/last registered frame of a video.
|
||||
- Export ``config.POINTS_PLY`` (binary little-endian, matching ``synthetic.write_ply``) and
|
||||
fill ``camera_poses`` via ``db.set_poses``.
|
||||
- **Failure handling** (spec M2): COLMAP optional at runtime (``shutil.which("colmap")``); if
|
||||
it registers < 60% of sampled frames or < 2 videos into one model, print a clear diagnostic
|
||||
and leave existing poses untouched — never corrupt the DB. The app degrades to "synced
|
||||
videos, no 3D".
|
||||
``registered=False``); never extrapolate past the first/last registered frame.
|
||||
- Export ``config.POINTS_PLY`` (binary little-endian, matching ``synthetic.write_ply``'s
|
||||
format) and fill ``camera_poses`` via ``db.set_poses``.
|
||||
- **Failure handling** (spec M2): COLMAP optional at runtime (``shutil.which("colmap")``);
|
||||
if it registers < 60% of frames or < 2 videos into one model, print a clear diagnostic and
|
||||
leave existing poses untouched — never corrupt the DB. The app degrades to "synced videos,
|
||||
no 3D".
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import functools
|
||||
import struct
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
||||
from festival4d import config, db, frames
|
||||
from festival4d.geometry import mat_to_quat, quat_to_mat, slerp_pose
|
||||
|
||||
log = logging.getLogger("festival4d.sfm")
|
||||
|
||||
# Minimum share of sampled frames COLMAP must register, and minimum videos in one model,
|
||||
# for the reconstruction to be trusted (spec M2 failure handling).
|
||||
MIN_REGISTERED_FRACTION = 0.60
|
||||
MIN_VIDEOS_IN_MODEL = 2
|
||||
NORMALIZED_SPHERE_RADIUS = 10.0
|
||||
|
||||
# How COLMAP camera models lay out PARAMS -> (fx, fy, cx, cy). Distortion terms are ignored:
|
||||
# the stored intrinsics are the pinhole part (exact after image_undistorter's PINHOLE output,
|
||||
# approximate if a distorted model is parsed directly). Covers the models COLMAP emits here.
|
||||
_INTRINSICS_FROM_PARAMS = {
|
||||
"SIMPLE_PINHOLE": lambda p: (p[0], p[0], p[1], p[2]),
|
||||
"PINHOLE": lambda p: (p[0], p[1], p[2], p[3]),
|
||||
"SIMPLE_RADIAL": lambda p: (p[0], p[0], p[1], p[2]),
|
||||
"RADIAL": lambda p: (p[0], p[0], p[1], p[2]),
|
||||
"SIMPLE_RADIAL_FISHEYE": lambda p: (p[0], p[0], p[1], p[2]),
|
||||
"RADIAL_FISHEYE": lambda p: (p[0], p[0], p[1], p[2]),
|
||||
"OPENCV": lambda p: (p[0], p[1], p[2], p[3]),
|
||||
"OPENCV_FISHEYE": lambda p: (p[0], p[1], p[2], p[3]),
|
||||
"FULL_OPENCV": lambda p: (p[0], p[1], p[2], p[3]),
|
||||
"FOV": lambda p: (p[0], p[1], p[2], p[3]),
|
||||
"THIN_PRISM_FISHEYE": lambda p: (p[0], p[1], p[2], p[3]),
|
||||
}
|
||||
|
||||
|
||||
def colmap_available() -> bool:
|
||||
"""Whether the COLMAP binary is on PATH (spec M2 optional-at-runtime rule)."""
|
||||
@ -64,416 +36,29 @@ def colmap_available() -> bool:
|
||||
return shutil.which("colmap") is not None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# COLMAP TXT-model parsers (hand-written; no pycolmap dependency)
|
||||
# ---------------------------------------------------------------------------
|
||||
def _iter_data_lines(path: Path):
|
||||
"""Yield non-empty, non-comment lines from a COLMAP TXT file."""
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if line and not line.startswith("#"):
|
||||
yield line
|
||||
|
||||
|
||||
def parse_images_txt(path: Path) -> list[dict]:
|
||||
"""Parse a COLMAP ``images.txt`` into per-image pose records (spec M2, lane B).
|
||||
|
||||
Format: two lines per image. The first is
|
||||
``IMAGE_ID QW QX QY QZ TX TY TZ CAMERA_ID NAME``; the second is the 2D keypoint list
|
||||
(ignored here). Returns one dict per image with the world->camera quaternion
|
||||
``[w,x,y,z]``, translation, camera id, and image name.
|
||||
"""
|
||||
records: list[dict] = []
|
||||
take_pose_line = True # image data alternates: pose line, then points2D line
|
||||
for line in _iter_data_lines(path):
|
||||
if take_pose_line:
|
||||
parts = line.split()
|
||||
if len(parts) < 10:
|
||||
raise ValueError(f"malformed images.txt pose line: {line!r}")
|
||||
records.append({
|
||||
"image_id": int(parts[0]),
|
||||
"qw": float(parts[1]), "qx": float(parts[2]),
|
||||
"qy": float(parts[3]), "qz": float(parts[4]),
|
||||
"tx": float(parts[5]), "ty": float(parts[6]), "tz": float(parts[7]),
|
||||
"camera_id": int(parts[8]),
|
||||
"name": " ".join(parts[9:]), # NAME may itself contain spaces
|
||||
})
|
||||
take_pose_line = not take_pose_line
|
||||
return records
|
||||
"""Parse a COLMAP ``images.txt`` into per-image pose records (spec M2, lane B)."""
|
||||
raise NotImplementedError("lane B (M2): implement parse_images_txt")
|
||||
|
||||
|
||||
def parse_cameras_txt(path: Path) -> dict[int, dict]:
|
||||
"""Parse a COLMAP ``cameras.txt`` into ``camera_id -> intrinsics`` (spec M2, lane B).
|
||||
|
||||
Format per line: ``CAMERA_ID MODEL WIDTH HEIGHT PARAMS[]``. Returns, per camera id, a
|
||||
dict with ``model``, ``width``, ``height``, pinhole ``fx, fy, cx, cy`` and the raw
|
||||
``params`` list.
|
||||
"""
|
||||
cameras: dict[int, dict] = {}
|
||||
for line in _iter_data_lines(path):
|
||||
parts = line.split()
|
||||
if len(parts) < 4:
|
||||
raise ValueError(f"malformed cameras.txt line: {line!r}")
|
||||
cam_id = int(parts[0])
|
||||
model = parts[1]
|
||||
width, height = int(parts[2]), int(parts[3])
|
||||
params = [float(x) for x in parts[4:]]
|
||||
if model not in _INTRINSICS_FROM_PARAMS:
|
||||
raise ValueError(f"unsupported COLMAP camera model {model!r}")
|
||||
try:
|
||||
fx, fy, cx, cy = _INTRINSICS_FROM_PARAMS[model](params)
|
||||
except IndexError:
|
||||
raise ValueError(
|
||||
f"cameras.txt: model {model} has too few PARAMS: {line!r}"
|
||||
) from None
|
||||
cameras[cam_id] = {
|
||||
"model": model, "width": width, "height": height,
|
||||
"fx": float(fx), "fy": float(fy), "cx": float(cx), "cy": float(cy),
|
||||
"params": params,
|
||||
}
|
||||
return cameras
|
||||
"""Parse a COLMAP ``cameras.txt`` into ``camera_id -> intrinsics`` (spec M2, lane B)."""
|
||||
raise NotImplementedError("lane B (M2): implement parse_cameras_txt")
|
||||
|
||||
|
||||
def parse_points3d_txt(path: Path) -> tuple[np.ndarray, np.ndarray]:
|
||||
"""Parse a COLMAP ``points3D.txt`` into ``(points Nx3 float32, colors Nx3 uint8)``.
|
||||
|
||||
Format per line: ``POINT3D_ID X Y Z R G B ERROR TRACK[]`` (track ignored).
|
||||
"""
|
||||
xyz: list[tuple[float, float, float]] = []
|
||||
rgb: list[tuple[int, int, int]] = []
|
||||
for line in _iter_data_lines(path):
|
||||
parts = line.split()
|
||||
if len(parts) < 7:
|
||||
raise ValueError(f"malformed points3D.txt line: {line!r}")
|
||||
xyz.append((float(parts[1]), float(parts[2]), float(parts[3])))
|
||||
rgb.append((int(parts[4]), int(parts[5]), int(parts[6])))
|
||||
if not xyz:
|
||||
return (np.zeros((0, 3), dtype=np.float32), np.zeros((0, 3), dtype=np.uint8))
|
||||
return (np.asarray(xyz, dtype=np.float32), np.asarray(rgb, dtype=np.uint8))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Scene normalization
|
||||
# ---------------------------------------------------------------------------
|
||||
def _rotation_aligning(a: np.ndarray, b: np.ndarray) -> np.ndarray:
|
||||
"""Shortest-arc rotation matrix that maps unit vector ``a`` onto unit vector ``b``."""
|
||||
a = a / np.linalg.norm(a)
|
||||
b = b / np.linalg.norm(b)
|
||||
v = np.cross(a, b)
|
||||
c = float(np.dot(a, b))
|
||||
s = float(np.linalg.norm(v))
|
||||
if s < 1e-12:
|
||||
if c > 0: # already aligned
|
||||
return np.eye(3)
|
||||
# antiparallel: rotate 180 deg about any axis perpendicular to a
|
||||
perp = np.array([1.0, 0.0, 0.0])
|
||||
if abs(a[0]) > 0.9:
|
||||
perp = np.array([0.0, 1.0, 0.0])
|
||||
axis = np.cross(a, perp)
|
||||
axis /= np.linalg.norm(axis)
|
||||
return 2.0 * np.outer(axis, axis) - np.eye(3)
|
||||
vx = np.array([[0.0, -v[2], v[1]], [v[2], 0.0, -v[0]], [-v[1], v[0], 0.0]])
|
||||
return np.eye(3) + vx + vx @ vx * ((1.0 - c) / (s * s))
|
||||
|
||||
|
||||
def _camera_center(pose: dict) -> np.ndarray:
|
||||
"""World-space camera center ``C = -R^T t`` for a world->camera pose dict."""
|
||||
R = quat_to_mat([pose["qw"], pose["qx"], pose["qy"], pose["qz"]])
|
||||
t = np.array([pose["tx"], pose["ty"], pose["tz"]], dtype=np.float64)
|
||||
return -R.T @ t
|
||||
|
||||
|
||||
def _camera_up(pose: dict) -> np.ndarray:
|
||||
"""World-space camera up vector. COLMAP camera +y is *down*, so up = R^T @ (0,-1,0)."""
|
||||
R = quat_to_mat([pose["qw"], pose["qx"], pose["qy"], pose["qz"]])
|
||||
return R.T @ np.array([0.0, -1.0, 0.0])
|
||||
def parse_points3d_txt(path: Path):
|
||||
"""Parse a COLMAP ``points3D.txt`` into ``(points Nx3, colors Nx3)`` (spec M2, lane B)."""
|
||||
raise NotImplementedError("lane B (M2): implement parse_points3d_txt")
|
||||
|
||||
|
||||
def normalize_scene(points, poses):
|
||||
"""Similarity transform: centroid->origin, camera sphere radius->10, up->+Y (spec M2).
|
||||
|
||||
``points`` is an ``(N,3)`` array; ``poses`` a list of world->camera pose dicts (keys
|
||||
``qw,qx,qy,qz,tx,ty,tz`` plus any extras, which are preserved). The single similarity
|
||||
transform ``X' = s * Rn @ (X - c)`` (translate to the point centroid, rotate the average
|
||||
camera-up onto +Y, scale so the camera bounding sphere has radius 10) is applied to both
|
||||
the points and the poses.
|
||||
|
||||
Returns ``(points_new float32 (N,3), poses_new list[dict])``.
|
||||
"""
|
||||
pts = np.asarray(points, dtype=np.float64).reshape(-1, 3)
|
||||
poses = list(poses)
|
||||
|
||||
# 1. Translate to the point-cloud centroid.
|
||||
centroid = pts.mean(axis=0) if len(pts) else np.zeros(3)
|
||||
|
||||
# 2. Rotate so the average camera-up aligns with +Y (phones held roughly upright).
|
||||
if poses:
|
||||
ups = np.array([_camera_up(p) for p in poses])
|
||||
up_avg = ups.mean(axis=0)
|
||||
if np.linalg.norm(up_avg) > 1e-9:
|
||||
Rn = _rotation_aligning(up_avg, np.array([0.0, 1.0, 0.0]))
|
||||
else:
|
||||
Rn = np.eye(3)
|
||||
else:
|
||||
Rn = np.eye(3)
|
||||
|
||||
# 3. Scale so the farthest camera center from the centroid sits at radius 10.
|
||||
if poses:
|
||||
centers = np.array([_camera_center(p) for p in poses])
|
||||
max_dist = float(np.max(np.linalg.norm(centers - centroid, axis=1)))
|
||||
scale = NORMALIZED_SPHERE_RADIUS / max_dist if max_dist > 1e-9 else 1.0
|
||||
else:
|
||||
scale = 1.0
|
||||
|
||||
# Apply X' = s * Rn @ (X - c) to points.
|
||||
points_new = (scale * (pts - centroid) @ Rn.T).astype(np.float32)
|
||||
|
||||
# Apply the matching transform to each world->camera pose:
|
||||
# R' = R @ Rn^T , t' = s * (t + R @ c) (derived so projections are unchanged).
|
||||
poses_new: list[dict] = []
|
||||
for p in poses:
|
||||
R = quat_to_mat([p["qw"], p["qx"], p["qy"], p["qz"]])
|
||||
t = np.array([p["tx"], p["ty"], p["tz"]], dtype=np.float64)
|
||||
R_new = R @ Rn.T
|
||||
t_new = scale * (t + R @ centroid)
|
||||
q_new = mat_to_quat(R_new)
|
||||
out = dict(p)
|
||||
out["qw"], out["qx"], out["qy"], out["qz"] = (
|
||||
float(q_new[0]), float(q_new[1]), float(q_new[2]), float(q_new[3]))
|
||||
out["tx"], out["ty"], out["tz"] = float(t_new[0]), float(t_new[1]), float(t_new[2])
|
||||
poses_new.append(out)
|
||||
|
||||
return points_new, poses_new
|
||||
"""Similarity transform: centroid->origin, camera sphere radius->10, up->+Y (spec M2)."""
|
||||
raise NotImplementedError("lane B (M2): implement normalize_scene")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pose interpolation
|
||||
# ---------------------------------------------------------------------------
|
||||
def interpolate_poses(registered: list[dict], sampled: list[dict]) -> list[dict]:
|
||||
"""Fill unregistered sampled frames of one video between registered ones (spec M2).
|
||||
|
||||
``registered`` are solved poses (with ``registered=True``); ``sampled`` are the frames
|
||||
that were fed to COLMAP as ``{frame_idx, t_video_s, ...}``. For each sampled frame that
|
||||
lies strictly between the first and last registered frame but wasn't solved, slerp the
|
||||
rotation and lerp the translation of the two bracketing registered poses (intrinsics
|
||||
inherited from the earlier neighbor), marking it ``registered=False``. Frames outside the
|
||||
registered span are dropped (no extrapolation). Returns all poses sorted by ``t_video_s``.
|
||||
"""
|
||||
reg = sorted(registered, key=lambda p: p["t_video_s"])
|
||||
if not reg:
|
||||
return []
|
||||
reg_frames = {int(p["frame_idx"]) for p in reg}
|
||||
t_first, t_last = reg[0]["t_video_s"], reg[-1]["t_video_s"]
|
||||
|
||||
out: list[dict] = list(reg)
|
||||
for sf in sampled:
|
||||
fidx = int(sf["frame_idx"])
|
||||
tv = float(sf["t_video_s"])
|
||||
if fidx in reg_frames:
|
||||
continue
|
||||
if tv <= t_first or tv >= t_last:
|
||||
continue # no extrapolation beyond the registered span
|
||||
# bracketing registered neighbors
|
||||
lo = max((p for p in reg if p["t_video_s"] <= tv), key=lambda p: p["t_video_s"])
|
||||
hi = min((p for p in reg if p["t_video_s"] >= tv), key=lambda p: p["t_video_s"])
|
||||
span = hi["t_video_s"] - lo["t_video_s"]
|
||||
alpha = 0.0 if span <= 1e-12 else (tv - lo["t_video_s"]) / span
|
||||
q, t = slerp_pose(
|
||||
[lo["qw"], lo["qx"], lo["qy"], lo["qz"]], [lo["tx"], lo["ty"], lo["tz"]],
|
||||
[hi["qw"], hi["qx"], hi["qy"], hi["qz"]], [hi["tx"], hi["ty"], hi["tz"]],
|
||||
alpha,
|
||||
)
|
||||
out.append({
|
||||
"frame_idx": fidx, "t_video_s": tv,
|
||||
"qw": float(q[0]), "qx": float(q[1]), "qy": float(q[2]), "qz": float(q[3]),
|
||||
"tx": float(t[0]), "ty": float(t[1]), "tz": float(t[2]),
|
||||
"fx": lo["fx"], "fy": lo["fy"], "cx": lo["cx"], "cy": lo["cy"],
|
||||
"registered": False,
|
||||
})
|
||||
out.sort(key=lambda p: p["t_video_s"])
|
||||
return out
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# COLMAP CLI orchestration
|
||||
# ---------------------------------------------------------------------------
|
||||
@functools.lru_cache(maxsize=None)
|
||||
def _colmap_help(command: str) -> str:
|
||||
"""Cached ``colmap <command> --help`` text, for build-aware option detection.
|
||||
|
||||
COLMAP prints its option list to stderr, so both streams are captured.
|
||||
"""
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
["colmap", command, "--help"], capture_output=True, text=True
|
||||
)
|
||||
return (proc.stdout or "") + "\n" + (proc.stderr or "")
|
||||
except Exception:
|
||||
return ""
|
||||
|
||||
|
||||
def _cpu_flag(command: str) -> list[str]:
|
||||
"""Force CPU SIFT for ``command`` if this COLMAP build exposes a ``use_gpu`` option.
|
||||
|
||||
The option prefix differs by version: COLMAP 3.x uses ``SiftExtraction`` /
|
||||
``SiftMatching``, 4.x uses ``FeatureExtraction`` / ``FeatureMatching``. We read the option
|
||||
name from ``--help`` so the pipeline runs headless on either (and on CPU-only builds that
|
||||
omit the option entirely, we pass nothing). Returns e.g. ``["--FeatureExtraction.use_gpu",
|
||||
"0"]`` or ``[]``.
|
||||
"""
|
||||
for line in _colmap_help(command).splitlines():
|
||||
line = line.strip()
|
||||
if line.startswith("--") and ".use_gpu" in line:
|
||||
return [line.split()[0], "0"]
|
||||
return []
|
||||
|
||||
|
||||
def _run_colmap_step(args: list[str]) -> bool:
|
||||
"""Run one ``colmap <args>`` step; return True on success, False (logged) on failure."""
|
||||
cmd = ["colmap", *args]
|
||||
log.info("colmap: %s", " ".join(args[:2]))
|
||||
try:
|
||||
proc = subprocess.run(cmd, capture_output=True, text=True)
|
||||
except FileNotFoundError:
|
||||
log.error("colmap binary vanished from PATH")
|
||||
return False
|
||||
if proc.returncode != 0:
|
||||
log.error("colmap %s failed (rc=%d): %s", args[0], proc.returncode,
|
||||
(proc.stderr or proc.stdout or "").strip()[-500:])
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _registered_image_count(model_dir: Path) -> int | None:
|
||||
"""Number of registered images in a COLMAP model dir, or None if it has no image list.
|
||||
|
||||
The mapper writes binary models by default; ``images.bin`` begins with a little-endian
|
||||
``uint64`` giving the registered-image count, which is the exact metric we want (file
|
||||
size would instead track total keypoint observations and mis-rank multi-component runs).
|
||||
Falls back to counting the TXT model's two-lines-per-image list.
|
||||
"""
|
||||
images_txt = model_dir / "images.txt"
|
||||
images_bin = model_dir / "images.bin"
|
||||
if images_txt.exists():
|
||||
return sum(1 for _ in _iter_data_lines(images_txt)) // 2
|
||||
if images_bin.exists():
|
||||
with open(images_bin, "rb") as f:
|
||||
header = f.read(8)
|
||||
return struct.unpack("<Q", header)[0] if len(header) == 8 else 0
|
||||
return None
|
||||
|
||||
|
||||
def _largest_model_dir(sparse_dir: Path) -> Path | None:
|
||||
"""Return the reconstruction subdir (``0``, ``1``, ...) with the most registered images."""
|
||||
candidates = [d for d in sparse_dir.iterdir() if d.is_dir()] if sparse_dir.exists() else []
|
||||
best, best_n = None, -1
|
||||
for d in candidates:
|
||||
n = _registered_image_count(d)
|
||||
if n is None:
|
||||
continue
|
||||
if n > best_n:
|
||||
best, best_n = d, n
|
||||
return best
|
||||
|
||||
|
||||
def run_colmap(frames_dir: Path, workspace: Path) -> Path | None:
|
||||
"""Drive the COLMAP CLI and return the exported TXT-model directory (spec M2, lane B).
|
||||
|
||||
``frames_dir`` holds one subfolder of JPEGs per video (single camera per folder). Runs
|
||||
feature extraction (CPU SIFT, OPENCV model), exhaustive matching, incremental mapping,
|
||||
then undistortion + TXT conversion. Returns the directory containing ``images.txt`` /
|
||||
``cameras.txt`` / ``points3D.txt``, or ``None`` if COLMAP produced no usable model
|
||||
(caller degrades gracefully).
|
||||
"""
|
||||
workspace = Path(workspace)
|
||||
# Start each run from a clean workspace: a stale database.db makes feature_extractor
|
||||
# fail ("images already exist"), and a leftover sparse/ model would poison the pick.
|
||||
if workspace.exists():
|
||||
import shutil
|
||||
shutil.rmtree(workspace)
|
||||
workspace.mkdir(parents=True, exist_ok=True)
|
||||
database = workspace / "database.db"
|
||||
sparse = workspace / "sparse"
|
||||
sparse.mkdir(exist_ok=True)
|
||||
|
||||
ok = _run_colmap_step([
|
||||
"feature_extractor",
|
||||
"--database_path", str(database),
|
||||
"--image_path", str(frames_dir),
|
||||
"--ImageReader.single_camera_per_folder", "1",
|
||||
"--ImageReader.camera_model", "OPENCV",
|
||||
*_cpu_flag("feature_extractor"),
|
||||
])
|
||||
if not ok:
|
||||
return None
|
||||
|
||||
# Exhaustive matching covers within- and cross-video pairs; frame counts are small.
|
||||
if not _run_colmap_step([
|
||||
"exhaustive_matcher",
|
||||
"--database_path", str(database),
|
||||
*_cpu_flag("exhaustive_matcher"),
|
||||
]):
|
||||
return None
|
||||
|
||||
if not _run_colmap_step([
|
||||
"mapper",
|
||||
"--database_path", str(database),
|
||||
"--image_path", str(frames_dir),
|
||||
"--output_path", str(sparse),
|
||||
]):
|
||||
return None
|
||||
|
||||
model = _largest_model_dir(sparse)
|
||||
if model is None:
|
||||
log.error("colmap mapper produced no reconstruction")
|
||||
return None
|
||||
|
||||
# Undistort -> PINHOLE model, then convert to TXT. Fall back to converting the raw
|
||||
# (possibly distorted) sparse model directly if undistortion fails.
|
||||
txt_dir = workspace / "model_txt"
|
||||
txt_dir.mkdir(exist_ok=True)
|
||||
dense = workspace / "dense"
|
||||
undistorted = _run_colmap_step([
|
||||
"image_undistorter",
|
||||
"--image_path", str(frames_dir),
|
||||
"--input_path", str(model),
|
||||
"--output_path", str(dense),
|
||||
"--output_type", "COLMAP",
|
||||
])
|
||||
convert_input = (dense / "sparse") if undistorted and (dense / "sparse").exists() else model
|
||||
if not _run_colmap_step([
|
||||
"model_converter",
|
||||
"--input_path", str(convert_input),
|
||||
"--output_path", str(txt_dir),
|
||||
"--output_type", "TXT",
|
||||
]):
|
||||
return None
|
||||
if not (txt_dir / "images.txt").exists():
|
||||
log.error("colmap model_converter produced no images.txt")
|
||||
return None
|
||||
return txt_dir
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Full reconstruction entrypoint
|
||||
# ---------------------------------------------------------------------------
|
||||
def _video_id_and_frame(name: str) -> tuple[int, int] | None:
|
||||
"""Recover ``(video_id, frame_idx)`` from a ``{video_id}_{frame_idx}.jpg`` image name."""
|
||||
stem = Path(name).stem
|
||||
if "_" not in stem:
|
||||
return None
|
||||
vid_s, frame_s = stem.rsplit("_", 1)
|
||||
try:
|
||||
return int(vid_s), int(frame_s)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def _diagnostic(msg: str) -> None:
|
||||
"""Emit a clear operator-facing diagnostic (spec M2 graceful degradation)."""
|
||||
log.warning("reconstruct: %s", msg)
|
||||
print(f"[reconstruct] {msg}")
|
||||
def run_colmap(frames_dir: Path, workspace: Path) -> Path:
|
||||
"""Drive the COLMAP CLI and return the exported TXT-model directory (spec M2, lane B)."""
|
||||
raise NotImplementedError("lane B (M2): implement run_colmap")
|
||||
|
||||
|
||||
def run_reconstruct() -> dict:
|
||||
@ -481,128 +66,6 @@ def run_reconstruct() -> dict:
|
||||
|
||||
Entrypoint for ``python -m festival4d reconstruct``. Degrades gracefully when COLMAP is
|
||||
absent or the reconstruction is too weak (spec M2 failure handling): existing poses are
|
||||
left untouched and the DB is never corrupted. Returns a summary dict with a ``status``.
|
||||
left untouched. Returns a summary dict.
|
||||
"""
|
||||
db.init_engine()
|
||||
db.init_db()
|
||||
videos = db.get_videos()
|
||||
if not videos:
|
||||
_diagnostic("no videos in the project — run `synthetic` or `ingest` first.")
|
||||
return {"status": "no_videos"}
|
||||
|
||||
if not colmap_available():
|
||||
_diagnostic(
|
||||
"COLMAP not found on PATH — skipping 3D reconstruction. Install it "
|
||||
"(`brew install colmap` on macOS) to enable pose/point-cloud solving. "
|
||||
"Existing (synthetic/previous) poses are left untouched; the app still runs as "
|
||||
"synced videos without 3D."
|
||||
)
|
||||
return {"status": "skipped_no_colmap", "videos": len(videos)}
|
||||
|
||||
# 1. Sample sharp frames per video into one subfolder each (single camera per folder).
|
||||
frames_root = config.FRAMES_DIR
|
||||
per_video_sampled: dict[int, list[dict]] = {}
|
||||
for v in videos:
|
||||
video_path = config.RAW_DIR / v.filename
|
||||
if not video_path.exists():
|
||||
_diagnostic(f"video file missing, skipping: {video_path}")
|
||||
continue
|
||||
out_dir = frames_root / str(v.id)
|
||||
paths = frames.sample_frames(video_path, v.id, out_dir)
|
||||
fps = float(v.fps) if v.fps and v.fps > 0 else 30.0
|
||||
sampled = []
|
||||
for p in paths:
|
||||
parsed = _video_id_and_frame(p.name)
|
||||
if parsed is None:
|
||||
continue
|
||||
_, fidx = parsed
|
||||
sampled.append({"frame_idx": fidx, "t_video_s": fidx / fps})
|
||||
per_video_sampled[v.id] = sorted(sampled, key=lambda s: s["frame_idx"])
|
||||
|
||||
total_sampled = sum(len(s) for s in per_video_sampled.values())
|
||||
if total_sampled == 0:
|
||||
_diagnostic("no frames could be sampled from any video — nothing to reconstruct.")
|
||||
return {"status": "no_frames"}
|
||||
|
||||
# 2. Run COLMAP.
|
||||
model_dir = run_colmap(frames_root, config.COLMAP_DIR)
|
||||
if model_dir is None:
|
||||
_diagnostic(
|
||||
"COLMAP produced no usable reconstruction. Likely causes: too-dark footage, "
|
||||
"motion blur, or insufficient view overlap between cameras. Existing poses left "
|
||||
"untouched — the app degrades to synced videos, no 3D."
|
||||
)
|
||||
return {"status": "failed_no_model", "sampled": total_sampled}
|
||||
|
||||
# 3. Parse the model.
|
||||
images = parse_images_txt(model_dir / "images.txt")
|
||||
cameras = parse_cameras_txt(model_dir / "cameras.txt")
|
||||
points, colors = parse_points3d_txt(model_dir / "points3D.txt")
|
||||
|
||||
# 4. Attach each registered image to its video + intrinsics.
|
||||
registered_by_video: dict[int, list[dict]] = {}
|
||||
for img in images:
|
||||
parsed = _video_id_and_frame(img["name"])
|
||||
if parsed is None:
|
||||
continue
|
||||
video_id, frame_idx = parsed
|
||||
cam = cameras.get(img["camera_id"])
|
||||
if cam is None:
|
||||
continue
|
||||
fps = next((float(v.fps) for v in videos if v.id == video_id), 30.0) or 30.0
|
||||
registered_by_video.setdefault(video_id, []).append({
|
||||
"frame_idx": frame_idx, "t_video_s": frame_idx / fps,
|
||||
"qw": img["qw"], "qx": img["qx"], "qy": img["qy"], "qz": img["qz"],
|
||||
"tx": img["tx"], "ty": img["ty"], "tz": img["tz"],
|
||||
"fx": cam["fx"], "fy": cam["fy"], "cx": cam["cx"], "cy": cam["cy"],
|
||||
"registered": True, "video_id": video_id,
|
||||
})
|
||||
|
||||
registered_count = sum(len(v) for v in registered_by_video.values())
|
||||
videos_in_model = len(registered_by_video)
|
||||
frac = registered_count / total_sampled if total_sampled else 0.0
|
||||
if frac < MIN_REGISTERED_FRACTION or videos_in_model < MIN_VIDEOS_IN_MODEL:
|
||||
_diagnostic(
|
||||
f"weak reconstruction: {registered_count}/{total_sampled} frames "
|
||||
f"({frac:.0%}) registered across {videos_in_model} video(s); need "
|
||||
f">= {MIN_REGISTERED_FRACTION:.0%} of frames and >= {MIN_VIDEOS_IN_MODEL} "
|
||||
"videos. Likely causes: too-dark footage, motion blur, or too little view "
|
||||
"overlap. Existing poses left untouched — degrading to synced videos, no 3D."
|
||||
)
|
||||
return {
|
||||
"status": "failed_weak", "sampled": total_sampled,
|
||||
"registered": registered_count, "videos_in_model": videos_in_model,
|
||||
}
|
||||
|
||||
# 5. Normalize the whole scene (points + all registered poses) with one transform.
|
||||
all_registered = [p for v in registered_by_video.values() for p in v]
|
||||
norm_points, norm_registered = normalize_scene(points, all_registered)
|
||||
norm_by_video: dict[int, list[dict]] = {}
|
||||
for p in norm_registered:
|
||||
norm_by_video.setdefault(p["video_id"], []).append(p)
|
||||
|
||||
# 6. Interpolate unregistered sampled frames, then write poses per video (atomic replace).
|
||||
interpolated_total = 0
|
||||
for video_id, reg in norm_by_video.items():
|
||||
sampled = per_video_sampled.get(video_id, [])
|
||||
final = interpolate_poses(reg, sampled)
|
||||
interpolated_total += sum(1 for p in final if not p.get("registered", True))
|
||||
db.set_poses(video_id, final)
|
||||
|
||||
# 7. Export the normalized point cloud in the frozen PLY format.
|
||||
from festival4d.synthetic import write_ply
|
||||
write_ply(config.POINTS_PLY, norm_points, colors)
|
||||
|
||||
summary = {
|
||||
"status": "ok",
|
||||
"videos_in_model": videos_in_model,
|
||||
"registered": registered_count,
|
||||
"interpolated": interpolated_total,
|
||||
"points": int(len(norm_points)),
|
||||
"points_ply": str(config.POINTS_PLY),
|
||||
}
|
||||
log.info("reconstruct: done — %s", summary)
|
||||
print(f"[reconstruct] reconstructed {videos_in_model} videos, "
|
||||
f"{registered_count} registered + {interpolated_total} interpolated poses, "
|
||||
f"{len(norm_points)} points -> {config.POINTS_PLY}")
|
||||
return summary
|
||||
raise NotImplementedError("lane B (M2): implement run_reconstruct")
|
||||
|
||||
@ -1,276 +0,0 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover" />
|
||||
<title>Festival 4D — Capture</title>
|
||||
<style>
|
||||
:root {
|
||||
color-scheme: dark;
|
||||
--bg: #0b0d12; --panel: #141824; --panel-2: #1c2233; --border: #232838;
|
||||
--text: #e6e8ee; --dim: #8a92a6; --ok: #59d499; --bad: #ff6b6b; --rec: #ff3b6b;
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
body {
|
||||
margin: 0; background: var(--bg); color: var(--text); min-height: 100vh;
|
||||
font: 15px/1.45 -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
|
||||
padding: env(safe-area-inset-top) 0.8rem calc(env(safe-area-inset-bottom) + 0.8rem);
|
||||
}
|
||||
h1 { font-size: 1.1rem; margin: 0.8rem 0 0.2rem; }
|
||||
.sub { color: var(--dim); font-size: 0.8rem; margin-bottom: 0.8rem; }
|
||||
.card {
|
||||
background: var(--panel); border: 1px solid var(--border); border-radius: 10px;
|
||||
padding: 0.7rem; margin-bottom: 0.7rem;
|
||||
}
|
||||
label { display: block; color: var(--dim); font-size: 0.76rem; margin-bottom: 0.2rem; }
|
||||
select, input[type="text"] {
|
||||
width: 100%; background: var(--panel-2); color: var(--text); font-size: 1rem;
|
||||
border: 1px solid var(--border); border-radius: 7px; padding: 0.5rem;
|
||||
margin-bottom: 0.55rem;
|
||||
}
|
||||
#preview {
|
||||
width: 100%; aspect-ratio: 16/9; background: #000; border-radius: 8px;
|
||||
object-fit: contain; display: block;
|
||||
}
|
||||
.row { display: flex; gap: 0.5rem; align-items: center; }
|
||||
button {
|
||||
flex: 1 1 auto; font-size: 1rem; font-weight: 600; padding: 0.75rem 0.6rem;
|
||||
border-radius: 9px; border: 1px solid var(--border); background: var(--panel-2);
|
||||
color: var(--text); cursor: pointer;
|
||||
}
|
||||
button:disabled { opacity: 0.45; cursor: default; }
|
||||
#rec.recording { background: var(--rec); border-color: var(--rec); color: #fff; }
|
||||
.pill {
|
||||
display: inline-flex; align-items: center; gap: 0.3rem; font-size: 0.75rem;
|
||||
color: var(--dim);
|
||||
}
|
||||
.dot { width: 8px; height: 8px; border-radius: 50%; background: var(--dim); }
|
||||
.dot.on { background: var(--rec); animation: pulse 1s infinite; }
|
||||
@keyframes pulse { 50% { opacity: 0.25; } }
|
||||
.msg { font-size: 0.82rem; margin-top: 0.5rem; line-height: 1.4; }
|
||||
.err { color: var(--bad); }
|
||||
.ok { color: var(--ok); }
|
||||
code {
|
||||
background: var(--panel-2); padding: 0.1rem 0.3rem; border-radius: 4px;
|
||||
font-size: 0.85em; word-break: break-all;
|
||||
}
|
||||
/* monitor */
|
||||
#mon { display: grid; grid-template-columns: repeat(auto-fill, minmax(104px, 1fr)); gap: 0.5rem; }
|
||||
.mcell { background: var(--panel-2); border: 1px solid var(--border); border-radius: 7px; overflow: hidden; }
|
||||
.mcell img { width: 100%; aspect-ratio: 16/9; object-fit: cover; display: block; background: #000; }
|
||||
.mcell .cap { padding: 0.25rem 0.35rem; font-size: 0.68rem; display: flex; align-items: center; gap: 0.25rem; }
|
||||
.mcell .nm { flex: 1 1 auto; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
||||
.mcell.rec { border-color: var(--rec); }
|
||||
#gate { display: none; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Festival 4D — Capture</h1>
|
||||
<div class="sub">Record this camera. Clips upload to the project and sync by audio.</div>
|
||||
|
||||
<!-- Shown when getUserMedia is unavailable (insecure origin) -->
|
||||
<div class="card" id="gate">
|
||||
<div class="msg err"><b>Camera blocked — this page isn't a secure context.</b></div>
|
||||
<div class="msg">
|
||||
Browsers only grant camera access over <b>HTTPS</b> (or localhost). On the machine running
|
||||
the backend:
|
||||
<div style="margin:0.4rem 0"><code>tailscale serve --bg 8000</code></div>
|
||||
then open the printed <code>https://<machine>.<tailnet>.ts.net/capture</code>
|
||||
URL on this device (join the tailnet first).
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div id="app">
|
||||
<div class="card">
|
||||
<label for="name">Device name (used in the filename)</label>
|
||||
<input type="text" id="name" placeholder="e.g. phone-left / logitech-c920" />
|
||||
<label for="cam">Camera</label>
|
||||
<select id="cam"></select>
|
||||
<label for="mic">Microphone <span style="color:var(--bad)">— required for sync</span></label>
|
||||
<select id="mic"></select>
|
||||
<video id="preview" playsinline muted autoplay></video>
|
||||
<div class="row" style="margin-top:0.55rem">
|
||||
<span class="pill"><span class="dot" id="recdot"></span><span id="status">idle</span></span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="row">
|
||||
<button id="rec">● Record</button>
|
||||
</div>
|
||||
<div class="msg" id="msg"></div>
|
||||
|
||||
<div class="card" style="margin-top:0.8rem">
|
||||
<label style="margin-bottom:0.4rem">Who's shooting what — live monitor</label>
|
||||
<div id="mon"></div>
|
||||
<div class="msg" id="monempty" style="color:var(--dim)">no other cameras yet</div>
|
||||
</div>
|
||||
|
||||
<div class="msg" style="color:var(--dim)">
|
||||
All cameras must hear the <b>same sound</b> — that's how clips get aligned. No clock sync
|
||||
needed; you don't even have to start together. When done, run
|
||||
<code>python -m festival4d ingest && python -m festival4d sync</code>.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
const $ = (id) => document.getElementById(id);
|
||||
const deviceId = (localStorage.f4dDevId ||= "dev-" + Math.random().toString(36).slice(2, 9));
|
||||
let stream = null, recorder = null, chunks = [], startedAt = 0, timer = null;
|
||||
|
||||
const secure = window.isSecureContext && navigator.mediaDevices?.getUserMedia;
|
||||
if (!secure) {
|
||||
$("gate").style.display = "block";
|
||||
$("app").style.display = "none";
|
||||
}
|
||||
|
||||
function msg(html, cls = "") { $("msg").className = "msg " + cls; $("msg").innerHTML = html; }
|
||||
|
||||
async function listDevices() {
|
||||
const devs = await navigator.mediaDevices.enumerateDevices();
|
||||
const fill = (sel, kind, fallback) => {
|
||||
const cur = sel.value;
|
||||
sel.innerHTML = devs
|
||||
.filter((d) => d.kind === kind)
|
||||
.map((d, i) => `<option value="${d.deviceId}">${d.label || `${fallback} ${i + 1}`}</option>`)
|
||||
.join("");
|
||||
if (cur) sel.value = cur;
|
||||
};
|
||||
fill($("cam"), "videoinput", "Camera");
|
||||
fill($("mic"), "audioinput", "Microphone");
|
||||
}
|
||||
|
||||
async function start(camId, micId) {
|
||||
stream?.getTracks().forEach((t) => t.stop());
|
||||
// Audio is NOT optional: the pipeline aligns clips by sound.
|
||||
stream = await navigator.mediaDevices.getUserMedia({
|
||||
video: camId ? { deviceId: { exact: camId } } : true,
|
||||
audio: micId ? { deviceId: { exact: micId } } : true,
|
||||
});
|
||||
$("preview").srcObject = stream;
|
||||
await listDevices(); // labels only populate after permission is granted
|
||||
}
|
||||
|
||||
function pickMime() {
|
||||
const want = [
|
||||
"video/webm;codecs=vp9,opus",
|
||||
"video/webm;codecs=vp8,opus",
|
||||
"video/webm",
|
||||
"video/mp4", // Safari / iOS
|
||||
];
|
||||
return want.find((m) => MediaRecorder.isTypeSupported?.(m)) || "";
|
||||
}
|
||||
|
||||
$("rec").addEventListener("click", () => (recorder ? stop() : record()));
|
||||
|
||||
function record() {
|
||||
if (!stream) return msg("no camera stream yet", "err");
|
||||
if (stream.getAudioTracks().length === 0)
|
||||
return msg("no audio track — clips without sound can't be synced.", "err");
|
||||
const mime = pickMime();
|
||||
chunks = [];
|
||||
recorder = new MediaRecorder(stream, mime ? { mimeType: mime } : undefined);
|
||||
recorder.ondataavailable = (e) => e.data.size && chunks.push(e.data);
|
||||
recorder.onstop = upload;
|
||||
recorder.start(1000);
|
||||
startedAt = Date.now();
|
||||
$("rec").textContent = "■ Stop";
|
||||
$("rec").classList.add("recording");
|
||||
$("recdot").classList.add("on");
|
||||
timer = setInterval(tick, 250);
|
||||
msg("recording — point at the stage and keep some static structure in frame.");
|
||||
}
|
||||
|
||||
function stop() {
|
||||
clearInterval(timer);
|
||||
recorder?.stop();
|
||||
recorder = null;
|
||||
$("rec").textContent = "● Record";
|
||||
$("rec").classList.remove("recording");
|
||||
$("recdot").classList.remove("on");
|
||||
tick();
|
||||
}
|
||||
|
||||
function elapsed() { return startedAt ? (Date.now() - startedAt) / 1000 : 0; }
|
||||
function tick() {
|
||||
$("status").textContent = recorder ? `recording ${elapsed().toFixed(0)}s` : "idle";
|
||||
}
|
||||
|
||||
async function upload() {
|
||||
const type = chunks[0]?.type || "video/webm";
|
||||
const blob = new Blob(chunks, { type });
|
||||
const ext = type.includes("mp4") ? "mp4" : "webm";
|
||||
const name = ($("name").value || "camera").trim();
|
||||
const fd = new FormData();
|
||||
fd.append("device_name", name);
|
||||
fd.append("file", blob, `clip.${ext}`);
|
||||
msg(`uploading ${(blob.size / 1e6).toFixed(1)} MB…`);
|
||||
try {
|
||||
const r = await fetch("api/capture/upload", { method: "POST", body: fd });
|
||||
const j = await r.json();
|
||||
if (!r.ok) throw new Error(j.detail || `HTTP ${r.status}`);
|
||||
msg(`saved <code>${j.filename}</code> (${(j.bytes / 1e6).toFixed(1)} MB). Record again or run the pipeline.`, "ok");
|
||||
} catch (e) {
|
||||
msg(`upload failed: ${e.message}`, "err");
|
||||
}
|
||||
startedAt = 0;
|
||||
tick();
|
||||
}
|
||||
|
||||
// --- heartbeat + monitor -------------------------------------------------------------
|
||||
const snapCanvas = document.createElement("canvas");
|
||||
function snapshot() {
|
||||
const v = $("preview");
|
||||
if (!v.videoWidth) return null;
|
||||
snapCanvas.width = 160;
|
||||
snapCanvas.height = Math.round((160 * v.videoHeight) / v.videoWidth);
|
||||
snapCanvas.getContext("2d").drawImage(v, 0, 0, snapCanvas.width, snapCanvas.height);
|
||||
return snapCanvas.toDataURL("image/jpeg", 0.4);
|
||||
}
|
||||
|
||||
async function beat() {
|
||||
try {
|
||||
await fetch("api/capture/heartbeat", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
device_id: deviceId,
|
||||
name: ($("name").value || "camera").trim(),
|
||||
status: recorder ? "recording" : "idle",
|
||||
elapsed_s: elapsed(),
|
||||
snapshot: snapshot(),
|
||||
}),
|
||||
});
|
||||
const devs = await (await fetch("api/capture/devices")).json();
|
||||
renderMonitor(devs.filter((d) => d.device_id !== deviceId));
|
||||
} catch { /* offline: monitor just goes stale */ }
|
||||
}
|
||||
|
||||
function renderMonitor(devs) {
|
||||
$("monempty").style.display = devs.length ? "none" : "block";
|
||||
$("mon").innerHTML = devs
|
||||
.map(
|
||||
(d) => `<div class="mcell ${d.status === "recording" ? "rec" : ""}">
|
||||
${d.snapshot ? `<img src="${d.snapshot}" alt="">` : `<img alt="">`}
|
||||
<div class="cap"><span class="dot ${d.status === "recording" ? "on" : ""}"></span>
|
||||
<span class="nm">${d.name}</span>
|
||||
<span>${d.status === "recording" ? d.elapsed_s.toFixed(0) + "s" : ""}</span></div>
|
||||
</div>`
|
||||
)
|
||||
.join("");
|
||||
}
|
||||
|
||||
// --- boot ----------------------------------------------------------------------------
|
||||
if (secure) {
|
||||
$("name").value = localStorage.f4dName || "";
|
||||
$("name").addEventListener("change", () => (localStorage.f4dName = $("name").value));
|
||||
$("cam").addEventListener("change", () => start($("cam").value, $("mic").value).catch((e) => msg(e.message, "err")));
|
||||
$("mic").addEventListener("change", () => start($("cam").value, $("mic").value).catch((e) => msg(e.message, "err")));
|
||||
start().then(() => msg("ready — name this device, then record.")).catch((e) =>
|
||||
msg(`camera error: ${e.message}`, "err")
|
||||
);
|
||||
setInterval(beat, 2000);
|
||||
navigator.mediaDevices.addEventListener?.("devicechange", listDevices);
|
||||
}
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@ -11,17 +11,11 @@ testable without real footage:
|
||||
- **Point cloud**: random points on the stage box + ground plane, written to
|
||||
``points.ply`` in the exact binary-little-endian format the real COLMAP path produces.
|
||||
- **Seeded events + anchors** so the timeline and overlays have data immediately.
|
||||
- **Two moving markers** (phase 6 / M18) composited into every camera's video *through the
|
||||
same camera projection the poses use*, plus ``track_truth.json`` — the 3D ground truth lane
|
||||
H recovers. Marker A is a solid magenta disc (color mode, ``hue:150``); marker B is a
|
||||
blinking bright disc speaking the frozen OOK protocol with ID 5 (``code:5``). See
|
||||
:func:`marker_position`, :func:`render_marker_overlay`, :func:`build_track_truth`.
|
||||
- **``ground_truth.json``**: offsets, drift, audio pulse (bang) times, events, and stage
|
||||
corners — part of the frozen contract (lanes A and D assert against it).
|
||||
|
||||
FROZEN CONTRACT after foundation. The module is factored so the numeric pieces
|
||||
(audio shifts, poses, PLY round-trip, marker geometry) are unit-testable without invoking
|
||||
ffmpeg.
|
||||
(audio shifts, poses, PLY round-trip) are unit-testable without invoking ffmpeg.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@ -37,8 +31,8 @@ from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
||||
from festival4d import config, db, tracker_detect
|
||||
from festival4d.geometry import mat_to_quat, quat_to_mat, slerp_pose
|
||||
from festival4d import config, db
|
||||
from festival4d.geometry import mat_to_quat
|
||||
|
||||
log = logging.getLogger("festival4d.synthetic")
|
||||
|
||||
@ -369,219 +363,6 @@ def read_ply(path: Path) -> tuple[np.ndarray, np.ndarray]:
|
||||
return points, colors
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Moving markers (phase 6 / M18). Two wearable markers travel known 3D paths across the
|
||||
# stage; each camera's frame shows them at the pixel its OWN pose projects the 3D point to,
|
||||
# so the rendered pixels and ``track_truth.json`` are consistent BY CONSTRUCTION — the marker
|
||||
# pixel is derived from the ground-truth 3D point through the same projection lane H inverts.
|
||||
#
|
||||
# Marker A — solid magenta (#ff00ff) disc, color mode, marker_key "hue:150". Slow circle
|
||||
# (radius 2, y≈1.5) centered on the stage.
|
||||
# Marker B — blinking bright disc, blink mode, ID 5, marker_key "code:5". A different path;
|
||||
# the LED follows the frozen OOK protocol (tracker_detect) in GLOBAL time, so
|
||||
# every camera sees the same on/off state at the same t_global.
|
||||
# Discs are small (radius ≈2.5% of frame height ⇒ ≪2% frame area) and composited AFTER all
|
||||
# existing content, so the 189-test floor (offsets/events/poses/SfM) is untouched (pitfall #6).
|
||||
# ---------------------------------------------------------------------------
|
||||
MARKER_A_KEY = tracker_detect.color_marker_key(tracker_detect.MAGENTA_OPENCV_HUE) # "hue:150"
|
||||
MARKER_A_HEX = "#ff00ff"
|
||||
MARKER_A_BGR = (255, 0, 255) # OpenCV BGR for magenta (drawn AFTER the hue filter)
|
||||
MARKER_B_ID = 5
|
||||
MARKER_B_KEY = tracker_detect.code_marker_key(MARKER_B_ID) # "code:5"
|
||||
MARKER_B_ON_BGR = (255, 255, 255) # bright white while the LED is ON
|
||||
|
||||
# testsrc2's base pattern contains solid full-saturation magenta AND cyan colour bars — the
|
||||
# EXACT two marker hues (contract #1) — plus other bright bars. Left as-is they swamp lane H's
|
||||
# colour detection (the magenta bar is a far bigger "magenta blob" than marker A's disc) and
|
||||
# starve marker B's white-blink luminance contrast. So the base is MUTED before the markers are
|
||||
# composited on top: saturation crushed (no high-saturation magenta/cyan survives) and dimmed
|
||||
# (the bright-white blink disc pops in luminance). The fully-saturated discs are then the only
|
||||
# high-S magenta/cyan and the only bright small blobs in frame — the fixture lane H is graded on
|
||||
# is actually solvable by construction. Tests assert geometry/audio/PLY, never base pixels.
|
||||
BASE_SATURATION = 0.28 # hue-filter saturation multiplier applied to the base
|
||||
BASE_BRIGHTNESS = -0.28 # eq brightness offset applied to the base (dim it)
|
||||
MARKER_DISC_HEIGHT_FRAC = 0.025 # disc radius / frame height (≈9 px @ 360 -> 0.11% area)
|
||||
MARKER_A_PERIOD_S = 20.0 # one slow loop across the fixture span
|
||||
MARKER_B_PERIOD_S = 16.0 # different path + rate so the two never lock together
|
||||
MARKER_TRUTH_DT_S = 0.05 # track_truth.json sampling step (t_global seconds)
|
||||
|
||||
|
||||
def marker_position(marker_key: str, t_global: float) -> tuple[float, float, float]:
|
||||
"""Ground-truth 3D position (Three.js scene space) of a marker at ``t_global`` seconds.
|
||||
|
||||
Pure and total over all real ``t_global`` (the markers exist for the whole span). This is
|
||||
THE ground truth: it is what ``track_truth.json`` records and what a correct solver
|
||||
recovers. FROZEN — lane H's acceptance is measured against these paths.
|
||||
"""
|
||||
if marker_key == MARKER_A_KEY:
|
||||
theta = 2.0 * np.pi * t_global / MARKER_A_PERIOD_S
|
||||
return (2.0 * float(np.cos(theta)), 1.5, 2.0 * float(np.sin(theta)))
|
||||
if marker_key == MARKER_B_KEY:
|
||||
phi = 2.0 * np.pi * t_global / MARKER_B_PERIOD_S
|
||||
return (
|
||||
2.4 * float(np.sin(phi)),
|
||||
1.0 + 0.4 * float(np.sin(2.0 * phi)),
|
||||
-0.8 + 1.2 * float(np.cos(phi)),
|
||||
)
|
||||
raise KeyError(f"unknown marker_key {marker_key!r}")
|
||||
|
||||
|
||||
MARKER_KEYS = (MARKER_A_KEY, MARKER_B_KEY)
|
||||
|
||||
|
||||
def _interp_pose(poses: list[dict], t_video: float):
|
||||
"""Interpolate a COLMAP pose ``(q, t, (fx,fy,cx,cy))`` at ``t_video`` from ``camera_track``
|
||||
output (ascending by ``t_video_s``). Mirrors ``resolve._pose_at`` / the frontend ``poseAt``
|
||||
exactly — slerp the rotation, lerp the translation via the FROZEN ``geometry.slerp_pose`` —
|
||||
so the marker pixels a solver back-projects land on the same 3D point. Clamps (no extrap.).
|
||||
"""
|
||||
def tup(p):
|
||||
return ([p["qw"], p["qx"], p["qy"], p["qz"]],
|
||||
[p["tx"], p["ty"], p["tz"]],
|
||||
(p["fx"], p["fy"], p["cx"], p["cy"]))
|
||||
|
||||
if t_video <= poses[0]["t_video_s"]:
|
||||
return tup(poses[0])
|
||||
if t_video >= poses[-1]["t_video_s"]:
|
||||
return tup(poses[-1])
|
||||
lo, hi = 0, len(poses) - 1
|
||||
while hi - lo > 1:
|
||||
mid = (lo + hi) // 2
|
||||
if poses[mid]["t_video_s"] <= t_video:
|
||||
lo = mid
|
||||
else:
|
||||
hi = mid
|
||||
a, b = poses[lo], poses[hi]
|
||||
span = b["t_video_s"] - a["t_video_s"]
|
||||
alpha = (t_video - a["t_video_s"]) / span if span > 1e-9 else 0.0
|
||||
q, t = slerp_pose(
|
||||
[a["qw"], a["qx"], a["qy"], a["qz"]], [a["tx"], a["ty"], a["tz"]],
|
||||
[b["qw"], b["qx"], b["qy"], b["qz"]], [b["tx"], b["ty"], b["tz"]],
|
||||
alpha,
|
||||
)
|
||||
return list(q), list(t), (a["fx"], a["fy"], a["cx"], a["cy"])
|
||||
|
||||
|
||||
def project_point(q, t, intr, X) -> tuple[float, float, float] | None:
|
||||
"""Project world point ``X`` through a COLMAP world->camera pose (the inverse of
|
||||
``geometry.ray_from_pixel``). Returns ``(px, py, z_cam)`` in pixels, or ``None`` if the
|
||||
point is behind the camera. COLMAP camera axes: +x right, +y down, +z forward.
|
||||
"""
|
||||
R = quat_to_mat(q) # world -> cam
|
||||
Xc = R @ np.asarray(X, dtype=np.float64).reshape(3) + np.asarray(t, dtype=np.float64).reshape(3)
|
||||
z = float(Xc[2])
|
||||
if z <= 1e-6:
|
||||
return None
|
||||
fx, fy, cx, cy = intr
|
||||
return fx * float(Xc[0]) / z + cx, fy * float(Xc[1]) / z + cy, z
|
||||
|
||||
|
||||
def _disc_radius_px(height: int) -> int:
|
||||
return max(6, int(round(MARKER_DISC_HEIGHT_FRAC * height)))
|
||||
|
||||
|
||||
def render_marker_overlay(out_dir: Path, poses: list[dict], offset_ms: float,
|
||||
width: int, height: int, fps: float, duration_s: float) -> int:
|
||||
"""Write a transparent BGRA PNG per frame with the two markers drawn at their projected
|
||||
pixels (phase 6 / M18). Composited over one camera's clip by :func:`render_video`.
|
||||
|
||||
For frame ``k`` at local ``t_video = k/fps``: convert to ``t_global`` with the FROZEN
|
||||
``config.t_global_from_video``, look up each marker's 3D truth at ``t_global``, and project
|
||||
it through this camera's pose interpolated at ``t_video``. Marker A is always drawn; marker
|
||||
B is drawn only when its LED is ON per the OOK schedule (``tracker_detect.led_on``, in
|
||||
global time). Off-frame / behind-camera projections are simply not drawn. Returns frame count.
|
||||
"""
|
||||
import cv2
|
||||
|
||||
out_dir = Path(out_dir)
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
radius = _disc_radius_px(height)
|
||||
n = int(round(duration_s * fps))
|
||||
for k in range(n):
|
||||
t_video = k / fps
|
||||
t_global = config.t_global_from_video(t_video, offset_ms, 0.0)
|
||||
q, t, intr = _interp_pose(poses, t_video)
|
||||
img = np.zeros((height, width, 4), dtype=np.uint8) # transparent BGRA
|
||||
_draw_marker_disc(img, q, t, intr, marker_position(MARKER_A_KEY, t_global),
|
||||
MARKER_A_BGR, radius)
|
||||
if tracker_detect.led_on(t_global, MARKER_B_ID):
|
||||
_draw_marker_disc(img, q, t, intr, marker_position(MARKER_B_KEY, t_global),
|
||||
MARKER_B_ON_BGR, radius)
|
||||
if not cv2.imwrite(str(out_dir / f"{k:05d}.png"), img):
|
||||
raise RuntimeError(f"failed to write marker overlay frame: {out_dir}/{k:05d}.png")
|
||||
return n
|
||||
|
||||
|
||||
def _draw_marker_disc(img, q, t, intr, X, bgr, radius) -> None:
|
||||
import cv2
|
||||
|
||||
proj = project_point(q, t, intr, X)
|
||||
if proj is None:
|
||||
return
|
||||
px, py, _ = proj
|
||||
h, w = img.shape[:2]
|
||||
if not (0.0 <= px < w and 0.0 <= py < h):
|
||||
return # off this camera's frame ⇒ not visible here
|
||||
cv2.circle(img, (int(round(px)), int(round(py))), radius,
|
||||
(int(bgr[0]), int(bgr[1]), int(bgr[2]), 255), thickness=-1, lineType=cv2.LINE_8)
|
||||
|
||||
|
||||
def build_track_truth(g_lo: float, g_hi: float, dt: float = MARKER_TRUTH_DT_S) -> dict:
|
||||
"""The ``track_truth.json`` body: each marker's 3D path sampled over ``[g_lo, g_hi]``.
|
||||
|
||||
Shape (spec M18): ``{"markers": [{"marker_key", "points": [{"t_global_s","x","y","z"}]}]}``
|
||||
in Three.js scene space. ``generated_by`` is added for provenance (extra keys are fine).
|
||||
"""
|
||||
n = max(1, int(round((g_hi - g_lo) / dt)) + 1)
|
||||
markers = []
|
||||
for key in MARKER_KEYS:
|
||||
points = []
|
||||
for i in range(n):
|
||||
tg = g_lo + i * dt
|
||||
x, y, z = marker_position(key, tg)
|
||||
points.append({"t_global_s": round(tg, 6), "x": x, "y": y, "z": z})
|
||||
markers.append({"marker_key": key, "points": points})
|
||||
return {"markers": markers, "generated_by": "festival4d.synthetic"}
|
||||
|
||||
|
||||
def render_blink_clip(out_path: Path, marker_id: int = MARKER_B_ID, fps: float = 30.0,
|
||||
duration_s: float = 3.0, width: int = 320, height: int = 240) -> Path:
|
||||
"""Render a small clip of a single blinking badge at frame center (phase 6 / M24 helper).
|
||||
|
||||
A fixed-position bright disc blinking ``marker_id`` in the frozen OOK protocol on a black
|
||||
background — no camera geometry, so lane K's ``badge_selftest --synthetic`` can round-trip
|
||||
the ID through ``tracker_detect``'s blink decoder without a full fixture. No audio track.
|
||||
"""
|
||||
import cv2
|
||||
|
||||
if shutil.which("ffmpeg") is None:
|
||||
raise RuntimeError("ffmpeg not found on PATH — required to render a blink clip")
|
||||
out_path = Path(out_path)
|
||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
radius = _disc_radius_px(height)
|
||||
cx, cy = width // 2, height // 2
|
||||
n = int(round(duration_s * fps))
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
frames_dir = Path(tmp)
|
||||
for k in range(n):
|
||||
t = k / fps
|
||||
img = np.zeros((height, width, 3), dtype=np.uint8) # black background
|
||||
if tracker_detect.led_on(t, marker_id):
|
||||
cv2.circle(img, (cx, cy), radius, (255, 255, 255), thickness=-1,
|
||||
lineType=cv2.LINE_8)
|
||||
cv2.imwrite(str(frames_dir / f"{k:05d}.png"), img)
|
||||
cmd = [
|
||||
"ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
|
||||
"-framerate", str(fps), "-start_number", "0",
|
||||
"-i", str(frames_dir / "%05d.png"),
|
||||
"-c:v", "libx264", "-pix_fmt", "yuv420p", "-preset", "veryfast", "-crf", "20",
|
||||
"-movflags", "+faststart",
|
||||
str(out_path),
|
||||
]
|
||||
subprocess.run(cmd, check=True)
|
||||
return out_path
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Video rendering (ffmpeg)
|
||||
# ---------------------------------------------------------------------------
|
||||
@ -615,30 +396,25 @@ def _has_drawtext() -> bool:
|
||||
|
||||
|
||||
def render_video(out_path: Path, wav_path: Path, cam_index: int,
|
||||
width: int, height: int, fps: float, duration_s: float,
|
||||
poses: list[dict] | None = None, offset_ms: float = 0.0) -> None:
|
||||
"""Render one synthetic clip: a distinct ``testsrc2`` visual muxed with ``wav_path``.
|
||||
|
||||
When ``poses`` is given (this camera's ``camera_track``), the two moving markers (M18) are
|
||||
composited on top via a per-frame RGBA overlay produced by :func:`render_marker_overlay`
|
||||
(drawn AFTER the base visual + label, so existing behavior/tests are untouched). ``offset_ms``
|
||||
aligns the marker schedule to global time for this camera.
|
||||
"""
|
||||
width: int, height: int, fps: float, duration_s: float) -> None:
|
||||
"""Render one synthetic clip: a distinct ``testsrc2`` visual muxed with ``wav_path``."""
|
||||
if shutil.which("ffmpeg") is None:
|
||||
raise RuntimeError("ffmpeg not found on PATH — required for the synthetic fixture")
|
||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Per-camera hue shift for distinctness, then mute saturation + brightness so the base
|
||||
# never collides with the fully-saturated markers composited on top (see BASE_* above).
|
||||
hue = cam_index * 47
|
||||
vf = f"hue=h={hue}:s={BASE_SATURATION},eq=brightness={BASE_BRIGHTNESS}"
|
||||
vf = f"hue=h={hue}"
|
||||
font = _find_font()
|
||||
if font and _has_drawtext():
|
||||
vf += (f",drawtext=fontfile='{font}':text='CAM {cam_index}':"
|
||||
"x=24:y=24:fontsize=44:fontcolor=white:box=1:boxcolor=black@0.5")
|
||||
|
||||
src = f"testsrc2=size={width}x{height}:rate={fps}:duration={duration_s}"
|
||||
common_out = [
|
||||
cmd = [
|
||||
"ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
|
||||
"-f", "lavfi", "-i", src,
|
||||
"-i", str(wav_path),
|
||||
"-vf", vf,
|
||||
"-c:v", "libx264", "-pix_fmt", "yuv420p", "-preset", "veryfast", "-crf", "28",
|
||||
"-profile:v", "baseline", "-level", "3.1",
|
||||
"-movflags", "+faststart",
|
||||
@ -646,33 +422,7 @@ def render_video(out_path: Path, wav_path: Path, cam_index: int,
|
||||
"-shortest",
|
||||
str(out_path),
|
||||
]
|
||||
|
||||
if poses is None:
|
||||
cmd = [
|
||||
"ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
|
||||
"-f", "lavfi", "-i", src,
|
||||
"-i", str(wav_path),
|
||||
"-vf", vf,
|
||||
*common_out,
|
||||
]
|
||||
subprocess.run(cmd, check=True)
|
||||
return
|
||||
|
||||
# Markers: pre-render the RGBA overlay frames, then composite them over the base visual.
|
||||
with tempfile.TemporaryDirectory() as ov:
|
||||
ov_dir = Path(ov)
|
||||
render_marker_overlay(ov_dir, poses, offset_ms, width, height, fps, duration_s)
|
||||
cmd = [
|
||||
"ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
|
||||
"-f", "lavfi", "-i", src, # 0: base video
|
||||
"-i", str(wav_path), # 1: audio
|
||||
"-framerate", str(fps), "-start_number", "0",
|
||||
"-i", str(ov_dir / "%05d.png"), # 2: marker overlay
|
||||
"-filter_complex", f"[0:v]{vf}[bg];[bg][2:v]overlay=0:0:eof_action=pass[vo]",
|
||||
"-map", "[vo]", "-map", "1:a",
|
||||
*common_out,
|
||||
]
|
||||
subprocess.run(cmd, check=True)
|
||||
subprocess.run(cmd, check=True)
|
||||
|
||||
|
||||
def ffprobe_video(path: Path) -> dict:
|
||||
@ -731,15 +481,11 @@ def build(base_dir: Path | str | None = None, duration_s: float | None = None,
|
||||
filename = f"cam{i}.mp4"
|
||||
out_path = raw / filename
|
||||
clip = slice_for_offset(master, sr, offset_ms, duration)
|
||||
poses = camera_track(i, duration, fps, w, h)
|
||||
|
||||
if run_ffmpeg:
|
||||
wav_path = tmp / f"cam{i}.wav"
|
||||
write_wav(wav_path, clip, sr)
|
||||
# Pass this camera's poses + offset so the M18 markers render through the same
|
||||
# projection lane H inverts (the marker pixel derives from its 3D truth).
|
||||
render_video(out_path, wav_path, i, w, h, fps, duration,
|
||||
poses=poses, offset_ms=float(offset_ms))
|
||||
render_video(out_path, wav_path, i, w, h, fps, duration)
|
||||
probed = ffprobe_video(out_path)
|
||||
vw, vh, vfps, vdur = (probed["width"], probed["height"],
|
||||
probed["fps"], probed["duration_s"])
|
||||
@ -750,7 +496,7 @@ def build(base_dir: Path | str | None = None, duration_s: float | None = None,
|
||||
filename=filename, duration_s=vdur, fps=vfps, width=vw, height=vh,
|
||||
offset_ms=float(offset_ms), drift_ppm=0.0, sync_confidence=1.0,
|
||||
)
|
||||
db.set_poses(video.id, poses)
|
||||
db.set_poses(video.id, camera_track(i, duration, fps, w, h))
|
||||
videos_gt.append({
|
||||
"video_id": video.id, "filename": filename,
|
||||
"offset_ms": float(offset_ms), "drift_ppm": 0.0,
|
||||
@ -791,28 +537,17 @@ def build(base_dir: Path | str | None = None, duration_s: float | None = None,
|
||||
gt_path = work / "ground_truth.json"
|
||||
gt_path.write_text(json.dumps(ground_truth, indent=2))
|
||||
|
||||
# Friend-track ground truth (M18): the two markers' 3D paths over the union of the
|
||||
# cameras' global coverage. Written for both ffmpeg and no-ffmpeg builds — it is pure
|
||||
# geometry (the same paths the markers are rendered from), so tests can check it cheaply.
|
||||
offsets_s = [o / 1000.0 for o in config.SYNTH_OFFSETS_MS]
|
||||
track_truth = build_track_truth(min(offsets_s), max(offsets_s) + duration)
|
||||
tt_path = work / "track_truth.json"
|
||||
tt_path.write_text(json.dumps(track_truth, indent=2))
|
||||
|
||||
summary = {
|
||||
"base_dir": str(base),
|
||||
"videos": [v["filename"] for v in videos_gt],
|
||||
"points_ply": str(work / "points.ply"),
|
||||
"point_count": int(len(points)),
|
||||
"ground_truth": str(gt_path),
|
||||
"track_truth": str(tt_path),
|
||||
"markers": [m["marker_key"] for m in track_truth["markers"]],
|
||||
"events": len(SEED_EVENTS),
|
||||
"anchors": len(STAGE_CORNERS),
|
||||
}
|
||||
log.info("synthetic: done — %d videos, %d points, %d events, %d anchors, %d markers",
|
||||
len(videos_gt), len(points), len(SEED_EVENTS), len(STAGE_CORNERS),
|
||||
len(track_truth["markers"]))
|
||||
log.info("synthetic: done — %d videos, %d points, %d events, %d anchors",
|
||||
len(videos_gt), len(points), len(SEED_EVENTS), len(STAGE_CORNERS))
|
||||
return summary
|
||||
|
||||
|
||||
|
||||
@ -1,171 +0,0 @@
|
||||
"""Marker detection (spec M19, lane H) — STUB with the frozen contracts baked in.
|
||||
|
||||
foundation3 ships this file with:
|
||||
|
||||
- the **concrete, foundation-owned protocol constants + pure encoders** every side of the
|
||||
system must agree on (the blink OOK protocol #4, the color-mode marker-key convention #1).
|
||||
``synthetic.py`` renders the fixture markers with these, ``scripts/badge_selftest.py``
|
||||
(lane K) imports this decoder path, and ``hardware/badge/protocol.h`` (lane K) duplicates
|
||||
the constants with a documented copy-check. **One source of truth lives here.**
|
||||
- a **stub** :func:`run_detect` that raises ``NotImplementedError``. Lane H fills the body
|
||||
(and adds ``backend/tests/test_tracker_detect.py``); it must not change the constants or
|
||||
the emitted ``marker_key`` conventions, only implement the detection logic.
|
||||
|
||||
Nothing in this module raises at import time — ``synthetic.py`` imports the constants and the
|
||||
``encode_word`` / ``led_on`` helpers, so the import must stay side-effect-free and light
|
||||
(no OpenCV at module scope; import ``cv2`` inside ``run_detect``).
|
||||
|
||||
================================================================================
|
||||
FROZEN CONTRACTS (plan/30-phase6.md — copy, do not reinterpret)
|
||||
================================================================================
|
||||
|
||||
Contract #1 — Marker hues (color mode)
|
||||
--------------------------------------
|
||||
Magenta ``#ff00ff`` and cyan ``#00ffff`` are the two supported fixture hues; real shoots may
|
||||
configure others via ``FESTIVAL4D_MARKER_HUES`` (comma-separated hex). Detection converts to
|
||||
HSV and gates on **saturation AND value before hue** (pitfall #2 — stage lighting lies about
|
||||
color). A color blob's ``marker_key`` is ``"hue:<H>"`` where ``<H>`` is the **OpenCV hue**
|
||||
(0..179) of the configured marker color, i.e. ``round(matplotlib_hue_degrees / 2)``:
|
||||
magenta → ``hue:150``, cyan → ``hue:90``. Use :func:`color_marker_key` — never format the
|
||||
string by hand.
|
||||
|
||||
Contract #2 — Schema (what the solver, not the detector, writes; here for context)
|
||||
----------------------------------------------------------------------------------
|
||||
``tracks(id INTEGER PK, marker_key TEXT NOT NULL, label TEXT, color TEXT, created_at TEXT)``
|
||||
``track_points(id INTEGER PK, track_id FK, t_global_s FLOAT, x FLOAT, y FLOAT, z FLOAT,
|
||||
quality FLOAT, views INTEGER)`` — points ordered by ``t_global_s``;
|
||||
``views`` = cameras used (1 ⇒ ground-plane fallback); coordinates in **Three.js scene space**.
|
||||
|
||||
Contract #3 — detections.json (this module's OUTPUT — write via `db`? No: a JSON file)
|
||||
--------------------------------------------------------------------------------------
|
||||
Write ``config.DETECTIONS_JSON`` (``data/work/detections.json``) with EXACTLY this shape::
|
||||
|
||||
{"videos": {"<video_id>": [{"t_video_s": float, "marker_key": str,
|
||||
"cx": float, "cy": float, "area": float,
|
||||
"conf": float 0..1}]},
|
||||
"generated_by": "festival4d.tracker_detect"}
|
||||
|
||||
``cx`` / ``cy`` are **normalized 0..1** (the bbox-annotation convention: ``cx = px / width``,
|
||||
``cy = py / height``). ``area`` is the blob area in pixels. Detections at ``t_video`` (the
|
||||
frame's local time) — the solver converts to ``t_global`` with the FROZEN
|
||||
``config.t_global_from_video`` helper (pitfall #1). No markers / dark / markerless video →
|
||||
a valid-empty ``{"videos": {}, "generated_by": ...}``, never a crash.
|
||||
|
||||
Contract #4 — Blink protocol (OOK, camera-decodable)
|
||||
----------------------------------------------------
|
||||
Bit period **133 ms** (≈4 frames @ 30 fps). One **word** = preamble ``11100`` + 6-bit ID
|
||||
(MSB first) + even parity = **12 bits**, repeated continuously (~1.6 s/word). ID **63** is
|
||||
reserved for the sync beacon. Decoding needs **≥24 fps** video and **≥1 full word** of
|
||||
visibility. Decode by **integrating brightness per bit window using the video's real fps**
|
||||
(``meta.fps``) — never by counting frames (pitfall #3). Emit ``marker_key = "code:<id>"``
|
||||
via :func:`code_marker_key`. The constants below are the byte-identical source for
|
||||
``hardware/badge/protocol.h`` (lane K).
|
||||
|
||||
Contract #5 — Ground plane (solver context)
|
||||
-------------------------------------------
|
||||
Synthetic ground is ``y = 0`` (scene space). Real projects estimate it as the
|
||||
5th-percentile-y horizontal plane of the point cloud. Single-view fallback intersects the
|
||||
pixel ray with this plane; such points get ``views = 1`` and ``quality ≤ 0.5``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
|
||||
log = logging.getLogger("festival4d.tracker_detect")
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Contract #4 — blink OOK protocol. FROZEN. Byte-identical to hardware/badge/protocol.h.
|
||||
# ---------------------------------------------------------------------------
|
||||
BLINK_BIT_PERIOD_S: float = 0.133 # 133 ms bit period (≈4 frames @ 30 fps)
|
||||
BLINK_PREAMBLE: tuple[int, ...] = (1, 1, 1, 0, 0) # start-of-word marker
|
||||
BLINK_ID_BITS: int = 6 # payload width (0..63)
|
||||
BLINK_WORD_BITS: int = len(BLINK_PREAMBLE) + BLINK_ID_BITS + 1 # +1 parity = 12
|
||||
BLINK_BEACON_ID: int = 63 # reserved: the sync/calibration beacon
|
||||
BLINK_MIN_FPS: float = 24.0 # decoder needs at least this frame rate
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Contract #1 — color mode marker hues. OpenCV hue is 0..179 (degrees / 2).
|
||||
# ---------------------------------------------------------------------------
|
||||
MAGENTA_OPENCV_HUE: int = 150 # #ff00ff
|
||||
CYAN_OPENCV_HUE: int = 90 # #00ffff
|
||||
DEFAULT_MARKER_HUES_HEX: tuple[str, ...] = ("#ff00ff", "#00ffff")
|
||||
|
||||
|
||||
def encode_word(marker_id: int) -> list[int]:
|
||||
"""The 12-bit OOK word for ``marker_id`` (contract #4). Pure; no I/O.
|
||||
|
||||
``preamble(11100) + 6-bit ID MSB-first + even-parity bit``. The parity bit is chosen so
|
||||
the **whole word has an even number of 1-bits** (parity over preamble+ID). This is the
|
||||
exact bit sequence a badge emits, ``synthetic.py`` renders, and lane H's decoder inverts.
|
||||
|
||||
>>> encode_word(5)
|
||||
[1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1]
|
||||
"""
|
||||
mid = int(marker_id)
|
||||
if not 0 <= mid < (1 << BLINK_ID_BITS):
|
||||
raise ValueError(f"marker_id {marker_id} out of range 0..{(1 << BLINK_ID_BITS) - 1}")
|
||||
id_bits = [(mid >> (BLINK_ID_BITS - 1 - i)) & 1 for i in range(BLINK_ID_BITS)]
|
||||
body = list(BLINK_PREAMBLE) + id_bits
|
||||
parity = sum(body) % 2 # even parity: word weight (incl. parity) is even
|
||||
return body + [parity]
|
||||
|
||||
|
||||
def led_on(t_seconds: float, marker_id: int, epoch: float = 0.0) -> bool:
|
||||
"""Whether the badge LED for ``marker_id`` is lit at wall-clock time ``t_seconds``.
|
||||
|
||||
The word repeats continuously from ``epoch``; a single physical badge blinks in real
|
||||
(global) time, so every camera that captures a frame at the same ``t_global`` sees the
|
||||
same LED state. Used by the synthetic fixture to render marker B faithfully.
|
||||
"""
|
||||
bit_index = int((t_seconds - epoch) // BLINK_BIT_PERIOD_S)
|
||||
word = encode_word(marker_id)
|
||||
return bool(word[bit_index % BLINK_WORD_BITS])
|
||||
|
||||
|
||||
def color_marker_key(opencv_hue: int) -> str:
|
||||
"""The ``marker_key`` for a color blob of the given OpenCV hue (contract #1)."""
|
||||
return f"hue:{int(opencv_hue)}"
|
||||
|
||||
|
||||
def code_marker_key(marker_id: int) -> str:
|
||||
"""The ``marker_key`` for a decoded blink ID (contract #4)."""
|
||||
return f"code:{int(marker_id)}"
|
||||
|
||||
|
||||
def marker_hues_hex() -> list[str]:
|
||||
"""Configured color-mode hues as hex strings (``FESTIVAL4D_MARKER_HUES`` or the default)."""
|
||||
env = os.environ.get("FESTIVAL4D_MARKER_HUES")
|
||||
if env:
|
||||
hues = [h.strip() for h in env.split(",") if h.strip()]
|
||||
if hues:
|
||||
return hues
|
||||
return list(DEFAULT_MARKER_HUES_HEX)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Lane H fills these. Signatures FROZEN.
|
||||
# ---------------------------------------------------------------------------
|
||||
def run_detect(sample_fps: float = 8.0) -> dict:
|
||||
"""Detect markers in every video and write ``detections.json`` (spec M19, contract #3).
|
||||
|
||||
Lane H implements:
|
||||
1. Sample frames per video with the ``frames.py`` machinery at ~``sample_fps`` fps
|
||||
(document the exact rate; ~8 fps comfortably oversamples the 133 ms bit period).
|
||||
2. **Color mode:** BGR→HSV; gate on saturation AND value *before* hue (pitfall #2);
|
||||
connected components per configured hue (:func:`marker_hues_hex`); blob centroid →
|
||||
normalized ``cx``/``cy``; ``marker_key = color_marker_key(<opencv_hue>)``.
|
||||
3. **Blink mode:** track bright blobs across the sampled window; integrate brightness
|
||||
per :data:`BLINK_BIT_PERIOD_S` window using the video's **real fps** (``meta.fps``,
|
||||
pitfall #3); correlate against :data:`BLINK_PREAMBLE`; check even parity; emit
|
||||
``marker_key = code_marker_key(<id>)``.
|
||||
4. Write ``config.DETECTIONS_JSON`` in the contract-#3 shape. No markers / dark /
|
||||
markerless → valid-empty ``{"videos": {}, "generated_by": "festival4d.tracker_detect"}``.
|
||||
|
||||
Returns a summary dict (per-video detection counts, decoded blink IDs). Never crash on a
|
||||
markerless or unreadable video — log and continue.
|
||||
"""
|
||||
raise NotImplementedError(
|
||||
"marker detection not implemented yet (lane H / M19) — fill tracker_detect.run_detect"
|
||||
)
|
||||
@ -1,72 +0,0 @@
|
||||
"""Track solving (spec M20, lane H) — STUB with the frozen contract in the docstring.
|
||||
|
||||
foundation3 ships the stub; lane H fills :func:`run_solve` (and adds
|
||||
``backend/tests/test_tracker_solve.py``). This is pure orchestration over the FROZEN
|
||||
geometry primitives — it must NEVER inline pose math (read ``resolve.py`` first; it is the
|
||||
exact per-timestep pattern), and it writes through the ``db`` track helpers only.
|
||||
|
||||
================================================================================
|
||||
FROZEN CONTRACTS (plan/30-phase6.md — copy, do not reinterpret)
|
||||
================================================================================
|
||||
|
||||
Input — ``config.DETECTIONS_JSON`` (contract #3, written by ``tracker_detect.run_detect``)::
|
||||
|
||||
{"videos": {"<video_id>": [{"t_video_s", "marker_key", "cx", "cy", "area", "conf"}]},
|
||||
"generated_by": "festival4d.tracker_detect"}
|
||||
|
||||
``cx``/``cy`` are normalized 0..1; multiply by the video's width/height for pixel coords.
|
||||
|
||||
Output — the ``tracks`` / ``track_points`` tables (contract #2), via ``db`` helpers::
|
||||
|
||||
tracks(id, marker_key, label, color, created_at)
|
||||
track_points(id, track_id, t_global_s, x, y, z, quality, views) # ordered by t_global_s
|
||||
# coordinates in Three.js scene space; views = cameras used (1 ⇒ ground-plane fallback)
|
||||
|
||||
Algorithm (spec M20 — the parts that bite):
|
||||
1. Load detections; group by ``marker_key``.
|
||||
2. **Time alignment is the whole game** (pitfall #1): a detection is found at ``t_video``;
|
||||
association happens at ``t_global``. Convert with the FROZEN
|
||||
``config.t_global_from_video(t_video, offset_ms, drift_ppm)`` ONLY — never re-derive it.
|
||||
3. Per time bucket (**≈0.25 s** of ``t_global``), collect same-``marker_key`` detections
|
||||
across cameras at matching ``t_global``:
|
||||
- **2+ views → triangulate** through ``geometry`` primitives exactly as ``resolve.py``
|
||||
does (``geometry.ray_from_pixel`` per view + ``geometry.triangulate_rays``, or a
|
||||
multi-ray least squares built on them). Record ``views`` and a ``quality`` derived
|
||||
from the residual/gap.
|
||||
- **1 view → ray ∩ ground plane** (contract #5): the synthetic ground is ``y = 0``;
|
||||
real projects estimate the 5th-percentile-y horizontal plane of the point cloud and
|
||||
record it in the result dict. Such points get ``views = 1`` and ``quality ≤ 0.5``.
|
||||
4. Smooth each track (moving average or Catmull-Rom — your call; **document which**).
|
||||
5. Solid color for hue markers, palette color for code markers.
|
||||
6. Write via ``db.add_track`` + ``db.set_track_points``. **Re-solve replaces that marker's
|
||||
track** (idempotent — never stack duplicates on repeated runs).
|
||||
7. Degradation: no detections → zero tracks + a note; never crash.
|
||||
|
||||
``POST /api/tracks/solve`` and ``python -m festival4d track`` both run detect→solve
|
||||
end-to-end once this and ``tracker_detect`` are filled.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
log = logging.getLogger("festival4d.tracker_solve")
|
||||
|
||||
# Poses are keyed every ~0.5 s; bucket association at 0.25 s (spec M20). Kept here so lane H
|
||||
# and any test reference one number.
|
||||
BUCKET_S: float = 0.25
|
||||
GROUND_PLANE_Y: float = 0.0 # synthetic fixture ground (contract #5, scene space)
|
||||
SINGLE_VIEW_MAX_QUALITY: float = 0.5 # ground-plane fallback quality cap (contract #5)
|
||||
|
||||
|
||||
def run_solve() -> dict:
|
||||
"""Solve detections into 3D friend tracks (spec M20, see module docstring).
|
||||
|
||||
Lane H implements the full detect-output → ``tracks``/``track_points`` pipeline described
|
||||
above. Returns a summary dict (tracks written, per-track point counts + coverage, the
|
||||
ground-plane used, and any degradation note). Idempotent: re-running replaces each
|
||||
marker's track rather than stacking duplicates.
|
||||
"""
|
||||
raise NotImplementedError(
|
||||
"track solving not implemented yet (lane H / M20) — fill tracker_solve.run_solve"
|
||||
)
|
||||
@ -32,15 +32,8 @@ def client():
|
||||
|
||||
def test_manifest_shape(client):
|
||||
data = client.get("/api/manifest").json()
|
||||
# Exact-set lock: additive manifest fields are fine, but must be added here consciously
|
||||
# (this assertion has already caught two silent drifts — has_splat and has_capture).
|
||||
# phase 6 (foundation3 / M18): has_tracks added — see plan/CHANGE_REQUESTS.md CR-6.
|
||||
assert set(data) == {"videos", "t_global_max", "has_poses", "has_splat", "has_capture",
|
||||
"capsule", "has_tracks"}
|
||||
assert data["has_tracks"] is False # synthetic build seeds no solved tracks (solver = lane H)
|
||||
assert set(data) == {"videos", "t_global_max", "has_poses"}
|
||||
assert data["has_poses"] is True
|
||||
assert data["capsule"] is False # live server is never a capsule (M17 bakes true)
|
||||
assert data["has_capture"] is False # capture routes are opt-in (FESTIVAL4D_CAPTURE)
|
||||
assert len(data["videos"]) == 3
|
||||
v = data["videos"][0]
|
||||
assert set(v) == {
|
||||
@ -85,19 +78,6 @@ def test_pointcloud_is_ply(client):
|
||||
assert resp.content[:3] == b"ply"
|
||||
|
||||
|
||||
def test_audio_track_extracted_and_cached(client):
|
||||
"""GET /api/audio/{id}: lazily-extracted AAC for the WebAudio master clock."""
|
||||
resp = client.get("/api/audio/1")
|
||||
assert resp.status_code == 200
|
||||
assert resp.headers["content-type"] == "audio/mp4"
|
||||
assert len(resp.content) > 1000
|
||||
# ftyp box near the start marks an MP4/M4A container.
|
||||
assert b"ftyp" in resp.content[:64]
|
||||
# Second request serves the cached file byte-identical (no re-extraction drift).
|
||||
assert client.get("/api/audio/1").content == resp.content
|
||||
assert client.get("/api/audio/999").status_code == 404
|
||||
|
||||
|
||||
def test_video_range_returns_206(client):
|
||||
"""Spec pitfall #3: <video> seeking needs HTTP Range -> 206."""
|
||||
resp = client.get("/media/cam0.mp4", headers={"Range": "bytes=0-100"})
|
||||
@ -123,26 +103,10 @@ def test_detect_events_endpoint(client):
|
||||
assert client.post("/api/events/detect", json={}).status_code == 200 # idempotent
|
||||
|
||||
|
||||
def test_annotation_resolves_to_anchor(client):
|
||||
# M8 (integration): a stored annotation now resolves to a 3D anchor. Single-view (no
|
||||
# event_id) falls back to the nearest point-cloud point / centroid-depth ray, so an
|
||||
# anchor + point come back. (Foundation asserted the stub contract anchor_id is None;
|
||||
# superseded by M8 landing — same situation as CR-1.)
|
||||
def test_annotation_stored(client):
|
||||
data = client.post(
|
||||
"/api/annotations",
|
||||
json={"video_id": 1, "t_video_s": 0.5, "bbox": [0.4, 0.4, 0.6, 0.6]},
|
||||
).json()
|
||||
assert data["annotation_id"] > 0
|
||||
assert data["anchor_id"] is not None
|
||||
assert data["point"] is not None and len(data["point"]) == 3
|
||||
assert data["method"] in {"nearest_point", "ray_depth", "triangulated"}
|
||||
assert data["anchor"]["id"] == data["anchor_id"]
|
||||
|
||||
|
||||
def test_event_patch_sets_user_source(client):
|
||||
events = client.get("/api/events").json()
|
||||
eid = events[0]["id"]
|
||||
patched = client.patch(f"/api/events/{eid}", json={"event_type": "pyro"}).json()
|
||||
assert patched["event_type"] == "pyro"
|
||||
assert patched["source"] == "user" # user correction attribution
|
||||
assert client.patch("/api/events/99999", json={"event_type": "x"}).status_code == 404
|
||||
assert data["anchor_id"] is None # M8 resolution is integration work
|
||||
|
||||
@ -1,171 +0,0 @@
|
||||
"""M10 acceptance tests (lane E): beat/onset analysis of the reference ingest WAV.
|
||||
|
||||
Built on the synthetic fixture + a real ingest run (per the phase-5 test pattern): the
|
||||
fixture's master audio carries a known 120 BPM click grid (``synthetic.BEAT_INTERVAL_S``
|
||||
= 0.5 s), so acceptance is objective — beats within ±50 ms of the grid, tempo within
|
||||
±2 BPM. Silence / short / missing audio must degrade per the house style, never crash.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shutil
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
pytestmark = pytest.mark.skipif(
|
||||
shutil.which("ffmpeg") is None or shutil.which("ffprobe") is None,
|
||||
reason="ffmpeg/ffprobe required to build the fixture + run ingest",
|
||||
)
|
||||
|
||||
BEAT_GRID_S = 0.5 # synthetic.BEAT_INTERVAL_S — the fixture's known pulse grid
|
||||
TRUE_TEMPO_BPM = 60.0 / BEAT_GRID_S # 120
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def project():
|
||||
"""Full synthetic project + ingest (produces the 16 kHz WAVs in config.AUDIO_DIR)."""
|
||||
from festival4d import config, ingest, synthetic
|
||||
|
||||
synthetic.build(run_ffmpeg=True) # default 20 s — enough signal for tempo tracking
|
||||
ingest.run_ingest()
|
||||
config.BEATS_JSON.unlink(missing_ok=True)
|
||||
yield config
|
||||
config.BEATS_JSON.unlink(missing_ok=True) # don't leak state into later test modules
|
||||
|
||||
|
||||
def _grid_error_s(t: float) -> float:
|
||||
"""Distance from t to the nearest 0.5 s grid line."""
|
||||
return abs(t - round(t / BEAT_GRID_S) * BEAT_GRID_S)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The real thing: known pulse grid recovered from the ingest WAV.
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_run_features_recovers_synthetic_pulse_grid(project):
|
||||
from festival4d import audio_features
|
||||
|
||||
summary = audio_features.run_features()
|
||||
assert summary["status"] == "ok"
|
||||
assert project.BEATS_JSON.exists()
|
||||
|
||||
data = json.loads(project.BEATS_JSON.read_text())
|
||||
# Contract #1 structure lock: exactly these keys.
|
||||
assert set(data) == {"tempo_bpm", "beats_s", "onsets", "generated_by"}
|
||||
assert data["generated_by"] == "festival4d.audio_features"
|
||||
|
||||
# Tempo within ±2 BPM of the known 120 BPM grid.
|
||||
assert data["tempo_bpm"] == pytest.approx(TRUE_TEMPO_BPM, abs=2.0)
|
||||
|
||||
# Beats: a healthy number over 20 s @ 120 BPM, each within ±50 ms of the grid,
|
||||
# strictly increasing, inside the reference window (t_global == reference local time).
|
||||
beats = data["beats_s"]
|
||||
assert len(beats) >= 20
|
||||
assert beats == sorted(beats)
|
||||
# AAC encode/decode pads the clip tail slightly, so allow a small margin past 20 s.
|
||||
assert all(0.0 <= b <= 20.5 for b in beats)
|
||||
worst = max(_grid_error_s(b) for b in beats)
|
||||
assert worst <= 0.050, f"worst beat off grid by {worst * 1000:.1f} ms"
|
||||
|
||||
# Onsets: present, valid strengths, times on the master timeline.
|
||||
onsets = data["onsets"]
|
||||
assert len(onsets) > 0
|
||||
assert all(set(o) == {"t_global_s", "strength"} for o in onsets)
|
||||
assert all(0.0 <= o["strength"] <= 1.0 for o in onsets)
|
||||
assert any(o["strength"] >= 0.9 for o in onsets) # normalized: the peak onset is ~1.0
|
||||
assert all(0.0 <= o["t_global_s"] <= 20.5 for o in onsets)
|
||||
|
||||
# The summary mirrors the file.
|
||||
assert summary["beats"] == len(beats)
|
||||
assert summary["onsets"] == len(onsets)
|
||||
assert summary["reference_video_id"] == 1 # cam0 (offset 0) is the reference
|
||||
|
||||
|
||||
def test_onsets_catch_the_ground_truth_bangs(project):
|
||||
"""The three loud synthetic bangs (3, 10, 17 s) must appear as strong onsets."""
|
||||
from festival4d import audio_features, synthetic
|
||||
|
||||
audio_features.run_features()
|
||||
data = json.loads(project.BEATS_JSON.read_text())
|
||||
onset_times = [o["t_global_s"] for o in data["onsets"]]
|
||||
for bang in synthetic.BANG_TIMES_S:
|
||||
assert any(abs(t - bang) <= 0.075 for t in onset_times), f"bang at {bang}s not detected"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Degradation: silence / short / missing input — valid output or clear note, never a crash.
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_silence_produces_valid_empty_json(project):
|
||||
from festival4d import audio_features, synthetic
|
||||
|
||||
wav = project.AUDIO_DIR / "1.wav"
|
||||
original = wav.read_bytes()
|
||||
try:
|
||||
synthetic.write_wav(wav, np.zeros(16_000 * 5, dtype=np.float32), 16_000)
|
||||
summary = audio_features.run_features()
|
||||
assert summary["status"] == "ok"
|
||||
data = json.loads(project.BEATS_JSON.read_text())
|
||||
assert data == {
|
||||
"tempo_bpm": None,
|
||||
"beats_s": [],
|
||||
"onsets": [],
|
||||
"generated_by": "festival4d.audio_features",
|
||||
}
|
||||
finally:
|
||||
wav.write_bytes(original)
|
||||
project.BEATS_JSON.unlink(missing_ok=True)
|
||||
|
||||
|
||||
def test_too_short_audio_produces_valid_empty_json(project):
|
||||
from festival4d import audio_features, synthetic
|
||||
|
||||
wav = project.AUDIO_DIR / "1.wav"
|
||||
original = wav.read_bytes()
|
||||
try:
|
||||
rng = np.random.default_rng(3)
|
||||
synthetic.write_wav(wav, rng.standard_normal(4_000).astype(np.float32) * 0.5, 16_000)
|
||||
audio_features.run_features()
|
||||
data = json.loads(project.BEATS_JSON.read_text())
|
||||
assert data["tempo_bpm"] is None and data["beats_s"] == [] and data["onsets"] == []
|
||||
finally:
|
||||
wav.write_bytes(original)
|
||||
project.BEATS_JSON.unlink(missing_ok=True)
|
||||
|
||||
|
||||
def test_missing_wav_degrades_with_note(project):
|
||||
from festival4d import audio_features
|
||||
|
||||
wav = project.AUDIO_DIR / "1.wav"
|
||||
moved = wav.with_suffix(".wav.bak")
|
||||
wav.rename(moved)
|
||||
try:
|
||||
summary = audio_features.run_features() # must not raise
|
||||
assert summary["status"] == "no_audio"
|
||||
assert "ingest" in summary["note"]
|
||||
assert not project.BEATS_JSON.exists() # nothing stale written
|
||||
finally:
|
||||
moved.rename(wav)
|
||||
|
||||
|
||||
def test_analyze_is_pure_and_beatless_noise_is_empty():
|
||||
"""Unit-level: analyze() on beatless noise returns the valid-empty shape."""
|
||||
from festival4d import audio_features
|
||||
|
||||
rng = np.random.default_rng(11)
|
||||
y = (rng.standard_normal(16_000 * 6) * 0.2).astype(np.float32) # 6 s of plain noise
|
||||
data = audio_features.analyze(y, 16_000)
|
||||
assert set(data) == {"tempo_bpm", "beats_s", "onsets", "generated_by"}
|
||||
# librosa may or may not hallucinate a weak pulse in noise; the hard requirement is a
|
||||
# valid, well-typed result — not a crash.
|
||||
assert isinstance(data["beats_s"], list)
|
||||
assert all(isinstance(b, float) for b in data["beats_s"])
|
||||
|
||||
|
||||
def test_cli_features_runs_end_to_end(project):
|
||||
"""`python -m festival4d features` dispatches into the real implementation now."""
|
||||
from festival4d import cli
|
||||
|
||||
assert cli.main(["features"]) == 0
|
||||
assert project.BEATS_JSON.exists()
|
||||
project.BEATS_JSON.unlink()
|
||||
@ -1,262 +0,0 @@
|
||||
"""Audio-sync tests (spec M1, lane A).
|
||||
|
||||
Pure-numpy where possible (no ffmpeg): GCC-PHAT sign/precision, pairwise+global solve
|
||||
recovering the M0 known offsets (±10 ms), disconnected components → None (pitfall #5),
|
||||
cycle-consistency rejection, drift recovery (±3 ppm), and the full ``run_sync`` persist +
|
||||
export path against synthesized WAVs. One ffmpeg-gated test runs the literal spec M1
|
||||
acceptance: ``synthetic -> ingest -> sync`` recovers ground truth.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shutil
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from festival4d import audio_sync, config, db, ingest, synthetic
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
def _broadband(n: int, seed: int = 0) -> np.ndarray:
|
||||
"""Seeded, mildly band-limited noise — a sharp, unambiguous correlation target."""
|
||||
rng = np.random.default_rng(seed)
|
||||
x = rng.standard_normal(n)
|
||||
return np.convolve(x, np.ones(4) / 4.0, mode="same")
|
||||
|
||||
|
||||
def _delay_samples(x: np.ndarray, d: int) -> np.ndarray:
|
||||
"""Return ``y`` with ``y[n] = x[n - d]`` (x delayed by d samples, zero-filled)."""
|
||||
y = np.zeros_like(x)
|
||||
if d >= 0:
|
||||
y[d:] = x[:len(x) - d] if d else x
|
||||
else:
|
||||
y[:d] = x[-d:]
|
||||
return y
|
||||
|
||||
|
||||
def _clean_audio_dir() -> None:
|
||||
"""Remove any WAVs left in the shared session audio dir (ids are reused after reset_db)."""
|
||||
config.AUDIO_DIR.mkdir(parents=True, exist_ok=True)
|
||||
for wav in config.AUDIO_DIR.glob("*.wav"):
|
||||
wav.unlink()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# GCC-PHAT
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_gcc_phat_recovers_known_delay_positive_and_negative():
|
||||
sr = 16_000
|
||||
ref = _broadband(sr * 4)
|
||||
for d in (0, 137, -211, 800):
|
||||
sig = _delay_samples(ref, d)
|
||||
offset_s, conf = audio_sync.gcc_phat(sig, ref, sr)
|
||||
# positive offset_s means sig lags ref, i.e. matches a positive sample delay
|
||||
assert abs(offset_s - d / sr) < 0.5 / sr, (d, offset_s * sr)
|
||||
assert conf > 5.0
|
||||
|
||||
|
||||
def test_gcc_phat_confidence_drops_with_noise():
|
||||
sr = 16_000
|
||||
ref = _broadband(sr * 4, seed=1)
|
||||
clean = _delay_samples(ref, 300)
|
||||
rng = np.random.default_rng(2)
|
||||
noisy = clean + 3.0 * rng.standard_normal(len(clean))
|
||||
_, c_clean = audio_sync.gcc_phat(clean, ref, sr)
|
||||
_, c_noisy = audio_sync.gcc_phat(noisy, ref, sr)
|
||||
assert c_clean > c_noisy
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# pairwise + global solve
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_pairwise_and_solve_recover_synthetic_offsets():
|
||||
"""The M0 known offsets (0/+1370/−842 ms) recovered within the spec's ±10 ms."""
|
||||
sr = config.AUDIO_SAMPLE_RATE
|
||||
master, _ = synthetic.synth_master_audio(sr)
|
||||
signals = {
|
||||
i + 1: synthetic.slice_for_offset(master, sr, off, config.SYNTH_DURATION_S)
|
||||
for i, off in enumerate(config.SYNTH_OFFSETS_MS)
|
||||
}
|
||||
edges = audio_sync.pairwise_offsets(signals, sr)
|
||||
solved = audio_sync.solve_global_offsets(edges, list(signals))
|
||||
for i, off in enumerate(config.SYNTH_OFFSETS_MS):
|
||||
assert solved[i + 1] is not None
|
||||
assert abs(solved[i + 1] - off) <= 10.0, (off, solved[i + 1])
|
||||
|
||||
|
||||
def test_pairwise_matches_direct_gcc_phat_with_unequal_lengths():
|
||||
"""The cached-FFT pairwise path agrees with a direct per-pair gcc_phat call.
|
||||
|
||||
Signals get different lengths on purpose: the shared padded size (2·max_len) then
|
||||
differs from the per-pair size (la+lb), which is exactly where the two paths could
|
||||
diverge if the lag windowing were wrong.
|
||||
"""
|
||||
sr = 16_000
|
||||
master = _broadband(sr * 8, seed=7)
|
||||
signals = {
|
||||
1: master[: sr * 6],
|
||||
2: _delay_samples(master, 900)[: sr * 4],
|
||||
3: _delay_samples(master, -1500)[: sr * 5],
|
||||
}
|
||||
edges = audio_sync.pairwise_offsets(signals, sr)
|
||||
assert len(edges) == 3
|
||||
for e in edges:
|
||||
direct_delay, direct_conf = audio_sync.gcc_phat(signals[e["a"]], signals[e["b"]], sr)
|
||||
assert abs(e["offset_s"] - -direct_delay) < 1.0 / sr, e
|
||||
assert e["confidence"] > 5.0 and direct_conf > 5.0
|
||||
|
||||
|
||||
def test_pairwise_edge_sign_is_relative_start_offset():
|
||||
"""Edge ``offset_s`` = offset_a − offset_b (seconds)."""
|
||||
sr = config.AUDIO_SAMPLE_RATE
|
||||
master, _ = synthetic.synth_master_audio(sr)
|
||||
a = synthetic.slice_for_offset(master, sr, 500.0, 6.0) # id 1, offset +500 ms
|
||||
b = synthetic.slice_for_offset(master, sr, -300.0, 6.0) # id 2, offset −300 ms
|
||||
edges = audio_sync.pairwise_offsets({1: a, 2: b}, sr)
|
||||
assert len(edges) == 1
|
||||
assert abs(edges[0]["offset_s"] * 1000.0 - (500.0 - -300.0)) <= 10.0
|
||||
|
||||
|
||||
def test_solve_disconnected_component_gets_none():
|
||||
"""A video with no usable edges is unsynced (offset None), not guessed (pitfall #5)."""
|
||||
edges = [
|
||||
{"a": 1, "b": 2, "offset_s": -0.5, "confidence": 30.0},
|
||||
{"a": 2, "b": 3, "offset_s": -0.4, "confidence": 30.0},
|
||||
]
|
||||
solved = audio_sync.solve_global_offsets(edges, [1, 2, 3, 4])
|
||||
assert solved[1] == 0.0
|
||||
assert abs(solved[2] - 500.0) < 1.0
|
||||
assert abs(solved[3] - 900.0) < 1.0
|
||||
assert solved[4] is None # isolated → None
|
||||
|
||||
|
||||
def test_solve_rejects_cycle_inconsistent_edge():
|
||||
"""A grossly inconsistent edge (>50 ms residual) is rejected; the rest still solve."""
|
||||
# True offsets: 1=0, 2=+500 ms, 3=+900 ms. Edge 1-3 is corrupted (should be −0.9 s).
|
||||
edges = [
|
||||
{"a": 1, "b": 2, "offset_s": -0.5, "confidence": 20.0},
|
||||
{"a": 2, "b": 3, "offset_s": -0.4, "confidence": 20.0},
|
||||
{"a": 1, "b": 3, "offset_s": +5.0, "confidence": 20.0}, # bad
|
||||
]
|
||||
solved = audio_sync.solve_global_offsets(edges, [1, 2, 3])
|
||||
assert solved[1] == 0.0
|
||||
assert abs(solved[2] - 500.0) < 1.0
|
||||
assert abs(solved[3] - 900.0) < 1.0 # recovered despite the bad edge
|
||||
|
||||
|
||||
def test_low_confidence_edges_are_dropped():
|
||||
"""Edges below the confidence floor don't connect a component."""
|
||||
edges = [{"a": 1, "b": 2, "offset_s": -0.5, "confidence": 1.01}]
|
||||
solved = audio_sync.solve_global_offsets(edges, [1, 2])
|
||||
# 1 is the lone reference at 0; 2 has only a rejected edge → None
|
||||
assert solved[1] == 0.0 and solved[2] is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# drift
|
||||
# ---------------------------------------------------------------------------
|
||||
def _make_drifted(ref: np.ndarray, sr: int, drift_ppm: float, offset_s: float = 0.0) -> np.ndarray:
|
||||
"""Build v from the config timebase: v(τ) = ref((1 − d)·τ + offset_s), d = ppm·1e−6."""
|
||||
d = drift_ppm * 1e-6
|
||||
local = np.arange(len(ref))
|
||||
src = (1.0 - d) * local + offset_s * sr
|
||||
return np.interp(src, np.arange(len(ref)), ref, left=0.0, right=0.0)
|
||||
|
||||
|
||||
def test_estimate_drift_recovers_known_ppm():
|
||||
sr = 8_000
|
||||
ref = _broadband(sr * 150, seed=5) # 150 s so several 30 s-spaced windows fit
|
||||
for ppm in (30.0, -22.0):
|
||||
v = _make_drifted(ref, sr, ppm)
|
||||
est = audio_sync.estimate_drift(v, ref, sr)
|
||||
assert abs(est - ppm) <= 3.0, (ppm, est)
|
||||
|
||||
|
||||
def test_estimate_drift_zero_within_deadband():
|
||||
sr = 8_000
|
||||
ref = _broadband(sr * 150, seed=6)
|
||||
assert audio_sync.estimate_drift(ref, ref, sr) == 0.0 # identical
|
||||
assert audio_sync.estimate_drift(_make_drifted(ref, sr, 1.5), ref, sr) == 0.0 # < 5 ppm
|
||||
|
||||
|
||||
def test_estimate_drift_returns_zero_when_too_short():
|
||||
sr = 16_000
|
||||
ref = _broadband(sr * 15) # < 2 windows at 10 s / 30 s hop
|
||||
assert audio_sync.estimate_drift(ref, ref, sr) == 0.0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# run_sync orchestration
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_run_sync_persists_and_exports():
|
||||
"""Full run_sync against synthesized WAVs (no ffmpeg): DB + sync.json, ±10 ms."""
|
||||
sr = config.AUDIO_SAMPLE_RATE
|
||||
master, _ = synthetic.synth_master_audio(sr)
|
||||
db.init_engine()
|
||||
db.reset_db()
|
||||
_clean_audio_dir()
|
||||
gt = {}
|
||||
for i, off in enumerate(config.SYNTH_OFFSETS_MS):
|
||||
v = db.add_video(filename=f"cam{i}.mp4", duration_s=config.SYNTH_DURATION_S,
|
||||
fps=30.0, width=640, height=360)
|
||||
clip = synthetic.slice_for_offset(master, sr, off, config.SYNTH_DURATION_S)
|
||||
synthetic.write_wav(ingest.audio_wav_path(v.id), clip, sr)
|
||||
gt[v.id] = off
|
||||
|
||||
solution = audio_sync.run_sync()
|
||||
|
||||
for v in db.get_videos():
|
||||
assert v.offset_ms is not None
|
||||
assert abs(v.offset_ms - gt[v.id]) <= 10.0, (gt[v.id], v.offset_ms)
|
||||
assert v.drift_ppm == 0.0
|
||||
assert 0.0 <= v.sync_confidence <= 1.0
|
||||
ref = min(gt)
|
||||
assert solution["reference_video_id"] == ref
|
||||
assert db.get_video(ref).sync_confidence == 1.0
|
||||
exported = json.loads(config.SYNC_JSON.read_text())
|
||||
assert exported["reference_video_id"] == ref
|
||||
assert len(exported["edges"]) == 3
|
||||
|
||||
|
||||
def test_run_sync_marks_video_without_audio_unsynced():
|
||||
"""A registered video with no extracted WAV is left offset=None, not guessed."""
|
||||
sr = config.AUDIO_SAMPLE_RATE
|
||||
master, _ = synthetic.synth_master_audio(sr)
|
||||
db.init_engine()
|
||||
db.reset_db()
|
||||
_clean_audio_dir()
|
||||
v0 = db.add_video(filename="cam0.mp4", duration_s=20.0, fps=30.0, width=640, height=360)
|
||||
v1 = db.add_video(filename="cam1.mp4", duration_s=20.0, fps=30.0, width=640, height=360)
|
||||
no_audio = db.add_video(filename="cam2.mp4", duration_s=20.0, fps=30.0, width=640, height=360)
|
||||
synthetic.write_wav(ingest.audio_wav_path(v0.id),
|
||||
synthetic.slice_for_offset(master, sr, 0.0, 20.0), sr)
|
||||
synthetic.write_wav(ingest.audio_wav_path(v1.id),
|
||||
synthetic.slice_for_offset(master, sr, 1370.0, 20.0), sr)
|
||||
# no WAV for `no_audio`
|
||||
|
||||
audio_sync.run_sync()
|
||||
assert db.get_video(no_audio.id).offset_ms is None
|
||||
assert db.get_video(no_audio.id).sync_confidence is None
|
||||
assert db.get_video(v0.id).offset_ms == 0.0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ffmpeg-gated: the literal spec M1 acceptance (synthetic -> ingest -> sync)
|
||||
# ---------------------------------------------------------------------------
|
||||
@pytest.mark.skipif(shutil.which("ffmpeg") is None or shutil.which("ffprobe") is None,
|
||||
reason="ffmpeg/ffprobe required for the end-to-end sync acceptance")
|
||||
def test_synthetic_ingest_sync_recovers_ground_truth():
|
||||
synthetic.build(duration_s=8.0, run_ffmpeg=True) # into the temp data dir (conftest)
|
||||
ingest.run_ingest()
|
||||
audio_sync.run_sync()
|
||||
|
||||
gt = {v["filename"]: v["offset_ms"]
|
||||
for v in json.loads(config.GROUND_TRUTH_JSON.read_text())["videos"]}
|
||||
for v in db.get_videos():
|
||||
assert v.offset_ms is not None, v.filename
|
||||
assert abs(v.offset_ms - gt[v.filename]) <= 10.0, (v.filename, v.offset_ms, gt[v.filename])
|
||||
assert abs(v.drift_ppm) <= 3.0
|
||||
@ -1,339 +0,0 @@
|
||||
"""M17 memory capsule (lane G): bundle structure, baked-JSON validity, serve.py Range.
|
||||
|
||||
Builds a real capsule from the synthetic project (temp data dir per conftest) against a
|
||||
fabricated frontend dist (spec allows this — running npm inside tests would be neither
|
||||
hermetic nor fast), then boots the bundled ``serve.py`` in a subprocess and exercises it
|
||||
over HTTP: 200s, JSON content types on the extension-less baked API files, and the 206
|
||||
Range behavior video seeking depends on (phase-5 pitfall #4).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shutil
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
import pytest
|
||||
|
||||
pytestmark = pytest.mark.skipif(
|
||||
shutil.which("ffmpeg") is None or shutil.which("ffprobe") is None,
|
||||
reason="ffmpeg/ffprobe required to build the synthetic capsule fixture",
|
||||
)
|
||||
|
||||
CAMPATH = {
|
||||
"version": 1,
|
||||
"keyframes": [
|
||||
{"t_global": 0.2, "pos": [0.0, 1.0, 5.0], "quat": [0.0, 0.0, 0.0, 1.0], "fov": 50.0},
|
||||
{"t_global": 0.8, "pos": [2.0, 1.0, 4.0], "quat": [0.0, 0.5, 0.0, 0.8660254], "fov": 42.0},
|
||||
],
|
||||
}
|
||||
BEATS = {"tempo_bpm": 120.0, "beats_s": [0.25, 0.75], "onsets": [],
|
||||
"generated_by": "festival4d.audio_features"}
|
||||
SPLAT_BYTES = b"ply\nfake-splat-for-tests\n"
|
||||
API_SNIPPET = '<script>window.__F4D_API_BASE__=""</script>'
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# ---------------------------------------------------------------------------
|
||||
@pytest.fixture(scope="module")
|
||||
def project():
|
||||
"""Synthetic project + every optional artifact seeded (beats, splat, a saved path)."""
|
||||
from festival4d import config, db, synthetic
|
||||
|
||||
synthetic.build(duration_s=1.0, run_ffmpeg=True) # into the temp data dir (conftest)
|
||||
db.init_engine()
|
||||
db.init_db()
|
||||
config.BEATS_JSON.parent.mkdir(parents=True, exist_ok=True)
|
||||
config.BEATS_JSON.write_text(json.dumps(BEATS))
|
||||
config.SPLAT_PLY.write_bytes(SPLAT_BYTES)
|
||||
saved = db.add_path("capsule sweep", json.dumps(CAMPATH))
|
||||
yield {"path_id": saved.id}
|
||||
# Leave the shared session data dir as later test modules expect to find it.
|
||||
config.BEATS_JSON.unlink(missing_ok=True)
|
||||
config.SPLAT_PLY.unlink(missing_ok=True)
|
||||
db.delete_path(saved.id)
|
||||
|
||||
|
||||
def _make_dist(root):
|
||||
"""A minimal Vite-shaped dist: index.html with a module <script> + one asset."""
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
(root / "assets").mkdir(exist_ok=True)
|
||||
(root / "assets" / "index.js").write_text("console.log('festival4d test dist');\n")
|
||||
(root / "index.html").write_text(
|
||||
"<!doctype html>\n<html>\n <head>\n <meta charset=\"UTF-8\" />\n"
|
||||
" <title>Festival 4D</title>\n"
|
||||
" <script type=\"module\" crossorigin src=\"/assets/index.js\"></script>\n"
|
||||
" </head>\n <body><div id=\"app\"></div></body>\n</html>\n"
|
||||
)
|
||||
return root
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def fake_dist(tmp_path_factory):
|
||||
return _make_dist(tmp_path_factory.mktemp("frontend") / "dist")
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def bundle(project, fake_dist, tmp_path_factory):
|
||||
from festival4d import capsule
|
||||
|
||||
out = tmp_path_factory.mktemp("capsule") / "bundle"
|
||||
summary = capsule.build_capsule(out_dir=out, dist_dir=fake_dist)
|
||||
return out, summary
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def server(bundle):
|
||||
"""The bundle's own serve.py, running as a real subprocess in the bundle root."""
|
||||
out, _ = bundle
|
||||
with socket.socket() as s:
|
||||
s.bind(("127.0.0.1", 0))
|
||||
port = s.getsockname()[1]
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "serve.py", "--port", str(port)],
|
||||
cwd=out, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
|
||||
)
|
||||
base = f"http://127.0.0.1:{port}"
|
||||
try:
|
||||
deadline = time.monotonic() + 15
|
||||
while True:
|
||||
try:
|
||||
urllib.request.urlopen(f"{base}/api/manifest", timeout=1).read()
|
||||
break
|
||||
except OSError:
|
||||
if proc.poll() is not None or time.monotonic() > deadline:
|
||||
raise RuntimeError(
|
||||
f"serve.py did not come up: {proc.stdout.read().decode(errors='replace')}"
|
||||
)
|
||||
time.sleep(0.1)
|
||||
yield base
|
||||
finally:
|
||||
proc.terminate()
|
||||
proc.wait(timeout=5)
|
||||
|
||||
|
||||
def _get(url, headers=None):
|
||||
"""GET returning (status, headers, body); headers stay a case-insensitive Message."""
|
||||
req = urllib.request.Request(url, headers=headers or {})
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=5) as resp:
|
||||
return resp.status, resp.headers, resp.read()
|
||||
except urllib.error.HTTPError as err:
|
||||
return err.code, err.headers, err.read()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Bundle structure
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_bundle_structure(bundle):
|
||||
from festival4d import db
|
||||
|
||||
out, summary = bundle
|
||||
for rel in ("index.html", "serve.py", "assets/index.js",
|
||||
"api/manifest", "api/events", "api/anchors",
|
||||
# phase 6 (foundation3 / M18): friend tracks baked read-only — CR-6.
|
||||
"api/tracks",
|
||||
"api/beats", "api/pointcloud", "api/splat",
|
||||
# the paths listing lives at paths/index.html so per-id files can
|
||||
# coexist under the same URL prefix (see capsule.py)
|
||||
"api/paths/index.html"):
|
||||
assert (out / rel).is_file(), f"missing {rel}"
|
||||
for v in db.get_videos():
|
||||
assert (out / "media" / v.filename).is_file()
|
||||
assert (out / "api" / "videos" / str(v.id) / "poses").is_file()
|
||||
assert (out / "api" / "audio" / str(v.id)).is_file()
|
||||
assert summary["bundle"] == str(out)
|
||||
assert summary["files"] >= 10
|
||||
assert summary["bytes"] > 0
|
||||
|
||||
|
||||
def test_baked_manifest_is_capsule_true_and_urls_resolve(bundle):
|
||||
out, _ = bundle
|
||||
manifest = json.loads((out / "api" / "manifest").read_text())
|
||||
assert manifest["capsule"] is True
|
||||
assert manifest["has_capture"] is False # capture routes never exist in a static bundle
|
||||
assert manifest["has_poses"] is True and manifest["has_splat"] is True
|
||||
assert manifest["t_global_max"] > 0
|
||||
assert len(manifest["videos"]) >= 2
|
||||
for v in manifest["videos"]:
|
||||
# Every manifest URL must resolve inside the bundle (API_BASE == "" ⇒ root-relative).
|
||||
assert v["url"].startswith("/media/")
|
||||
assert (out / v["url"].lstrip("/")).is_file()
|
||||
|
||||
|
||||
def test_baked_json_matches_live_api_shapes(bundle, project):
|
||||
from festival4d import api, db
|
||||
|
||||
out, _ = bundle
|
||||
manifest = json.loads((out / "api" / "manifest").read_text())
|
||||
live = api.manifest()
|
||||
assert set(manifest) == set(live) # key set identical to the live route
|
||||
assert json.loads((out / "api" / "events").read_text()) == api.get_events()
|
||||
assert json.loads((out / "api" / "anchors").read_text()) == api.get_anchors()
|
||||
for v in db.get_videos():
|
||||
baked = json.loads((out / "api" / "videos" / str(v.id) / "poses").read_text())
|
||||
assert baked == api.video_poses(v.id)
|
||||
assert len(baked) > 0
|
||||
|
||||
assert json.loads((out / "api" / "beats").read_text()) == BEATS
|
||||
assert (out / "api" / "splat").read_bytes() == SPLAT_BYTES
|
||||
assert (out / "api" / "pointcloud").read_bytes()[:3] == b"ply"
|
||||
|
||||
# Saved camera paths (M16) baked read-only so the load dropdown works zero-backend.
|
||||
listing = json.loads((out / "api" / "paths" / "index.html").read_text())
|
||||
assert [p["name"] for p in listing] == ["capsule sweep"]
|
||||
pid = project["path_id"]
|
||||
assert json.loads((out / "api" / "paths" / str(pid)).read_text())["json"] == CAMPATH
|
||||
|
||||
|
||||
def test_index_html_injects_api_base_before_app_script(bundle):
|
||||
out, _ = bundle
|
||||
html = (out / "index.html").read_text()
|
||||
assert html.count(API_SNIPPET) == 1
|
||||
assert html.index(API_SNIPPET) < html.index('type="module"')
|
||||
|
||||
|
||||
def test_optional_artifacts_omitted_when_absent(project, fake_dist, tmp_path):
|
||||
"""Degrade, don't block: no beats/splat ⇒ those files are simply not baked."""
|
||||
from festival4d import capsule, config
|
||||
|
||||
beats, splat = config.BEATS_JSON.read_text(), config.SPLAT_PLY.read_bytes()
|
||||
config.BEATS_JSON.unlink()
|
||||
config.SPLAT_PLY.unlink()
|
||||
try:
|
||||
out = tmp_path / "bare"
|
||||
capsule.build_capsule(out_dir=out, dist_dir=fake_dist)
|
||||
assert not (out / "api" / "beats").exists()
|
||||
assert not (out / "api" / "splat").exists()
|
||||
manifest = json.loads((out / "api" / "manifest").read_text())
|
||||
assert manifest["has_splat"] is False
|
||||
assert manifest["capsule"] is True
|
||||
finally:
|
||||
config.BEATS_JSON.write_text(beats)
|
||||
config.SPLAT_PLY.write_bytes(splat)
|
||||
|
||||
|
||||
def test_missing_dist_errors_clearly(tmp_path):
|
||||
from festival4d import capsule
|
||||
|
||||
with pytest.raises(RuntimeError, match="npm run build"):
|
||||
capsule.build_capsule(out_dir=tmp_path / "never", dist_dir=tmp_path / "no-dist")
|
||||
assert not (tmp_path / "never").exists()
|
||||
|
||||
|
||||
def test_rebuild_overwrites_previous_capsule_but_not_strangers(bundle, fake_dist, project, tmp_path):
|
||||
from festival4d import capsule
|
||||
|
||||
# A non-capsule, non-empty directory is refused (don't eat someone's folder).
|
||||
stranger = tmp_path / "stranger"
|
||||
stranger.mkdir()
|
||||
(stranger / "precious.txt").write_text("do not delete")
|
||||
with pytest.raises(RuntimeError, match="refusing"):
|
||||
capsule.build_capsule(out_dir=stranger, dist_dir=fake_dist)
|
||||
assert (stranger / "precious.txt").exists()
|
||||
|
||||
# Rebuilding over a previous capsule works and never double-injects the snippet.
|
||||
out, _ = bundle
|
||||
capsule.build_capsule(out_dir=out, dist_dir=fake_dist)
|
||||
assert (out / "index.html").read_text().count(API_SNIPPET) == 1
|
||||
|
||||
|
||||
def test_cli_capsule_dispatch(project, fake_dist, tmp_path, monkeypatch):
|
||||
"""`python -m festival4d capsule` (CR-4: the stub exit-2 contract is retired)."""
|
||||
from festival4d import cli, config
|
||||
|
||||
# The CLI has no --dist flag; point the repo-root lookup at a fabricated tree.
|
||||
fake_root = tmp_path / "repo"
|
||||
_make_dist(fake_root / "frontend" / "dist")
|
||||
monkeypatch.setattr(config, "REPO_ROOT", fake_root)
|
||||
|
||||
out = tmp_path / "cli-bundle"
|
||||
assert cli.main(["capsule", "--out", str(out)]) == 0
|
||||
assert (out / "serve.py").is_file()
|
||||
assert json.loads((out / "api" / "manifest").read_text())["capsule"] is True
|
||||
|
||||
# And the no-dist error path is a clear message, not a stub exit.
|
||||
monkeypatch.setattr(config, "REPO_ROOT", tmp_path / "empty-repo")
|
||||
with pytest.raises(RuntimeError, match="npm run build"):
|
||||
cli.main(["capsule", "--out", str(tmp_path / "nope")])
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# serve.py over real HTTP
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_serve_index_and_json_content_types(server):
|
||||
status, headers, body = _get(f"{server}/")
|
||||
assert status == 200
|
||||
assert headers["Content-Type"].startswith("text/html")
|
||||
assert API_SNIPPET.encode() in body
|
||||
|
||||
status, headers, body = _get(f"{server}/api/manifest")
|
||||
assert status == 200
|
||||
assert headers["Content-Type"] == "application/json" # extension-less baked file
|
||||
assert json.loads(body)["capsule"] is True
|
||||
|
||||
for rel, ctype in (("api/events", "application/json"),
|
||||
("api/paths", "application/json"),
|
||||
("api/pointcloud", "application/octet-stream"),
|
||||
("api/splat", "application/octet-stream")):
|
||||
status, headers, _ = _get(f"{server}/{rel}")
|
||||
assert (status, headers["Content-Type"]) == (200, ctype), rel
|
||||
|
||||
|
||||
def test_serve_video_range_206(server, bundle):
|
||||
out, _ = bundle
|
||||
manifest = json.loads((out / "api" / "manifest").read_text())
|
||||
url = manifest["videos"][0]["url"] # "/media/<filename>"
|
||||
size = (out / url.lstrip("/")).stat().st_size
|
||||
|
||||
# Plain GET: 200, full body, and Range support advertised (browsers probe this).
|
||||
status, headers, body = _get(f"{server}{url}")
|
||||
assert status == 200
|
||||
assert headers["Accept-Ranges"] == "bytes"
|
||||
assert int(headers["Content-Length"]) == size == len(body)
|
||||
|
||||
# bytes=0-100 ⇒ 206 with exactly 101 bytes and a correct Content-Range.
|
||||
status, headers, body = _get(f"{server}{url}", {"Range": "bytes=0-100"})
|
||||
assert status == 206
|
||||
assert headers["Content-Range"] == f"bytes 0-100/{size}"
|
||||
assert len(body) == 101
|
||||
assert body[4:8] == b"ftyp" # mp4 magic — it's really the video's first bytes
|
||||
|
||||
# Open-ended seek (bytes=N-) ⇒ the tail from N.
|
||||
mid = size // 2
|
||||
status, headers, body = _get(f"{server}{url}", {"Range": f"bytes={mid}-"})
|
||||
assert status == 206
|
||||
assert headers["Content-Range"] == f"bytes {mid}-{size - 1}/{size}"
|
||||
assert len(body) == size - mid
|
||||
|
||||
# Suffix form (bytes=-N) ⇒ the final N bytes (mp4 moov probing does this).
|
||||
status, headers, body = _get(f"{server}{url}", {"Range": "bytes=-50"})
|
||||
assert status == 206
|
||||
assert headers["Content-Range"] == f"bytes {size - 50}-{size - 1}/{size}"
|
||||
assert len(body) == 50
|
||||
|
||||
# Start past EOF ⇒ 416 with the total size.
|
||||
status, headers, _ = _get(f"{server}{url}", {"Range": f"bytes={size + 10}-"})
|
||||
assert status == 416
|
||||
assert headers["Content-Range"] == f"bytes */{size}"
|
||||
|
||||
|
||||
def test_serve_audio_bytes_and_range(server, bundle):
|
||||
status, headers, body = _get(f"{server}/api/audio/1")
|
||||
assert status == 200
|
||||
assert headers["Content-Type"] == "audio/mp4"
|
||||
assert body[4:8] == b"ftyp" # m4a container magic
|
||||
# The WebAudio clock fetches whole files, but Range must still work on audio.
|
||||
status, headers, part = _get(f"{server}/api/audio/1", {"Range": "bytes=0-15"})
|
||||
assert status == 206 and part == body[:16]
|
||||
|
||||
|
||||
def test_serve_missing_file_404(server):
|
||||
assert _get(f"{server}/api/nope")[0] == 404
|
||||
assert _get(f"{server}/media/ghost.mp4")[0] == 404
|
||||
@ -1,137 +0,0 @@
|
||||
"""Live capture tests (Phase 5) — upload safety, the env gate, and the device monitor.
|
||||
|
||||
The router is exercised directly on a throwaway app: ``capture_enabled()`` is read at
|
||||
``api`` import time, so mounting it here is both simpler and keeps the gate test honest
|
||||
(the real app must NOT expose these routes unless FESTIVAL4D_CAPTURE is set).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from festival4d import capture, config
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def client():
|
||||
app = FastAPI()
|
||||
app.include_router(capture.router)
|
||||
config.RAW_DIR.mkdir(parents=True, exist_ok=True)
|
||||
with TestClient(app) as c:
|
||||
yield c
|
||||
|
||||
|
||||
def _upload(client, name: str, filename: str, data: bytes = b"fake-video-bytes"):
|
||||
return client.post(
|
||||
"/api/capture/upload",
|
||||
data={"device_name": name},
|
||||
files={"file": (filename, data, "video/webm")},
|
||||
)
|
||||
|
||||
|
||||
# --- the gate ---------------------------------------------------------------------------------
|
||||
|
||||
def test_capture_disabled_by_default():
|
||||
"""FESTIVAL4D_CAPTURE is unset in tests => the real app exposes no upload surface."""
|
||||
assert capture.capture_enabled() is False
|
||||
from festival4d import api
|
||||
|
||||
paths = {r.path for r in api.app.routes}
|
||||
assert "/capture" not in paths
|
||||
assert not any(p.startswith("/api/capture") for p in paths)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("val,expected", [("1", True), ("true", True), ("on", True),
|
||||
("0", False), ("", False), ("no", False)])
|
||||
def test_capture_enabled_parsing(monkeypatch, val, expected):
|
||||
monkeypatch.setenv("FESTIVAL4D_CAPTURE", val)
|
||||
assert capture.capture_enabled() is expected
|
||||
|
||||
|
||||
# --- upload -----------------------------------------------------------------------------------
|
||||
|
||||
def test_upload_saves_sanitized_file(client):
|
||||
r = _upload(client, "Phone Left!", "clip.webm")
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
saved = config.RAW_DIR / body["filename"]
|
||||
assert saved.is_file() and saved.read_bytes() == b"fake-video-bytes"
|
||||
assert body["filename"].startswith("phone-left-") # slugified, timestamped
|
||||
assert body["filename"].endswith(".webm")
|
||||
assert body["bytes"] == len(b"fake-video-bytes")
|
||||
saved.unlink()
|
||||
|
||||
|
||||
def test_upload_name_cannot_traverse(client):
|
||||
r = _upload(client, "../../etc/passwd", "clip.webm")
|
||||
assert r.status_code == 200
|
||||
saved = config.RAW_DIR / r.json()["filename"]
|
||||
# The slug strips separators entirely; the file lands inside RAW_DIR, nowhere else.
|
||||
assert saved.resolve().parent == config.RAW_DIR.resolve()
|
||||
assert ".." not in r.json()["filename"]
|
||||
saved.unlink()
|
||||
|
||||
|
||||
def test_upload_rejects_bad_container(client):
|
||||
r = _upload(client, "cam", "payload.exe")
|
||||
assert r.status_code == 400
|
||||
assert "unsupported container" in r.json()["detail"]
|
||||
|
||||
|
||||
def test_upload_rejects_empty(client):
|
||||
r = _upload(client, "cam", "clip.webm", data=b"")
|
||||
assert r.status_code == 400
|
||||
assert "empty" in r.json()["detail"]
|
||||
|
||||
|
||||
def test_upload_rejects_oversize(client, monkeypatch):
|
||||
monkeypatch.setattr(capture, "MAX_UPLOAD_BYTES", 8)
|
||||
r = _upload(client, "cam", "clip.webm", data=b"x" * 64)
|
||||
assert r.status_code == 413
|
||||
# The partial file must not be left behind for ingest to trip on.
|
||||
assert not any(p.name.startswith("cam-") for p in config.RAW_DIR.glob("cam-*.webm"))
|
||||
|
||||
|
||||
# --- monitor ----------------------------------------------------------------------------------
|
||||
|
||||
def test_heartbeat_and_devices_monitor(client):
|
||||
capture._devices.clear()
|
||||
client.post("/api/capture/heartbeat", json={
|
||||
"device_id": "d1", "name": "phone-left", "status": "recording",
|
||||
"elapsed_s": 3.5, "snapshot": "data:image/jpeg;base64,AAAA",
|
||||
})
|
||||
client.post("/api/capture/heartbeat", json={"device_id": "d2", "name": "laptop"})
|
||||
devs = client.get("/api/capture/devices").json()
|
||||
assert [d["name"] for d in devs] == ["laptop", "phone-left"] # sorted by name
|
||||
d1 = next(d for d in devs if d["device_id"] == "d1")
|
||||
assert d1["status"] == "recording" and d1["snapshot"].startswith("data:image/jpeg")
|
||||
assert "_seen" not in d1 # internals not leaked
|
||||
capture._devices.clear()
|
||||
|
||||
|
||||
def test_oversized_snapshot_dropped_but_device_kept(client):
|
||||
capture._devices.clear()
|
||||
client.post("/api/capture/heartbeat", json={
|
||||
"device_id": "big", "name": "cam", "snapshot": "data:image/jpeg;base64," + "A" * 400_000,
|
||||
})
|
||||
devs = client.get("/api/capture/devices").json()
|
||||
assert len(devs) == 1 and "snapshot" not in devs[0]
|
||||
capture._devices.clear()
|
||||
|
||||
|
||||
def test_stale_devices_pruned(client, monkeypatch):
|
||||
capture._devices.clear()
|
||||
client.post("/api/capture/heartbeat", json={"device_id": "old", "name": "cam"})
|
||||
assert len(client.get("/api/capture/devices").json()) == 1
|
||||
monkeypatch.setattr(capture, "DEVICE_STALE_S", -1.0) # everything is instantly stale
|
||||
assert client.get("/api/capture/devices").json() == []
|
||||
capture._devices.clear()
|
||||
|
||||
|
||||
def test_capture_page_served(client):
|
||||
r = client.get("/capture")
|
||||
assert r.status_code == 200
|
||||
assert "Festival 4D — Capture" in r.text
|
||||
assert "getUserMedia" in r.text # the page really is the capture app
|
||||
@ -1,242 +0,0 @@
|
||||
"""M11 acceptance tests (lane E): deterministic auto-director -> frozen camPath JSON.
|
||||
|
||||
Runs on the synthetic fixture WITHOUT ffmpeg (poses + events + DB only): generate_path
|
||||
never touches media files. The synthetic seed events have known times/confidences, so
|
||||
event selection, keyframe placement, beat snapping, and the pose conversion are all
|
||||
checked against ground truth. The quaternion test locks the [x,y,z,w] Three.js order via
|
||||
the frozen geometry.colmap_to_threejs — this repo's #1 historical bug source.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
# Synthetic seed events (synthetic.SEED_EVENTS): (t_global, type, confidence)
|
||||
# 3.0 bass_drop .94 | 6.0 crowd_wave .72 | 8.0 quiet .61 | 10.0 pyro .88
|
||||
# 13.0 light_show .90 | 17.0 confetti .81 | 19.0 artist .77
|
||||
TOP3_TIMES = [3.0, 10.0, 13.0] # confidences .94, .88, .90 -> top 3
|
||||
KEYFRAME_KEYS = {"t_global", "pos", "quat", "fov"}
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def project():
|
||||
"""Fresh synthetic project (no ffmpeg) per test — several tests mutate the DB."""
|
||||
from festival4d import config, synthetic
|
||||
|
||||
synthetic.build(run_ffmpeg=False)
|
||||
config.BEATS_JSON.unlink(missing_ok=True) # unsnapped by default; tests opt in
|
||||
yield config
|
||||
config.BEATS_JSON.unlink(missing_ok=True)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Structure lock: the frozen camPath shape (contract #2).
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_output_matches_frozen_campath_shape(project):
|
||||
from festival4d import director
|
||||
|
||||
path = director.generate_path()
|
||||
assert set(path) == {"version", "keyframes"} # exact-set lock (no stray fields on success)
|
||||
assert path["version"] == 1
|
||||
assert len(path["keyframes"]) > 0
|
||||
for kf in path["keyframes"]:
|
||||
assert set(kf) == KEYFRAME_KEYS # exact-set lock per keyframe
|
||||
assert isinstance(kf["t_global"], float)
|
||||
assert len(kf["pos"]) == 3 and all(isinstance(v, float) for v in kf["pos"])
|
||||
assert len(kf["quat"]) == 4 and all(isinstance(v, float) for v in kf["quat"])
|
||||
assert abs(math.hypot(*kf["quat"][:3], kf["quat"][3]) - 1.0) < 1e-6 # unit quaternion
|
||||
assert 10.0 < kf["fov"] < 120.0
|
||||
times = [kf["t_global"] for kf in path["keyframes"]]
|
||||
assert times == sorted(times)
|
||||
assert len(set(times)) == len(times) # strictly monotone (dedup after snapping)
|
||||
# JSON-serializable end to end (the API returns this dict verbatim).
|
||||
json.dumps(path)
|
||||
|
||||
|
||||
def test_covers_the_top_n_events_with_lead_and_duration(project):
|
||||
from festival4d import director
|
||||
|
||||
path = director.generate_path(top_n=3, lead_s=2.0)
|
||||
times = [kf["t_global"] for kf in path["keyframes"]]
|
||||
# Each top-3 event contributes keys at t-2.0 and t+duration (1.0 in the fixture):
|
||||
# 3.0 -> {1.0, 4.0}; 10.0 -> {8.0, 11.0}; 13.0 -> {11.0, 14.0}. The shared 11.0
|
||||
# collapses to one keyframe -> exactly 5, and every event is bracketed.
|
||||
assert times == [1.0, 4.0, 8.0, 11.0, 14.0]
|
||||
for t_event in TOP3_TIMES:
|
||||
assert any(t <= t_event for t in times) and any(t >= t_event for t in times)
|
||||
|
||||
|
||||
def test_full_top_n_covers_every_event(project):
|
||||
from festival4d import director
|
||||
|
||||
path = director.generate_path(top_n=8, lead_s=2.0) # fixture has 7 events
|
||||
times = [kf["t_global"] for kf in path["keyframes"]]
|
||||
for t_event in [3.0, 6.0, 8.0, 10.0, 13.0, 17.0, 19.0]:
|
||||
assert min(abs(t - (t_event - 2.0)) for t in times) < 1e-9
|
||||
assert min(abs(t - (t_event + 1.0)) for t in times) < 1e-9
|
||||
|
||||
|
||||
def test_confidence_ties_break_to_the_earlier_event(project):
|
||||
from festival4d import db, director
|
||||
|
||||
db.clear_events()
|
||||
db.add_event(t_global_s=15.0, event_type="pyro", source="ai", duration_s=1.0, confidence=0.9)
|
||||
db.add_event(t_global_s=5.0, event_type="pyro", source="ai", duration_s=1.0, confidence=0.9)
|
||||
path = director.generate_path(top_n=1, lead_s=2.0)
|
||||
assert [kf["t_global"] for kf in path["keyframes"]] == [3.0, 6.0] # the t=5 event won
|
||||
|
||||
|
||||
def test_lead_time_clamps_at_zero(project):
|
||||
from festival4d import db, director
|
||||
|
||||
db.clear_events()
|
||||
db.add_event(t_global_s=0.5, event_type="pyro", source="ai", duration_s=1.0, confidence=0.9)
|
||||
path = director.generate_path(top_n=1, lead_s=2.0)
|
||||
assert [kf["t_global"] for kf in path["keyframes"]] == [0.0, 1.5]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pose conversion: through the frozen geometry.colmap_to_threejs, quat order [x,y,z,w].
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_keyframe_pose_matches_frozen_conversion(project):
|
||||
from festival4d import db, director
|
||||
from festival4d.geometry import colmap_to_threejs, mat_to_quat, quat_to_mat
|
||||
|
||||
path = director.generate_path(top_n=1, lead_s=2.0) # top event: bass_drop @ 3.0
|
||||
kf = path["keyframes"][0] # t_global = 1.0
|
||||
|
||||
# Every synthetic event time is an exact cam0 pose time (offset 0), so the chosen
|
||||
# camera is video 1 and the keyframe pose is cam0's registered row at t_video_s=1.0.
|
||||
pose = next(p for p in db.get_poses(1) if abs(p.t_video_s - 1.0) < 1e-9)
|
||||
expected_pos, expected_rot = colmap_to_threejs(
|
||||
[pose.qw, pose.qx, pose.qy, pose.qz], [pose.tx, pose.ty, pose.tz]
|
||||
)
|
||||
assert np.allclose(kf["pos"], expected_pos, atol=1e-9)
|
||||
|
||||
# Quaternion order lock: reordering [x,y,z,w] -> [w,x,y,z] must reproduce the Three.js
|
||||
# world rotation matrix (up to the quaternion double cover).
|
||||
x, y, z, w = kf["quat"]
|
||||
assert np.allclose(quat_to_mat([w, x, y, z]), expected_rot, atol=1e-9)
|
||||
expected_q = mat_to_quat(expected_rot) # [w,x,y,z], canonical w >= 0
|
||||
q_out = np.array([w, x, y, z])
|
||||
if np.dot(q_out, expected_q) < 0:
|
||||
q_out = -q_out
|
||||
assert np.allclose(q_out, expected_q, atol=1e-9)
|
||||
|
||||
# FOV from the chosen camera's intrinsics: 2*atan(height / (2*fy)) in degrees.
|
||||
video = db.get_video(1)
|
||||
expected_fov = math.degrees(2.0 * math.atan(video.height / (2.0 * pose.fy)))
|
||||
assert kf["fov"] == pytest.approx(expected_fov, abs=1e-9)
|
||||
|
||||
|
||||
def test_unregistered_poses_are_excluded(project):
|
||||
from festival4d import db, director
|
||||
|
||||
# Mark every cam0 pose unregistered: the director must cut to cam1/cam2 instead.
|
||||
poses = db.get_poses(1)
|
||||
db.set_poses(1, [
|
||||
{"frame_idx": p.frame_idx, "t_video_s": p.t_video_s,
|
||||
"qw": p.qw, "qx": p.qx, "qy": p.qy, "qz": p.qz,
|
||||
"tx": p.tx, "ty": p.ty, "tz": p.tz,
|
||||
"fx": p.fx, "fy": p.fy, "cx": p.cx, "cy": p.cy, "registered": False}
|
||||
for p in poses
|
||||
])
|
||||
path = director.generate_path(top_n=1, lead_s=2.0)
|
||||
assert len(path["keyframes"]) == 2
|
||||
cam0_centers = {
|
||||
tuple(np.round(np.asarray(_center(p)), 6)) for p in poses
|
||||
}
|
||||
for kf in path["keyframes"]:
|
||||
assert tuple(np.round(kf["pos"], 6)) not in cam0_centers
|
||||
|
||||
|
||||
def _center(pose):
|
||||
from festival4d.geometry import colmap_to_threejs
|
||||
|
||||
pos, _ = colmap_to_threejs([pose.qw, pose.qx, pose.qy, pose.qz], [pose.tx, pose.ty, pose.tz])
|
||||
return pos
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Beat snapping (M10 -> M11), and degradation without it.
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_keyframes_snap_to_beats_when_beats_exist(project):
|
||||
from festival4d import director
|
||||
|
||||
grid = [round(k * 0.75, 4) for k in range(0, 28)] # 0.75 s grid != the keyframe times
|
||||
project.BEATS_JSON.parent.mkdir(parents=True, exist_ok=True)
|
||||
project.BEATS_JSON.write_text(json.dumps({
|
||||
"tempo_bpm": 80.0, "beats_s": grid, "onsets": [],
|
||||
"generated_by": "festival4d.audio_features",
|
||||
}))
|
||||
path = director.generate_path(top_n=3, lead_s=2.0)
|
||||
times = [kf["t_global"] for kf in path["keyframes"]]
|
||||
assert len(times) > 0
|
||||
assert all(any(abs(t - b) < 1e-9 for b in grid) for t in times) # every key ON a beat
|
||||
assert times != [1.0, 4.0, 8.0, 11.0, 14.0] # actually moved vs the unsnapped grid
|
||||
|
||||
|
||||
def test_no_beats_file_means_no_snapping(project):
|
||||
from festival4d import director
|
||||
|
||||
assert not project.BEATS_JSON.exists()
|
||||
times = [kf["t_global"] for kf in director.generate_path(top_n=3, lead_s=2.0)["keyframes"]]
|
||||
assert times == [1.0, 4.0, 8.0, 11.0, 14.0]
|
||||
|
||||
|
||||
def test_corrupt_beats_file_degrades_to_unsnapped(project):
|
||||
from festival4d import director
|
||||
|
||||
project.BEATS_JSON.parent.mkdir(parents=True, exist_ok=True)
|
||||
project.BEATS_JSON.write_text("{ not json")
|
||||
times = [kf["t_global"] for kf in director.generate_path(top_n=3, lead_s=2.0)["keyframes"]]
|
||||
assert times == [1.0, 4.0, 8.0, 11.0, 14.0]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Degradation: empty inputs -> valid empty path (still camPath.fromJSON-safe).
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_no_events_yields_valid_empty_path(project):
|
||||
from festival4d import db, director
|
||||
|
||||
db.clear_events()
|
||||
path = director.generate_path()
|
||||
assert path["version"] == 1 and path["keyframes"] == []
|
||||
assert "events" in path["note"]
|
||||
|
||||
|
||||
def test_no_registered_poses_yields_valid_empty_path_with_note(project):
|
||||
from festival4d import db, director
|
||||
|
||||
for video in db.get_videos():
|
||||
db.set_poses(video.id, [])
|
||||
path = director.generate_path()
|
||||
assert path["version"] == 1 and path["keyframes"] == []
|
||||
assert "poses" in path["note"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Dispatch: the API route and CLI now run the real implementation.
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_api_director_route_returns_real_path(project):
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from festival4d import api
|
||||
|
||||
with TestClient(api.app) as client:
|
||||
data = client.post("/api/director", json={"top_n": 3, "lead_s": 2.0}).json()
|
||||
assert data["version"] == 1
|
||||
assert [kf["t_global"] for kf in data["keyframes"]] == [1.0, 4.0, 8.0, 11.0, 14.0]
|
||||
assert all(set(kf) == KEYFRAME_KEYS for kf in data["keyframes"])
|
||||
|
||||
|
||||
def test_cli_direct_prints_campath_json(project, capsys):
|
||||
from festival4d import cli
|
||||
|
||||
assert cli.main(["direct", "--top-n", "2", "--lead-s", "1.0"]) == 0
|
||||
printed = json.loads(capsys.readouterr().out)
|
||||
assert printed["version"] == 1 and len(printed["keyframes"]) > 0
|
||||
@ -108,226 +108,3 @@ def test_quat_to_mat_matches_scipy_oracle():
|
||||
np.testing.assert_allclose(
|
||||
quat_to_mat(q_wxyz), Rotation.from_quat(q_xyzw).as_matrix(), atol=1e-9
|
||||
)
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Lane B (spec M2 + M8) tests — below the frozen block. These exercise the
|
||||
# stubs foundation left: slerp_pose, ray_from_pixel, triangulate_rays,
|
||||
# nearest_point_on_ray. They do NOT touch the frozen colmap_to_threejs cases above.
|
||||
# ===========================================================================
|
||||
from scipy.spatial.transform import Slerp # noqa: E402
|
||||
|
||||
from festival4d.geometry import ( # noqa: E402
|
||||
colmap_to_threejs,
|
||||
nearest_point_on_ray,
|
||||
ray_from_pixel,
|
||||
slerp_pose,
|
||||
triangulate_rays,
|
||||
)
|
||||
|
||||
|
||||
def _wxyz_to_xyzw(q):
|
||||
q = np.asarray(q, dtype=float)
|
||||
return np.array([q[1], q[2], q[3], q[0]])
|
||||
|
||||
|
||||
def _rand_unit_quat(rng):
|
||||
q = rng.normal(size=4)
|
||||
return q / np.linalg.norm(q)
|
||||
|
||||
|
||||
# --- slerp_pose ------------------------------------------------------------
|
||||
def test_slerp_pose_endpoints():
|
||||
q0 = _rand_unit_quat(np.random.default_rng(1))
|
||||
q1 = _rand_unit_quat(np.random.default_rng(2))
|
||||
t0 = np.array([1.0, -2.0, 3.0])
|
||||
t1 = np.array([-4.0, 5.0, 6.0])
|
||||
|
||||
q_a, t_a = slerp_pose(q0, t0, q1, t1, 0.0)
|
||||
q_b, t_b = slerp_pose(q0, t0, q1, t1, 1.0)
|
||||
# endpoints recover the endpoint *rotations* (quaternion up to sign) and translations
|
||||
np.testing.assert_allclose(quat_to_mat(q_a), quat_to_mat(q0), atol=1e-12)
|
||||
np.testing.assert_allclose(quat_to_mat(q_b), quat_to_mat(q1), atol=1e-12)
|
||||
np.testing.assert_allclose(t_a, t0, atol=1e-12)
|
||||
np.testing.assert_allclose(t_b, t1, atol=1e-12)
|
||||
|
||||
|
||||
def test_slerp_pose_matches_scipy_oracle():
|
||||
rng = np.random.default_rng(20260716)
|
||||
for _ in range(30):
|
||||
q0 = _rand_unit_quat(rng)
|
||||
q1 = _rand_unit_quat(rng)
|
||||
oracle = Slerp([0.0, 1.0], Rotation.from_quat(
|
||||
[_wxyz_to_xyzw(q0), _wxyz_to_xyzw(q1)]))
|
||||
for alpha in (0.1, 0.25, 0.5, 0.73, 0.9):
|
||||
q, _ = slerp_pose(q0, [0, 0, 0], q1, [0, 0, 0], alpha)
|
||||
np.testing.assert_allclose(
|
||||
quat_to_mat(q), oracle(alpha).as_matrix(), atol=1e-9)
|
||||
|
||||
|
||||
def test_slerp_pose_translation_is_linear():
|
||||
q = np.array([1.0, 0.0, 0.0, 0.0])
|
||||
t0 = np.array([0.0, 0.0, 0.0])
|
||||
t1 = np.array([10.0, -4.0, 2.0])
|
||||
for alpha in (0.0, 0.3, 0.5, 1.0):
|
||||
_, t = slerp_pose(q, t0, q, t1, alpha)
|
||||
np.testing.assert_allclose(t, (1 - alpha) * t0 + alpha * t1, atol=1e-12)
|
||||
|
||||
|
||||
def test_slerp_pose_double_cover_takes_short_arc():
|
||||
"""q1 and -q1 are the same rotation; slerp must yield the same (short-arc) result."""
|
||||
rng = np.random.default_rng(99)
|
||||
q0 = _rand_unit_quat(rng)
|
||||
q1 = _rand_unit_quat(rng)
|
||||
for alpha in (0.2, 0.5, 0.8):
|
||||
qa, _ = slerp_pose(q0, [0, 0, 0], q1, [0, 0, 0], alpha)
|
||||
qb, _ = slerp_pose(q0, [0, 0, 0], -q1, [0, 0, 0], alpha)
|
||||
np.testing.assert_allclose(quat_to_mat(qa), quat_to_mat(qb), atol=1e-12)
|
||||
|
||||
|
||||
def test_slerp_pose_midpoint_is_half_angle():
|
||||
"""A 180-degree-ish pair: the midpoint rotation angle is half the endpoint angle."""
|
||||
q0 = np.array([1.0, 0.0, 0.0, 0.0]) # identity
|
||||
ang = np.radians(100.0)
|
||||
q1 = np.array([np.cos(ang / 2), 0.0, np.sin(ang / 2), 0.0]) # yaw 100 deg about Y
|
||||
qm, _ = slerp_pose(q0, [0, 0, 0], q1, [0, 0, 0], 0.5)
|
||||
# rotation angle of qm relative to identity should be ~50 deg
|
||||
angle = 2.0 * np.arccos(min(1.0, abs(qm[0])))
|
||||
assert abs(np.degrees(angle) - 50.0) < 1e-6
|
||||
|
||||
|
||||
# --- ray_from_pixel --------------------------------------------------------
|
||||
def test_ray_from_pixel_origin_is_camera_center():
|
||||
"""The ray origin equals the Three.js camera center from the frozen contract."""
|
||||
rng = np.random.default_rng(5)
|
||||
for _ in range(20):
|
||||
q = _rand_unit_quat(rng)
|
||||
t = rng.uniform(-5, 5, size=3)
|
||||
position, _ = colmap_to_threejs(q, t)
|
||||
origin, _ = ray_from_pixel(q, t, 600, 600, 320, 180, 320, 180)
|
||||
np.testing.assert_allclose(origin, position, atol=1e-9)
|
||||
|
||||
|
||||
def test_ray_from_pixel_center_is_forward_axis():
|
||||
"""The center pixel unprojects along the camera forward (+z) axis, in world coords."""
|
||||
q = np.array([1.0, 0.0, 0.0, 0.0]) # identity world->cam
|
||||
t = np.array([0.0, 0.0, -10.0]) # camera center at (0,0,10)
|
||||
origin, direction = ray_from_pixel(q, t, 500, 500, 320, 180, 320, 180)
|
||||
np.testing.assert_allclose(origin, [0.0, 0.0, 10.0], atol=1e-9)
|
||||
np.testing.assert_allclose(direction, [0.0, 0.0, 1.0], atol=1e-9)
|
||||
|
||||
|
||||
def _project_colmap(q, t, fx, fy, cx, cy, X):
|
||||
"""Forward COLMAP pinhole projection of a world point X -> pixel (px, py)."""
|
||||
R = quat_to_mat(q)
|
||||
xc = R @ np.asarray(X, float) + np.asarray(t, float)
|
||||
return fx * xc[0] / xc[2] + cx, fy * xc[1] / xc[2] + cy, xc[2]
|
||||
|
||||
|
||||
def test_ray_from_pixel_is_projection_inverse():
|
||||
"""A world point in front of the camera lies exactly on the ray through its pixel."""
|
||||
rng = np.random.default_rng(7)
|
||||
for _ in range(50):
|
||||
q = _rand_unit_quat(rng)
|
||||
t = rng.uniform(-3, 3, size=3)
|
||||
fx = fy = rng.uniform(400, 800)
|
||||
cx, cy = 320.0, 180.0
|
||||
C = -quat_to_mat(q).T @ t # camera center
|
||||
forward = quat_to_mat(q).T @ np.array([0, 0, 1.0])
|
||||
X = C + rng.uniform(2, 8) * forward + rng.uniform(-1, 1, size=3) # in front
|
||||
px, py, zc = _project_colmap(q, t, fx, fy, cx, cy, X)
|
||||
if zc <= 0.1:
|
||||
continue
|
||||
origin, direction = ray_from_pixel(q, t, fx, fy, cx, cy, px, py)
|
||||
to_X = X - origin
|
||||
# X - origin must be parallel to direction and in front (positive projection)
|
||||
cross = np.cross(to_X, direction)
|
||||
assert np.linalg.norm(cross) < 1e-6 * (1 + np.linalg.norm(to_X))
|
||||
assert np.dot(to_X, direction) > 0
|
||||
|
||||
|
||||
# --- triangulate_rays ------------------------------------------------------
|
||||
def test_triangulate_rays_intersecting():
|
||||
P = np.array([1.0, 2.0, 3.0])
|
||||
oa = np.array([0.0, 0.0, 0.0])
|
||||
ob = np.array([4.0, 0.0, 0.0])
|
||||
point, gap = triangulate_rays(oa, P - oa, ob, P - ob)
|
||||
np.testing.assert_allclose(point, P, atol=1e-9)
|
||||
assert gap < 1e-9
|
||||
|
||||
|
||||
def test_triangulate_rays_skew_known_geometry():
|
||||
"""Ray A along +x at z=0; ray B along +y at z=1. Closest points (0,0,0),(0,0,1)."""
|
||||
oa = np.array([0.0, 0.0, 0.0]); da = np.array([1.0, 0.0, 0.0])
|
||||
ob = np.array([0.0, 0.0, 1.0]); db = np.array([0.0, 1.0, 0.0])
|
||||
point, gap = triangulate_rays(oa, da, ob, db)
|
||||
np.testing.assert_allclose(point, [0.0, 0.0, 0.5], atol=1e-9)
|
||||
assert abs(gap - 1.0) < 1e-9
|
||||
|
||||
|
||||
def test_triangulate_rays_parallel_is_safe():
|
||||
"""Parallel rays must not divide-by-zero; gap ~ their separation."""
|
||||
oa = np.array([0.0, 0.0, 0.0]); da = np.array([1.0, 0.0, 0.0])
|
||||
ob = np.array([0.0, 2.0, 0.0]); db = np.array([1.0, 0.0, 0.0])
|
||||
point, gap = triangulate_rays(oa, da, ob, db)
|
||||
assert np.all(np.isfinite(point))
|
||||
assert abs(gap - 2.0) < 1e-6
|
||||
|
||||
|
||||
def test_triangulate_rays_from_two_cameras_recovers_point():
|
||||
"""Two cameras looking at a world point; triangulating their pixel rays recovers it."""
|
||||
rng = np.random.default_rng(123)
|
||||
P = np.array([0.5, 1.0, -0.5])
|
||||
fx = fy = 600.0; cx, cy = 320.0, 180.0
|
||||
rays = []
|
||||
for center in ([6.0, 2.0, 6.0], [-6.0, 2.5, 6.0]):
|
||||
C = np.array(center, float)
|
||||
z = P - C; z /= np.linalg.norm(z)
|
||||
up = np.array([0.0, 1.0, 0.0])
|
||||
up = up - np.dot(up, z) * z; up /= np.linalg.norm(up)
|
||||
x = np.cross(z, up); y = -up
|
||||
R_c2w = np.column_stack([x, y, z]); R = R_c2w.T
|
||||
t = -R @ C
|
||||
q = mat_to_quat(R)
|
||||
px, py, _ = _project_colmap(q, t, fx, fy, cx, cy, P)
|
||||
rays.append(ray_from_pixel(q, t, fx, fy, cx, cy, px, py))
|
||||
point, gap = triangulate_rays(rays[0][0], rays[0][1], rays[1][0], rays[1][1])
|
||||
assert np.linalg.norm(point - P) < 0.2 # spec M8 acceptance tolerance
|
||||
assert gap < 1e-6
|
||||
|
||||
|
||||
# --- nearest_point_on_ray --------------------------------------------------
|
||||
def test_nearest_point_on_ray_hits_within_radius():
|
||||
o = np.array([0.0, 0.0, 0.0]); d = np.array([0.0, 0.0, 1.0])
|
||||
points = np.array([
|
||||
[0.1, 0.0, 5.0], # perp 0.1, in front -> candidate
|
||||
[0.05, 0.0, 3.0], # perp 0.05, in front -> closest
|
||||
[2.0, 0.0, 5.0], # perp 2.0 -> outside radius
|
||||
])
|
||||
got = nearest_point_on_ray(o, d, points, radius=0.3)
|
||||
np.testing.assert_allclose(got, [0.05, 0.0, 3.0], atol=1e-12)
|
||||
|
||||
|
||||
def test_nearest_point_on_ray_excludes_outside_and_behind():
|
||||
o = np.array([0.0, 0.0, 0.0]); d = np.array([0.0, 0.0, 1.0])
|
||||
behind = np.array([[0.05, 0.0, -3.0]]) # in cylinder but behind origin
|
||||
outside = np.array([[1.0, 0.0, 3.0]]) # in front but outside radius
|
||||
assert nearest_point_on_ray(o, d, behind, radius=0.3) is None
|
||||
assert nearest_point_on_ray(o, d, outside, radius=0.3) is None
|
||||
assert nearest_point_on_ray(o, d, np.zeros((0, 3)), radius=0.3) is None
|
||||
|
||||
|
||||
def test_nearest_point_on_ray_synthetic_stage_corner():
|
||||
"""M8 single-view fallback: a ray toward a stage corner selects that corner point."""
|
||||
from festival4d import synthetic
|
||||
|
||||
corner = np.array(synthetic.STAGE_CORNERS["Stage FL"], float)
|
||||
points, _ = synthetic.generate_point_cloud()
|
||||
# a camera looking straight at the corner; ray through the image center hits it
|
||||
q, t = synthetic.look_at_colmap((5.0, 3.0, 9.0), corner)
|
||||
intr = synthetic.intrinsics(640, 360)
|
||||
origin, direction = ray_from_pixel(
|
||||
q, t, intr["fx"], intr["fy"], intr["cx"], intr["cy"], intr["cx"], intr["cy"])
|
||||
got = nearest_point_on_ray(origin, direction, points.astype(float), radius=0.3)
|
||||
assert got is not None
|
||||
np.testing.assert_allclose(got, corner, atol=1e-4)
|
||||
|
||||
@ -1,93 +0,0 @@
|
||||
"""Ingest tests (spec M1, lane A).
|
||||
|
||||
Pure unit tests for the path/fps helpers, plus ffmpeg-gated tests that probe a real
|
||||
rendered clip, extract audio, and run the full idempotent ``run_ingest``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import shutil
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from festival4d import config, db, ingest, synthetic
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# pure helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_audio_wav_path_convention():
|
||||
assert ingest.audio_wav_path(7) == config.AUDIO_DIR / "7.wav"
|
||||
|
||||
|
||||
def test_parse_fps():
|
||||
assert ingest._parse_fps("30/1") == 30.0
|
||||
assert abs(ingest._parse_fps("30000/1001") - 29.97) < 0.01
|
||||
assert ingest._parse_fps("25") == 25.0
|
||||
assert ingest._parse_fps(None) is None
|
||||
assert ingest._parse_fps("0/0") is None
|
||||
assert ingest._parse_fps("N/A") is None
|
||||
|
||||
|
||||
def test_run_ingest_no_files_is_empty(tmp_path, monkeypatch):
|
||||
"""An empty raw dir yields an empty summary, not a crash."""
|
||||
monkeypatch.setattr(config, "RAW_DIR", tmp_path / "empty_raw")
|
||||
(tmp_path / "empty_raw").mkdir()
|
||||
db.init_engine()
|
||||
db.reset_db()
|
||||
assert ingest.run_ingest() == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ffmpeg-gated
|
||||
# ---------------------------------------------------------------------------
|
||||
def _render_clip(tmp_path, duration_s=2.0, cam_index=0):
|
||||
sr = config.AUDIO_SAMPLE_RATE
|
||||
master, _ = synthetic.synth_master_audio(sr)
|
||||
clip = synthetic.slice_for_offset(master, sr, 0.0, duration_s)
|
||||
wav = tmp_path / "src.wav"
|
||||
synthetic.write_wav(wav, clip, sr)
|
||||
mp4 = tmp_path / f"cam{cam_index}.mp4"
|
||||
synthetic.render_video(mp4, wav, cam_index, 640, 360, 30, duration_s)
|
||||
return mp4
|
||||
|
||||
|
||||
@pytest.mark.skipif(shutil.which("ffmpeg") is None or shutil.which("ffprobe") is None,
|
||||
reason="ffmpeg/ffprobe required")
|
||||
def test_probe_video(tmp_path):
|
||||
mp4 = _render_clip(tmp_path, duration_s=2.0)
|
||||
info = ingest.probe_video(mp4)
|
||||
assert info["width"] == 640 and info["height"] == 360
|
||||
assert abs(info["fps"] - 30.0) < 1.0
|
||||
assert 1.5 < info["duration_s"] < 2.6
|
||||
|
||||
|
||||
@pytest.mark.skipif(shutil.which("ffmpeg") is None or shutil.which("ffprobe") is None,
|
||||
reason="ffmpeg/ffprobe required")
|
||||
def test_extract_audio(tmp_path):
|
||||
mp4 = _render_clip(tmp_path, duration_s=2.0)
|
||||
out = ingest.extract_audio(mp4, tmp_path / "out.wav", sample_rate=16_000)
|
||||
assert out.exists()
|
||||
samples, sr = synthetic.read_wav(out)
|
||||
assert sr == 16_000
|
||||
assert samples.ndim == 1 # mono
|
||||
assert abs(len(samples) / sr - 2.0) < 0.3 # ~2 s
|
||||
|
||||
|
||||
@pytest.mark.skipif(shutil.which("ffmpeg") is None or shutil.which("ffprobe") is None,
|
||||
reason="ffmpeg/ffprobe required")
|
||||
def test_run_ingest_registers_and_is_idempotent():
|
||||
synthetic.build(duration_s=2.0, run_ffmpeg=True) # into the temp data dir (conftest)
|
||||
summary = ingest.run_ingest()
|
||||
assert len(summary) == 3
|
||||
for s in summary:
|
||||
assert ingest.audio_wav_path(s["video_id"]).exists()
|
||||
samples, sr = synthetic.read_wav(ingest.audio_wav_path(s["video_id"]))
|
||||
assert sr == 16_000 and samples.ndim == 1
|
||||
|
||||
# Second run reuses the existing rows (no unique-constraint crash) and re-extracts.
|
||||
again = ingest.run_ingest()
|
||||
assert len(again) == 3
|
||||
assert all(s["reused_db_row"] for s in again)
|
||||
assert len(db.get_videos()) == 3 # no duplicate rows
|
||||
@ -1,198 +0,0 @@
|
||||
"""Deepened coverage of the M16 saved camera-path routes (lane G).
|
||||
|
||||
foundation2's basic shape tests live in test_phase5_api.py; this module hardens the
|
||||
contract: deep-equal roundtrips (including float fidelity and large keyframe lists),
|
||||
multiple coexisting paths, name handling, 404s, and the 422 validation surface of the
|
||||
frozen camPath JSON (plan/20-phase5.md contract #2).
|
||||
|
||||
Pure DB/API surface — no synthetic fixture or ffmpeg needed. The test session's data
|
||||
dir is a temp dir (conftest), and every test cleans up the rows it created so later
|
||||
modules (test_phase5_api.py runs after this one alphabetically) still see an empty table.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
def _campath(keyframes):
|
||||
return {"version": 1, "keyframes": keyframes}
|
||||
|
||||
|
||||
def _kf(t, pos=(0.0, 1.0, 5.0), quat=(0.0, 0.0, 0.0, 1.0), fov=50.0):
|
||||
return {"t_global": t, "pos": list(pos), "quat": list(quat), "fov": fov}
|
||||
|
||||
|
||||
SIMPLE = _campath([_kf(1.0), _kf(4.0, pos=(2, 1, 4), quat=(0, 0.3826834, 0, 0.9238795), fov=42)])
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def client():
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from festival4d import api
|
||||
|
||||
with TestClient(api.app) as c:
|
||||
yield c
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean_paths(client):
|
||||
"""Leave the paths table exactly as found (empty) — later modules assert on it."""
|
||||
yield
|
||||
for p in client.get("/api/paths").json():
|
||||
client.delete(f"/api/paths/{p['id']}")
|
||||
|
||||
|
||||
def _post(client, name, obj_or_text):
|
||||
text = obj_or_text if isinstance(obj_or_text, str) else json.dumps(obj_or_text)
|
||||
return client.post("/api/paths", json={"name": name, "json": text})
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Roundtrip fidelity
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_roundtrip_preserves_float_precision(client):
|
||||
# Awkward floats must survive POST -> store-as-text -> GET deep-equal (the live UI
|
||||
# depends on "reload the page, load the path, land on the exact same keyframes").
|
||||
path = _campath([
|
||||
_kf(0.123456789012345, pos=(-1.5e-7, 2.0000000001, math.pi), fov=49.99999999),
|
||||
_kf(1e3 + 1e-9, quat=(0.5, -0.5, 0.5, -0.5), fov=1e2 / 3),
|
||||
])
|
||||
created = _post(client, "precise", path).json()
|
||||
assert created["json"] == path
|
||||
assert client.get(f"/api/paths/{created['id']}").json()["json"] == path
|
||||
|
||||
|
||||
def test_roundtrip_large_keyframe_list(client):
|
||||
# A long recorded flythrough: hundreds of keyframes with non-round values.
|
||||
path = _campath([
|
||||
_kf(i * 0.0333 + 0.0001 * i * i,
|
||||
pos=(math.sin(i), 1 + 0.01 * i, math.cos(i)),
|
||||
quat=(0.0, math.sin(i / 500), 0.0, math.cos(i / 500)),
|
||||
fov=30 + (i % 40) * 0.77)
|
||||
for i in range(500)
|
||||
])
|
||||
created = _post(client, "marathon", path).json()
|
||||
got = client.get(f"/api/paths/{created['id']}").json()["json"]
|
||||
assert got == path
|
||||
assert len(got["keyframes"]) == 500
|
||||
|
||||
|
||||
def test_empty_keyframes_is_a_valid_path(client):
|
||||
# The director degrades to {"version":1,"keyframes":[]} — that must be saveable.
|
||||
resp = _post(client, "empty", _campath([]))
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["json"] == _campath([])
|
||||
|
||||
|
||||
def test_extra_keys_in_campath_survive_roundtrip(client):
|
||||
# Validation is a floor, not a strainer: unknown fields (e.g. the director's "note")
|
||||
# ride along verbatim because the JSON is stored as text.
|
||||
path = {**_campath([_kf(2.0)]), "note": "auto-director v1"}
|
||||
created = _post(client, "annotated", path).json()
|
||||
assert created["json"] == path
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Listing / multiple paths / name handling
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_multiple_paths_coexist_and_list_in_id_order(client):
|
||||
ids = [_post(client, name, SIMPLE).json()["id"] for name in ("a", "b", "c")]
|
||||
listing = client.get("/api/paths").json()
|
||||
assert [p["id"] for p in listing] == sorted(ids)
|
||||
assert [p["name"] for p in listing] == ["a", "b", "c"]
|
||||
# Listing is the summary shape only — the (potentially large) JSON stays off it.
|
||||
assert all(set(p) == {"id", "name", "created_at"} for p in listing)
|
||||
|
||||
# Deleting the middle one leaves the neighbors intact and loadable.
|
||||
assert client.delete(f"/api/paths/{ids[1]}").status_code == 200
|
||||
left = client.get("/api/paths").json()
|
||||
assert [p["id"] for p in left] == [ids[0], ids[2]]
|
||||
assert client.get(f"/api/paths/{ids[2]}").json()["json"] == SIMPLE
|
||||
|
||||
|
||||
def test_duplicate_names_are_allowed_distinct_rows(client):
|
||||
a = _post(client, "same name", SIMPLE).json()
|
||||
b = _post(client, "same name", SIMPLE).json()
|
||||
assert a["id"] != b["id"]
|
||||
assert [p["name"] for p in client.get("/api/paths").json()] == ["same name", "same name"]
|
||||
|
||||
|
||||
def test_names_preserved_verbatim(client):
|
||||
# Unicode, emoji, quotes, leading/trailing whitespace — stored and returned as-is.
|
||||
for name in ("🎥 drop №5 — “final”", " padded ", "a" * 300):
|
||||
created = _post(client, name, SIMPLE).json()
|
||||
assert created["name"] == name
|
||||
assert client.get(f"/api/paths/{created['id']}").json()["name"] == name
|
||||
|
||||
|
||||
def test_created_at_is_an_iso_timestamp(client):
|
||||
from datetime import datetime
|
||||
|
||||
created = _post(client, "stamped", SIMPLE).json()
|
||||
# Parses as ISO-8601; the frontend shows it as a tooltip.
|
||||
assert isinstance(datetime.fromisoformat(created["created_at"]), datetime)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 404s
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_get_and_delete_unknown_id_404(client):
|
||||
assert client.get("/api/paths/424242").status_code == 404
|
||||
assert client.delete("/api/paths/424242").status_code == 404
|
||||
|
||||
|
||||
def test_deleted_path_stays_deleted(client):
|
||||
pid = _post(client, "doomed", SIMPLE).json()["id"]
|
||||
assert client.delete(f"/api/paths/{pid}").json() == {"deleted": pid}
|
||||
assert client.get(f"/api/paths/{pid}").status_code == 404
|
||||
assert client.delete(f"/api/paths/{pid}").status_code == 404
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Validation (422) — the frozen camPath JSON contract, edge by edge
|
||||
# ---------------------------------------------------------------------------
|
||||
@pytest.mark.parametrize(
|
||||
"body",
|
||||
[
|
||||
"", # empty text
|
||||
"not json at all {",
|
||||
json.dumps(None),
|
||||
json.dumps([_kf(0.0)]), # top-level array, not object
|
||||
json.dumps("a string"),
|
||||
json.dumps({"keyframes": []}), # version missing
|
||||
json.dumps({"version": "1", "keyframes": []}), # version wrong type
|
||||
json.dumps({"version": 2, "keyframes": []}), # wrong version
|
||||
json.dumps({"version": 1}), # keyframes missing
|
||||
json.dumps({"version": 1, "keyframes": {}}), # keyframes not a list
|
||||
json.dumps(_campath(["not a dict"])),
|
||||
json.dumps(_campath([{k: v for k, v in _kf(0.0).items() if k != "t_global"}])),
|
||||
json.dumps(_campath([{k: v for k, v in _kf(0.0).items() if k != "fov"}])),
|
||||
json.dumps(_campath([{**_kf(0.0), "t_global": "0.0"}])), # time as string
|
||||
json.dumps(_campath([{**_kf(0.0), "pos": [0.0, 1.0]}])), # pos wrong arity
|
||||
json.dumps(_campath([{**_kf(0.0), "quat": [0, 0, 0, 1, 0]}])), # quat wrong arity
|
||||
json.dumps(_campath([{**_kf(0.0), "pos": 5}])), # pos not a list
|
||||
json.dumps(_campath([_kf(0.0), {**_kf(1.0), "quat": None}])), # one bad among good
|
||||
],
|
||||
)
|
||||
def test_invalid_campath_json_422(client, body):
|
||||
assert client.post("/api/paths", json={"name": "bad", "json": body}).status_code == 422
|
||||
|
||||
|
||||
def test_validation_failure_creates_no_row(client):
|
||||
assert client.get("/api/paths").json() == []
|
||||
assert _post(client, "bad", "{").status_code == 422
|
||||
assert client.get("/api/paths").json() == []
|
||||
|
||||
|
||||
def test_missing_request_fields_422(client):
|
||||
assert client.post("/api/paths", json={"json": json.dumps(SIMPLE)}).status_code == 422
|
||||
assert client.post("/api/paths", json={"name": "x"}).status_code == 422
|
||||
# "json" is the wire key (contract #3) — the internal alias must not be accepted.
|
||||
assert client.post(
|
||||
"/api/paths", json={"name": "x", "path_json": json.dumps(SIMPLE)}
|
||||
).status_code == 422
|
||||
@ -1,106 +0,0 @@
|
||||
"""Phase-5 foundation2 contract tests: the new route shapes lanes E/F/G depend on
|
||||
(plan/20-phase5.md "Contracts"). Lane G deepens paths coverage in test_paths.py;
|
||||
lane E owns the real beats/director behavior tests.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shutil
|
||||
|
||||
import pytest
|
||||
|
||||
pytestmark = pytest.mark.skipif(
|
||||
shutil.which("ffmpeg") is None or shutil.which("ffprobe") is None,
|
||||
reason="ffmpeg/ffprobe required to build the API test fixture",
|
||||
)
|
||||
|
||||
CAMPATH = {
|
||||
"version": 1,
|
||||
"keyframes": [
|
||||
{"t_global": 1.0, "pos": [0, 1, 5], "quat": [0, 0, 0, 1], "fov": 50},
|
||||
{"t_global": 4.0, "pos": [2, 1, 4], "quat": [0, 0.3826834, 0, 0.9238795], "fov": 42},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def client():
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from festival4d import synthetic
|
||||
|
||||
synthetic.build(duration_s=1.0, run_ffmpeg=True) # into the temp data dir (conftest)
|
||||
|
||||
from festival4d import api
|
||||
|
||||
with TestClient(api.app) as c:
|
||||
yield c
|
||||
|
||||
|
||||
def test_beats_404_then_serves_file(client):
|
||||
from festival4d import config
|
||||
|
||||
config.BEATS_JSON.unlink(missing_ok=True)
|
||||
assert client.get("/api/beats").status_code == 404
|
||||
body = {"tempo_bpm": 120.0, "beats_s": [0.5, 1.0], "onsets": [], "generated_by": "festival4d.audio_features"}
|
||||
config.BEATS_JSON.parent.mkdir(parents=True, exist_ok=True)
|
||||
config.BEATS_JSON.write_text(json.dumps(body))
|
||||
try:
|
||||
assert client.get("/api/beats").json() == body
|
||||
finally:
|
||||
config.BEATS_JSON.unlink()
|
||||
|
||||
|
||||
def test_director_returns_valid_campath(client):
|
||||
# CR-5: lane E's M11 landed, so on the synthetic fixture (events + poses present) the
|
||||
# response is a REAL path — assert the frozen camPath shape (holds stubbed or implemented).
|
||||
# The empty+note degradation (no events / no poses) is covered in test_director.py.
|
||||
data = client.post("/api/director", json={}).json()
|
||||
assert data["version"] == 1
|
||||
assert isinstance(data["keyframes"], list)
|
||||
for kf in data["keyframes"]:
|
||||
assert set(kf) == {"t_global", "pos", "quat", "fov"}
|
||||
assert client.post("/api/director", json={"top_n": 3, "lead_s": 1.0}).status_code == 200
|
||||
|
||||
|
||||
def test_paths_crud_roundtrip(client):
|
||||
assert client.get("/api/paths").json() == []
|
||||
created = client.post(
|
||||
"/api/paths", json={"name": "sweep", "json": json.dumps(CAMPATH)}
|
||||
).json()
|
||||
assert set(created) == {"id", "name", "created_at", "json"}
|
||||
assert created["json"] == CAMPATH # deep-equal roundtrip
|
||||
pid = created["id"]
|
||||
|
||||
listing = client.get("/api/paths").json()
|
||||
assert [set(p) for p in listing] == [{"id", "name", "created_at"}]
|
||||
assert client.get(f"/api/paths/{pid}").json()["json"] == CAMPATH
|
||||
|
||||
assert client.delete(f"/api/paths/{pid}").json() == {"deleted": pid}
|
||||
assert client.get(f"/api/paths/{pid}").status_code == 404
|
||||
assert client.delete(f"/api/paths/{pid}").status_code == 404
|
||||
|
||||
|
||||
def test_paths_post_validates_campath(client):
|
||||
post = lambda body: client.post("/api/paths", json={"name": "bad", "json": body}).status_code
|
||||
assert post("not json at all {") == 422
|
||||
assert post(json.dumps({"version": 2, "keyframes": []})) == 422
|
||||
assert post(json.dumps({"version": 1})) == 422
|
||||
assert post(json.dumps({"version": 1, "keyframes": [{"t_global": 0}]})) == 422
|
||||
|
||||
|
||||
def test_anchor_patch_label_and_color(client):
|
||||
a = client.post("/api/anchors", json={"label": "F", "x": 0, "y": 0, "z": 0}).json()
|
||||
patched = client.patch(f"/api/anchors/{a['id']}", json={"label": "Fred", "color": "#4dd0ff"}).json()
|
||||
assert patched["label"] == "Fred" and patched["color"] == "#4dd0ff"
|
||||
assert (patched["x"], patched["y"], patched["z"]) == (0, 0, 0) # position untouched
|
||||
# partial patch: color-only leaves the label alone
|
||||
assert client.patch(f"/api/anchors/{a['id']}", json={"color": "#fff"}).json()["label"] == "Fred"
|
||||
assert client.patch("/api/anchors/99999", json={"label": "x"}).status_code == 404
|
||||
|
||||
|
||||
# foundation2's test_cli_dispatches_stubs_gracefully asserted `cli.main(<cmd>) == 2` while the
|
||||
# phase-5 subcommands were stubs. Every one has now landed, so the test retired (CR-4 lane G for
|
||||
# `capsule`, CR-5 lane E for `features`/`direct`): the real CLI dispatches are covered in
|
||||
# test_capsule.py, test_audio_features.py, and test_director.py respectively.
|
||||
@ -1,186 +0,0 @@
|
||||
"""M8 annotation -> 3D resolution tests (integration phase).
|
||||
|
||||
The headline acceptance (spec M8): annotating the *same* stage corner from two videos
|
||||
produces an anchor within 0.2 scene units of the true corner. We build the synthetic fixture
|
||||
without ffmpeg (``run_ffmpeg=False`` populates the DB + point cloud only), forward-project a
|
||||
known corner into each camera to get faithful bbox centers, then resolve and check the error.
|
||||
|
||||
Also covers: the single-view nearest-point fallback, the near-parallel/gap rejection path,
|
||||
and the ``update_event`` correction helper.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from festival4d import config, db, resolve, synthetic
|
||||
from festival4d.geometry import quat_to_mat
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def fixture():
|
||||
# No ffmpeg: this writes videos/poses/anchors/events into the DB and points.ply to WORK_DIR.
|
||||
synthetic.build(run_ffmpeg=False)
|
||||
return synthetic.STAGE_CORNERS
|
||||
|
||||
|
||||
def _project(video, pose, world_pt):
|
||||
"""Forward-project a world point into a video, returning (px, py, z_cam).
|
||||
|
||||
Mirrors the COLMAP pinhole the fixture uses: x_cam = R@X + t; px = fx*x/z + cx (y-down).
|
||||
"""
|
||||
q = [pose.qw, pose.qx, pose.qy, pose.qz]
|
||||
t = np.array([pose.tx, pose.ty, pose.tz])
|
||||
R = quat_to_mat(q)
|
||||
x_cam = R @ np.asarray(world_pt, dtype=float) + t
|
||||
z = x_cam[2]
|
||||
px = pose.fx * x_cam[0] / z + pose.cx
|
||||
py = pose.fy * x_cam[1] / z + pose.cy
|
||||
return px, py, z
|
||||
|
||||
|
||||
def _bbox_center(px, py, W, H, half=0.02):
|
||||
"""A small normalized bbox centered on the projected pixel (clamped to [0,1])."""
|
||||
cx, cy = px / W, py / H
|
||||
return [
|
||||
max(0.0, cx - half), max(0.0, cy - half),
|
||||
min(1.0, cx + half), min(1.0, cy + half),
|
||||
]
|
||||
|
||||
|
||||
def _views_seeing(corner):
|
||||
"""Videos whose first pose projects `corner` in front of the camera and inside the frame."""
|
||||
seen = []
|
||||
for v in db.get_videos():
|
||||
poses = db.get_poses(v.id)
|
||||
if not poses:
|
||||
continue
|
||||
p0 = poses[0]
|
||||
px, py, z = _project(v, p0, corner)
|
||||
if z > 0 and 0 <= px < v.width and 0 <= py < v.height:
|
||||
seen.append((v, p0, px, py))
|
||||
return seen
|
||||
|
||||
|
||||
def test_two_view_triangulation_recovers_corner(fixture):
|
||||
corners = fixture
|
||||
# Pick a corner visible in at least two cameras.
|
||||
chosen = None
|
||||
for label, world in corners.items():
|
||||
views = _views_seeing(world)
|
||||
if len(views) >= 2:
|
||||
chosen = (label, np.asarray(world, dtype=float), views)
|
||||
break
|
||||
assert chosen is not None, "no stage corner is visible in >=2 synthetic cameras"
|
||||
label, world, views = chosen
|
||||
|
||||
event = db.add_event(t_global_s=5.0, event_type="artist_moment", source="ai",
|
||||
description=f"located {label}")
|
||||
|
||||
# Annotate the corner from the first two cameras that see it, then resolve the second one
|
||||
# (which triangulates against the first).
|
||||
(v_a, pose_a, pxa, pya), (v_b, pose_b, pxb, pyb) = views[0], views[1]
|
||||
bbox_a = _bbox_center(pxa, pya, v_a.width, v_a.height)
|
||||
bbox_b = _bbox_center(pxb, pyb, v_b.width, v_b.height)
|
||||
|
||||
db.add_annotation(v_a.id, pose_a.t_video_s, *bbox_a, event_id=event.id)
|
||||
res = resolve.resolve_annotation(v_b.id, pose_b.t_video_s, bbox_b, event_id=event.id)
|
||||
|
||||
assert res.method == "triangulated", f"expected triangulation, got {res.method}"
|
||||
err = float(np.linalg.norm(np.asarray(res.point) - world))
|
||||
assert err < 0.2, f"triangulated {label} off by {err:.3f} units (>0.2)"
|
||||
# Sanity: the mutual-approach gap is tight for exact projections.
|
||||
assert res.gap is not None and res.gap < 0.2
|
||||
|
||||
|
||||
def test_single_view_falls_back_to_point_cloud(fixture):
|
||||
corners = fixture
|
||||
world = np.asarray(corners["Stage FL"], dtype=float)
|
||||
views = _views_seeing(world)
|
||||
assert views, "Stage FL should be visible in at least one camera"
|
||||
v, pose, px, py = views[0]
|
||||
bbox = _bbox_center(px, py, v.width, v.height)
|
||||
|
||||
# No event_id => no other view to triangulate => nearest point-cloud point (corner is in
|
||||
# the cloud) or centroid-depth ray. Either way a point comes back.
|
||||
res = resolve.resolve_annotation(v.id, pose.t_video_s, bbox, event_id=None)
|
||||
assert res.method in {"nearest_point", "ray_depth"}
|
||||
assert res.point is not None and len(res.point) == 3
|
||||
if res.method == "nearest_point":
|
||||
# The exact corner is seeded into the cloud, so the nearest hit should be very close.
|
||||
assert float(np.linalg.norm(np.asarray(res.point) - world)) < resolve.NEAREST_RADIUS
|
||||
|
||||
|
||||
def test_update_event_sets_user_source(fixture):
|
||||
ev = db.add_event(t_global_s=9.0, event_type="candidate", source="audio_auto",
|
||||
description="unlabeled")
|
||||
updated = db.update_event(ev.id, event_type="pyro", source="user")
|
||||
assert updated.event_type == "pyro"
|
||||
assert updated.source == "user"
|
||||
# Unspecified fields are preserved.
|
||||
assert updated.description == "unlabeled"
|
||||
|
||||
|
||||
def test_missing_video_raises(fixture):
|
||||
with pytest.raises(KeyError):
|
||||
resolve.resolve_annotation(9999, 0.0, [0.4, 0.4, 0.6, 0.6], event_id=None)
|
||||
|
||||
|
||||
# --- Phase 4: anchor supersede + delete (API-level, no ffmpeg) --------------------------------
|
||||
|
||||
def _views_and_bbox(corner):
|
||||
views = _views_seeing(corner)
|
||||
out = []
|
||||
for (v, p, px, py) in views:
|
||||
out.append((v, p, _bbox_center(px, py, v.width, v.height)))
|
||||
return out
|
||||
|
||||
|
||||
def test_two_view_annotation_supersedes_to_single_anchor(fixture):
|
||||
"""Two annotations of the same event from two views yield ONE anchor (the fallback is
|
||||
superseded in place by the triangulation), not a sibling pair (the Round-3 wart)."""
|
||||
from festival4d.api import AnnotationIn, create_annotation
|
||||
|
||||
corner = np.asarray(fixture["Stage FR"], dtype=float)
|
||||
vb = _views_and_bbox(corner)
|
||||
assert len(vb) >= 2, "Stage FR should be visible in >=2 cameras"
|
||||
event = db.add_event(t_global_s=5.0, event_type="pyro", source="ai", description="p4 pyro")
|
||||
|
||||
n0 = len(db.get_anchors())
|
||||
(va, pa, bba), (vb2, pb, bbb) = vb[0], vb[1]
|
||||
r1 = create_annotation(AnnotationIn(video_id=va.id, t_video_s=pa.t_video_s, bbox=bba, event_id=event.id))
|
||||
r2 = create_annotation(AnnotationIn(video_id=vb2.id, t_video_s=pb.t_video_s, bbox=bbb, event_id=event.id))
|
||||
|
||||
assert r2["method"] == "triangulated" and r2["superseded"] is True
|
||||
assert r1["anchor_id"] == r2["anchor_id"] # same anchor, updated in place
|
||||
assert len(db.get_anchors()) - n0 == 1 # ONE anchor, not two
|
||||
# Both annotations link to that single anchor.
|
||||
links = {a.resolved_anchor_id for a in db.get_annotations(event_id=event.id)}
|
||||
assert links == {r2["anchor_id"]}
|
||||
# Superseded position is the accurate triangulation.
|
||||
anchor = db.get_anchor(r2["anchor_id"])
|
||||
assert float(np.linalg.norm([anchor.x - corner[0], anchor.y - corner[1], anchor.z - corner[2]])) < 0.2
|
||||
|
||||
|
||||
def test_delete_anchor_unlinks_annotations_and_404(fixture):
|
||||
from fastapi import HTTPException
|
||||
|
||||
from festival4d.api import AnnotationIn, create_annotation, delete_anchor
|
||||
|
||||
corner = np.asarray(fixture["Stage BL"], dtype=float)
|
||||
vb = _views_and_bbox(corner)
|
||||
assert vb, "Stage BL should be visible in >=1 camera"
|
||||
event = db.add_event(t_global_s=12.0, event_type="confetti", source="ai", description="p4 del")
|
||||
v, p, bb = vb[0]
|
||||
r = create_annotation(AnnotationIn(video_id=v.id, t_video_s=p.t_video_s, bbox=bb, event_id=event.id))
|
||||
aid = r["anchor_id"]
|
||||
assert aid is not None
|
||||
|
||||
assert delete_anchor(aid) == {"deleted": aid}
|
||||
assert db.get_anchor(aid) is None
|
||||
# The annotation that resolved to it is unlinked, not orphaned with a dangling FK.
|
||||
assert all(a.resolved_anchor_id != aid for a in db.get_annotations(event_id=event.id))
|
||||
with pytest.raises(HTTPException) as exc:
|
||||
delete_anchor(aid)
|
||||
assert exc.value.status_code == 404
|
||||
@ -1,494 +0,0 @@
|
||||
"""Tests for lane B: frames.py (sampling) + sfm.py (parsers, normalization,
|
||||
interpolation, and the graceful-degradation reconstruct pipeline).
|
||||
|
||||
Everything here is verifiable without COLMAP: parsers run on hand-written TXT snippets, the
|
||||
export path runs on a hand-built (synthetic-pose) COLMAP model injected in place of a real
|
||||
COLMAP run, and the failure paths are exercised by monkeypatching. Spec M2 acceptance:
|
||||
"on the synthetic fixture (which skips COLMAP and injects known poses) the whole export path
|
||||
runs; if COLMAP is installed, the pipeline runs end-to-end without crashing."
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import struct
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from festival4d import config, db, frames, sfm, synthetic
|
||||
from festival4d.geometry import quat_to_mat
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# COLMAP TXT parsers (hand-written snippets)
|
||||
# ===========================================================================
|
||||
IMAGES_TXT = """\
|
||||
# Image list with two lines of data per image:
|
||||
# IMAGE_ID, QW, QX, QY, QZ, TX, TY, TZ, CAMERA_ID, NAME
|
||||
# POINTS2D[] as (X, Y, POINT3D_ID)
|
||||
# Number of images: 2, mean observations per image: 2
|
||||
1 0.9998 0.0 0.02 0.0 -1.5 0.3 8.0 1 1/1_0.jpg
|
||||
480.0 270.0 5 500.1 260.2 -1
|
||||
2 0.7071 0.0 0.7071 0.0 2.0 0.1 7.5 2 2/2_30.jpg
|
||||
100.0 120.0 -1 300.0 200.0 5
|
||||
"""
|
||||
|
||||
CAMERAS_TXT = """\
|
||||
# Camera list with one line of data per camera:
|
||||
# CAMERA_ID, MODEL, WIDTH, HEIGHT, PARAMS[]
|
||||
# Number of cameras: 2
|
||||
1 PINHOLE 640 360 500.0 500.0 320.0 180.0
|
||||
2 OPENCV 640 360 510.0 511.0 321.0 181.0 0.01 -0.02 0.0 0.0
|
||||
"""
|
||||
|
||||
POINTS3D_TXT = """\
|
||||
# 3D point list with one line of data per point:
|
||||
# POINT3D_ID, X, Y, Z, R, G, B, ERROR, TRACK[] as (IMAGE_ID, POINT2D_IDX)
|
||||
# Number of points: 2, mean track length: 2
|
||||
5 1.0 2.0 3.0 200 100 50 0.5 1 0 2 1
|
||||
9 -1.0 0.0 4.0 10 20 30 0.8 1 1
|
||||
"""
|
||||
|
||||
|
||||
def test_parse_images_txt(tmp_path):
|
||||
p = tmp_path / "images.txt"
|
||||
p.write_text(IMAGES_TXT)
|
||||
recs = sfm.parse_images_txt(p)
|
||||
assert len(recs) == 2
|
||||
assert recs[0]["image_id"] == 1
|
||||
assert recs[0]["camera_id"] == 1
|
||||
assert recs[0]["name"] == "1/1_0.jpg"
|
||||
np.testing.assert_allclose(
|
||||
[recs[0]["qw"], recs[0]["qx"], recs[0]["qy"], recs[0]["qz"]],
|
||||
[0.9998, 0.0, 0.02, 0.0])
|
||||
np.testing.assert_allclose(
|
||||
[recs[1]["tx"], recs[1]["ty"], recs[1]["tz"]], [2.0, 0.1, 7.5])
|
||||
|
||||
|
||||
def test_parse_cameras_txt(tmp_path):
|
||||
p = tmp_path / "cameras.txt"
|
||||
p.write_text(CAMERAS_TXT)
|
||||
cams = sfm.parse_cameras_txt(p)
|
||||
assert set(cams) == {1, 2}
|
||||
assert cams[1]["model"] == "PINHOLE"
|
||||
assert (cams[1]["fx"], cams[1]["fy"], cams[1]["cx"], cams[1]["cy"]) == \
|
||||
(500.0, 500.0, 320.0, 180.0)
|
||||
# OPENCV: first 4 params are fx, fy, cx, cy; distortion ignored
|
||||
assert (cams[2]["fx"], cams[2]["fy"], cams[2]["cx"], cams[2]["cy"]) == \
|
||||
(510.0, 511.0, 321.0, 181.0)
|
||||
|
||||
|
||||
def test_parse_points3d_txt(tmp_path):
|
||||
p = tmp_path / "points3D.txt"
|
||||
p.write_text(POINTS3D_TXT)
|
||||
points, colors = sfm.parse_points3d_txt(p)
|
||||
assert points.shape == (2, 3) and colors.shape == (2, 3)
|
||||
np.testing.assert_allclose(points[0], [1.0, 2.0, 3.0])
|
||||
assert tuple(colors[0]) == (200, 100, 50)
|
||||
assert points.dtype == np.float32 and colors.dtype == np.uint8
|
||||
|
||||
|
||||
def test_parse_points3d_empty(tmp_path):
|
||||
p = tmp_path / "points3D.txt"
|
||||
p.write_text("# header only\n")
|
||||
points, colors = sfm.parse_points3d_txt(p)
|
||||
assert points.shape == (0, 3) and colors.shape == (0, 3)
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Scene normalization
|
||||
# ===========================================================================
|
||||
def _all_synth_poses():
|
||||
poses = []
|
||||
for i in range(3):
|
||||
poses.extend(synthetic.camera_track(i, 20.0, 30, 640, 360))
|
||||
return poses
|
||||
|
||||
|
||||
def test_normalize_scene_invariants():
|
||||
points, _ = synthetic.generate_point_cloud()
|
||||
poses = _all_synth_poses()
|
||||
npoints, nposes = sfm.normalize_scene(points.astype(float), poses)
|
||||
|
||||
# centroid at origin
|
||||
assert np.linalg.norm(npoints.mean(axis=0)) < 1e-3
|
||||
# camera bounding sphere (from origin = point centroid) has radius 10
|
||||
centers = np.array([sfm._camera_center(p) for p in nposes])
|
||||
assert abs(float(np.max(np.linalg.norm(centers, axis=1))) - 10.0) < 1e-6
|
||||
# average camera-up aligns with +Y
|
||||
ups = np.array([sfm._camera_up(p) for p in nposes])
|
||||
up_avg = ups.mean(axis=0)
|
||||
up_avg /= np.linalg.norm(up_avg)
|
||||
np.testing.assert_allclose(up_avg, [0.0, 1.0, 0.0], atol=1e-6)
|
||||
|
||||
|
||||
def test_normalize_scene_preserves_projection():
|
||||
"""A world point projects to the same pixel before and after normalization
|
||||
(a similarity transform of the world must not change any image)."""
|
||||
points, _ = synthetic.generate_point_cloud()
|
||||
poses = _all_synth_poses()
|
||||
intr = synthetic.intrinsics(640, 360)
|
||||
for p in poses:
|
||||
p.update(intr)
|
||||
X = np.array([0.7, 0.9, 0.3]) # arbitrary world point
|
||||
|
||||
def project(pose, Xw):
|
||||
R = quat_to_mat([pose["qw"], pose["qx"], pose["qy"], pose["qz"]])
|
||||
t = np.array([pose["tx"], pose["ty"], pose["tz"]])
|
||||
xc = R @ Xw + t
|
||||
return np.array([pose["fx"] * xc[0] / xc[2] + pose["cx"],
|
||||
pose["fy"] * xc[1] / xc[2] + pose["cy"]])
|
||||
|
||||
npoints, nposes = sfm.normalize_scene(points.astype(float), poses)
|
||||
# transform the same world point by X' = s*Rn@(X-c): recover s,Rn,c from the point map.
|
||||
# Easier: the invariant is projection equality, so map X through the same transform used
|
||||
# on the cloud by fitting it — instead, just check pixel equality using each pose pair.
|
||||
# Reconstruct transform from three non-collinear cloud points is overkill; use the fact
|
||||
# that projection is invariant, so compare original point's pixel to the transformed
|
||||
# point's pixel where the transform is inferred from the point cloud centroid+scale.
|
||||
c = points.astype(float).mean(axis=0)
|
||||
centers = np.array([sfm._camera_center(p) for p in poses])
|
||||
scale = 10.0 / float(np.max(np.linalg.norm(centers - c, axis=1)))
|
||||
ups = np.array([sfm._camera_up(p) for p in poses])
|
||||
up_avg = ups.mean(axis=0)
|
||||
Rn = sfm._rotation_aligning(up_avg, np.array([0.0, 1.0, 0.0]))
|
||||
Xn = scale * Rn @ (X - c)
|
||||
|
||||
for p_old, p_new in zip(poses, nposes):
|
||||
np.testing.assert_allclose(project(p_old, X), project(p_new, Xn), atol=1e-6)
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Pose interpolation
|
||||
# ===========================================================================
|
||||
def _pose(frame_idx, t, q, t3, registered=True):
|
||||
return {"frame_idx": frame_idx, "t_video_s": t,
|
||||
"qw": q[0], "qx": q[1], "qy": q[2], "qz": q[3],
|
||||
"tx": t3[0], "ty": t3[1], "tz": t3[2],
|
||||
"fx": 500.0, "fy": 500.0, "cx": 320.0, "cy": 180.0,
|
||||
"registered": registered, "video_id": 1}
|
||||
|
||||
|
||||
def test_interpolate_poses_fills_gaps_no_extrapolation():
|
||||
q0 = [1.0, 0.0, 0.0, 0.0]
|
||||
ang = np.radians(60.0)
|
||||
q1 = [np.cos(ang / 2), 0.0, np.sin(ang / 2), 0.0] # 60 deg yaw about Y
|
||||
registered = [_pose(0, 0.0, q0, [0, 0, 0]), _pose(60, 2.0, q1, [6, 0, 0])]
|
||||
sampled = [{"frame_idx": f, "t_video_s": f / 30.0} for f in (0, 30, 60, 90)]
|
||||
|
||||
out = sfm.interpolate_poses(registered, sampled)
|
||||
by_frame = {p["frame_idx"]: p for p in out}
|
||||
assert set(by_frame) == {0, 30, 60} # frame 90 dropped (beyond last registered)
|
||||
assert by_frame[30]["registered"] is False # interpolated
|
||||
assert by_frame[0]["registered"] is True and by_frame[60]["registered"] is True
|
||||
# midpoint translation is the lerp; rotation is ~30 deg
|
||||
np.testing.assert_allclose(
|
||||
[by_frame[30]["tx"], by_frame[30]["ty"], by_frame[30]["tz"]], [3, 0, 0], atol=1e-9)
|
||||
mid_angle = 2.0 * np.degrees(np.arccos(min(1.0, abs(by_frame[30]["qw"]))))
|
||||
assert abs(mid_angle - 30.0) < 1e-6
|
||||
|
||||
|
||||
def test_interpolate_poses_empty_registered():
|
||||
assert sfm.interpolate_poses([], [{"frame_idx": 0, "t_video_s": 0.0}]) == []
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# frames.py — sharpness + windowed sampling
|
||||
# ===========================================================================
|
||||
def test_sharpness_sharp_beats_blurred():
|
||||
import cv2
|
||||
|
||||
rng = np.random.default_rng(0)
|
||||
noise = (rng.integers(0, 256, size=(120, 120), dtype=np.uint8)) # high-frequency
|
||||
flat = np.full((120, 120), 128, dtype=np.uint8) # no edges
|
||||
blurred = cv2.GaussianBlur(noise, (0, 0), sigmaX=5)
|
||||
assert frames.sharpness(noise) > frames.sharpness(blurred) > frames.sharpness(flat)
|
||||
# accepts BGR too
|
||||
bgr = cv2.cvtColor(noise, cv2.COLOR_GRAY2BGR)
|
||||
assert frames.sharpness(bgr) > 0
|
||||
|
||||
|
||||
def test_sample_frames_picks_sharpest_per_window(tmp_path):
|
||||
import cv2
|
||||
|
||||
w = h = 128
|
||||
fps = 30.0
|
||||
n_frames = 30 # 1.0 s -> two 0.5 s windows
|
||||
vid_path = tmp_path / "clip.mp4"
|
||||
writer = cv2.VideoWriter(str(vid_path), cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
|
||||
if not writer.isOpened():
|
||||
pytest.skip("no usable VideoWriter codec in this environment")
|
||||
checker = np.indices((h, w)).sum(axis=0) % 2 # fine checkerboard -> sharp
|
||||
sharp = (checker * 255).astype(np.uint8)
|
||||
flat = np.full((h, w), 128, dtype=np.uint8)
|
||||
sharp_frames = {7, 22}
|
||||
for i in range(n_frames):
|
||||
g = sharp if i in sharp_frames else flat
|
||||
writer.write(cv2.cvtColor(g, cv2.COLOR_GRAY2BGR))
|
||||
writer.release()
|
||||
|
||||
out = frames.sample_frames(vid_path, video_id=5, out_dir=tmp_path / "frames")
|
||||
got_frames = sorted(int(Path(p).stem.split("_")[1]) for p in out)
|
||||
|
||||
# ground-truth: decode the written stream and take the per-window argmax ourselves
|
||||
cap = cv2.VideoCapture(str(vid_path))
|
||||
sh, idx = [], 0
|
||||
while True:
|
||||
ok, fr = cap.read()
|
||||
if not ok:
|
||||
break
|
||||
sh.append(frames.sharpness(fr)); idx += 1
|
||||
cap.release()
|
||||
bucket = {}
|
||||
for i, s in enumerate(sh):
|
||||
win = int((i / fps) / 0.5)
|
||||
if win not in bucket or s > bucket[win][0]:
|
||||
bucket[win] = (s, i)
|
||||
expected = sorted(v[1] for v in bucket.values())
|
||||
|
||||
assert got_frames == expected
|
||||
assert all(Path(p).name.startswith("5_") for p in out)
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# run_reconstruct — graceful degradation + full export path (no real COLMAP)
|
||||
# ===========================================================================
|
||||
def _snapshot_poses():
|
||||
return {
|
||||
v.id: [(p.frame_idx, round(p.qw, 9), round(p.tx, 9), p.registered)
|
||||
for p in db.get_poses(v.id)]
|
||||
for v in db.get_videos()
|
||||
}
|
||||
|
||||
|
||||
def _write_colmap_model(model_dir: Path, records: list[dict], cameras: dict,
|
||||
points, colors) -> None:
|
||||
"""Write a COLMAP TXT model (images/cameras/points3D) from pose records."""
|
||||
model_dir.mkdir(parents=True, exist_ok=True)
|
||||
with open(model_dir / "images.txt", "w") as f:
|
||||
f.write("# image list\n")
|
||||
for i, r in enumerate(records, start=1):
|
||||
f.write(f"{i} {r['qw']} {r['qx']} {r['qy']} {r['qz']} "
|
||||
f"{r['tx']} {r['ty']} {r['tz']} {r['camera_id']} {r['name']}\n")
|
||||
f.write("100.0 100.0 -1\n") # dummy points2D line
|
||||
with open(model_dir / "cameras.txt", "w") as f:
|
||||
f.write("# camera list\n")
|
||||
for cid, cam in cameras.items():
|
||||
f.write(f"{cid} PINHOLE {cam['width']} {cam['height']} "
|
||||
f"{cam['fx']} {cam['fy']} {cam['cx']} {cam['cy']}\n")
|
||||
with open(model_dir / "points3D.txt", "w") as f:
|
||||
f.write("# point list\n")
|
||||
for j, (p, c) in enumerate(zip(points, colors), start=1):
|
||||
f.write(f"{j} {p[0]} {p[1]} {p[2]} {int(c[0])} {int(c[1])} {int(c[2])} 0.5 1 0\n")
|
||||
|
||||
|
||||
def test_reconstruct_no_videos():
|
||||
db.init_engine()
|
||||
db.reset_db()
|
||||
result = sfm.run_reconstruct()
|
||||
assert result["status"] == "no_videos"
|
||||
|
||||
|
||||
def test_reconstruct_skips_without_colmap(monkeypatch):
|
||||
synthetic.build(run_ffmpeg=False)
|
||||
monkeypatch.setattr(sfm, "colmap_available", lambda: False)
|
||||
before = _snapshot_poses()
|
||||
result = sfm.run_reconstruct()
|
||||
assert result["status"] == "skipped_no_colmap"
|
||||
assert _snapshot_poses() == before # poses untouched
|
||||
|
||||
|
||||
def _fake_sampling(monkeypatch, frames_per_video):
|
||||
"""Make sampling hermetic: placeholder raw files + fake sample_frames output."""
|
||||
for v in db.get_videos():
|
||||
(config.RAW_DIR / v.filename).parent.mkdir(parents=True, exist_ok=True)
|
||||
(config.RAW_DIR / v.filename).write_bytes(b"stub")
|
||||
|
||||
def fake_sample(video_path, video_id, out_dir, **kw):
|
||||
return [Path(f"{video_id}_{f}.jpg") for f in frames_per_video]
|
||||
|
||||
monkeypatch.setattr(sfm.frames, "sample_frames", fake_sample)
|
||||
monkeypatch.setattr(sfm, "colmap_available", lambda: True)
|
||||
|
||||
|
||||
def test_reconstruct_failed_no_model(monkeypatch):
|
||||
synthetic.build(run_ffmpeg=False)
|
||||
_fake_sampling(monkeypatch, [0, 30, 60])
|
||||
monkeypatch.setattr(sfm, "run_colmap", lambda frames_dir, workspace: None)
|
||||
before = _snapshot_poses()
|
||||
result = sfm.run_reconstruct()
|
||||
assert result["status"] == "failed_no_model"
|
||||
assert _snapshot_poses() == before # poses untouched
|
||||
|
||||
|
||||
def test_reconstruct_failed_weak(monkeypatch, tmp_path):
|
||||
synthetic.build(run_ffmpeg=False)
|
||||
_fake_sampling(monkeypatch, [0, 30, 60]) # 3 videos x 3 = 9 sampled
|
||||
# model registers only one image of one video -> ~11%, < 60% and < 2 videos
|
||||
track = synthetic.camera_track(0, 20.0, 30, 640, 360)
|
||||
rec = dict(track[0]); rec["camera_id"] = 1; rec["name"] = "1_0.jpg"
|
||||
cams = {1: {"width": 640, "height": 360, **synthetic.intrinsics(640, 360)}}
|
||||
pts, cols = synthetic.generate_point_cloud()
|
||||
model_dir = tmp_path / "weak"
|
||||
_write_colmap_model(model_dir, [rec], cams, pts[:50], cols[:50])
|
||||
monkeypatch.setattr(sfm, "run_colmap", lambda frames_dir, workspace: model_dir)
|
||||
before = _snapshot_poses()
|
||||
result = sfm.run_reconstruct()
|
||||
assert result["status"] == "failed_weak"
|
||||
assert _snapshot_poses() == before # poses untouched
|
||||
|
||||
|
||||
def test_reconstruct_full_export_path(monkeypatch, tmp_path):
|
||||
"""The whole export path runs on injected known poses (spec M2 acceptance)."""
|
||||
synthetic.build(run_ffmpeg=False)
|
||||
videos = db.get_videos()
|
||||
id_by_cam = {i: videos[i].id for i in range(len(videos))} # cam index -> db id
|
||||
sampled_frames = [0, 15, 30, 45, 60]
|
||||
_fake_sampling(monkeypatch, sampled_frames)
|
||||
|
||||
# Build a model: videos for cam0/cam1 fully registered; cam2 registers a subset so
|
||||
# interpolation must fill its gaps. All frame indices come from the synthetic tracks.
|
||||
records, cameras = [], {}
|
||||
reg_plan = {0: sampled_frames, 1: sampled_frames, 2: [0, 30, 60]}
|
||||
for cam_idx, frame_list in reg_plan.items():
|
||||
vid = id_by_cam[cam_idx]
|
||||
cameras[vid] = {"width": 640, "height": 360, **synthetic.intrinsics(640, 360)}
|
||||
track = {p["frame_idx"]: p for p in synthetic.camera_track(cam_idx, 20.0, 30, 640, 360)}
|
||||
for f in frame_list:
|
||||
p = track[f]
|
||||
records.append({**p, "camera_id": vid, "name": f"{vid}_{f}.jpg"})
|
||||
pts, cols = synthetic.generate_point_cloud()
|
||||
model_dir = tmp_path / "good"
|
||||
_write_colmap_model(model_dir, records, cameras, pts, cols)
|
||||
monkeypatch.setattr(sfm, "run_colmap", lambda frames_dir, workspace: model_dir)
|
||||
|
||||
result = sfm.run_reconstruct()
|
||||
assert result["status"] == "ok"
|
||||
assert result["registered"] == 13 # 5 + 5 + 3
|
||||
assert result["videos_in_model"] == 3
|
||||
assert result["interpolated"] == 2 # cam2 frames 15 and 45
|
||||
|
||||
# cam2 poses: 5 total (3 registered + 2 interpolated), sorted, with interp flags
|
||||
cam2_poses = db.get_poses(id_by_cam[2])
|
||||
assert len(cam2_poses) == 5
|
||||
assert sum(1 for p in cam2_poses if not p.registered) == 2
|
||||
assert [p.frame_idx for p in cam2_poses] == [0, 15, 30, 45, 60]
|
||||
|
||||
# normalized point cloud written in the frozen PLY format; camera sphere radius ~ 10
|
||||
assert config.POINTS_PLY.exists()
|
||||
npoints, _ = synthetic.read_ply(config.POINTS_PLY)
|
||||
assert len(npoints) == len(pts)
|
||||
all_centers = []
|
||||
for v in videos:
|
||||
for p in db.get_poses(v.id):
|
||||
R = quat_to_mat([p.qw, p.qx, p.qy, p.qz])
|
||||
all_centers.append(-R.T @ np.array([p.tx, p.ty, p.tz]))
|
||||
max_r = float(np.max(np.linalg.norm(np.array(all_centers), axis=1)))
|
||||
assert abs(max_r - 10.0) < 0.5 # registered cams normalized to radius 10
|
||||
|
||||
|
||||
def test_colmap_model_parser_roundtrip(tmp_path):
|
||||
"""Writing then parsing a model recovers the poses/intrinsics/points."""
|
||||
track = synthetic.camera_track(0, 20.0, 30, 640, 360)[:3]
|
||||
records = [{**p, "camera_id": 1, "name": f"1_{p['frame_idx']}.jpg"} for p in track]
|
||||
cams = {1: {"width": 640, "height": 360, **synthetic.intrinsics(640, 360)}}
|
||||
pts, cols = synthetic.generate_point_cloud()
|
||||
model_dir = tmp_path / "rt"
|
||||
_write_colmap_model(model_dir, records, cams, pts[:20], cols[:20])
|
||||
|
||||
images = sfm.parse_images_txt(model_dir / "images.txt")
|
||||
cameras = sfm.parse_cameras_txt(model_dir / "cameras.txt")
|
||||
ppoints, pcolors = sfm.parse_points3d_txt(model_dir / "points3D.txt")
|
||||
assert len(images) == 3
|
||||
np.testing.assert_allclose(images[0]["qw"], track[0]["qw"], atol=1e-6)
|
||||
assert cameras[1]["fx"] == pytest.approx(synthetic.intrinsics(640, 360)["fx"])
|
||||
assert len(ppoints) == 20
|
||||
|
||||
|
||||
def test_parse_cameras_txt_truncated_params(tmp_path):
|
||||
"""A camera line with too few PARAMS gives a clean ValueError, not a raw IndexError."""
|
||||
p = tmp_path / "cameras.txt"
|
||||
p.write_text("# header\n7 PINHOLE 640 360 500.0 500.0 320.0\n") # PINHOLE needs 4 params
|
||||
with pytest.raises(ValueError, match="too few PARAMS"):
|
||||
sfm.parse_cameras_txt(p)
|
||||
|
||||
|
||||
def test_largest_model_dir_picks_by_registered_count(tmp_path):
|
||||
"""Binary models are ranked by the images.bin header count, not file size."""
|
||||
sparse = tmp_path / "sparse"
|
||||
# model 0: 10 registered images but a LARGE images.bin (many keypoint observations)
|
||||
(sparse / "0").mkdir(parents=True)
|
||||
(sparse / "0" / "images.bin").write_bytes(struct.pack("<Q", 10) + b"\x00" * 5000)
|
||||
# model 1: 15 registered images but a SMALL images.bin (few keypoints)
|
||||
(sparse / "1").mkdir(parents=True)
|
||||
(sparse / "1" / "images.bin").write_bytes(struct.pack("<Q", 15) + b"\x00" * 100)
|
||||
assert sfm._registered_image_count(sparse / "1") == 15
|
||||
assert sfm._largest_model_dir(sparse).name == "1" # more images wins despite smaller file
|
||||
|
||||
|
||||
def test_interpolate_poses_drops_before_first_registered():
|
||||
"""Lower no-extrapolation guard: a sampled frame before the first registered one is dropped."""
|
||||
q = [1.0, 0.0, 0.0, 0.0]
|
||||
registered = [_pose(30, 1.0, q, [3, 0, 0]), _pose(60, 2.0, q, [6, 0, 0])]
|
||||
sampled = [{"frame_idx": f, "t_video_s": f / 30.0} for f in (0, 30, 45, 60)]
|
||||
out = sfm.interpolate_poses(registered, sampled)
|
||||
frames_out = {p["frame_idx"] for p in out}
|
||||
assert 0 not in frames_out # tv=0 < t_first=1.0 -> dropped, no extrapolation
|
||||
assert frames_out == {30, 45, 60}
|
||||
by = {p["frame_idx"]: p for p in out}
|
||||
assert by[45]["registered"] is False # interior gap still interpolated
|
||||
|
||||
|
||||
def test_reconstruct_subset_leaves_others_untouched(monkeypatch, tmp_path):
|
||||
"""When only a subset of videos registers, the rest keep their existing poses (DB safety)."""
|
||||
synthetic.build(run_ffmpeg=False)
|
||||
videos = db.get_videos()
|
||||
id_by_cam = {i: videos[i].id for i in range(len(videos))}
|
||||
sampled_frames = [0, 15, 30, 45, 60]
|
||||
_fake_sampling(monkeypatch, sampled_frames)
|
||||
|
||||
# Register cam0 + cam1 fully (10/15 = 67% >= 60%, 2 videos -> ok); cam2 NOT in the model.
|
||||
records, cameras = [], {}
|
||||
for cam_idx in (0, 1):
|
||||
vid = id_by_cam[cam_idx]
|
||||
cameras[vid] = {"width": 640, "height": 360, **synthetic.intrinsics(640, 360)}
|
||||
track = {p["frame_idx"]: p for p in synthetic.camera_track(cam_idx, 20.0, 30, 640, 360)}
|
||||
for f in sampled_frames:
|
||||
records.append({**track[f], "camera_id": vid, "name": f"{vid}_{f}.jpg"})
|
||||
pts, cols = synthetic.generate_point_cloud()
|
||||
model_dir = tmp_path / "subset"
|
||||
_write_colmap_model(model_dir, records, cameras, pts, cols)
|
||||
monkeypatch.setattr(sfm, "run_colmap", lambda frames_dir, workspace: model_dir)
|
||||
|
||||
cam2_before = [(p.frame_idx, p.qw, p.tx, p.registered) for p in db.get_poses(id_by_cam[2])]
|
||||
result = sfm.run_reconstruct()
|
||||
assert result["status"] == "ok"
|
||||
assert result["videos_in_model"] == 2
|
||||
# cam2 absent from the model: its rows must be byte-identical afterward
|
||||
cam2_after = [(p.frame_idx, p.qw, p.tx, p.registered) for p in db.get_poses(id_by_cam[2])]
|
||||
assert cam2_after == cam2_before
|
||||
assert len(cam2_before) == 41 # original synthetic poses intact
|
||||
assert len(db.get_poses(id_by_cam[0])) == 5 # cam0 replaced with sampled frames
|
||||
|
||||
|
||||
def test_reconstruct_no_frames_when_videos_missing(monkeypatch):
|
||||
"""All raw files absent -> videos skipped -> no_frames, poses untouched."""
|
||||
synthetic.build(run_ffmpeg=False)
|
||||
for f in config.RAW_DIR.glob("*.mp4"): # clear stubs left by other tests
|
||||
f.unlink()
|
||||
monkeypatch.setattr(sfm, "colmap_available", lambda: True)
|
||||
before = _snapshot_poses()
|
||||
result = sfm.run_reconstruct()
|
||||
assert result["status"] == "no_frames"
|
||||
assert _snapshot_poses() == before
|
||||
|
||||
|
||||
def test_reconstruct_no_frames_empty_sampling(monkeypatch):
|
||||
"""Raw files exist but sampling yields nothing -> no_frames, poses untouched."""
|
||||
synthetic.build(run_ffmpeg=False)
|
||||
_fake_sampling(monkeypatch, []) # sample_frames returns []
|
||||
before = _snapshot_poses()
|
||||
result = sfm.run_reconstruct()
|
||||
assert result["status"] == "no_frames"
|
||||
assert _snapshot_poses() == before
|
||||
@ -1,351 +0,0 @@
|
||||
"""foundation3 (phase 6 / M18) contract tests — the ground three lanes stand on.
|
||||
|
||||
Covers what foundation3 owns and freezes, NOT the lane implementations (detection = lane H's
|
||||
test_tracker_detect.py; solving = test_tracker_solve.py; ribbons = lane J's live-browser
|
||||
evidence):
|
||||
|
||||
A. tracks / track_points DB helpers (contract #2) — round-trips, replace + delete semantics.
|
||||
B. /api/tracks routes (contract #2) + POST /api/tracks/solve degradation + manifest.has_tracks.
|
||||
C. The frozen OOK blink protocol + color marker_key convention (contracts #1, #4) — pure,
|
||||
no OpenCV. These constants are byte-identical to hardware/badge/protocol.h (lane K).
|
||||
D. The synthetic fixture (M18 step 1): track_truth.json shape, marker paths, and — the one
|
||||
that matters most — a projection ROUND-TRIP proving the rendered marker pixels
|
||||
back-project to the ground-truth 3D point through the frozen geometry primitives. If this
|
||||
passes, lane H's M20 recovery (median 3D error < 0.3) is achievable by construction.
|
||||
|
||||
Hermetic: no ffmpeg (geometry only), temp data dir from conftest. DB-touching tests clean up
|
||||
after themselves so later modules see the tables as they found them.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import shutil
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from festival4d import config, db, synthetic, tracker_detect
|
||||
|
||||
_HAS_FFMPEG = shutil.which("ffmpeg") is not None and shutil.which("ffprobe") is not None
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# A. DB helpers (contract #2)
|
||||
# ===========================================================================
|
||||
def _use_temp_db(tmp_path):
|
||||
db.init_engine(tmp_path / "t.db")
|
||||
db.reset_db()
|
||||
|
||||
|
||||
def test_add_and_get_tracks_ordered_by_id(tmp_path):
|
||||
_use_temp_db(tmp_path)
|
||||
a = db.add_track("hue:150", label="Alice", color="#ff00ff")
|
||||
b = db.add_track("code:5")
|
||||
assert db.has_tracks() is True
|
||||
tracks = db.get_tracks()
|
||||
assert [t.id for t in tracks] == sorted(t.id for t in tracks) == [a.id, b.id]
|
||||
assert tracks[0].marker_key == "hue:150" and tracks[0].label == "Alice"
|
||||
assert tracks[1].label is None and tracks[1].color is None # optional fields default None
|
||||
assert db.get_track(a.id).marker_key == "hue:150"
|
||||
assert db.get_track(999999) is None
|
||||
|
||||
|
||||
def test_has_tracks_false_on_empty(tmp_path):
|
||||
_use_temp_db(tmp_path)
|
||||
assert db.has_tracks() is False
|
||||
assert db.get_tracks() == []
|
||||
|
||||
|
||||
def test_set_track_points_replaces_and_orders(tmp_path):
|
||||
_use_temp_db(tmp_path)
|
||||
t = db.add_track("code:5")
|
||||
# Insert out of order; helper must return them sorted by t_global_s (contract #2).
|
||||
n = db.set_track_points(t.id, [
|
||||
{"t_global_s": 2.0, "x": 1, "y": 2, "z": 3, "quality": 0.9, "views": 2},
|
||||
{"t_global_s": 1.0, "x": 0, "y": 0, "z": 0, "quality": 0.4, "views": 1},
|
||||
])
|
||||
assert n == 2
|
||||
pts = db.get_track_points(t.id)
|
||||
assert [p.t_global_s for p in pts] == [1.0, 2.0]
|
||||
assert pts[0].views == 1 and pts[0].quality == 0.4
|
||||
assert pts[1].views == 2
|
||||
# Re-solve replaces (idempotent — never stacks duplicates, per M20 contract).
|
||||
db.set_track_points(t.id, [{"t_global_s": 5.0, "x": 9, "y": 9, "z": 9}])
|
||||
pts = db.get_track_points(t.id)
|
||||
assert len(pts) == 1 and pts[0].t_global_s == 5.0
|
||||
assert pts[0].quality is None and pts[0].views is None # omitted optionals -> None
|
||||
|
||||
|
||||
def test_set_track_points_unknown_track_raises(tmp_path):
|
||||
_use_temp_db(tmp_path)
|
||||
with pytest.raises(KeyError):
|
||||
db.set_track_points(424242, [{"t_global_s": 0.0, "x": 0, "y": 0, "z": 0}])
|
||||
|
||||
|
||||
def test_patch_track_partial_and_missing(tmp_path):
|
||||
_use_temp_db(tmp_path)
|
||||
t = db.add_track("hue:150", label="Old", color="#111111")
|
||||
db.patch_track(t.id, label="New") # color untouched
|
||||
got = db.get_track(t.id)
|
||||
assert got.label == "New" and got.color == "#111111"
|
||||
db.patch_track(t.id, color="#00ffff")
|
||||
assert db.get_track(t.id).color == "#00ffff"
|
||||
with pytest.raises(KeyError):
|
||||
db.patch_track(424242, label="x")
|
||||
|
||||
|
||||
def test_delete_track_removes_points(tmp_path):
|
||||
_use_temp_db(tmp_path)
|
||||
t = db.add_track("code:5")
|
||||
db.set_track_points(t.id, [{"t_global_s": 0.0, "x": 0, "y": 0, "z": 0}])
|
||||
assert db.delete_track(t.id) is True
|
||||
assert db.get_track(t.id) is None
|
||||
assert db.get_track_points(t.id) == [] # points cascade (SQLite doesn't by default)
|
||||
assert db.delete_track(t.id) is False # already gone
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# B. /api/tracks routes (contract #2)
|
||||
# ===========================================================================
|
||||
@pytest.fixture(scope="module")
|
||||
def client():
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from festival4d import api
|
||||
|
||||
with TestClient(api.app) as c:
|
||||
yield c
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def _api_db(client):
|
||||
"""Point the API at a clean per-test db and restore the shared engine afterwards."""
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
d = Path(tempfile.mkdtemp(prefix="tracks-api-"))
|
||||
db.init_engine(d / "api.db")
|
||||
db.reset_db()
|
||||
yield
|
||||
db.init_engine(config.DB_PATH)
|
||||
|
||||
|
||||
def test_get_tracks_shape(client, _api_db):
|
||||
t = db.add_track("hue:150", label="Alice", color="#ff00ff")
|
||||
db.set_track_points(t.id, [
|
||||
{"t_global_s": 1.0, "x": 0.1, "y": 1.5, "z": 2.0, "quality": 0.9, "views": 2},
|
||||
{"t_global_s": 1.25, "x": 0.2, "y": 1.5, "z": 1.9, "quality": 0.5, "views": 1},
|
||||
])
|
||||
listing = client.get("/api/tracks").json()
|
||||
assert len(listing) == 1
|
||||
row = listing[0]
|
||||
assert set(row) == {"id", "marker_key", "label", "color", "points"}
|
||||
assert row["marker_key"] == "hue:150" and row["label"] == "Alice"
|
||||
assert [p["t_global_s"] for p in row["points"]] == [1.0, 1.25] # ordered
|
||||
assert set(row["points"][0]) == {"t_global_s", "x", "y", "z", "quality", "views"}
|
||||
assert row["points"][1]["views"] == 1
|
||||
|
||||
|
||||
def test_patch_and_delete_routes(client, _api_db):
|
||||
t = db.add_track("code:5")
|
||||
patched = client.patch(f"/api/tracks/{t.id}", json={"label": "Bob", "color": "#00ffff"}).json()
|
||||
assert patched["label"] == "Bob" and patched["color"] == "#00ffff"
|
||||
# partial patch leaves the other field
|
||||
assert client.patch(f"/api/tracks/{t.id}", json={"color": "#fff"}).json()["label"] == "Bob"
|
||||
assert client.patch("/api/tracks/999999", json={"label": "x"}).status_code == 404
|
||||
assert client.delete(f"/api/tracks/{t.id}").json() == {"deleted": t.id}
|
||||
assert client.get("/api/tracks").json() == []
|
||||
assert client.delete(f"/api/tracks/{t.id}").status_code == 404
|
||||
|
||||
|
||||
def test_solve_degrades_while_stubbed(client, _api_db):
|
||||
# Lane H is a stub (NotImplementedError). The frozen route must NOT 500 — it returns a
|
||||
# valid empty result + a note (the house pattern, mirrors POST /api/events/detect).
|
||||
resp = client.post("/api/tracks/solve")
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body["result"] is None
|
||||
assert "not implemented" in body["note"].lower()
|
||||
assert body["tracks"] == [] # no tracks solved yet
|
||||
|
||||
|
||||
def test_manifest_has_tracks_reflects_db(client, _api_db):
|
||||
assert client.get("/api/manifest").json()["has_tracks"] is False
|
||||
db.add_track("hue:150")
|
||||
assert client.get("/api/manifest").json()["has_tracks"] is True
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# C. Frozen blink OOK protocol + color marker_key (contracts #1, #4)
|
||||
# ===========================================================================
|
||||
def test_protocol_constants_match_spec():
|
||||
assert tracker_detect.BLINK_BIT_PERIOD_S == 0.133
|
||||
assert tracker_detect.BLINK_PREAMBLE == (1, 1, 1, 0, 0)
|
||||
assert tracker_detect.BLINK_ID_BITS == 6
|
||||
assert tracker_detect.BLINK_WORD_BITS == 12 # 5 preamble + 6 id + 1 parity
|
||||
assert tracker_detect.BLINK_BEACON_ID == 63
|
||||
assert tracker_detect.BLINK_MIN_FPS == 24.0
|
||||
assert tracker_detect.MAGENTA_OPENCV_HUE == 150 # #ff00ff, hue 300°/2
|
||||
assert tracker_detect.CYAN_OPENCV_HUE == 90 # #00ffff, hue 180°/2
|
||||
|
||||
|
||||
def test_encode_word_id5_exact_and_even_parity():
|
||||
word = tracker_detect.encode_word(5)
|
||||
assert word == [1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1] # preamble + 000101 + parity
|
||||
assert len(word) == 12
|
||||
for mid in range(0, 64):
|
||||
w = tracker_detect.encode_word(mid)
|
||||
assert tuple(w[:5]) == tracker_detect.BLINK_PREAMBLE
|
||||
assert sum(w) % 2 == 0 # even parity over the whole word
|
||||
# 6-bit ID MSB-first recovers the value
|
||||
recovered = sum(bit << (5 - i) for i, bit in enumerate(w[5:11]))
|
||||
assert recovered == mid
|
||||
|
||||
|
||||
def test_encode_word_rejects_out_of_range():
|
||||
for bad in (-1, 64, 100):
|
||||
with pytest.raises(ValueError):
|
||||
tracker_detect.encode_word(bad)
|
||||
|
||||
|
||||
def test_led_on_matches_encoded_word_windows():
|
||||
# The LED state at bit-window k equals bit k of the repeating word (global time).
|
||||
word = tracker_detect.encode_word(5)
|
||||
period = tracker_detect.BLINK_BIT_PERIOD_S
|
||||
for k in range(len(word) * 2): # two full words
|
||||
t_mid = (k + 0.5) * period # sample mid-window
|
||||
assert tracker_detect.led_on(t_mid, 5) == bool(word[k % len(word)])
|
||||
|
||||
|
||||
def test_marker_key_helpers_and_hue_env(monkeypatch):
|
||||
assert tracker_detect.color_marker_key(150) == "hue:150"
|
||||
assert tracker_detect.code_marker_key(5) == "code:5"
|
||||
monkeypatch.delenv("FESTIVAL4D_MARKER_HUES", raising=False)
|
||||
assert tracker_detect.marker_hues_hex() == ["#ff00ff", "#00ffff"]
|
||||
monkeypatch.setenv("FESTIVAL4D_MARKER_HUES", "#ff0000, #00ff00")
|
||||
assert tracker_detect.marker_hues_hex() == ["#ff0000", "#00ff00"]
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# D. Synthetic fixture: track_truth shape, marker paths, projection round-trip
|
||||
# ===========================================================================
|
||||
def test_track_truth_shape_and_both_markers():
|
||||
tt = synthetic.build_track_truth(0.0, 2.0, dt=0.05)
|
||||
assert set(tt) >= {"markers"}
|
||||
keys = {m["marker_key"] for m in tt["markers"]}
|
||||
assert keys == {"hue:150", "code:5"} # marker A (color) + marker B (blink)
|
||||
for m in tt["markers"]:
|
||||
ts = [p["t_global_s"] for p in m["points"]]
|
||||
assert ts == sorted(ts) # monotonically increasing
|
||||
assert len(ts) == len(set(ts)) # no dup timesteps
|
||||
assert set(m["points"][0]) == {"t_global_s", "x", "y", "z"} # Three.js scene space
|
||||
|
||||
|
||||
def test_marker_paths_and_unknown_key():
|
||||
# Marker A: a circle radius 2 at y=1.5 (contract, spec M18).
|
||||
for tg in (0.0, 3.7, 11.2):
|
||||
x, y, z = synthetic.marker_position("hue:150", tg)
|
||||
assert y == pytest.approx(1.5)
|
||||
assert math.hypot(x, z) == pytest.approx(2.0, abs=1e-9)
|
||||
# Marker B follows a different path (never identical to A) so the two never lock.
|
||||
assert synthetic.marker_position("code:5", 3.7) != synthetic.marker_position("hue:150", 3.7)
|
||||
with pytest.raises(KeyError):
|
||||
synthetic.marker_position("code:99", 0.0)
|
||||
|
||||
|
||||
def test_marker_discs_are_small():
|
||||
# Both discs together must be well under 2% of frame area (spec M18: ≤2%, pitfall #6).
|
||||
w, h = config.SYNTH_VIDEO_W, config.SYNTH_VIDEO_H
|
||||
r = synthetic._disc_radius_px(h)
|
||||
two_discs = 2 * math.pi * r * r
|
||||
assert two_discs / (w * h) < 0.02
|
||||
|
||||
|
||||
def test_projection_round_trip_recovers_ground_truth():
|
||||
"""THE foundation contract: rendered marker pixels back-project to the 3D truth.
|
||||
|
||||
For each marker at several t_global: project its ground-truth 3D point through every
|
||||
camera's interpolated pose (exactly as render_marker_overlay draws the pixel), then treat
|
||||
the in-frame pixels as a perfect detector's output and triangulate them back through the
|
||||
FROZEN geometry primitives. Recovery must be ~exact on the noise-free fixture — this is
|
||||
what makes lane H's M20 (median 3D error < 0.3) reachable by construction.
|
||||
"""
|
||||
from festival4d import geometry
|
||||
|
||||
w, h = config.SYNTH_VIDEO_W, config.SYNTH_VIDEO_H
|
||||
fps, dur = float(config.SYNTH_FPS), config.SYNTH_DURATION_S
|
||||
# Poses per camera, generated the same way build() does (no DB, no ffmpeg needed).
|
||||
cam_poses = [synthetic.camera_track(i, dur, fps, w, h)
|
||||
for i in range(len(config.SYNTH_OFFSETS_MS))]
|
||||
|
||||
worst = 0.0
|
||||
checked = 0
|
||||
for key in synthetic.MARKER_KEYS:
|
||||
for t_global in (3.0, 6.5, 10.0, 14.25, 17.5): # inside every camera's valid span
|
||||
truth = np.array(synthetic.marker_position(key, t_global))
|
||||
rays = []
|
||||
for i, offset_ms in enumerate(config.SYNTH_OFFSETS_MS):
|
||||
t_video = config.t_video_from_global(t_global, offset_ms, 0.0)
|
||||
q, t, intr = synthetic._interp_pose(cam_poses[i], t_video)
|
||||
proj = synthetic.project_point(q, t, intr, truth)
|
||||
if proj is None:
|
||||
continue
|
||||
px, py, _ = proj
|
||||
if not (0.0 <= px < w and 0.0 <= py < h):
|
||||
continue # off this camera's frame
|
||||
fx, fy, cx, cy = intr
|
||||
rays.append(geometry.ray_from_pixel(q, t, fx, fy, cx, cy, px, py))
|
||||
assert len(rays) >= 2, f"{key} @ {t_global}s: need 2+ views, got {len(rays)}"
|
||||
point, gap = geometry.triangulate_rays(rays[0][0], rays[0][1], rays[1][0], rays[1][1])
|
||||
err = float(np.linalg.norm(point - truth))
|
||||
worst = max(worst, err)
|
||||
checked += 1
|
||||
assert checked >= 8
|
||||
# Noise-free synthetic geometry: recovery is essentially exact, far under the 0.3 M20 bar.
|
||||
assert worst < 1e-3, f"worst round-trip error {worst:.2e} (>1e-3)"
|
||||
|
||||
|
||||
@pytest.mark.skipif(not _HAS_FFMPEG, reason="ffmpeg/ffprobe required to render the fixture video")
|
||||
def test_rendered_marker_a_is_the_dominant_magenta_blob(tmp_path):
|
||||
"""The base-muting contract (BASE_SATURATION / BASE_BRIGHTNESS) actually renders a solvable
|
||||
fixture: testsrc2's own solid magenta/cyan colour bars must NOT out-mass marker A's disc, or
|
||||
lane H's colour detector locks onto a bar and M19/M20 fail. This emulates that detector —
|
||||
largest saturated-magenta blob in a real rendered frame — and asserts it lands on the disc.
|
||||
Guards against a future un-muting of the base swamping detection.
|
||||
"""
|
||||
import cv2
|
||||
|
||||
dur = 2.0
|
||||
synthetic.build(base_dir=tmp_path, duration_s=dur, run_ffmpeg=True)
|
||||
try:
|
||||
w, h = config.SYNTH_VIDEO_W, config.SYNTH_VIDEO_H
|
||||
poses = synthetic.camera_track(0, dur, float(config.SYNTH_FPS), w, h)
|
||||
t_video = 1.0
|
||||
q, t, intr = synthetic._interp_pose(poses, t_video)
|
||||
t_global = config.t_global_from_video(t_video, config.SYNTH_OFFSETS_MS[0], 0.0)
|
||||
proj = synthetic.project_point(q, t, intr,
|
||||
synthetic.marker_position(synthetic.MARKER_A_KEY, t_global))
|
||||
assert proj and 0 <= proj[0] < w and 0 <= proj[1] < h, "marker A not in cam0 frame at t=1s"
|
||||
|
||||
cap = cv2.VideoCapture(str(tmp_path / "raw" / "cam0.mp4"))
|
||||
cap.set(cv2.CAP_PROP_POS_MSEC, t_video * 1000.0)
|
||||
ok, img = cap.read()
|
||||
cap.release()
|
||||
assert ok, "could not read cam0 frame"
|
||||
|
||||
hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
|
||||
hh, ss, vv = hsv[:, :, 0].astype(int), hsv[:, :, 1], hsv[:, :, 2]
|
||||
# OpenCV-hue magenta ≈150, gated on high saturation AND value (pitfall #2).
|
||||
mask = (((hh >= 145) & (hh <= 155)) & (ss > 170) & (vv > 170)).astype(np.uint8)
|
||||
n, _lab, stats, cent = cv2.connectedComponentsWithStats(mask, 8)
|
||||
assert n > 1, "no saturated-magenta blob at all — marker A not rendered?"
|
||||
biggest = 1 + int(np.argmax(stats[1:, cv2.CC_STAT_AREA]))
|
||||
cx, cy = cent[biggest]
|
||||
dist = math.hypot(cx - proj[0], cy - proj[1])
|
||||
assert dist < 10.0, (
|
||||
f"dominant magenta blob at ({cx:.0f},{cy:.0f}) is {dist:.0f}px from marker A's "
|
||||
f"projected centre ({proj[0]:.0f},{proj[1]:.0f}) — base bars are swamping the disc"
|
||||
)
|
||||
finally:
|
||||
db.init_engine(config.DB_PATH) # build() rebound the engine; restore the shared one
|
||||
118
deploy/DEPLOY.md
118
deploy/DEPLOY.md
@ -1,118 +0,0 @@
|
||||
# Deploying Festival 4D to digalot.fyi/festifun
|
||||
|
||||
Target: `dealgod@100.94.195.115` (tailscale), served under `https://digalot.fyi/festifun` by the
|
||||
existing web server that already terminates TLS for the domain.
|
||||
|
||||
The app runs as two pieces behind that web server:
|
||||
- **Static SPA** — the built frontend (`frontend/dist`), served under `/festifun/`.
|
||||
- **API backend** — uvicorn on `127.0.0.1:8000` (localhost-only), reached via `/festifun/api`
|
||||
and `/festifun/media`.
|
||||
|
||||
This topology was validated locally end-to-end against a proxy that mirrors the nginx config in
|
||||
this directory (static + prefix-stripping API/media proxy with Range) — the app loads, plays
|
||||
video, and renders the 3D scene correctly under the `/festifun` prefix.
|
||||
|
||||
---
|
||||
|
||||
## Deploy mode: fully open
|
||||
|
||||
Per the owner's decision, **all features are public and unauthenticated** — playback, 3D, and the
|
||||
mutating endpoints (annotate, correct event type, add/delete anchors). Be aware this means anyone
|
||||
who finds the URL can edit or delete the project's annotations/anchors/events. If that becomes a
|
||||
problem, add HTTP basic-auth on the `/festifun/` location, or run the backend read-only.
|
||||
|
||||
### AI classification & API keys — deliberately OFF on the public box
|
||||
`POST /api/events/detect` runs ffmpeg and, if a classifier key is present, a paid AI call. To
|
||||
avoid an unauthenticated public endpoint that spends real credits, **do not put `GEMINI_API_KEY`
|
||||
or the OpenRouter creds on the server.** Instead bake the moment labels in *before* deploy:
|
||||
|
||||
```bash
|
||||
# locally, with keys loaded (set -a; . ./.env; set +a):
|
||||
python -m festival4d events # writes AI labels into data/project.db
|
||||
```
|
||||
|
||||
Then ship that DB. On the server, `detect` still works but degrades to candidate-only (no spend),
|
||||
exactly as designed. If you *want* live classification on the public site, set the key in the
|
||||
systemd unit — but add rate-limiting first (nginx `limit_req`), or it's a bill-run-up vector.
|
||||
|
||||
---
|
||||
|
||||
## One-time server setup
|
||||
|
||||
```bash
|
||||
# on dealgod@100.94.195.115
|
||||
sudo mkdir -p /var/www/festifun
|
||||
sudo chown dealgod:dealgod /var/www/festifun
|
||||
# Python venv for the backend
|
||||
python3 -m venv /var/www/festifun/.venv
|
||||
```
|
||||
|
||||
## Build + push (run locally)
|
||||
|
||||
```bash
|
||||
# 1. Build the frontend for the /festifun prefix + same-origin API
|
||||
cd frontend
|
||||
FESTIVAL4D_BASE=/festifun/ VITE_API_BASE=/festifun npm run build
|
||||
|
||||
# 2. Generate the project to ship (synthetic demo, or your real footage pipeline first),
|
||||
# with labels baked in if you loaded keys:
|
||||
cd ..
|
||||
python -m festival4d synthetic # or: ingest -> sync -> reconstruct -> events
|
||||
|
||||
# 3. Push build + backend + data to the server
|
||||
rsync -az --delete frontend/dist/ dealgod@100.94.195.115:/var/www/festifun/dist/
|
||||
rsync -az --delete backend/ dealgod@100.94.195.115:/var/www/festifun/backend/
|
||||
rsync -az pyproject.toml dealgod@100.94.195.115:/var/www/festifun/
|
||||
rsync -az --delete data/ dealgod@100.94.195.115:/var/www/festifun/data/
|
||||
```
|
||||
|
||||
## Install backend deps + service (on the server)
|
||||
|
||||
```bash
|
||||
cd /var/www/festifun
|
||||
.venv/bin/pip install -e ".[dev]" # or a runtime-only extra if defined
|
||||
sudo cp backend/../deploy/festifun-api.service /etc/systemd/system/ # adjust path to the repo copy
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now festifun-api
|
||||
curl -s localhost:8000/api/health # -> {"status":"ok"}
|
||||
```
|
||||
|
||||
## Wire up the web server
|
||||
|
||||
Append the blocks from `deploy/festifun.nginx.conf` into the existing `server { }` for
|
||||
digalot.fyi, adjusting `alias` paths to `/var/www/festifun/dist/`. Then:
|
||||
|
||||
```bash
|
||||
sudo nginx -t && sudo systemctl reload nginx
|
||||
```
|
||||
|
||||
Visit `https://digalot.fyi/festifun/`.
|
||||
|
||||
> Using Caddy instead of nginx? The equivalent is a `handle_path /festifun/*` block: `file_server`
|
||||
> for the SPA, `reverse_proxy 127.0.0.1:8000` for `/festifun/api/*` and `/festifun/media/*`
|
||||
> (Caddy strips the matched prefix with `handle_path`). Ask and I'll write the Caddyfile once I
|
||||
> can see which server is actually running there.
|
||||
|
||||
---
|
||||
|
||||
## Updating later
|
||||
|
||||
Re-run the build + rsync steps, then `sudo systemctl restart festifun-api` (only needed if the
|
||||
backend or data changed; a frontend-only change just needs the `dist` rsync).
|
||||
|
||||
## Notes / caveats
|
||||
|
||||
- **CORS** is irrelevant in this topology — everything is same-origin under `/festifun`. The
|
||||
dev-only wide CORS in `config.py` stays as-is; it doesn't affect the hosted site.
|
||||
- **Single project.** The app assumes one project at a time (SQLite at `data/project.db`).
|
||||
- The backend binds `127.0.0.1` only; nginx is the sole public entry point.
|
||||
|
||||
---
|
||||
|
||||
## Live capture on the public box: leave it OFF
|
||||
|
||||
`FESTIVAL4D_CAPTURE` is **not** set in `deploy/festifun-api.service`, so `/capture` and
|
||||
`/api/capture/*` are never mounted publicly — an open upload endpoint on a public host would let
|
||||
anyone write files to the server. Capture is for your local/tailnet machine (see the README's
|
||||
Capture section: `FESTIVAL4D_CAPTURE=1` + `tailscale serve`). Record locally, run the pipeline,
|
||||
then deploy the resulting project.
|
||||
@ -1,28 +0,0 @@
|
||||
# Festival 4D backend — systemd unit (uvicorn on 127.0.0.1:8000, localhost-only)
|
||||
#
|
||||
# Install: sudo cp deploy/festifun-api.service /etc/systemd/system/
|
||||
# sudo systemctl daemon-reload && sudo systemctl enable --now festifun-api
|
||||
# Logs: journalctl -u festifun-api -f
|
||||
#
|
||||
# The service binds 127.0.0.1 only — nginx is the sole public entry point. Adjust User,
|
||||
# paths, and the venv location to match the server.
|
||||
|
||||
[Unit]
|
||||
Description=Festival 4D API (uvicorn)
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=dealgod
|
||||
WorkingDirectory=/var/www/festifun/backend
|
||||
# The app reads FESTIVAL4D_DATA_DIR for the project (db + media + point cloud).
|
||||
Environment=FESTIVAL4D_DATA_DIR=/var/www/festifun/data
|
||||
# NOTE: no GEMINI_API_KEY / OPENROUTER creds here on purpose — see DEPLOY.md "AI classification".
|
||||
# Moment labels are baked into the shipped DB by running `events` locally before deploy, so the
|
||||
# public box needs no keys and POST /api/events/detect degrades to candidates-only (no spend).
|
||||
ExecStart=/var/www/festifun/.venv/bin/python -m uvicorn festival4d.api:app --host 127.0.0.1 --port 8000
|
||||
Restart=on-failure
|
||||
RestartSec=3
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@ -1,46 +0,0 @@
|
||||
# Festival 4D — nginx location block for hosting under digalot.fyi/festifun
|
||||
#
|
||||
# Drop these `location` blocks inside the existing `server { }` for digalot.fyi (the one that
|
||||
# already terminates TLS for the domain). The app is served entirely under the /festifun prefix:
|
||||
# /festifun/ -> static SPA build (frontend/dist)
|
||||
# /festifun/api/... -> uvicorn backend on 127.0.0.1:8000 (prefix stripped)
|
||||
# /festifun/media/... -> uvicorn backend (video files, Range-capable)
|
||||
#
|
||||
# Adjust FESTIFUN_ROOT to wherever you rsync the build (see deploy/DEPLOY.md).
|
||||
|
||||
# --- static SPA (built with FESTIVAL4D_BASE=/festifun/) ---
|
||||
location /festifun/ {
|
||||
alias /var/www/festifun/dist/;
|
||||
try_files $uri $uri/ /festifun/index.html; # SPA fallback
|
||||
}
|
||||
|
||||
# bare /festifun -> /festifun/
|
||||
location = /festifun {
|
||||
return 301 /festifun/;
|
||||
}
|
||||
|
||||
# --- API: strip the /festifun prefix, proxy to uvicorn ---
|
||||
# The trailing slash on proxy_pass performs the prefix strip:
|
||||
# /festifun/api/manifest -> http://127.0.0.1:8000/api/manifest
|
||||
location /festifun/api/ {
|
||||
proxy_pass http://127.0.0.1:8000/api/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_read_timeout 300s; # POST /api/events/detect runs ffmpeg + (optional) an AI call
|
||||
}
|
||||
|
||||
# --- media: video files, Range-capable (required for <video> seeking) ---
|
||||
location /festifun/media/ {
|
||||
proxy_pass http://127.0.0.1:8000/media/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header Range $http_range; # forward Range for seeking
|
||||
proxy_set_header If-Range $http_if_range;
|
||||
proxy_force_ranges on;
|
||||
}
|
||||
|
||||
# Optional, faster alternative for /festifun/media/: serve the files directly instead of
|
||||
# proxying (uvicorn StaticFiles already does Range, but nginx is faster for large media):
|
||||
# location /festifun/media/ { alias /var/www/festifun/data/raw/; }
|
||||
@ -1,77 +0,0 @@
|
||||
# Festival 4D — Future Ideas
|
||||
|
||||
Parking lot for post-v0.1.0 extensions (spec §5 future-extensions). Not committed work — a place
|
||||
to capture direction so the prototype's scope stays honest. Roughly ordered by payoff-to-effort.
|
||||
|
||||
> **Status 2026-07-17 (evening):** splatting ✅ shipped (see `modelbeast-crossover.md`);
|
||||
> spatial audio ✅ shipped (🎧, WebAudio master clock). Cinematic export ✅ **M12**,
|
||||
> multi-modal audio analysis ✅ **M10/M11**, anchor manager ✅ **M13**, path persistence ✅
|
||||
> **M16** — all SHIPPED in phase 5 (`plan/20-phase5.md`, suite floor 189). Still parked here:
|
||||
> realtime ingest, object/artist tracking, per-project workspaces, sync-graph visualization,
|
||||
> per-moment splats (crossover doc §3), and the new hardware ideas below.
|
||||
|
||||
## Neural rendering — Gaussian Splatting
|
||||
Replace the sparse COLMAP point cloud with a **3D Gaussian Splatting** model for photorealistic
|
||||
free-roam instead of a dot cloud. COLMAP poses + images already feed splatting trainers
|
||||
(e.g. `gsplat`, Inria 3DGS) directly, so the reconstruction step's output is reusable. The viewer
|
||||
would swap `Points` for a splat renderer (a WebGL splat viewer, or bake to a mesh). Biggest visual
|
||||
upgrade; heaviest compute (needs a GPU training pass per project).
|
||||
|
||||
## Cinematic export — render the camera path to MP4
|
||||
M9 already produces keyframed god's-eye paths. Add a headless render: step `t_global` at a fixed
|
||||
fps, drive the Three.js camera along the path, capture frames (offscreen canvas → `captureStream`
|
||||
or server-side `puppeteer` + `ffmpeg`), and mux to MP4. Turns the app from a viewer into a
|
||||
"director's cut" clip exporter. Path JSON export/import is already in place, so this is additive.
|
||||
|
||||
## Real-time / streaming ingest
|
||||
Instead of pre-recorded files, ingest live phone feeds (WebRTC/RTMP). Requires: rolling audio
|
||||
sync on a sliding window (GCC-PHAT already windowed for drift — reuse it), incremental SfM or
|
||||
pose tracking against a pre-built map, and a streaming transport in the frontend. Large effort;
|
||||
the offline pipeline is the right foundation.
|
||||
|
||||
## Multi-modal audio analysis for moment detection
|
||||
Today's `events` step finds candidates from RMS + spectral flux, then a VLM classifies video.
|
||||
Add **audio-native** analysis in parallel: beat/tempo tracking, drop detection from
|
||||
spectrograms, song-boundary segmentation. Fuse audio-derived and vision-derived labels for
|
||||
higher-confidence events and beat-aligned auto camera cuts. Cheap; complements the existing
|
||||
classifier providers.
|
||||
|
||||
## Object / artist tracking
|
||||
Extend M8 (manual bbox → 3D anchor) with automatic per-frame tracking (e.g. a detector +
|
||||
tracker), so an anchor follows the lead performer across time instead of being static. Enables
|
||||
a "follow artist" auto-camera mode in the 3D viewer.
|
||||
|
||||
## Stereo / spatial-video ingest (split the eyes, pin the scale)
|
||||
Spatial video (iPhone 15 Pro+ MV-HEVC, QooCam, Canon dual-fisheye) is two synced views with a
|
||||
**factory-known baseline**. Ingest tweak: detect multi-view files and split L/R into two
|
||||
pipeline videos (ffmpeg can demux the streams; identical offset, shared audio). Payoff beyond
|
||||
one extra viewpoint: COLMAP reconstructions are scale-free, and a known stereo baseline anchors
|
||||
**metric scale** for the whole scene — real-meter units for anchors, paths, and spatial audio
|
||||
distances. Cheap ingest change, high leverage.
|
||||
|
||||
## Depth-camera seeding (small venues only)
|
||||
RGB-D cams (ZED 2i, OAK-D, LiDAR phones) can seed/regularize splat training (depth-supervised
|
||||
3DGS → fewer floaters, faster convergence) and densify the point cloud without SfM guessing.
|
||||
Honest constraint: consumer depth dies past ~10–15 m (LiDAR ~5 m), so this only pays at club /
|
||||
small-venue range, not a festival main stage. Ingest: accept a per-video depth track or
|
||||
per-frame depth PNGs alongside the RGB.
|
||||
|
||||
## Microcontroller rig kit (Arduino as crew, not camera)
|
||||
An ESP32-CAM is not concert-grade — but a microcontroller on the rig earns its place three ways:
|
||||
- **Sync/calibration beacon:** an LED blinking a known pattern visible to all cameras =
|
||||
visual ground truth to validate the audio sync (and a free calibration target).
|
||||
- **Servo pan/tilt or slider mount:** one slowly-sweeping fixed camera contributes rich
|
||||
parallax from a single device; the per-frame pose track already handles moving cameras.
|
||||
- **Record start/stop trigger** for the whole rig.
|
||||
DIY camera *nodes* should be Raspberry Pi + **Global Shutter** camera module (kills
|
||||
rolling-shutter wobble under fast stage lights) pushing into the existing `/capture` endpoint
|
||||
(`FESTIVAL4D_CAPTURE=1`) — which was built for exactly this.
|
||||
|
||||
## Quality-of-life
|
||||
- **Standalone anchor manager** — the delete UI currently lives in the event correction panel;
|
||||
a dedicated always-available anchor list would decouple anchor management from events.
|
||||
- **Persist camera paths server-side** — paths are JSON export/import only; a `paths` table +
|
||||
endpoints would let them live with the project.
|
||||
- **Per-project workspaces** — the app assumes one project at a time (SQLite at `data/project.db`).
|
||||
- **Sync-graph visualization** — show which videos aligned into which connected component when
|
||||
some clips share no audio.
|
||||
@ -1,24 +0,0 @@
|
||||
# Festival 4D × MODELBEAST — crossover map
|
||||
|
||||
*2026-07-16. Both projects live on the same fleet (M3 Ultra 256GB primary, M1 Ultra 128GB, M4 Pro 24GB — see MODELBEAST's `CLUSTER.md`). They stay separate products — asset factory vs. experience viewer — but share infrastructure and pipelines.*
|
||||
|
||||
## Shipped: the splat upgrade (ideas.md #1) ✅
|
||||
|
||||
`docs/ideas.md`'s top item ("replace the sparse point cloud with 3D Gaussian Splatting") is now wired:
|
||||
|
||||
- **Backend:** `GET /api/splat` serves `data/work/splat.ply` when present; `/api/manifest` reports `has_splat`.
|
||||
- **Viewer:** `scene3d.js` lazy-loads `@mkkellogg/gaussian-splats-3d` and renders the splat as a `DropInViewer` in the existing scene (camera rigs, anchors, and fly-paths overlay as before). Falls back to the point cloud when absent or on load error.
|
||||
- **Training:** `scripts/splat_via_modelbeast.sh` sends this project's frames to the MODELBEAST queue (`colmap_poses` → `brush_train` on whichever GPU node is free) and drops the trained `.ply` into place. No local GPU blocking; the farm handles it.
|
||||
|
||||
## The bigger crossover map
|
||||
|
||||
1. **Fleet compute for every heavy step.** MODELBEAST's queue pools gpu (1 job/Mac) and cpu lanes across all three nodes. Festival 4D's COLMAP/splat/classification steps can submit there (`mb` CLI + token) instead of hogging the local machine.
|
||||
2. **Poor-man's volumetric capture** (with CorridorKey): several phones film one subject on a green screen → *our GCC-PHAT audio sync* aligns them with no clapboard → CorridorKey (neural keyer, tuned on this fleet — see MODELBEAST's `CORRIDORKEY.md`) extracts true-edge mattes per synced frame → per-instant multi-view cutouts → 3D-gen or per-moment splats. DIY volumetric video from phones; every part exists in-house today.
|
||||
3. **Per-moment splats = pragmatic "4D".** True temporal 4DGS has no credible Mac port yet (honest limit). But we already detect *moments*: pool all cameras' frames in a ±1s window around each moment → COLMAP → splat → one splat per moment → scrub between them on the timeline. Static splats, time-indexed — feels 4D where it counts. Caveat: needs decent multi-camera coverage; 3 fan phones is thin.
|
||||
4. **Back-flow to MODELBEAST:** our audio-sync module is the missing piece for any multi-camera capture rig there; our moment detection could auto-pick hero frames for video→3D.
|
||||
5. **GVM wildcard:** CorridorKey's ~80GB auto-matting module (untested on Metal, fits only the M3 256GB) would key *real concert footage* — performer/crowd separation without green screens — feeding use case 2 with actual festival material.
|
||||
|
||||
## Practical notes
|
||||
- The `mb` CLI needs `MB_HOST` + `MB_TOKEN` (MODELBEAST Settings → Users). The M3 primary queue is at `100.89.131.57:8777` on the tailnet.
|
||||
- Splat training cost: a ~40s 4K room clip → ~190 frames → COLMAP (~1 min, GLOMAP) → Brush 30k steps (~15 min on the M3). Concert scenes with crowds will be harder than rooms — expect floaters; SuperSplat can crop/clean.
|
||||
- License note for use case 2: CorridorKey is CC BY-NC-SA (non-commercial).
|
||||
@ -5,295 +5,39 @@
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>Festival 4D</title>
|
||||
<style>
|
||||
:root {
|
||||
color-scheme: dark;
|
||||
--bg: #0b0d12;
|
||||
--panel: #141824;
|
||||
--panel-2: #1c2233;
|
||||
--border: #232838;
|
||||
--text: #e6e8ee;
|
||||
--dim: #8a92a6;
|
||||
--ok: #59d499;
|
||||
--bad: #ff6b6b;
|
||||
--accent: #59d499;
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
html, body { height: 100%; }
|
||||
:root { color-scheme: dark; }
|
||||
body {
|
||||
margin: 0;
|
||||
font: 14px/1.45 -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
|
||||
background: var(--bg);
|
||||
color: var(--text);
|
||||
overflow: hidden;
|
||||
font: 15px/1.5 -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
|
||||
background: #0b0d12;
|
||||
color: #e6e8ee;
|
||||
padding: 2rem;
|
||||
}
|
||||
#app { display: flex; flex-direction: column; height: 100vh; }
|
||||
|
||||
/* Top bar */
|
||||
#topbar {
|
||||
display: flex; align-items: baseline; gap: 0.6rem;
|
||||
padding: 0.5rem 0.9rem; border-bottom: 1px solid var(--border); flex: 0 0 auto;
|
||||
h1 { margin: 0 0 0.25rem; font-size: 1.6rem; }
|
||||
.sub { color: #8a92a6; margin-bottom: 1.5rem; }
|
||||
.card {
|
||||
background: #141824;
|
||||
border: 1px solid #232838;
|
||||
border-radius: 10px;
|
||||
padding: 1rem 1.25rem;
|
||||
margin-bottom: 1rem;
|
||||
max-width: 720px;
|
||||
}
|
||||
#topbar h1 { margin: 0; font-size: 1.05rem; font-weight: 700; letter-spacing: 0.2px; }
|
||||
#topbar .sub { color: var(--dim); font-size: 0.8rem; }
|
||||
|
||||
/* Stage: 3D on the left, video grid on the right */
|
||||
#stage { flex: 1 1 auto; display: flex; min-height: 0; }
|
||||
#scene-pane { flex: 1 1 55%; position: relative; min-width: 0; background: #0a0c11; }
|
||||
#scene3d { position: absolute; inset: 0; width: 100%; height: 100%; display: block; }
|
||||
#scene-controls {
|
||||
position: absolute; top: 0.6rem; left: 0.6rem; display: flex; gap: 0.4rem; z-index: 5;
|
||||
align-items: center; flex-wrap: wrap; max-width: calc(100% - 260px);
|
||||
}
|
||||
#scene-controls .btn { padding: 0.25rem 0.5rem; font-size: 0.78rem; }
|
||||
#key-count { color: var(--accent); font-variant-numeric: tabular-nums; }
|
||||
.ctl-sep { width: 1px; height: 18px; background: var(--border); margin: 0 0.15rem; }
|
||||
#scene-hint {
|
||||
position: absolute; bottom: 0.5rem; left: 0.6rem; color: var(--dim);
|
||||
font-size: 0.72rem; z-index: 5; pointer-events: none;
|
||||
background: rgba(10,12,17,0.6); padding: 0.2rem 0.45rem; border-radius: 5px;
|
||||
}
|
||||
#grid-pane {
|
||||
flex: 1 1 45%; min-width: 0; border-left: 1px solid var(--border);
|
||||
overflow: auto; background: #0d1017;
|
||||
}
|
||||
#video-grid {
|
||||
display: grid; gap: 6px; padding: 6px;
|
||||
grid-template-columns: repeat(auto-fit, minmax(240px, 1fr));
|
||||
}
|
||||
|
||||
.cell {
|
||||
position: relative; aspect-ratio: 16 / 9; background: #000;
|
||||
border: 1px solid var(--border); border-radius: 8px; overflow: hidden;
|
||||
transition: opacity 0.15s ease, border-color 0.15s ease;
|
||||
}
|
||||
.cell.out { opacity: 0.32; }
|
||||
.cell.audio-on { border-color: var(--accent); }
|
||||
.cell.followed { box-shadow: inset 0 0 0 2px var(--accent); }
|
||||
.cell-video { position: absolute; inset: 0; width: 100%; height: 100%; object-fit: contain; }
|
||||
.cell-overlay { position: absolute; inset: 0; pointer-events: none; }
|
||||
.cell-bar {
|
||||
position: absolute; left: 0; right: 0; bottom: 0; display: flex; align-items: center;
|
||||
gap: 0.35rem; padding: 0.25rem 0.4rem; font-size: 0.72rem;
|
||||
background: linear-gradient(transparent, rgba(6,8,14,0.85));
|
||||
}
|
||||
.cell-label { font-weight: 600; }
|
||||
.cell-status { color: var(--dim); margin-left: 0.1rem; }
|
||||
.cell-status.ok { color: var(--ok); }
|
||||
.cell-status.bad { color: var(--bad); }
|
||||
.cell-btn {
|
||||
margin-left: auto; background: rgba(255,255,255,0.08); color: var(--text);
|
||||
border: 1px solid var(--border); border-radius: 5px; cursor: pointer;
|
||||
width: 22px; height: 20px; font-size: 0.72rem; line-height: 1; padding: 0;
|
||||
}
|
||||
.cell-btn + .cell-btn { margin-left: 0.25rem; }
|
||||
.cell-btn:hover { background: rgba(255,255,255,0.16); }
|
||||
.cell.audio-on .audio-btn { background: var(--accent); color: #071018; border-color: var(--accent); }
|
||||
|
||||
/* Bottom bar: transport + timeline */
|
||||
#bottombar {
|
||||
flex: 0 0 auto; border-top: 1px solid var(--border); padding: 0.5rem 0.9rem 0.7rem;
|
||||
background: var(--panel);
|
||||
}
|
||||
#controls { display: flex; align-items: center; gap: 0.5rem; margin-bottom: 0.5rem; }
|
||||
.btn {
|
||||
background: var(--panel-2); color: var(--text); border: 1px solid var(--border);
|
||||
border-radius: 6px; cursor: pointer; padding: 0.3rem 0.6rem; font-size: 0.85rem;
|
||||
}
|
||||
.btn:hover { background: #262d42; }
|
||||
.btn.active { background: var(--accent); color: #071018; border-color: var(--accent); }
|
||||
#btn-play { min-width: 2.4rem; font-size: 1rem; }
|
||||
#time-display { font-variant-numeric: tabular-nums; color: var(--dim); margin-left: 0.2rem; }
|
||||
select.btn { padding-right: 0.4rem; }
|
||||
.spacer { flex: 1 1 auto; }
|
||||
.ctl-label { color: var(--dim); font-size: 0.78rem; }
|
||||
|
||||
/* Timeline */
|
||||
#timeline { position: relative; }
|
||||
.tl-track {
|
||||
position: relative; height: 26px; background: var(--panel-2);
|
||||
border: 1px solid var(--border); border-radius: 6px; cursor: pointer; overflow: visible;
|
||||
touch-action: none;
|
||||
}
|
||||
.tl-fill { position: absolute; left: 0; top: 0; bottom: 0; width: 0;
|
||||
background: rgba(89,212,153,0.14); border-radius: 6px 0 0 6px; }
|
||||
.tl-markers { position: absolute; inset: 0; }
|
||||
.tl-marker {
|
||||
position: absolute; top: 0; bottom: 0; width: 3px; margin-left: -1.5px;
|
||||
border-radius: 2px; cursor: pointer; box-shadow: 0 0 0 1px rgba(0,0,0,0.35);
|
||||
}
|
||||
.tl-marker:hover { width: 5px; margin-left: -2.5px; }
|
||||
.tl-playhead {
|
||||
position: absolute; top: -3px; bottom: -3px; width: 2px; margin-left: -1px;
|
||||
background: #fff; box-shadow: 0 0 6px rgba(255,255,255,0.6); pointer-events: none;
|
||||
}
|
||||
.tl-tooltip {
|
||||
position: absolute; bottom: 34px; transform: translateX(-50%);
|
||||
background: #0f1420; border: 1px solid var(--border); border-radius: 7px;
|
||||
padding: 0.4rem 0.55rem; font-size: 0.78rem; max-width: 260px; z-index: 20;
|
||||
pointer-events: none; box-shadow: 0 6px 20px rgba(0,0,0,0.5);
|
||||
}
|
||||
.tl-tooltip strong { text-transform: capitalize; }
|
||||
.tl-conf { color: var(--dim); font-size: 0.72rem; }
|
||||
.tl-legend {
|
||||
display: flex; flex-wrap: wrap; gap: 0.5rem 0.9rem; margin-top: 0.45rem;
|
||||
font-size: 0.72rem; color: var(--dim);
|
||||
}
|
||||
.tl-legend-item { display: inline-flex; align-items: center; gap: 0.3rem; text-transform: capitalize; }
|
||||
.tl-legend-item i { width: 10px; height: 10px; border-radius: 3px; display: inline-block; }
|
||||
|
||||
/* Dev sync overlay */
|
||||
#dev-overlay {
|
||||
position: fixed; top: 0.6rem; right: 0.6rem; z-index: 30;
|
||||
background: rgba(15,20,32,0.9); border: 1px solid var(--border); border-radius: 8px;
|
||||
padding: 0.45rem 0.6rem; font-size: 0.72rem; min-width: 132px;
|
||||
backdrop-filter: blur(4px);
|
||||
}
|
||||
#dev-overlay .hd { color: var(--dim); font-weight: 600; margin-bottom: 0.3rem; }
|
||||
.dev-row { display: flex; justify-content: space-between; gap: 0.8rem; font-variant-numeric: tabular-nums; }
|
||||
.ok { color: var(--ok); }
|
||||
.bad { color: var(--bad); }
|
||||
.dim { color: var(--dim); }
|
||||
|
||||
/* M8 annotation correction panel */
|
||||
#annot-panel {
|
||||
position: absolute; top: 0.6rem; right: 0.6rem; width: 232px; z-index: 8;
|
||||
background: rgba(15,20,32,0.94); border: 1px solid var(--border); border-radius: 9px;
|
||||
font-size: 0.78rem; box-shadow: 0 8px 26px rgba(0,0,0,0.5); backdrop-filter: blur(5px);
|
||||
}
|
||||
.ap-head {
|
||||
display: flex; align-items: center; gap: 0.4rem; padding: 0.45rem 0.55rem;
|
||||
border-bottom: 1px solid var(--border);
|
||||
}
|
||||
.ap-dot { width: 9px; height: 9px; border-radius: 3px; flex: 0 0 auto; }
|
||||
.ap-head strong { flex: 1 1 auto; font-size: 0.8rem; }
|
||||
.ap-x {
|
||||
background: none; border: none; color: var(--dim); cursor: pointer; font-size: 0.85rem;
|
||||
padding: 0 0.2rem;
|
||||
}
|
||||
.ap-x:hover { color: var(--text); }
|
||||
.ap-body { padding: 0.5rem 0.55rem 0.6rem; display: flex; flex-direction: column; gap: 0.45rem; }
|
||||
.ap-row { display: flex; align-items: center; justify-content: space-between; gap: 0.4rem; color: var(--dim); }
|
||||
.ap-type { flex: 1 1 auto; margin-left: 0.4rem; background: var(--panel-2); color: var(--text);
|
||||
border: 1px solid var(--border); border-radius: 5px; padding: 0.2rem 0.3rem; text-transform: capitalize; }
|
||||
.ap-desc { color: var(--text); line-height: 1.35; }
|
||||
.ap-meta { text-transform: capitalize; }
|
||||
.ap-annotate { width: 100%; text-align: center; }
|
||||
.ap-hint { line-height: 1.35; min-height: 1.1em; }
|
||||
.ap-anchors { border-top: 1px solid var(--border); margin-top: 0.15rem; padding-top: 0.4rem; }
|
||||
.ap-anchors-hd { font-size: 0.72rem; margin-bottom: 0.25rem; }
|
||||
.ap-anchors-list { max-height: 132px; overflow-y: auto; display: flex; flex-direction: column; gap: 0.15rem; }
|
||||
.ap-anchors-empty { font-size: 0.74rem; }
|
||||
.ap-anchor-row { display: flex; align-items: center; gap: 0.35rem; font-size: 0.74rem; }
|
||||
.ap-anchor-dot { width: 8px; height: 8px; border-radius: 3px; flex: 0 0 auto; }
|
||||
.ap-anchor-label { flex: 1 1 auto; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
|
||||
.ap-anchor-del {
|
||||
background: none; border: none; color: var(--dim); cursor: pointer; font-size: 0.75rem;
|
||||
padding: 0 0.15rem; flex: 0 0 auto;
|
||||
}
|
||||
.ap-anchor-del:hover { color: var(--bad); }
|
||||
|
||||
.annot-rect {
|
||||
position: absolute; border: 1.5px solid var(--accent); background: rgba(89,212,153,0.14);
|
||||
pointer-events: none; z-index: 6; border-radius: 2px;
|
||||
}
|
||||
|
||||
/* Capsule mode (M17, phase-5 contract #5): a static baked bundle has no write API, so
|
||||
everything marked .write-ui disappears when main.js sets body.capsule from the manifest. */
|
||||
body.capsule .write-ui { display: none !important; }
|
||||
|
||||
/* Loading / error */
|
||||
#loading {
|
||||
position: fixed; inset: 0; display: flex; align-items: center; justify-content: center;
|
||||
background: var(--bg); z-index: 100; color: var(--dim);
|
||||
}
|
||||
#loading .err { text-align: center; color: var(--text); }
|
||||
code { background: var(--panel-2); padding: 0.1rem 0.35rem; border-radius: 4px; }
|
||||
.ok { color: #59d499; }
|
||||
.bad { color: #ff6b6b; }
|
||||
table { border-collapse: collapse; width: 100%; }
|
||||
th, td { text-align: left; padding: 0.35rem 0.6rem; border-bottom: 1px solid #232838; }
|
||||
th { color: #8a92a6; font-weight: 600; }
|
||||
code { background: #1c2233; padding: 0.1rem 0.35rem; border-radius: 4px; }
|
||||
.pill { display: inline-block; padding: 0.1rem 0.5rem; border-radius: 999px; font-size: 0.8rem; }
|
||||
.pill.ok { background: rgba(89,212,153,0.15); }
|
||||
.pill.bad { background: rgba(255,107,107,0.15); }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div id="app">
|
||||
<div id="topbar">
|
||||
<h1>Festival 4D</h1>
|
||||
<span class="sub">synchronized multi-cam replay · 3D viewer · x-ray overlays</span>
|
||||
<span class="spacer"></span>
|
||||
<!-- shown only when the backend has capture enabled (FESTIVAL4D_CAPTURE=1) -->
|
||||
<a id="capture-link" class="btn" style="display:none; text-decoration:none"
|
||||
title="Record from webcams / phones into this project" target="_blank">📹 Capture</a>
|
||||
</div>
|
||||
|
||||
<div id="stage">
|
||||
<div id="scene-pane">
|
||||
<canvas id="scene3d"></canvas>
|
||||
<div id="scene-controls">
|
||||
<button class="btn active" id="btn-roam" title="Free roam (Esc)">Free roam</button>
|
||||
<span class="ctl-sep"></span>
|
||||
<button class="btn" id="btn-key" title="Add camera keyframe here (K)">＋ Key <b id="key-count">0</b></button>
|
||||
<button class="btn" id="btn-path" title="Play the keyframed camera path">▶ Path</button>
|
||||
<button class="btn" id="btn-path-export" title="Export path as JSON">⤓</button>
|
||||
<button class="btn" id="btn-path-import" title="Import path JSON">⤒</button>
|
||||
<button class="btn" id="btn-path-clear" title="Clear all keyframes">✕</button>
|
||||
<!-- Phase 5: hidden until the owning lane's module reports ready (plan/20-phase5.md) -->
|
||||
<button class="btn" id="btn-director" style="display:none"
|
||||
title="Auto-director: build a camera path from the detected events (M11)">🎬 Auto-path</button>
|
||||
<button class="btn write-ui" id="btn-path-save" style="display:none"
|
||||
title="Save the current camera path to the project (M16)">💾</button>
|
||||
<select class="btn" id="path-load-select" style="display:none"
|
||||
title="Load a saved camera path (M16)"></select>
|
||||
<button class="btn write-ui" id="btn-path-del" style="display:none"
|
||||
title="Delete the selected saved path (M16)">🗑</button>
|
||||
<span class="ctl-sep"></span>
|
||||
<button class="btn" id="btn-spatial" style="display:none"
|
||||
title="Spatial audio: hear every camera from its position in the scene">🎧 3D audio</button>
|
||||
<button class="btn" id="btn-export" style="display:none"
|
||||
title="Export the camera path as a cinematic webm (M12)">⏺ Export</button>
|
||||
<button class="btn" id="btn-photo" style="display:none"
|
||||
title="Photo mode: hi-res still of the 3D scene (M14, P)">📷</button>
|
||||
<button class="btn" id="btn-fx" style="display:none"
|
||||
title="Moment FX: bursts and pulses on event markers (M15)">✨</button>
|
||||
<button class="btn" id="btn-anchors" style="display:none"
|
||||
title="Anchors & friend tags (M13)">⚓</button>
|
||||
<button class="btn" id="btn-friends" style="display:none"
|
||||
title="Friend tracks: ribbons, moving labels, follow-a-friend (M21/M22)">👣 Friends</button>
|
||||
<input type="file" id="path-file" accept="application/json,.json" hidden />
|
||||
</div>
|
||||
<div id="scene-hint">drag to orbit · 1–9 snap to camera · K add keyframe · Esc detach</div>
|
||||
<div id="annot-panel" class="write-ui"></div>
|
||||
<!-- M13 anchor manager mounts here (lane F); hidden until anchorPanel.js is filled -->
|
||||
<div id="anchor-panel" style="display:none"></div>
|
||||
<!-- M21 friends panel mounts here (lane J); hidden until friendTracks.js is filled -->
|
||||
<div id="friends-panel" style="display:none"></div>
|
||||
</div>
|
||||
<div id="grid-pane">
|
||||
<div id="video-grid"></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div id="bottombar">
|
||||
<div id="controls">
|
||||
<button class="btn" id="btn-back" title="Back 1s (←)">⏪</button>
|
||||
<button class="btn" id="btn-play" title="Play/Pause (Space)">▶</button>
|
||||
<button class="btn" id="btn-fwd" title="Forward 1s (→)">⏩</button>
|
||||
<span id="time-display">0:00.0 / 0:00.0</span>
|
||||
<span class="spacer"></span>
|
||||
<span class="ctl-label">speed</span>
|
||||
<select class="btn" id="rate-select">
|
||||
<option value="0.25">0.25×</option>
|
||||
<option value="0.5">0.5×</option>
|
||||
<option value="1" selected>1×</option>
|
||||
<option value="2">2×</option>
|
||||
</select>
|
||||
</div>
|
||||
<div id="timeline"></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div id="dev-overlay">
|
||||
<div class="hd">sync error</div>
|
||||
<div id="dev-rows"></div>
|
||||
</div>
|
||||
|
||||
<div id="loading">Loading project…</div>
|
||||
|
||||
<h1>Festival 4D</h1>
|
||||
<div class="sub">Foundation hello page — proves the API contract, CORS, and the frozen pose helper.</div>
|
||||
<div id="app"><div class="card">Loading…</div></div>
|
||||
<script type="module" src="/src/main.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
|
||||
10
frontend/package-lock.json
generated
10
frontend/package-lock.json
generated
@ -8,7 +8,6 @@
|
||||
"name": "festival4d-frontend",
|
||||
"version": "0.1.0",
|
||||
"dependencies": {
|
||||
"@mkkellogg/gaussian-splats-3d": "^0.4.7",
|
||||
"three": "^0.170.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
@ -457,15 +456,6 @@
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@mkkellogg/gaussian-splats-3d": {
|
||||
"version": "0.4.7",
|
||||
"resolved": "https://registry.npmjs.org/@mkkellogg/gaussian-splats-3d/-/gaussian-splats-3d-0.4.7.tgz",
|
||||
"integrity": "sha512-0vy9/i9sJLFH/v3WJZ4axCsqjkToe8UsV3xY7bvK5EUC0akiRsWZODoCiSzpxhTLNyzSKTsyQKozIFeNA5RWRA==",
|
||||
"license": "MIT",
|
||||
"peerDependencies": {
|
||||
"three": ">=0.160.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@rollup/rollup-android-arm-eabi": {
|
||||
"version": "4.62.2",
|
||||
"resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm-eabi/-/rollup-android-arm-eabi-4.62.2.tgz",
|
||||
|
||||
@ -9,7 +9,6 @@
|
||||
"preview": "vite preview"
|
||||
},
|
||||
"dependencies": {
|
||||
"@mkkellogg/gaussian-splats-3d": "^0.4.7",
|
||||
"three": "^0.170.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
|
||||
@ -1,332 +0,0 @@
|
||||
// Anchor manager & friend tags (spec M13, lane F).
|
||||
//
|
||||
// A collapsible panel (pre-wired container #anchor-panel, toggle #btn-anchors) listing every
|
||||
// anchor in state.anchors: color dot (recolor), label (rename), jump-to (scene3d.focusOn),
|
||||
// delete. Mutations go through PATCH/DELETE ${state.apiBase}/api/anchors/{id}; after any
|
||||
// mutation we update state.anchors and emit("anchors-changed") so scene3d + the video
|
||||
// overlays refresh (main.js wires that event to scene3d.refreshAnchors; overlays read
|
||||
// state.anchors every frame).
|
||||
//
|
||||
// ＋ Tag (friend tags): drives the FROZEN M8 annotation flow WITHOUT an event.
|
||||
// annotate.js was written null-event safe on purpose: startAnnotateMode() only needs the
|
||||
// registered cells, and _submit POSTs /api/annotations with event_id: null, which the backend
|
||||
// resolves to an INDEPENDENT anchor ("Annotations with no event stay independent").
|
||||
// Flow: prompt for the friend's name -> annotator.close() (clears any currentEvent) ->
|
||||
// annotator.startAnnotateMode() -> the user drags a bbox over the friend on any video ->
|
||||
// the resolved anchor arrives via emit("anchors-changed", anchor) -> we PATCH the pending
|
||||
// name + a palette color onto it and stop annotate mode. Esc (or ＋ Tag again) cancels.
|
||||
//
|
||||
// Capsule mode (contract #5): every MUTATING control carries class="write-ui" so a baked
|
||||
// bundle hides it; the read-only list + jump-to stay available.
|
||||
|
||||
import { state, emit, on } from "./state.js";
|
||||
import { scene3d } from "./scene3d.js";
|
||||
import { annotator } from "./annotate.js";
|
||||
|
||||
const DEFAULT_COLOR = "#59d499";
|
||||
// Friendly tag palette (mirrors the scene's per-video accent hues).
|
||||
const TAG_COLORS = ["#ff5d73", "#4dd0ff", "#c9a2ff", "#ffcf5d", "#7dffb0", "#ff9d5d"];
|
||||
|
||||
const CSS = `
|
||||
#anchor-panel {
|
||||
position: absolute; top: 3.1rem; left: 0.6rem; width: 244px; z-index: 7;
|
||||
background: rgba(15,20,32,0.94); border: 1px solid var(--border); border-radius: 9px;
|
||||
font-size: 0.78rem; box-shadow: 0 8px 26px rgba(0,0,0,0.5); backdrop-filter: blur(5px);
|
||||
max-height: min(60vh, 420px); display: flex; flex-direction: column;
|
||||
}
|
||||
.anch-head {
|
||||
display: flex; align-items: center; gap: 0.4rem; padding: 0.45rem 0.55rem;
|
||||
border-bottom: 1px solid var(--border); flex: 0 0 auto;
|
||||
}
|
||||
.anch-head strong { flex: 1 1 auto; font-size: 0.8rem; }
|
||||
.anch-n { color: var(--dim); font-weight: 400; }
|
||||
.anch-x {
|
||||
background: none; border: none; color: var(--dim); cursor: pointer;
|
||||
font-size: 0.85rem; padding: 0 0.2rem;
|
||||
}
|
||||
.anch-x:hover { color: var(--text); }
|
||||
.anch-hint { padding: 0.3rem 0.55rem 0; line-height: 1.35; min-height: 0; flex: 0 0 auto; }
|
||||
.anch-hint:empty { display: none; }
|
||||
.anch-list {
|
||||
padding: 0.4rem 0.45rem 0.5rem; overflow-y: auto; display: flex;
|
||||
flex-direction: column; gap: 0.12rem; flex: 1 1 auto;
|
||||
}
|
||||
.anch-empty { color: var(--dim); padding: 0.1rem 0.15rem 0.2rem; line-height: 1.4; }
|
||||
.anch-row {
|
||||
display: flex; align-items: center; gap: 0.4rem; padding: 0.22rem 0.3rem;
|
||||
border-radius: 6px;
|
||||
}
|
||||
.anch-row:hover { background: rgba(255,255,255,0.05); }
|
||||
.anch-dot {
|
||||
width: 12px; height: 12px; border-radius: 4px; flex: 0 0 auto; position: relative;
|
||||
box-shadow: 0 0 0 1px rgba(0,0,0,0.4);
|
||||
}
|
||||
.anch-color {
|
||||
position: absolute; inset: 0; opacity: 0; width: 100%; height: 100%;
|
||||
cursor: pointer; padding: 0; border: none;
|
||||
}
|
||||
.anch-label {
|
||||
flex: 1 1 auto; white-space: nowrap; overflow: hidden; text-overflow: ellipsis;
|
||||
cursor: pointer;
|
||||
}
|
||||
.anch-label:hover { color: var(--accent); }
|
||||
.anch-btn {
|
||||
background: none; border: none; color: var(--dim); cursor: pointer;
|
||||
font-size: 0.78rem; padding: 0 0.15rem; flex: 0 0 auto; line-height: 1;
|
||||
}
|
||||
.anch-btn:hover { color: var(--text); }
|
||||
.anch-del:hover { color: var(--bad); }
|
||||
`;
|
||||
|
||||
function esc(s) {
|
||||
return String(s ?? "").replace(
|
||||
/[&<>"']/g,
|
||||
(c) => ({ "&": "&", "<": "<", ">": ">", '"': """, "'": "'" }[c])
|
||||
);
|
||||
}
|
||||
|
||||
async function patchAnchor(id, body) {
|
||||
const resp = await fetch(`${state.apiBase}/api/anchors/${id}`, {
|
||||
method: "PATCH",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify(body),
|
||||
});
|
||||
if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
|
||||
return resp.json();
|
||||
}
|
||||
|
||||
/** Replace (or append) an anchor in state.anchors and notify every consumer. */
|
||||
function upsertIntoState(anchor) {
|
||||
const i = state.anchors.findIndex((a) => a.id === anchor.id);
|
||||
if (i >= 0) state.anchors[i] = anchor;
|
||||
else state.anchors.push(anchor);
|
||||
emit("anchors-changed", anchor);
|
||||
}
|
||||
|
||||
export const anchorPanel = {
|
||||
ready: false,
|
||||
open: false,
|
||||
_panel: null,
|
||||
_btn: null,
|
||||
_pendingTag: null, // { name, color, knownIds:Set<number> } while a friend tag is in flight
|
||||
|
||||
init() {
|
||||
this._btn = document.getElementById("btn-anchors");
|
||||
this._panel = document.getElementById("anchor-panel");
|
||||
if (!this._btn || !this._panel) return; // pre-wired DOM missing — stay hidden, degrade
|
||||
|
||||
const style = document.createElement("style");
|
||||
style.textContent = CSS;
|
||||
document.head.appendChild(style);
|
||||
|
||||
this._panel.innerHTML = `
|
||||
<div class="anch-head">
|
||||
<strong>⚓ Anchors <span class="anch-n"></span></strong>
|
||||
<button class="btn anch-tag write-ui" title="Tag a friend: drag a box over them on any video">＋ Tag</button>
|
||||
<button class="anch-x" title="Collapse">✕</button>
|
||||
</div>
|
||||
<div class="anch-hint dim"></div>
|
||||
<div class="anch-list"></div>`;
|
||||
|
||||
this._panel.querySelector(".anch-x").addEventListener("click", () => this.toggle(false));
|
||||
this._panel.querySelector(".anch-tag").addEventListener("click", () => {
|
||||
if (this._pendingTag) this._cancelTag();
|
||||
else this._startTag();
|
||||
});
|
||||
this._btn.addEventListener("click", () => this.toggle());
|
||||
|
||||
// Any anchor mutation anywhere (this panel, annotate.js, a pending friend tag resolving).
|
||||
on("anchors-changed", (payload) => this._onAnchorsChanged(payload));
|
||||
|
||||
// Esc cancels a pending tag (main.js also maps Esc to free-roam; both are fine together).
|
||||
window.addEventListener("keydown", (e) => {
|
||||
if (e.code === "Escape" && this._pendingTag) this._cancelTag();
|
||||
});
|
||||
|
||||
this._render();
|
||||
this._btn.style.display = ""; // unhide: the module is live
|
||||
this.ready = true;
|
||||
},
|
||||
|
||||
toggle(force) {
|
||||
if (!this._panel) return;
|
||||
this.open = force != null ? !!force : !this.open;
|
||||
this._panel.style.display = this.open ? "" : "none";
|
||||
this._btn?.classList.toggle("active", this.open);
|
||||
if (this.open) this._render();
|
||||
},
|
||||
|
||||
// --- friend tag flow ------------------------------------------------------------------
|
||||
|
||||
_startTag() {
|
||||
let name;
|
||||
try {
|
||||
name = window.prompt("Friend's name:", "");
|
||||
} catch {
|
||||
name = null; // prompt() unavailable (headless) — nothing to do
|
||||
}
|
||||
name = (name || "").trim();
|
||||
if (!name) return;
|
||||
const color = TAG_COLORS[state.anchors.length % TAG_COLORS.length];
|
||||
this._pendingTag = { name, color, knownIds: new Set(state.anchors.map((a) => a.id)) };
|
||||
try {
|
||||
annotator.close(); // clear any currentEvent so the annotation POSTs event_id: null
|
||||
annotator.startAnnotateMode();
|
||||
} catch (err) {
|
||||
console.warn("[anchorPanel] could not start annotate mode", err);
|
||||
this._pendingTag = null;
|
||||
this._hint(`<span class="bad">tag mode failed: ${esc(err.message)}</span>`);
|
||||
return;
|
||||
}
|
||||
this._tagButton(true);
|
||||
this._hint(`drag a box over <b>${esc(name)}</b> on any video · Esc cancels`);
|
||||
this.toggle(true);
|
||||
},
|
||||
|
||||
_cancelTag() {
|
||||
this._pendingTag = null;
|
||||
try {
|
||||
annotator.stopAnnotateMode();
|
||||
} catch {
|
||||
/* already stopped */
|
||||
}
|
||||
this._tagButton(false);
|
||||
this._hint("");
|
||||
},
|
||||
|
||||
_tagButton(active) {
|
||||
const b = this._panel?.querySelector(".anch-tag");
|
||||
if (!b) return;
|
||||
b.textContent = active ? "✕ Cancel" : "＋ Tag";
|
||||
b.classList.toggle("active", active);
|
||||
},
|
||||
|
||||
_onAnchorsChanged(payload) {
|
||||
const tag = this._pendingTag;
|
||||
if (tag && payload && payload.id != null && !tag.knownIds.has(payload.id)) {
|
||||
// The bbox the user just drew resolved into a brand-new anchor: name + color it.
|
||||
this._pendingTag = null;
|
||||
this._finishTag(tag, payload);
|
||||
}
|
||||
this._render();
|
||||
},
|
||||
|
||||
async _finishTag(tag, anchor) {
|
||||
try {
|
||||
annotator.stopAnnotateMode();
|
||||
} catch {
|
||||
/* fine */
|
||||
}
|
||||
this._tagButton(false);
|
||||
try {
|
||||
const updated = await patchAnchor(anchor.id, { label: tag.name, color: tag.color });
|
||||
upsertIntoState(updated);
|
||||
this._hint(`tagged <b>${esc(tag.name)}</b> — visible in 3D and on every video`);
|
||||
} catch (err) {
|
||||
this._hint(`<span class="bad">naming the tag failed: ${esc(err.message)}</span>`);
|
||||
}
|
||||
},
|
||||
|
||||
// --- list rendering + row actions ------------------------------------------------------
|
||||
|
||||
_hint(html) {
|
||||
const h = this._panel?.querySelector(".anch-hint");
|
||||
if (h) h.innerHTML = html;
|
||||
},
|
||||
|
||||
_render() {
|
||||
if (!this._panel) return;
|
||||
const n = this._panel.querySelector(".anch-n");
|
||||
if (n) n.textContent = `(${state.anchors.length})`;
|
||||
const list = this._panel.querySelector(".anch-list");
|
||||
if (!list) return;
|
||||
if (state.anchors.length === 0) {
|
||||
list.innerHTML =
|
||||
`<div class="anch-empty">no anchors yet — <b>＋ Tag</b> a friend,` +
|
||||
` or annotate an event on the timeline</div>`;
|
||||
return;
|
||||
}
|
||||
list.innerHTML = state.anchors
|
||||
.map((a) => {
|
||||
const color = a.color || DEFAULT_COLOR;
|
||||
return (
|
||||
`<div class="anch-row" data-id="${a.id}">` +
|
||||
`<span class="anch-dot" style="background:${esc(color)}">` +
|
||||
`<input type="color" class="anch-color write-ui" value="${esc(color)}" title="Recolor"></span>` +
|
||||
`<span class="anch-label" title="Jump to ${esc(a.label || "anchor")}">${esc(a.label) || "(unnamed)"}</span>` +
|
||||
`<button class="anch-btn anch-jump" title="Jump to in 3D">◎</button>` +
|
||||
`<button class="anch-btn anch-rename write-ui" title="Rename">✎</button>` +
|
||||
`<button class="anch-btn anch-del write-ui" title="Delete">✕</button></div>`
|
||||
);
|
||||
})
|
||||
.join("");
|
||||
for (const row of list.querySelectorAll(".anch-row")) {
|
||||
const id = Number(row.dataset.id);
|
||||
row.querySelector(".anch-jump").addEventListener("click", () => this._jump(id));
|
||||
row.querySelector(".anch-label").addEventListener("click", () => this._jump(id));
|
||||
row.querySelector(".anch-rename").addEventListener("click", () => this._rename(id));
|
||||
row.querySelector(".anch-del").addEventListener("click", () => this._delete(id));
|
||||
const colorInput = row.querySelector(".anch-color");
|
||||
// Live-preview the dot while the picker is open; PATCH once on commit.
|
||||
colorInput.addEventListener("input", () => {
|
||||
row.querySelector(".anch-dot").style.background = colorInput.value;
|
||||
});
|
||||
colorInput.addEventListener("change", () => this._recolor(id, colorInput.value));
|
||||
}
|
||||
},
|
||||
|
||||
_find(id) {
|
||||
return state.anchors.find((a) => a.id === id);
|
||||
},
|
||||
|
||||
_jump(id) {
|
||||
const a = this._find(id);
|
||||
if (!a) return;
|
||||
try {
|
||||
scene3d.focusOn(a.x, a.y, a.z);
|
||||
} catch (err) {
|
||||
console.warn("[anchorPanel] focusOn failed", err);
|
||||
}
|
||||
},
|
||||
|
||||
async _rename(id) {
|
||||
const a = this._find(id);
|
||||
if (!a) return;
|
||||
let name;
|
||||
try {
|
||||
name = window.prompt("Rename anchor:", a.label || "");
|
||||
} catch {
|
||||
name = null;
|
||||
}
|
||||
if (name == null) return; // cancelled
|
||||
name = name.trim();
|
||||
if (!name || name === a.label) return;
|
||||
try {
|
||||
upsertIntoState(await patchAnchor(id, { label: name }));
|
||||
} catch (err) {
|
||||
this._hint(`<span class="bad">rename failed: ${esc(err.message)}</span>`);
|
||||
}
|
||||
},
|
||||
|
||||
async _recolor(id, color) {
|
||||
if (!this._find(id)) return;
|
||||
try {
|
||||
upsertIntoState(await patchAnchor(id, { color }));
|
||||
} catch (err) {
|
||||
this._hint(`<span class="bad">recolor failed: ${esc(err.message)}</span>`);
|
||||
this._render(); // roll the previewed dot back to the stored color
|
||||
}
|
||||
},
|
||||
|
||||
async _delete(id) {
|
||||
try {
|
||||
const resp = await fetch(`${state.apiBase}/api/anchors/${id}`, { method: "DELETE" });
|
||||
if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
|
||||
const i = state.anchors.findIndex((a) => a.id === id);
|
||||
if (i >= 0) state.anchors.splice(i, 1);
|
||||
emit("anchors-changed", null);
|
||||
} catch (err) {
|
||||
this._hint(`<span class="bad">delete failed: ${esc(err.message)}</span>`);
|
||||
}
|
||||
},
|
||||
};
|
||||
@ -1,313 +1,6 @@
|
||||
// Correction panel + bbox annotation → 3D (spec M8, integration phase).
|
||||
//
|
||||
// Clicking a timeline event opens a panel: it shows the classification, a dropdown to correct
|
||||
// the event_type (PATCH /api/events/{id}, source→'user'), and an "annotate location" mode.
|
||||
// In annotate mode the user drags a rectangle on ANY video's overlay; the normalized bbox is
|
||||
// POSTed to /api/annotations. The backend casts a ray through the bbox center and resolves a
|
||||
// 3D point (triangulate across two views of the same event, else nearest point-cloud point /
|
||||
// centroid-depth ray). The resolved anchor is pushed live into state so overlays + the 3D
|
||||
// scene show it immediately.
|
||||
// bbox drawing + submission → POST /api/annotations (spec M8).
|
||||
// STUB — this is phase-3 (integration) work; lane C leaves it as a clean seam. Drag a
|
||||
// rectangle on a video's overlay canvas → normalized bbox → POST /api/annotations; the
|
||||
// resolved anchor then appears live in overlays + the 3D scene.
|
||||
|
||||
import { state, emit, API_BASE } from "./state.js";
|
||||
import { contentRect } from "./videoGrid.js";
|
||||
import { eventColor } from "./timeline.js";
|
||||
|
||||
const EVENT_TYPES = [
|
||||
"bass_drop", "pyro", "confetti", "crowd_wave",
|
||||
"artist_moment", "light_show", "quiet_moment", "candidate", "other",
|
||||
];
|
||||
|
||||
export class Annotator {
|
||||
constructor() {
|
||||
this.panel = null;
|
||||
this.cells = [];
|
||||
this.currentEvent = null;
|
||||
this.annotating = false;
|
||||
this._rubber = null; // { cell, div, sx, sy }
|
||||
this._cleanup = [];
|
||||
}
|
||||
|
||||
init(panelEl, cells) {
|
||||
this.panel = panelEl;
|
||||
this.cells = cells;
|
||||
this.panel.style.display = "none";
|
||||
}
|
||||
|
||||
/** Open the correction panel for a timeline event. */
|
||||
openForEvent(ev) {
|
||||
this.currentEvent = ev;
|
||||
this.stopAnnotateMode();
|
||||
this._render();
|
||||
this.panel.style.display = "block";
|
||||
}
|
||||
|
||||
close() {
|
||||
this.stopAnnotateMode();
|
||||
this.currentEvent = null;
|
||||
if (this.panel) this.panel.style.display = "none";
|
||||
}
|
||||
|
||||
_render() {
|
||||
const ev = this.currentEvent;
|
||||
if (!ev) return;
|
||||
const options = EVENT_TYPES.map(
|
||||
(t) => `<option value="${t}"${t === ev.event_type ? " selected" : ""}>${t.replace(/_/g, " ")}</option>`
|
||||
).join("");
|
||||
this.panel.innerHTML = `
|
||||
<div class="ap-head">
|
||||
<span class="ap-dot" style="background:${eventColor(ev.event_type)}"></span>
|
||||
<strong>Event @ ${ev.t_global_s.toFixed(1)}s</strong>
|
||||
<button class="ap-x" title="Close">✕</button>
|
||||
</div>
|
||||
<div class="ap-body">
|
||||
<label class="ap-row">type
|
||||
<select class="ap-type">${options}</select>
|
||||
</label>
|
||||
<div class="ap-desc">${ev.description || "<span class='dim'>no description</span>"}</div>
|
||||
<div class="ap-meta dim">${ev.source}${ev.confidence != null ? ` · ${(ev.confidence * 100).toFixed(0)}%` : ""}</div>
|
||||
<button class="ap-annotate btn">📍 Annotate location</button>
|
||||
<div class="ap-hint dim"></div>
|
||||
<div class="ap-anchors">
|
||||
<div class="ap-anchors-hd dim">anchors <span class="ap-anchors-n"></span></div>
|
||||
<div class="ap-anchors-list"></div>
|
||||
</div>
|
||||
</div>`;
|
||||
|
||||
this.panel.querySelector(".ap-x").addEventListener("click", () => this.close());
|
||||
this.panel.querySelector(".ap-type").addEventListener("change", (e) =>
|
||||
this._correctType(e.target.value)
|
||||
);
|
||||
this.panel.querySelector(".ap-annotate").addEventListener("click", () =>
|
||||
this.annotating ? this.stopAnnotateMode() : this.startAnnotateMode()
|
||||
);
|
||||
this._renderAnchorList();
|
||||
}
|
||||
|
||||
/** Render the deletable anchor list in the panel (M8 / Phase 4). */
|
||||
_renderAnchorList() {
|
||||
const list = this.panel?.querySelector(".ap-anchors-list");
|
||||
const n = this.panel?.querySelector(".ap-anchors-n");
|
||||
if (!list) return;
|
||||
if (n) n.textContent = `(${state.anchors.length})`;
|
||||
if (state.anchors.length === 0) {
|
||||
list.innerHTML = `<div class="dim ap-anchors-empty">none yet</div>`;
|
||||
return;
|
||||
}
|
||||
list.innerHTML = state.anchors
|
||||
.map(
|
||||
(a) =>
|
||||
`<div class="ap-anchor-row" data-id="${a.id}">` +
|
||||
`<span class="ap-anchor-dot" style="background:${a.color || "#59d499"}"></span>` +
|
||||
`<span class="ap-anchor-label" title="${a.label || ""}">${a.label || "(unnamed)"}</span>` +
|
||||
`<button class="ap-anchor-del" title="Delete anchor">✕</button></div>`
|
||||
)
|
||||
.join("");
|
||||
for (const row of list.querySelectorAll(".ap-anchor-row")) {
|
||||
const id = Number(row.dataset.id);
|
||||
row.querySelector(".ap-anchor-del").addEventListener("click", () => this._deleteAnchor(id));
|
||||
}
|
||||
}
|
||||
|
||||
async _deleteAnchor(id) {
|
||||
try {
|
||||
const resp = await fetch(`${API_BASE}/api/anchors/${id}`, { method: "DELETE" });
|
||||
if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
|
||||
const i = state.anchors.findIndex((a) => a.id === id);
|
||||
if (i >= 0) state.anchors.splice(i, 1);
|
||||
emit("anchors-changed", null); // scene3d refresh; overlays read state.anchors each frame
|
||||
this._renderAnchorList();
|
||||
} catch (err) {
|
||||
this._hint(`<span class="bad">delete failed: ${err.message}</span>`);
|
||||
}
|
||||
}
|
||||
|
||||
_hint(msg) {
|
||||
const h = this.panel?.querySelector(".ap-hint");
|
||||
if (h) h.innerHTML = msg;
|
||||
}
|
||||
|
||||
async _correctType(newType) {
|
||||
const ev = this.currentEvent;
|
||||
if (!ev) return;
|
||||
try {
|
||||
const resp = await fetch(`${API_BASE}/api/events/${ev.id}`, {
|
||||
method: "PATCH",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ event_type: newType }),
|
||||
});
|
||||
if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
|
||||
const updated = await resp.json();
|
||||
// Reflect into state + timeline marker color.
|
||||
const i = state.events.findIndex((e) => e.id === ev.id);
|
||||
if (i >= 0) state.events[i] = updated;
|
||||
this.currentEvent = updated;
|
||||
emit("events-changed", updated);
|
||||
this._hint(`corrected → <b>${newType.replace(/_/g, " ")}</b> (source: user)`);
|
||||
} catch (err) {
|
||||
this._hint(`<span class="bad">correction failed: ${err.message}</span>`);
|
||||
}
|
||||
}
|
||||
|
||||
startAnnotateMode() {
|
||||
this.annotating = true;
|
||||
const btn = this.panel?.querySelector(".ap-annotate");
|
||||
if (btn) {
|
||||
btn.textContent = "✓ Done annotating";
|
||||
btn.classList.add("active");
|
||||
}
|
||||
this._hint("drag a box on any video over the object. Repeat on a 2nd video to triangulate.");
|
||||
for (const cell of this.cells) {
|
||||
cell.canvas.style.pointerEvents = "auto";
|
||||
cell.canvas.style.cursor = "crosshair";
|
||||
const down = (e) => this._onDown(cell, e);
|
||||
cell.canvas.addEventListener("pointerdown", down);
|
||||
this._cleanup.push(() => cell.canvas.removeEventListener("pointerdown", down));
|
||||
}
|
||||
}
|
||||
|
||||
stopAnnotateMode() {
|
||||
this.annotating = false;
|
||||
const btn = this.panel?.querySelector(".ap-annotate");
|
||||
if (btn) {
|
||||
btn.textContent = "📍 Annotate location";
|
||||
btn.classList.remove("active");
|
||||
}
|
||||
for (const cell of this.cells) {
|
||||
cell.canvas.style.pointerEvents = "none";
|
||||
cell.canvas.style.cursor = "";
|
||||
}
|
||||
this._teardownRubber();
|
||||
for (const fn of this._cleanup) fn();
|
||||
this._cleanup = [];
|
||||
}
|
||||
|
||||
_onDown(cell, e) {
|
||||
e.preventDefault();
|
||||
cell.canvas.setPointerCapture(e.pointerId);
|
||||
const rect = cell.el.getBoundingClientRect();
|
||||
const sx = e.clientX - rect.left;
|
||||
const sy = e.clientY - rect.top;
|
||||
const div = document.createElement("div");
|
||||
div.className = "annot-rect";
|
||||
cell.el.appendChild(div);
|
||||
this._rubber = { cell, div, sx, sy, rect };
|
||||
this._positionRubber(sx, sy);
|
||||
|
||||
const move = (ev) => this._onMove(ev);
|
||||
const up = (ev) => this._onUp(ev, move, up);
|
||||
cell.canvas.addEventListener("pointermove", move);
|
||||
cell.canvas.addEventListener("pointerup", up);
|
||||
cell.canvas.addEventListener("pointercancel", up);
|
||||
}
|
||||
|
||||
_positionRubber(cx, cy) {
|
||||
const r = this._rubber;
|
||||
if (!r) return;
|
||||
const x0 = Math.min(r.sx, cx);
|
||||
const y0 = Math.min(r.sy, cy);
|
||||
r.div.style.left = `${x0}px`;
|
||||
r.div.style.top = `${y0}px`;
|
||||
r.div.style.width = `${Math.abs(cx - r.sx)}px`;
|
||||
r.div.style.height = `${Math.abs(cy - r.sy)}px`;
|
||||
}
|
||||
|
||||
_onMove(e) {
|
||||
const r = this._rubber;
|
||||
if (!r) return;
|
||||
const rect = r.cell.el.getBoundingClientRect();
|
||||
this._positionRubber(e.clientX - rect.left, e.clientY - rect.top);
|
||||
}
|
||||
|
||||
async _onUp(e, move, up) {
|
||||
const r = this._rubber;
|
||||
if (!r) return;
|
||||
const { cell } = r;
|
||||
cell.canvas.removeEventListener("pointermove", move);
|
||||
cell.canvas.removeEventListener("pointerup", up);
|
||||
cell.canvas.removeEventListener("pointercancel", up);
|
||||
|
||||
const rect = cell.el.getBoundingClientRect();
|
||||
const ex = e.clientX - rect.left;
|
||||
const ey = e.clientY - rect.top;
|
||||
const bbox = this._toNormalizedBbox(cell, r.sx, r.sy, ex, ey);
|
||||
this._teardownRubber();
|
||||
if (!bbox) {
|
||||
this._hint('<span class="bad">draw the box over the video content area.</span>');
|
||||
return;
|
||||
}
|
||||
await this._submit(cell, bbox);
|
||||
}
|
||||
|
||||
_teardownRubber() {
|
||||
if (this._rubber?.div?.parentNode) this._rubber.div.parentNode.removeChild(this._rubber.div);
|
||||
this._rubber = null;
|
||||
}
|
||||
|
||||
/** Element-px rectangle → normalized [0,1] bbox in the video's CONTENT space (letterbox-aware). */
|
||||
_toNormalizedBbox(cell, ax, ay, bx, by) {
|
||||
const cr = contentRect(cell.video);
|
||||
const nx = (v) => (v - cr.ox) / cr.cw;
|
||||
const ny = (v) => (v - cr.oy) / cr.ch;
|
||||
let x0 = nx(Math.min(ax, bx));
|
||||
let y0 = ny(Math.min(ay, by));
|
||||
let x1 = nx(Math.max(ax, bx));
|
||||
let y1 = ny(Math.max(ay, by));
|
||||
x0 = Math.max(0, Math.min(1, x0));
|
||||
y0 = Math.max(0, Math.min(1, y0));
|
||||
x1 = Math.max(0, Math.min(1, x1));
|
||||
y1 = Math.max(0, Math.min(1, y1));
|
||||
// A near-zero drag (a click) becomes a tiny box centered on the point.
|
||||
if (x1 - x0 < 1e-4 && y1 - y0 < 1e-4) {
|
||||
const cx = (x0 + x1) / 2;
|
||||
const cy = (y0 + y1) / 2;
|
||||
x0 = Math.max(0, cx - 0.01); x1 = Math.min(1, cx + 0.01);
|
||||
y0 = Math.max(0, cy - 0.01); y1 = Math.min(1, cy + 0.01);
|
||||
}
|
||||
if (x1 <= x0 || y1 <= y0) return null;
|
||||
return [x0, y0, x1, y1];
|
||||
}
|
||||
|
||||
async _submit(cell, bbox) {
|
||||
const ev = this.currentEvent;
|
||||
this._hint("resolving 3D position…");
|
||||
try {
|
||||
const resp = await fetch(`${API_BASE}/api/annotations`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
video_id: cell.id,
|
||||
t_video_s: cell.video.currentTime,
|
||||
bbox,
|
||||
event_id: ev ? ev.id : null,
|
||||
}),
|
||||
});
|
||||
if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
|
||||
const res = await resp.json();
|
||||
if (res.anchor) {
|
||||
// Upsert: a supersede returns an EXISTING anchor id (moved in place) — replace, don't
|
||||
// duplicate. A fresh annotation returns a new id — append.
|
||||
const i = state.anchors.findIndex((a) => a.id === res.anchor.id);
|
||||
if (i >= 0) state.anchors[i] = res.anchor;
|
||||
else state.anchors.push(res.anchor);
|
||||
emit("anchors-changed", res.anchor);
|
||||
this._renderAnchorList();
|
||||
const how =
|
||||
res.method === "triangulated"
|
||||
? `triangulated (gap ${res.gap != null ? res.gap.toFixed(2) : "?"})`
|
||||
: res.method === "nearest_point"
|
||||
? "nearest point-cloud point"
|
||||
: "ray at scene depth";
|
||||
const note = res.superseded
|
||||
? "refined the anchor in place."
|
||||
: "Annotate another view to refine.";
|
||||
this._hint(`anchor placed via <b>${how}</b>. ${note}`);
|
||||
} else {
|
||||
this._hint('<span class="bad">could not resolve a 3D point (no poses?).</span>');
|
||||
}
|
||||
} catch (err) {
|
||||
this._hint(`<span class="bad">annotation failed: ${err.message}</span>`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export const annotator = new Annotator();
|
||||
export {};
|
||||
|
||||
@ -1,167 +1,6 @@
|
||||
// Keyframed "god's-eye" camera paths (spec M9, integration phase).
|
||||
//
|
||||
// "Add keyframe" captures the current free-roam camera pose at the current t_global. Keyframes
|
||||
// render as octahedron gizmos with a smooth preview line. On playback (while the transport
|
||||
// plays) the viewer camera follows the path: position is interpolated with a Catmull-Rom curve
|
||||
// through the keyframe centers, orientation with quaternion slerp, both parameterized by
|
||||
// t_global between the first and last keyframe. Paths export/import as JSON.
|
||||
// Keyframed camera paths (spec M9).
|
||||
// STUB — this is phase-3 (integration) work; lane C leaves it as a clean seam. "Add
|
||||
// keyframe" captures the current free-roam pose at the current t_global; playback
|
||||
// interpolates position (Catmull-Rom) + orientation (slerp); export/import path as JSON.
|
||||
|
||||
import * as THREE from "three";
|
||||
import { state } from "./state.js";
|
||||
import { scene3d } from "./scene3d.js";
|
||||
|
||||
const _qa = new THREE.Quaternion();
|
||||
const _qb = new THREE.Quaternion();
|
||||
const _pos = new THREE.Vector3();
|
||||
|
||||
export class CamPath {
|
||||
constructor() {
|
||||
this.keyframes = []; // [{ t_global, pos:[x,y,z], quat:[x,y,z,w], fov }]
|
||||
this.playing = false;
|
||||
this.group = null; // gizmos + preview line
|
||||
this._curve = null;
|
||||
this.onChange = null; // optional UI callback (count)
|
||||
}
|
||||
|
||||
_ensureGroup() {
|
||||
if (!this.group) {
|
||||
this.group = new THREE.Group();
|
||||
scene3d.addObject(this.group);
|
||||
scene3d.registerHelper(this.group); // keyframe gizmos hide with the helpers (M14)
|
||||
}
|
||||
}
|
||||
|
||||
/** Capture the current viewer-camera pose at the current master time. */
|
||||
addKeyframe() {
|
||||
const cam = scene3d.camera;
|
||||
if (!cam) return;
|
||||
this.keyframes.push({
|
||||
t_global: state.tGlobal,
|
||||
pos: [cam.position.x, cam.position.y, cam.position.z],
|
||||
quat: [cam.quaternion.x, cam.quaternion.y, cam.quaternion.z, cam.quaternion.w],
|
||||
fov: cam.fov,
|
||||
});
|
||||
this.keyframes.sort((a, b) => a.t_global - b.t_global);
|
||||
this._rebuild();
|
||||
}
|
||||
|
||||
clear() {
|
||||
this.stop();
|
||||
this.keyframes = [];
|
||||
this._rebuild();
|
||||
}
|
||||
|
||||
count() {
|
||||
return this.keyframes.length;
|
||||
}
|
||||
|
||||
_rebuild() {
|
||||
this._ensureGroup();
|
||||
for (const child of [...this.group.children]) {
|
||||
this.group.remove(child);
|
||||
child.geometry?.dispose?.();
|
||||
child.material?.dispose?.();
|
||||
}
|
||||
this._curve = null;
|
||||
|
||||
for (const k of this.keyframes) {
|
||||
const giz = new THREE.Mesh(
|
||||
new THREE.OctahedronGeometry(0.16),
|
||||
new THREE.MeshBasicMaterial({ color: 0xffd23b, wireframe: true })
|
||||
);
|
||||
giz.position.set(k.pos[0], k.pos[1], k.pos[2]);
|
||||
this.group.add(giz);
|
||||
}
|
||||
|
||||
if (this.keyframes.length >= 2) {
|
||||
this._curve = new THREE.CatmullRomCurve3(
|
||||
this.keyframes.map((k) => new THREE.Vector3(k.pos[0], k.pos[1], k.pos[2]))
|
||||
);
|
||||
const pts = this._curve.getPoints(Math.max(20, this.keyframes.length * 12));
|
||||
const line = new THREE.Line(
|
||||
new THREE.BufferGeometry().setFromPoints(pts),
|
||||
new THREE.LineBasicMaterial({ color: 0xffd23b, transparent: true, opacity: 0.6 })
|
||||
);
|
||||
this.group.add(line);
|
||||
}
|
||||
this.onChange?.(this.keyframes.length);
|
||||
}
|
||||
|
||||
/** Begin following the path (requires >= 2 keyframes). Caller should also start the transport. */
|
||||
play() {
|
||||
if (this.keyframes.length < 2) return false;
|
||||
this.playing = true;
|
||||
scene3d.enterPathMode();
|
||||
return true;
|
||||
}
|
||||
|
||||
stop() {
|
||||
if (!this.playing) return;
|
||||
this.playing = false;
|
||||
scene3d.exitPathMode();
|
||||
}
|
||||
|
||||
/** Per-frame: if playing, drive the viewer camera from the path at the current master time. */
|
||||
update(tGlobal) {
|
||||
if (!this.playing) return;
|
||||
// If something else took the camera (snap-to-camera exits path mode), stop driving it.
|
||||
if (!scene3d.pathMode) {
|
||||
this.playing = false;
|
||||
this.onChange?.(this.keyframes.length);
|
||||
return;
|
||||
}
|
||||
if (!this._curve || this.keyframes.length < 2) return;
|
||||
const kf = this.keyframes;
|
||||
const n = kf.length;
|
||||
const t0 = kf[0].t_global;
|
||||
const t1 = kf[n - 1].t_global;
|
||||
|
||||
// Locate the bracketing keyframes by time; clamp outside the path's time span.
|
||||
let i = 0;
|
||||
if (tGlobal <= t0) {
|
||||
i = 0;
|
||||
} else if (tGlobal >= t1) {
|
||||
i = n - 2;
|
||||
} else {
|
||||
while (i < n - 2 && kf[i + 1].t_global <= tGlobal) i++;
|
||||
}
|
||||
const a = kf[i];
|
||||
const b = kf[i + 1];
|
||||
const span = b.t_global - a.t_global;
|
||||
let f = span > 1e-6 ? (tGlobal - a.t_global) / span : 0;
|
||||
f = Math.max(0, Math.min(1, f));
|
||||
|
||||
// Position: Catmull-Rom, parameterized by (segment index + local fraction) / (n - 1).
|
||||
const u = Math.max(0, Math.min(1, (i + f) / (n - 1)));
|
||||
this._curve.getPoint(u, _pos);
|
||||
|
||||
// Orientation: slerp the two bracketing keyframe quaternions.
|
||||
_qa.set(a.quat[0], a.quat[1], a.quat[2], a.quat[3]);
|
||||
_qb.set(b.quat[0], b.quat[1], b.quat[2], b.quat[3]);
|
||||
_qa.slerp(_qb, f);
|
||||
|
||||
const fov = a.fov + (b.fov - a.fov) * f;
|
||||
scene3d.applyCameraPose(_pos, _qa, fov);
|
||||
}
|
||||
|
||||
toJSON() {
|
||||
return JSON.stringify({ version: 1, keyframes: this.keyframes }, null, 2);
|
||||
}
|
||||
|
||||
fromJSON(text) {
|
||||
const data = typeof text === "string" ? JSON.parse(text) : text;
|
||||
if (!data || !Array.isArray(data.keyframes)) throw new Error("invalid camera-path file");
|
||||
this.keyframes = data.keyframes
|
||||
.filter((k) => Array.isArray(k.pos) && Array.isArray(k.quat))
|
||||
.map((k) => ({
|
||||
t_global: +k.t_global || 0,
|
||||
pos: k.pos.slice(0, 3).map(Number),
|
||||
quat: k.quat.slice(0, 4).map(Number),
|
||||
fov: +k.fov || 50,
|
||||
}))
|
||||
.sort((a, b) => a.t_global - b.t_global);
|
||||
this._rebuild();
|
||||
}
|
||||
}
|
||||
|
||||
export const camPath = new CamPath();
|
||||
export {};
|
||||
|
||||
@ -1,70 +0,0 @@
|
||||
// Auto-director UI (spec M11, lane E).
|
||||
//
|
||||
// Pre-wired by foundation2: the hidden 🎬 button (#btn-director in index.html) and this
|
||||
// module's import + init() call in main.js. This module wires the click: POST /api/director
|
||||
// -> the response is ALWAYS a valid camPath JSON (frozen contract #2) -> load it with
|
||||
// camPath.fromJSON and play it via the existing #btn-path flow (never edit camPath.js).
|
||||
// Zero keyframes means "no events / no poses yet" — surfaced on the button, never a crash
|
||||
// (pitfall #5). Failures degrade the same way; the button always recovers.
|
||||
|
||||
import { state } from "./state.js";
|
||||
import { camPath } from "./camPath.js";
|
||||
|
||||
const BTN_LABEL = "🎬 Auto-path";
|
||||
let flashTimer = null;
|
||||
|
||||
/** Show a transient message on the button, then restore the label. */
|
||||
function flash(btn, text, ms = 2200) {
|
||||
btn.textContent = text;
|
||||
clearTimeout(flashTimer);
|
||||
flashTimer = setTimeout(() => (btn.textContent = BTN_LABEL), ms);
|
||||
}
|
||||
|
||||
async function requestPath(btn) {
|
||||
btn.disabled = true;
|
||||
btn.textContent = "🎬 …";
|
||||
try {
|
||||
const resp = await fetch(`${state.apiBase}/api/director`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({}), // server defaults: top_n=8, lead_s=2.0
|
||||
});
|
||||
if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
|
||||
const path = await resp.json();
|
||||
|
||||
if (!Array.isArray(path.keyframes) || path.keyframes.length === 0) {
|
||||
// Valid-empty path: no events/poses yet (or still stubbed). Surface, don't crash.
|
||||
console.info("[director] empty path from /api/director", path.note ?? "");
|
||||
flash(btn, "🎬 no events yet");
|
||||
return;
|
||||
}
|
||||
|
||||
camPath.fromJSON(path); // replaces any existing keyframes; updates the ＋ Key counter
|
||||
flash(btn, `🎬 ${path.keyframes.length} keys`);
|
||||
|
||||
// Play through the existing #btn-path flow so its UI state (⏹ label, transport
|
||||
// auto-start) stays consistent with a manually loaded path.
|
||||
const playBtn = document.getElementById("btn-path");
|
||||
if (playBtn && !camPath.playing && path.keyframes.length >= 2) playBtn.click();
|
||||
} catch (err) {
|
||||
console.error("[director] /api/director failed", err);
|
||||
flash(btn, "🎬 error — see console");
|
||||
} finally {
|
||||
btn.disabled = false;
|
||||
}
|
||||
}
|
||||
|
||||
export const director = {
|
||||
ready: false,
|
||||
|
||||
init() {
|
||||
const btn = document.getElementById("btn-director");
|
||||
if (!btn) return; // markup missing: stay hidden, degrade silently
|
||||
btn.addEventListener("click", () => {
|
||||
if (btn.disabled) return;
|
||||
requestPath(btn);
|
||||
});
|
||||
btn.style.display = ""; // unhide: this lane has landed
|
||||
this.ready = true;
|
||||
},
|
||||
};
|
||||
@ -1,211 +0,0 @@
|
||||
// Cinematic export (spec M12, lane E).
|
||||
//
|
||||
// Pre-wired by foundation2: the hidden ⏺ button (#btn-export in index.html), this module's
|
||||
// import + init() call in main.js, and the transport hooks transport.captureAudioStream()
|
||||
// (a MediaStream tapping the live WebAudio mix — post-gain, spatial included) +
|
||||
// transport.releaseAudioStream() when done.
|
||||
//
|
||||
// Flow: seek to the first camPath keyframe, play the path via the existing #btn-path flow
|
||||
// (so its UI state stays truthful), record scene3d's canvas captureStream(30) muxed with the
|
||||
// audio tap through MediaRecorder (vp9/opus webm, fallbacks below), stop at the last
|
||||
// keyframe, download `festival4d-cut.webm`, then restore ALL prior UI/transport state.
|
||||
// Export runs at rate 1 regardless of the user's speed (the recording is wall-clock, so the
|
||||
// file duration must equal the path's time span); the prior rate is restored afterwards.
|
||||
// Degradation: fewer than 2 keyframes -> button disabled; no decoded WebAudio track -> the
|
||||
// tap is silent but the export still works (pitfall #5); clicking ⏺ mid-export stops early
|
||||
// and still produces a valid file. NOTE (pitfall #2): canvas.captureStream only delivers
|
||||
// real frames in a FOCUSED tab — a hidden/backgrounded pane records a frozen image.
|
||||
|
||||
import { state } from "./state.js";
|
||||
import { transport } from "./transport.js";
|
||||
import { camPath } from "./camPath.js";
|
||||
import { scene3d } from "./scene3d.js";
|
||||
|
||||
const BTN_LABEL = "⏺ Export";
|
||||
const MIME_CANDIDATES = [
|
||||
"video/webm;codecs=vp9,opus", // the spec'd target
|
||||
"video/webm;codecs=vp8,opus",
|
||||
"video/webm",
|
||||
];
|
||||
|
||||
function pickMime() {
|
||||
if (typeof MediaRecorder === "undefined") return null;
|
||||
return MIME_CANDIDATES.find((m) => MediaRecorder.isTypeSupported(m)) ?? null;
|
||||
}
|
||||
|
||||
function download(blob, filename) {
|
||||
const a = document.createElement("a");
|
||||
a.href = URL.createObjectURL(blob);
|
||||
a.download = filename;
|
||||
a.click();
|
||||
URL.revokeObjectURL(a.href);
|
||||
}
|
||||
|
||||
/** Toggle the shared #btn-path flow (main.js owns its UI state) into the desired state. */
|
||||
function setPathPlaying(playing) {
|
||||
if (camPath.playing === playing) return;
|
||||
document.getElementById("btn-path")?.click();
|
||||
}
|
||||
|
||||
export const exportVideo = {
|
||||
ready: false,
|
||||
exporting: false,
|
||||
|
||||
_btn: null,
|
||||
_recorder: null,
|
||||
_chunks: [],
|
||||
_canvasStream: null,
|
||||
_endT: 0,
|
||||
_prior: null,
|
||||
_finishing: false,
|
||||
|
||||
init() {
|
||||
const btn = document.getElementById("btn-export");
|
||||
if (!btn) return; // markup missing: stay hidden, degrade silently
|
||||
this._btn = btn;
|
||||
|
||||
btn.addEventListener("click", () => {
|
||||
if (this.exporting) this._finish("stopped early");
|
||||
else this._start();
|
||||
});
|
||||
|
||||
// Enabled only with a playable path (>= 2 keyframes). main.js assigned camPath.onChange
|
||||
// before lane modules init, so chain it rather than replace it (camPath.js is frozen).
|
||||
const chained = camPath.onChange;
|
||||
camPath.onChange = (n) => {
|
||||
chained?.(n);
|
||||
if (!this.exporting) btn.disabled = n < 2;
|
||||
};
|
||||
btn.disabled = camPath.count() < 2;
|
||||
|
||||
btn.style.display = ""; // unhide: this lane has landed
|
||||
this.ready = true;
|
||||
},
|
||||
|
||||
_start() {
|
||||
if (this.exporting || camPath.keyframes.length < 2) return;
|
||||
const mime = pickMime();
|
||||
if (!mime) {
|
||||
console.error("[export] MediaRecorder/webm unsupported in this browser");
|
||||
this._flash("⏺ unsupported");
|
||||
return;
|
||||
}
|
||||
|
||||
const kf = camPath.keyframes; // sorted by t_global (camPath invariant)
|
||||
const startT = kf[0].t_global;
|
||||
this._endT = kf[kf.length - 1].t_global;
|
||||
|
||||
// Snapshot everything we mutate, so _finish can restore it exactly.
|
||||
this._prior = {
|
||||
tGlobal: state.tGlobal,
|
||||
playing: state.playing,
|
||||
rate: state.rate,
|
||||
pathPlaying: camPath.playing,
|
||||
followCameraId: state.followCameraId,
|
||||
};
|
||||
|
||||
// Park the world at the first keyframe, real-time rate, path engaged.
|
||||
setPathPlaying(false);
|
||||
if (state.playing) transport.pause();
|
||||
transport.setRate(1);
|
||||
transport.seek(startT);
|
||||
|
||||
// Canvas frames + the live WebAudio mix tap (foundation2 hook; silent when no track).
|
||||
const canvas = scene3d.renderer.domElement;
|
||||
this._canvasStream = canvas.captureStream(30);
|
||||
const audioStream = transport.captureAudioStream();
|
||||
const mixed = new MediaStream([
|
||||
...this._canvasStream.getVideoTracks(),
|
||||
...audioStream.getAudioTracks(),
|
||||
]);
|
||||
|
||||
this._chunks = [];
|
||||
this._recorder = new MediaRecorder(mixed, {
|
||||
mimeType: mime,
|
||||
videoBitsPerSecond: 8_000_000,
|
||||
});
|
||||
this._recorder.addEventListener("dataavailable", (e) => {
|
||||
if (e.data && e.data.size > 0) this._chunks.push(e.data);
|
||||
});
|
||||
this._recorder.addEventListener("stop", () => this._deliver(mime));
|
||||
|
||||
this.exporting = true;
|
||||
this._finishing = false;
|
||||
this._btn.classList.add("active");
|
||||
this._btn.textContent = "⏺ REC — click to stop";
|
||||
|
||||
// Roll: recorder + playback in the same tick so file duration ~= the path span.
|
||||
setPathPlaying(true); // #btn-path flow: camPath.play() + transport starts the clock
|
||||
this._recorder.start(250);
|
||||
this._tick = this._tick.bind(this);
|
||||
requestAnimationFrame(this._tick);
|
||||
console.info(`[export] recording ${startT.toFixed(2)}s -> ${this._endT.toFixed(2)}s (${mime})`);
|
||||
},
|
||||
|
||||
_tick() {
|
||||
if (!this.exporting || this._finishing) return;
|
||||
// Done when the playhead crosses the last keyframe — or when anything halts the ride
|
||||
// (end of timeline pauses the transport; snap-to-camera exits path mode).
|
||||
if (state.tGlobal >= this._endT - 1e-3 || !state.playing || !camPath.playing) {
|
||||
this._finish("end of path");
|
||||
return;
|
||||
}
|
||||
requestAnimationFrame(this._tick);
|
||||
},
|
||||
|
||||
_finish(reason) {
|
||||
if (!this.exporting || this._finishing) return;
|
||||
this._finishing = true;
|
||||
console.info(`[export] finishing (${reason})`);
|
||||
try {
|
||||
if (this._recorder && this._recorder.state !== "inactive") this._recorder.stop();
|
||||
else this._deliver(null); // recorder never really started: still restore the UI
|
||||
} catch (err) {
|
||||
console.error("[export] recorder stop failed", err);
|
||||
this._deliver(null);
|
||||
}
|
||||
},
|
||||
|
||||
/** MediaRecorder "stop" handler: build + download the file, then restore prior state. */
|
||||
_deliver(mime) {
|
||||
if (this._chunks.length > 0) {
|
||||
const blob = new Blob(this._chunks, { type: mime ?? "video/webm" });
|
||||
download(blob, "festival4d-cut.webm");
|
||||
console.info(`[export] festival4d-cut.webm — ${(blob.size / 1e6).toFixed(2)} MB`);
|
||||
} else {
|
||||
console.warn("[export] no data recorded (hidden tab? — captureStream needs focus)");
|
||||
}
|
||||
this._chunks = [];
|
||||
this._recorder = null;
|
||||
|
||||
// Release the shared hooks.
|
||||
transport.releaseAudioStream();
|
||||
this._canvasStream?.getTracks().forEach((t) => t.stop());
|
||||
this._canvasStream = null;
|
||||
|
||||
// Restore ALL prior UI/transport state (spec M12).
|
||||
const prior = this._prior ?? {};
|
||||
this._prior = null;
|
||||
setPathPlaying(false); // resets #btn-path label via main.js's own flow
|
||||
if (state.playing) transport.pause();
|
||||
transport.setRate(prior.rate ?? 1);
|
||||
transport.seek(prior.tGlobal ?? 0);
|
||||
if (prior.followCameraId != null) scene3d.snapTo(prior.followCameraId);
|
||||
if (prior.pathPlaying) setPathPlaying(true);
|
||||
if (prior.playing && !state.playing) transport.play();
|
||||
|
||||
this.exporting = false;
|
||||
this._finishing = false;
|
||||
this._btn.classList.remove("active");
|
||||
this._btn.textContent = BTN_LABEL;
|
||||
this._btn.disabled = camPath.count() < 2;
|
||||
},
|
||||
|
||||
_flash(text, ms = 2200) {
|
||||
const btn = this._btn;
|
||||
btn.textContent = text;
|
||||
setTimeout(() => {
|
||||
if (!this.exporting) btn.textContent = BTN_LABEL;
|
||||
}, ms);
|
||||
},
|
||||
};
|
||||
@ -1,41 +0,0 @@
|
||||
// Friend tracks (spec M21 ribbons + moving labels + panel, M22 follow-a-friend cam; lane J).
|
||||
// STUB shipped by foundation3 — lane J fills the body.
|
||||
//
|
||||
// main.js imports this, calls friendTracks.init() in buildUI (alongside the other lane
|
||||
// modules) and friendTracks.update(state.tGlobal) every animation frame — the fx.js pattern
|
||||
// (after transport.tick(), next to camPath.update / fx.update, before scene3d.update()).
|
||||
// While stubbed, both are safe no-ops and the hidden #btn-friends button stays hidden; a
|
||||
// broken or unfilled module degrades to no UI, exactly like the phase-5 stubs.
|
||||
//
|
||||
// Contract lane J must honor (frozen by foundation3):
|
||||
// - state.tracks is loaded at boot by main.js when manifest.has_tracks, in the
|
||||
// GET /api/tracks shape (contract #2):
|
||||
// [{ id, marker_key, label, color,
|
||||
// points: [{ t_global_s, x, y, z, quality, views }] }] // points sorted by t_global_s
|
||||
// - Scene objects go through scene3d.addObject / scene3d.removeObject ONLY, and every
|
||||
// helper is registered with scene3d.registerHelper so photo mode (M14) hides ribbons +
|
||||
// labels. (Read anchorPanel.js for the panel style/UX; do not edit it.)
|
||||
// - Rename / recolor / delete round-trip through PATCH / DELETE /api/tracks/{id}; those
|
||||
// controls carry the `write-ui` class so the capsule (M17) hides them while ribbons +
|
||||
// follow still work.
|
||||
// - Follow-a-friend (M22) shares scene3d pathMode with camPath — mirror camPath's "someone
|
||||
// else took the camera" contention checks exactly (pitfall #4), or two drivers fight.
|
||||
// - update(tGlobal) MUST NEVER THROW: it runs inside the master loop and a throw kills
|
||||
// playback. Guard the body and disable the module on error (fx.js does this).
|
||||
|
||||
export const friendTracks = {
|
||||
ready: false,
|
||||
|
||||
/** Called once by main.js (buildUI). Lane J (M21): unhide #btn-friends, build the friends
|
||||
* panel in #friends-panel, wire the toggle that renders ribbons + moving labels. */
|
||||
init() {
|
||||
// Stub: #btn-friends stays hidden until lane J fills M21. No-op is the safe degrade.
|
||||
},
|
||||
|
||||
/** Called by main.js every animation frame with the master clock. Lane J (M21/M22):
|
||||
* interpolate each track's position at tGlobal, move ribbons/labels, drive the chase cam.
|
||||
* MUST NEVER THROW. */
|
||||
update(_tGlobal) {
|
||||
// Stub: no tracks rendered yet (M21/M22).
|
||||
},
|
||||
};
|
||||
@ -1,281 +0,0 @@
|
||||
// Moment FX (spec M15, lane F).
|
||||
//
|
||||
// main.js calls fx.update(tGlobal) every animation frame (after transport.tick(), before
|
||||
// scene3d.update()). While the ✨ toggle is on we watch the playhead cross event markers
|
||||
// (state.events) by tracking the previous tGlobal:
|
||||
// - crossing works in play AND scrub (scrub = a stream of small forward seeks, so plain
|
||||
// prev<t_e<=t detection covers it);
|
||||
// - a backward jump just resets the tracker (no retro-fire);
|
||||
// - a big forward jump (> JUMP_GAP_S) is a seek, not a scrub — reset, don't machine-gun
|
||||
// every event in between.
|
||||
// pyro/confetti -> a ~1 s additive particle burst at the event's resolved anchor. Events
|
||||
// don't carry an anchor id in state, but the backend names an event's resolved anchor
|
||||
// `event.description or event.event_type`, so we map event->anchor by that label; when no
|
||||
// anchor matches we fall back to the stage centroid (0, 0.5, 0) (documented fallback).
|
||||
// bass_drop -> pulse the point cloud / splat scale for ~a beat (60/tempo from
|
||||
// GET /api/beats when present, else 0.5 s — pitfall #5), restoring the exact prior scale.
|
||||
//
|
||||
// Scene access is ONLY through scene3d.addObject/removeObject (plus read-only looks at
|
||||
// scene3d.points/.splat for the pulse). Hot-path discipline: burst slots + their typed
|
||||
// arrays are allocated once, spawns recycle slots, update() allocates nothing, and the whole
|
||||
// body is guarded — a throw here would kill the app's master loop, so on error FX disables
|
||||
// itself instead of ever throwing.
|
||||
|
||||
import * as THREE from "three";
|
||||
import { state } from "./state.js";
|
||||
import { scene3d } from "./scene3d.js";
|
||||
|
||||
const MAX_BURSTS = 6; // concurrent burst cap
|
||||
const PARTICLES = 160; // per burst
|
||||
const BURST_LIFE_S = 1.0;
|
||||
const JUMP_GAP_S = 2.0; // forward delta above this = seek/jump, not playback or scrub
|
||||
const MAX_FIRE_PER_FRAME = 2;
|
||||
const STAGE_CENTROID = { x: 0, y: 0.5, z: 0 };
|
||||
const PYRO_COLORS = [0xffd27a, 0xff9d5d, 0xff5d3c, 0xfff3c4];
|
||||
const CONFETTI_COLORS = [0xff5d73, 0x4dd0ff, 0xc9a2ff, 0xffcf5d, 0x7dffb0, 0xff9d5d];
|
||||
|
||||
export const fx = {
|
||||
ready: false,
|
||||
enabled: false,
|
||||
_prevT: null,
|
||||
_lastNow: 0,
|
||||
_bursts: null, // preallocated slot pool (lazy, first enable)
|
||||
_pulse: null, // { obj, base, t0, dur } while a bass_drop pulse is live
|
||||
_beatLen: 0.5, // seconds; refined from /api/beats when available
|
||||
_dead: false, // a runtime error tripped the breaker — stay inert
|
||||
|
||||
init() {
|
||||
const btn = document.getElementById("btn-fx");
|
||||
if (!btn) return; // pre-wired DOM missing — degrade to hidden
|
||||
btn.addEventListener("click", () => {
|
||||
this.enabled = !this.enabled;
|
||||
btn.classList.toggle("active", this.enabled);
|
||||
if (this.enabled) {
|
||||
this._ensurePool();
|
||||
this._loadBeats();
|
||||
} else {
|
||||
this._clearAll();
|
||||
}
|
||||
});
|
||||
btn.style.display = ""; // unhide: the module is live
|
||||
this.ready = true;
|
||||
},
|
||||
|
||||
/** Called by main.js every animation frame. MUST NEVER THROW (it would kill the loop). */
|
||||
update(tGlobal) {
|
||||
if (this._dead) return;
|
||||
try {
|
||||
const now = performance.now() / 1000;
|
||||
const dt = Math.min(0.1, Math.max(0, now - this._lastNow)); // clamp tab-stall spikes
|
||||
this._lastNow = now;
|
||||
|
||||
if (!this.enabled || typeof tGlobal !== "number") {
|
||||
this._prevT = tGlobal;
|
||||
return;
|
||||
}
|
||||
|
||||
// --- playhead crossing detection ---
|
||||
const prev = this._prevT;
|
||||
this._prevT = tGlobal;
|
||||
if (prev != null && tGlobal > prev && tGlobal - prev <= JUMP_GAP_S) {
|
||||
let fired = 0;
|
||||
const events = state.events || [];
|
||||
for (let i = 0; i < events.length && fired < MAX_FIRE_PER_FRAME; i++) {
|
||||
const ev = events[i];
|
||||
const te = ev?.t_global_s;
|
||||
if (typeof te !== "number" || te <= prev || te > tGlobal) continue;
|
||||
if (ev.event_type === "pyro" || ev.event_type === "confetti") {
|
||||
this._spawnBurst(ev.event_type, this._anchorForEvent(ev));
|
||||
fired++;
|
||||
} else if (ev.event_type === "bass_drop") {
|
||||
this._startPulse(now);
|
||||
fired++;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
this._animateBursts(now, dt);
|
||||
this._animatePulse(now);
|
||||
} catch (err) {
|
||||
// Breaker: never let FX take the app down. Clean up best-effort and go inert.
|
||||
console.warn("[fx] runtime error — disabling FX", err);
|
||||
this._dead = true;
|
||||
this.enabled = false;
|
||||
try {
|
||||
this._clearAll();
|
||||
document.getElementById("btn-fx")?.classList.remove("active");
|
||||
} catch {
|
||||
/* stay inert */
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
// --- event -> anchor -----------------------------------------------------------------
|
||||
|
||||
/** Best-effort event->anchor from state alone: the backend labels an event's resolved
|
||||
* anchor with `description or event_type`. Fallback: stage centroid. */
|
||||
_anchorForEvent(ev) {
|
||||
const anchors = state.anchors || [];
|
||||
for (const a of anchors) {
|
||||
if (a.label && (a.label === ev.description || a.label === ev.event_type)) return a;
|
||||
}
|
||||
return STAGE_CENTROID;
|
||||
},
|
||||
|
||||
// --- particle bursts -------------------------------------------------------------------
|
||||
|
||||
_ensurePool() {
|
||||
if (this._bursts) return;
|
||||
this._bursts = [];
|
||||
for (let i = 0; i < MAX_BURSTS; i++) {
|
||||
const positions = new Float32Array(PARTICLES * 3);
|
||||
const colors = new Float32Array(PARTICLES * 3);
|
||||
const geometry = new THREE.BufferGeometry();
|
||||
geometry.setAttribute("position", new THREE.BufferAttribute(positions, 3));
|
||||
geometry.setAttribute("color", new THREE.BufferAttribute(colors, 3));
|
||||
const material = new THREE.PointsMaterial({
|
||||
size: 0.09,
|
||||
vertexColors: true,
|
||||
transparent: true,
|
||||
opacity: 1,
|
||||
blending: THREE.AdditiveBlending,
|
||||
depthWrite: false,
|
||||
sizeAttenuation: true,
|
||||
});
|
||||
const points = new THREE.Points(geometry, material);
|
||||
points.frustumCulled = false; // positions mutate in place; skip stale-bounds culling
|
||||
this._bursts.push({
|
||||
points,
|
||||
positions,
|
||||
velocities: new Float32Array(PARTICLES * 3),
|
||||
active: false,
|
||||
t0: 0,
|
||||
gravity: 0,
|
||||
});
|
||||
}
|
||||
},
|
||||
|
||||
_spawnBurst(type, at) {
|
||||
if (!this._bursts) this._ensurePool();
|
||||
const slot = this._bursts.find((b) => !b.active);
|
||||
if (!slot) return; // concurrency cap reached — drop, don't grow
|
||||
const pyro = type === "pyro";
|
||||
const palette = pyro ? PYRO_COLORS : CONFETTI_COLORS;
|
||||
const { positions, velocities } = slot;
|
||||
const color = slot.points.geometry.getAttribute("color");
|
||||
const c = new THREE.Color();
|
||||
for (let i = 0; i < PARTICLES; i++) {
|
||||
const j = i * 3;
|
||||
positions[j] = at.x + (Math.random() - 0.5) * 0.1;
|
||||
positions[j + 1] = at.y + (Math.random() - 0.5) * 0.1;
|
||||
positions[j + 2] = at.z + (Math.random() - 0.5) * 0.1;
|
||||
// Random direction; pyro shoots up-and-out fast, confetti pops gently and flutters down.
|
||||
const theta = Math.random() * Math.PI * 2;
|
||||
const up = Math.random();
|
||||
const speed = pyro ? 2.5 + Math.random() * 3 : 1 + Math.random() * 1.6;
|
||||
const horiz = pyro ? 0.55 : 0.9;
|
||||
velocities[j] = Math.cos(theta) * speed * horiz;
|
||||
velocities[j + 1] = (pyro ? 0.5 + up * 0.8 : 0.4 + up * 0.6) * speed;
|
||||
velocities[j + 2] = Math.sin(theta) * speed * horiz;
|
||||
c.setHex(palette[(Math.random() * palette.length) | 0]);
|
||||
color.array[j] = c.r;
|
||||
color.array[j + 1] = c.g;
|
||||
color.array[j + 2] = c.b;
|
||||
}
|
||||
slot.points.geometry.getAttribute("position").needsUpdate = true;
|
||||
color.needsUpdate = true;
|
||||
slot.points.material.opacity = 1;
|
||||
slot.gravity = pyro ? 2.2 : 4.5;
|
||||
slot.t0 = performance.now() / 1000;
|
||||
slot.active = true;
|
||||
scene3d.addObject(slot.points);
|
||||
},
|
||||
|
||||
_animateBursts(now, dt) {
|
||||
if (!this._bursts) return;
|
||||
for (const b of this._bursts) {
|
||||
if (!b.active) continue;
|
||||
const age = now - b.t0;
|
||||
if (age >= BURST_LIFE_S) {
|
||||
b.active = false;
|
||||
scene3d.removeObject(b.points);
|
||||
continue;
|
||||
}
|
||||
const { positions, velocities } = b;
|
||||
const g = b.gravity * dt;
|
||||
for (let j = 0; j < positions.length; j += 3) {
|
||||
positions[j] += velocities[j] * dt;
|
||||
positions[j + 1] += velocities[j + 1] * dt;
|
||||
positions[j + 2] += velocities[j + 2] * dt;
|
||||
velocities[j + 1] -= g;
|
||||
}
|
||||
b.points.geometry.getAttribute("position").needsUpdate = true;
|
||||
b.points.material.opacity = 1 - age / BURST_LIFE_S;
|
||||
}
|
||||
},
|
||||
|
||||
// --- bass-drop pulse ---------------------------------------------------------------------
|
||||
|
||||
_startPulse(now) {
|
||||
const obj = scene3d.splat ?? scene3d.points; // read-only handles; scale restored after
|
||||
if (!obj) return; // nothing loaded (yet) — degrade silently
|
||||
if (this._pulse && this._pulse.obj === obj) {
|
||||
this._pulse.t0 = now; // retrigger: keep the ORIGINAL base scale, restart the envelope
|
||||
return;
|
||||
}
|
||||
if (this._pulse) this._endPulse(); // different object (fallback swapped) — restore old one
|
||||
this._pulse = { obj, base: obj.scale.x, t0: now, dur: this._beatLen };
|
||||
},
|
||||
|
||||
_animatePulse(now) {
|
||||
const p = this._pulse;
|
||||
if (!p) return;
|
||||
const k = (now - p.t0) / p.dur;
|
||||
if (k >= 1) {
|
||||
this._endPulse();
|
||||
return;
|
||||
}
|
||||
// One smooth swell-and-settle over the beat.
|
||||
const amp = 0.16 * Math.sin(Math.PI * Math.min(1, Math.max(0, k)));
|
||||
p.obj.scale.setScalar(p.base * (1 + amp));
|
||||
},
|
||||
|
||||
_endPulse() {
|
||||
const p = this._pulse;
|
||||
if (p) {
|
||||
try {
|
||||
p.obj.scale.setScalar(p.base); // restore the exact prior scale
|
||||
} catch {
|
||||
/* object may have been torn down */
|
||||
}
|
||||
}
|
||||
this._pulse = null;
|
||||
},
|
||||
|
||||
_clearAll() {
|
||||
if (this._bursts) {
|
||||
for (const b of this._bursts) {
|
||||
if (b.active) {
|
||||
b.active = false;
|
||||
scene3d.removeObject(b.points);
|
||||
}
|
||||
}
|
||||
}
|
||||
this._endPulse();
|
||||
},
|
||||
|
||||
async _loadBeats() {
|
||||
if (this._beatsLoaded) return;
|
||||
this._beatsLoaded = true;
|
||||
try {
|
||||
const resp = await fetch(`${state.apiBase}/api/beats`);
|
||||
if (!resp.ok) return; // 404 until features runs — keep the 0.5 s default (pitfall #5)
|
||||
const beats = await resp.json();
|
||||
if (typeof beats?.tempo_bpm === "number" && beats.tempo_bpm > 30) {
|
||||
this._beatLen = 60 / beats.tempo_bpm;
|
||||
}
|
||||
} catch {
|
||||
/* no beats — default stands */
|
||||
}
|
||||
},
|
||||
};
|
||||
@ -1,57 +0,0 @@
|
||||
// Pose lookup + interpolation over a video's `camera_poses` track (from GET /api/videos/{id}/poses).
|
||||
//
|
||||
// Poses arrive keyed by frame_idx / t_video_s (every ~0.5 s on the synthetic fixture, all
|
||||
// registered). To render a smooth frustum / follow-cam at an arbitrary t_video we interpolate
|
||||
// BETWEEN stored poses in COLMAP space — slerp the world->camera quaternion, lerp the
|
||||
// translation — then hand the interpolated (q, t) to the FROZEN colmapToThreejs() helper.
|
||||
// We interpolate the CONTRACT INPUTS; we never reimplement the COLMAP->Three.js conversion.
|
||||
|
||||
import * as THREE from "three";
|
||||
|
||||
const _qa = new THREE.Quaternion();
|
||||
const _qb = new THREE.Quaternion();
|
||||
|
||||
/**
|
||||
* Interpolated pose at local video time `tv` (seconds). Returns a COLMAP-convention pose
|
||||
* `{ q:[w,x,y,z], t:[x,y,z], intrinsics, registered, t_video_s }` ready for colmapToThreejs,
|
||||
* or null if the track is empty. Clamps to the endpoints (no extrapolation).
|
||||
* @param {Array} poses ascending-by-t_video_s pose list
|
||||
* @param {number} tv local video time (seconds)
|
||||
*/
|
||||
export function poseAt(poses, tv) {
|
||||
if (!poses || poses.length === 0) return null;
|
||||
const n = poses.length;
|
||||
if (tv <= poses[0].t_video_s) return poses[0];
|
||||
if (tv >= poses[n - 1].t_video_s) return poses[n - 1];
|
||||
|
||||
// binary search for the bracketing pair [lo, hi] with poses[lo].t <= tv < poses[hi].t
|
||||
let lo = 0;
|
||||
let hi = n - 1;
|
||||
while (hi - lo > 1) {
|
||||
const mid = (lo + hi) >> 1;
|
||||
if (poses[mid].t_video_s <= tv) lo = mid;
|
||||
else hi = mid;
|
||||
}
|
||||
const a = poses[lo];
|
||||
const b = poses[hi];
|
||||
const span = b.t_video_s - a.t_video_s;
|
||||
const f = span > 1e-9 ? (tv - a.t_video_s) / span : 0;
|
||||
|
||||
// COLMAP quaternion order is [w, x, y, z]; THREE.Quaternion is (x, y, z, w).
|
||||
_qa.set(a.q[1], a.q[2], a.q[3], a.q[0]);
|
||||
_qb.set(b.q[1], b.q[2], b.q[3], b.q[0]);
|
||||
_qa.slerp(_qb, f);
|
||||
const q = [_qa.w, _qa.x, _qa.y, _qa.z];
|
||||
const t = [
|
||||
a.t[0] + (b.t[0] - a.t[0]) * f,
|
||||
a.t[1] + (b.t[1] - a.t[1]) * f,
|
||||
a.t[2] + (b.t[2] - a.t[2]) * f,
|
||||
];
|
||||
return {
|
||||
q,
|
||||
t,
|
||||
intrinsics: a.intrinsics,
|
||||
registered: a.registered && b.registered,
|
||||
t_video_s: tv,
|
||||
};
|
||||
}
|
||||
@ -1,18 +0,0 @@
|
||||
// Timebase convention (FROZEN CONTRACT) — the JS mirror of backend/festival4d/config.py
|
||||
// (`t_video_from_global` / `t_global_from_video`). Keep these identical to the Python side.
|
||||
//
|
||||
// t_global is the master timeline in seconds. For a video v:
|
||||
// t_video = (t_global - offset_ms/1000) * (1 + drift_ppm * 1e-6)
|
||||
// The reference video has offset_ms == 0 (and drift_ppm == 0). A positive offset_ms means
|
||||
// the video started recording later than the master zero, so at a given t_global its local
|
||||
// playhead is earlier. Never re-derive this algebra inline — call these helpers everywhere.
|
||||
|
||||
/** Master-timeline seconds -> a video's local playhead seconds (FROZEN). */
|
||||
export function tVideoFromGlobal(tGlobal, offsetMs, driftPpm = 0) {
|
||||
return (tGlobal - offsetMs / 1000) * (1 + driftPpm * 1e-6);
|
||||
}
|
||||
|
||||
/** Inverse of tVideoFromGlobal (FROZEN). */
|
||||
export function tGlobalFromVideo(tVideo, offsetMs, driftPpm = 0) {
|
||||
return tVideo / (1 + driftPpm * 1e-6) + offsetMs / 1000;
|
||||
}
|
||||
@ -1,330 +1,91 @@
|
||||
// Festival 4D viewer — composition root (spec M4/M5/M6 + timeline markers).
|
||||
//
|
||||
// Boots by loading the frozen synthetic API (manifest, poses, anchors, events), builds the
|
||||
// video grid + 3D scene + timeline, wires transport controls + keyboard shortcuts, and runs a
|
||||
// single master animation loop that: advances the master clock & corrects every video
|
||||
// (transport), redraws each overlay for its displayed frame, updates the 3D frusta / follow-cam,
|
||||
// moves the timeline playhead, and refreshes the dev sync overlay. The master clock lives in
|
||||
// transport.js and is derived from performance.now() — never from a <video> element.
|
||||
// Foundation hello page. Fetches /api/manifest (proving the API + CORS work cross-origin)
|
||||
// and runs the frozen pose.js self-test in the browser. Lane C (M4–M6) replaces this with
|
||||
// the real app; the modules alongside it are stubs to fill in.
|
||||
|
||||
import { state, on, API_BASE, referenceVideoId } from "./state.js";
|
||||
import { transport } from "./transport.js";
|
||||
import { createVideoGrid } from "./videoGrid.js";
|
||||
import { drawOverlay, projectWorldToVideoPx } from "./overlays.js";
|
||||
import { poseAt } from "./lib/poseTrack.js";
|
||||
import { scene3d, videoColor } from "./scene3d.js";
|
||||
import { timeline } from "./timeline.js";
|
||||
import { annotator } from "./annotate.js";
|
||||
import { camPath } from "./camPath.js";
|
||||
// Phase-5 lane modules (stubs until lanes E/F/G fill them; each wires its own hidden button).
|
||||
import { director } from "./director.js";
|
||||
import { exportVideo } from "./exportVideo.js";
|
||||
import { anchorPanel } from "./anchorPanel.js";
|
||||
import { photoMode } from "./photoMode.js";
|
||||
import { fx } from "./fx.js";
|
||||
import { pathsStore } from "./pathsStore.js";
|
||||
// Phase-6 lane module (stub until lane J fills it — M21 ribbons/panel, M22 follow-a-friend).
|
||||
import { friendTracks } from "./friendTracks.js";
|
||||
import { selfTest as poseSelfTest } from "./lib/pose.js";
|
||||
|
||||
const laneModules = { director, exportVideo, anchorPanel, photoMode, fx, pathsStore, friendTracks };
|
||||
// The FastAPI backend (`python -m festival4d serve`). The Vite dev server runs on :5173;
|
||||
// this cross-origin fetch exercises the CORS config in api.py.
|
||||
const API_BASE = "http://localhost:8000";
|
||||
|
||||
async function getJSON(path) {
|
||||
const resp = await fetch(API_BASE + path);
|
||||
if (!resp.ok) throw new Error(`${path} -> HTTP ${resp.status}`);
|
||||
return resp.json();
|
||||
const app = document.getElementById("app");
|
||||
|
||||
function card(html) {
|
||||
const el = document.createElement("div");
|
||||
el.className = "card";
|
||||
el.innerHTML = html;
|
||||
return el;
|
||||
}
|
||||
|
||||
function fmtTime(s) {
|
||||
s = Math.max(0, s);
|
||||
const m = Math.floor(s / 60);
|
||||
const sec = s - m * 60;
|
||||
return `${m}:${sec.toFixed(1).padStart(4, "0")}`;
|
||||
function pill(ok, text) {
|
||||
return `<span class="pill ${ok ? "ok" : "bad"}">${text}</span>`;
|
||||
}
|
||||
|
||||
function el(id) {
|
||||
return document.getElementById(id);
|
||||
}
|
||||
|
||||
async function boot() {
|
||||
const loading = el("loading");
|
||||
function renderPoseSelfTest() {
|
||||
try {
|
||||
const manifest = await getJSON("/api/manifest");
|
||||
state.manifest = manifest;
|
||||
state.videos = manifest.videos;
|
||||
state.tGlobalMax = manifest.t_global_max;
|
||||
state.hasPoses = manifest.has_poses;
|
||||
// Capture is opt-in on the backend; only surface the link when it's actually mounted.
|
||||
if (manifest.has_capture) {
|
||||
const link = el("capture-link");
|
||||
link.href = `${API_BASE}/capture`;
|
||||
link.style.display = "";
|
||||
}
|
||||
state.hasSplat = !!manifest.has_splat;
|
||||
state.hasTracks = !!manifest.has_tracks; // phase 6 (M18): any solved friend track
|
||||
// Capsule (M17): a static baked bundle has no write API — hide everything .write-ui.
|
||||
state.capsule = !!manifest.capsule;
|
||||
document.body.classList.toggle("capsule", state.capsule);
|
||||
for (const v of state.videos) state.enabled[v.id] = true;
|
||||
state.audioSourceId = referenceVideoId();
|
||||
|
||||
const [anchors, events] = await Promise.all([
|
||||
getJSON("/api/anchors"),
|
||||
getJSON("/api/events"),
|
||||
]);
|
||||
state.anchors = anchors;
|
||||
state.events = events;
|
||||
|
||||
// Friend tracks (M18): only fetched when the backend reports solved tracks, so a project
|
||||
// without them pays nothing. Failure is non-fatal — tracks are an overlay, not core.
|
||||
if (state.hasTracks) {
|
||||
try {
|
||||
state.tracks = await getJSON("/api/tracks");
|
||||
} catch (err) {
|
||||
console.warn("[festival4d] friend tracks failed to load", err);
|
||||
}
|
||||
}
|
||||
|
||||
if (state.hasPoses) {
|
||||
const lists = await Promise.all(
|
||||
state.videos.map((v) => getJSON(`/api/videos/${v.id}/poses`))
|
||||
);
|
||||
state.videos.forEach((v, i) => (state.poses[v.id] = lists[i]));
|
||||
}
|
||||
|
||||
buildUI();
|
||||
startLoop();
|
||||
loading.style.display = "none";
|
||||
|
||||
// Debug hook so the transport/scene can be driven in tests (e.g. a setTimeout pump when
|
||||
// a headless pane throttles rAF). Dev builds always; production only when opted in via
|
||||
// localStorage.setItem("f4dDebug", "1").
|
||||
if (import.meta.env?.DEV || localStorage.getItem("f4dDebug")) {
|
||||
window.__f4d = {
|
||||
state, transport, scene3d, timeline, camPath, cells: () => cells,
|
||||
poseAt, projectWorldToVideoPx, ...laneModules,
|
||||
};
|
||||
}
|
||||
poseSelfTest();
|
||||
return card(
|
||||
`<strong>Pose contract</strong> ${pill(true, "self-test passed")}<br />` +
|
||||
`<span class="sub">frontend/src/lib/pose.js matches the frozen COLMAP→Three.js vectors.</span>`
|
||||
);
|
||||
} catch (err) {
|
||||
console.error("[festival4d] boot failed", err);
|
||||
loading.innerHTML = `<div class="err">Failed to load the project.<br><code>${err.message}</code>` +
|
||||
`<br><span class="dim">Is the backend running? <code>python -m festival4d serve</code></span></div>`;
|
||||
return card(
|
||||
`<strong>Pose contract</strong> ${pill(false, "self-test FAILED")}<br />` +
|
||||
`<code>${err.message}</code>`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
let cells = [];
|
||||
|
||||
function buildUI() {
|
||||
// Video grid
|
||||
const grid = el("video-grid");
|
||||
cells = createVideoGrid(grid, state.videos, API_BASE, transport);
|
||||
transport.register(cells);
|
||||
|
||||
// Per-video snap buttons -> scene3d (grid module leaves them unwired).
|
||||
cells.forEach((cell, i) => {
|
||||
cell.el.style.setProperty("--accent", videoColor(i));
|
||||
cell.snapBtn.addEventListener("click", (e) => {
|
||||
e.stopPropagation();
|
||||
scene3d.snapTo(cell.id);
|
||||
});
|
||||
});
|
||||
|
||||
// 3D scene
|
||||
scene3d.init(el("scene3d"));
|
||||
|
||||
// Timeline
|
||||
timeline.init(el("timeline"), transport);
|
||||
|
||||
// M8: correction panel + bbox annotation → 3D anchors.
|
||||
annotator.init(el("annot-panel"), cells);
|
||||
on("event-selected", (ev) => annotator.openForEvent(ev));
|
||||
on("anchors-changed", () => scene3d.refreshAnchors());
|
||||
on("events-changed", () => timeline.refresh());
|
||||
|
||||
wireTransportControls();
|
||||
wireCamPath();
|
||||
wireKeyboard();
|
||||
positionVideosWhenReady();
|
||||
|
||||
// Phase-5 lane modules: each init() wires its own (hidden) controls and unhides them when
|
||||
// the lane has filled the stub. A broken module degrades to its button staying hidden.
|
||||
for (const [name, mod] of Object.entries(laneModules)) {
|
||||
try {
|
||||
mod.init();
|
||||
} catch (err) {
|
||||
console.warn(`[festival4d] lane module "${name}" init failed`, err);
|
||||
}
|
||||
}
|
||||
|
||||
// Reflect follow/roam state on the free-roam button; snapping cancels a playing path.
|
||||
on("follow", (id) => {
|
||||
el("btn-roam").classList.toggle("active", id == null);
|
||||
if (id != null) resetPathButton();
|
||||
});
|
||||
function renderManifest(m) {
|
||||
const rows = m.videos
|
||||
.map(
|
||||
(v) => `
|
||||
<tr>
|
||||
<td>${v.id}</td>
|
||||
<td><code>${v.filename}</code></td>
|
||||
<td>${v.duration_s.toFixed(2)} s</td>
|
||||
<td>${v.width}×${v.height} @ ${v.fps}</td>
|
||||
<td>${v.offset_ms === null ? "—" : v.offset_ms + " ms"}</td>
|
||||
</tr>`
|
||||
)
|
||||
.join("");
|
||||
return card(`
|
||||
<strong>API manifest</strong> ${pill(true, "fetched (CORS ok)")}<br />
|
||||
<span class="sub">
|
||||
t_global_max = ${m.t_global_max.toFixed(2)} s ·
|
||||
has_poses = ${m.has_poses ? "yes" : "no"} ·
|
||||
${m.videos.length} videos
|
||||
</span>
|
||||
<table>
|
||||
<thead><tr><th>id</th><th>file</th><th>duration</th><th>frame</th><th>offset</th></tr></thead>
|
||||
<tbody>${rows}</tbody>
|
||||
</table>
|
||||
<div class="sub" style="margin-top:0.75rem">
|
||||
Videos served with HTTP Range from <code>${API_BASE}/media/…</code>
|
||||
</div>
|
||||
`);
|
||||
}
|
||||
|
||||
function resetPathButton() {
|
||||
const b = el("btn-path");
|
||||
b.textContent = "▶ Path";
|
||||
b.classList.remove("active");
|
||||
}
|
||||
async function main() {
|
||||
app.innerHTML = "";
|
||||
app.appendChild(renderPoseSelfTest());
|
||||
|
||||
function wireCamPath() {
|
||||
camPath.onChange = (n) => {
|
||||
el("key-count").textContent = String(n);
|
||||
if (!camPath.playing) resetPathButton();
|
||||
};
|
||||
el("btn-key").addEventListener("click", () => camPath.addKeyframe());
|
||||
el("btn-path").addEventListener("click", () => {
|
||||
const b = el("btn-path");
|
||||
if (camPath.playing) {
|
||||
camPath.stop();
|
||||
resetPathButton();
|
||||
} else if (camPath.play()) {
|
||||
b.textContent = "⏹ Path";
|
||||
b.classList.add("active");
|
||||
if (!state.playing) transport.toggle(); // advance master time so the path animates
|
||||
}
|
||||
});
|
||||
el("btn-path-export").addEventListener("click", () => {
|
||||
const blob = new Blob([camPath.toJSON()], { type: "application/json" });
|
||||
const a = document.createElement("a");
|
||||
a.href = URL.createObjectURL(blob);
|
||||
a.download = "festival4d-campath.json";
|
||||
a.click();
|
||||
URL.revokeObjectURL(a.href);
|
||||
});
|
||||
const fileInput = el("path-file");
|
||||
el("btn-path-import").addEventListener("click", () => fileInput.click());
|
||||
fileInput.addEventListener("change", async () => {
|
||||
const f = fileInput.files?.[0];
|
||||
if (f) {
|
||||
try {
|
||||
camPath.fromJSON(await f.text());
|
||||
} catch (err) {
|
||||
console.error("[festival4d] camera-path import failed", err);
|
||||
}
|
||||
}
|
||||
fileInput.value = "";
|
||||
});
|
||||
el("btn-path-clear").addEventListener("click", () => {
|
||||
camPath.clear();
|
||||
resetPathButton();
|
||||
});
|
||||
}
|
||||
|
||||
function wireTransportControls() {
|
||||
el("btn-play").addEventListener("click", () => transport.toggle());
|
||||
el("btn-back").addEventListener("click", () => transport.seekBy(-1));
|
||||
el("btn-fwd").addEventListener("click", () => transport.seekBy(1));
|
||||
el("rate-select").addEventListener("change", (e) =>
|
||||
transport.setRate(parseFloat(e.target.value))
|
||||
);
|
||||
el("btn-roam").addEventListener("click", () => {
|
||||
if (camPath.playing) {
|
||||
camPath.stop();
|
||||
resetPathButton();
|
||||
}
|
||||
scene3d.freeRoam();
|
||||
});
|
||||
|
||||
// 🎧 spatial audio — only offered when there are camera poses to place emitters at.
|
||||
const spatialBtn = el("btn-spatial");
|
||||
if (state.hasPoses) spatialBtn.style.display = "";
|
||||
spatialBtn.addEventListener("click", () => transport.setSpatial(!state.spatialAudio));
|
||||
on("spatial", (enabled) => spatialBtn.classList.toggle("active", enabled));
|
||||
}
|
||||
|
||||
function wireKeyboard() {
|
||||
window.addEventListener("keydown", (e) => {
|
||||
if (e.target && /^(INPUT|SELECT|TEXTAREA)$/.test(e.target.tagName)) return;
|
||||
if (e.code === "Space") {
|
||||
e.preventDefault();
|
||||
transport.toggle();
|
||||
} else if (e.code === "ArrowLeft") {
|
||||
e.preventDefault();
|
||||
transport.seekBy(e.shiftKey ? -5 : -1);
|
||||
} else if (e.code === "ArrowRight") {
|
||||
e.preventDefault();
|
||||
transport.seekBy(e.shiftKey ? 5 : 1);
|
||||
} else if (e.code === "Escape" || e.code === "Digit0") {
|
||||
if (camPath.playing) {
|
||||
camPath.stop();
|
||||
resetPathButton();
|
||||
}
|
||||
scene3d.freeRoam();
|
||||
} else if (e.code === "KeyK") {
|
||||
e.preventDefault();
|
||||
camPath.addKeyframe();
|
||||
} else if (/^Digit[1-9]$/.test(e.code)) {
|
||||
const n = parseInt(e.code.slice(5), 10) - 1;
|
||||
const v = state.videos[n];
|
||||
if (v) scene3d.snapTo(v.id);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
function positionVideosWhenReady() {
|
||||
let remaining = cells.length;
|
||||
if (remaining === 0) return;
|
||||
const done = () => {
|
||||
if (--remaining === 0) transport.seek(0); // park every video on its t=0 frame
|
||||
};
|
||||
for (const { video } of cells) {
|
||||
if (video.readyState >= 1) done();
|
||||
else video.addEventListener("loadedmetadata", done, { once: true });
|
||||
try {
|
||||
const resp = await fetch(`${API_BASE}/api/manifest`);
|
||||
if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
|
||||
const manifest = await resp.json();
|
||||
app.appendChild(renderManifest(manifest));
|
||||
console.log("[festival4d] manifest loaded", manifest);
|
||||
} catch (err) {
|
||||
app.appendChild(
|
||||
card(
|
||||
`<strong>API manifest</strong> ${pill(false, "fetch failed")}<br />` +
|
||||
`<code>${err.message}</code><br />` +
|
||||
`<span class="sub">Is the backend running? <code>python -m festival4d serve</code></span>`
|
||||
)
|
||||
);
|
||||
console.error("[festival4d] manifest fetch failed", err);
|
||||
}
|
||||
}
|
||||
|
||||
function startLoop() {
|
||||
const frame = () => {
|
||||
transport.tick();
|
||||
for (const cell of cells) drawOverlay(cell);
|
||||
fx.update(state.tGlobal); // M15: playhead-crossing effects (no-op while stubbed)
|
||||
friendTracks.update(state.tGlobal); // M21/M22: move ribbons/labels + chase cam (no-op while stubbed)
|
||||
camPath.update(state.tGlobal); // M9: drive the viewer camera before the scene renders
|
||||
scene3d.update();
|
||||
timeline.update(state.tGlobal);
|
||||
updateHud();
|
||||
requestAnimationFrame(frame);
|
||||
};
|
||||
requestAnimationFrame(frame);
|
||||
}
|
||||
|
||||
function updateHud() {
|
||||
el("btn-play").textContent = state.playing ? "⏸" : "▶";
|
||||
el("time-display").textContent = `${fmtTime(state.tGlobal)} / ${fmtTime(state.tGlobalMax)}`;
|
||||
|
||||
// Dev sync overlay + per-cell status.
|
||||
let devRows = "";
|
||||
for (const cell of cells) {
|
||||
const { id, meta } = cell;
|
||||
const err = state.syncErrorMs[id];
|
||||
const inRange = state.inRange[id];
|
||||
const enabled = state.enabled[id] !== false;
|
||||
let label, cls;
|
||||
if (!enabled) {
|
||||
label = "disabled";
|
||||
cls = "dim";
|
||||
} else if (!inRange) {
|
||||
label = "out of range";
|
||||
cls = "dim";
|
||||
} else if (err == null) {
|
||||
label = "—";
|
||||
cls = "dim";
|
||||
} else {
|
||||
label = `${err >= 0 ? "+" : ""}${err.toFixed(0)} ms`;
|
||||
cls = Math.abs(err) < 50 ? "ok" : "bad";
|
||||
}
|
||||
devRows += `<div class="dev-row"><span>${meta.filename}</span><span class="${cls}">${label}</span></div>`;
|
||||
|
||||
// Cell overlay state (dim out-of-range / disabled, mark audio + follow).
|
||||
cell.el.classList.toggle("out", !inRange || !enabled);
|
||||
cell.el.classList.toggle("audio-on", state.audioSourceId === id);
|
||||
cell.el.classList.toggle("followed", state.followCameraId === id);
|
||||
cell.statusEl.textContent = enabled ? (inRange ? label : "waiting…") : "off";
|
||||
cell.statusEl.className = `cell-status ${cls}`;
|
||||
}
|
||||
el("dev-rows").innerHTML = devRows;
|
||||
}
|
||||
|
||||
boot();
|
||||
main();
|
||||
|
||||
@ -1,137 +1,6 @@
|
||||
// 3D -> 2D anchor projection onto each video's overlay canvas — the "x-ray" HUD (spec M6).
|
||||
//
|
||||
// For each visible video we build a Three.js PerspectiveCamera at that video's CURRENT pose
|
||||
// (from the FROZEN colmapToThreejs helper, matrixWorld path) with an fov derived from the
|
||||
// stored intrinsics, project every anchor, drop anchors behind the camera (camera-space z >= 0),
|
||||
// convert NDC -> native video pixels -> canvas pixels through the letterbox content rect, and
|
||||
// draw a dot + halo + label. Anchors are drawn regardless of real-world occlusion — that IS the
|
||||
// x-ray feature. We consume pose.js; we never reimplement the COLMAP->Three.js conversion.
|
||||
// 3D→2D anchor projection onto each video's overlay canvas — the "x-ray" HUD (spec M6).
|
||||
// STUB — lane C fills this in. For each anchor + visible video: build the video's current
|
||||
// Three.js camera (pose.js helper), project the anchor, convert NDC→canvas px, skip if
|
||||
// behind camera (check z_cam < 0). Anchors render regardless of occlusion — that's the feature.
|
||||
|
||||
import * as THREE from "three";
|
||||
import { colmapToThreejs } from "./lib/pose.js";
|
||||
import { poseAt } from "./lib/poseTrack.js";
|
||||
import { contentRect } from "./videoGrid.js";
|
||||
import { state } from "./state.js";
|
||||
|
||||
// Reused across all videos/frames (every synthetic video shares intrinsics + resolution, and
|
||||
// we fully overwrite the matrices each call, so a single scratch camera is safe and cheap).
|
||||
const _cam = new THREE.PerspectiveCamera();
|
||||
_cam.matrixAutoUpdate = false;
|
||||
const _m4 = new THREE.Matrix4();
|
||||
const _v = new THREE.Vector3();
|
||||
|
||||
function configureProjector(pose, W, H) {
|
||||
// Centered-principal-point pinhole -> PerspectiveCamera (spec M5/M6 prototype assumption;
|
||||
// the synthetic fixture has cx=W/2, cy=H/2 exactly).
|
||||
const fy = pose.intrinsics.fy;
|
||||
_cam.fov = (2 * Math.atan(H / (2 * fy)) * 180) / Math.PI;
|
||||
_cam.aspect = W / H;
|
||||
_cam.near = 0.01;
|
||||
_cam.far = 2000;
|
||||
_cam.updateProjectionMatrix();
|
||||
|
||||
const { matrixWorld } = colmapToThreejs(pose.q, pose.t);
|
||||
_m4.fromArray(matrixWorld);
|
||||
_cam.matrixWorld.copy(_m4);
|
||||
_cam.matrixWorldInverse.copy(_m4).invert();
|
||||
}
|
||||
|
||||
/** Project a world point to native video pixels, or null if behind the camera. */
|
||||
function projectToVideoPx(x, y, z, W, H) {
|
||||
_v.set(x, y, z);
|
||||
// Behind-camera test in camera space: Three.js cameras look down -z, so a visible point has
|
||||
// camera-space z < 0. Do this BEFORE .project() (which mutates _v into NDC).
|
||||
_v.applyMatrix4(_cam.matrixWorldInverse);
|
||||
if (_v.z >= 0) return null;
|
||||
_v.applyMatrix4(_cam.projectionMatrix); // perspective divide happens inside .project(); do it manually
|
||||
// _v is now clip space already divided (applyMatrix4 on a Vector3 divides by w). NDC in [-1,1].
|
||||
const u = (_v.x * 0.5 + 0.5) * W;
|
||||
const vpx = (1 - (_v.y * 0.5 + 0.5)) * H; // flip Y: NDC +y is up, pixels grow downward
|
||||
return { u, v: vpx };
|
||||
}
|
||||
|
||||
/**
|
||||
* Project a world point to native video pixels through the FROZEN pose helper for a given
|
||||
* COLMAP pose (as returned by poseAt) and video resolution. Returns {u, v} or null (behind).
|
||||
* Single source of truth for overlay projection — used by drawOverlay and by tests.
|
||||
*/
|
||||
export function projectWorldToVideoPx(pose, x, y, z, W, H) {
|
||||
configureProjector(pose, W, H);
|
||||
return projectToVideoPx(x, y, z, W, H);
|
||||
}
|
||||
|
||||
function drawMarker(ctx, x, y, anchor) {
|
||||
const color = anchor.color || "#59d499";
|
||||
// halo
|
||||
ctx.beginPath();
|
||||
ctx.arc(x, y, 5.5, 0, Math.PI * 2);
|
||||
ctx.fillStyle = "rgba(255,255,255,0.85)";
|
||||
ctx.fill();
|
||||
// dot
|
||||
ctx.beginPath();
|
||||
ctx.arc(x, y, 4, 0, Math.PI * 2);
|
||||
ctx.fillStyle = color;
|
||||
ctx.fill();
|
||||
// label
|
||||
const text = anchor.label || "";
|
||||
if (text) {
|
||||
ctx.font = "600 11px -apple-system, system-ui, sans-serif";
|
||||
ctx.textBaseline = "middle";
|
||||
const tx = x + 9;
|
||||
const ty = y;
|
||||
const w = ctx.measureText(text).width;
|
||||
ctx.fillStyle = "rgba(6,8,14,0.72)";
|
||||
ctx.fillRect(tx - 3, ty - 8, w + 6, 16);
|
||||
ctx.fillStyle = "#e6e8ee";
|
||||
ctx.fillText(text, tx, ty + 0.5);
|
||||
}
|
||||
}
|
||||
|
||||
/** Redraw one cell's overlay for the frame currently displayed by its <video>. */
|
||||
export function drawOverlay(cell) {
|
||||
const { video, canvas } = cell;
|
||||
const ctx = canvas.getContext("2d");
|
||||
const dpr = window.devicePixelRatio || 1;
|
||||
const elW = video.clientWidth;
|
||||
const elH = video.clientHeight;
|
||||
if (elW === 0 || elH === 0) return;
|
||||
|
||||
// Resolution-match the canvas backing store to the element (dpr-aware).
|
||||
const bw = Math.round(elW * dpr);
|
||||
const bh = Math.round(elH * dpr);
|
||||
if (canvas.width !== bw || canvas.height !== bh) {
|
||||
canvas.width = bw;
|
||||
canvas.height = bh;
|
||||
canvas.style.width = elW + "px";
|
||||
canvas.style.height = elH + "px";
|
||||
}
|
||||
ctx.setTransform(dpr, 0, 0, dpr, 0, 0);
|
||||
ctx.clearRect(0, 0, elW, elH);
|
||||
|
||||
if (!state.hasPoses || !state.inRange[cell.id]) return;
|
||||
if (video.videoWidth === 0) return;
|
||||
const poses = state.poses[cell.id];
|
||||
if (!poses || poses.length === 0 || state.anchors.length === 0) return;
|
||||
|
||||
// Use the frame the <video> is ACTUALLY showing (currentTime), not the master target, so the
|
||||
// overlay tracks the displayed picture even mid-correction.
|
||||
const pose = poseAt(poses, video.currentTime);
|
||||
if (!pose) return;
|
||||
|
||||
const W = video.videoWidth;
|
||||
const H = video.videoHeight;
|
||||
configureProjector(pose, W, H);
|
||||
const rect = contentRect(video);
|
||||
|
||||
for (const a of state.anchors) {
|
||||
const p = projectToVideoPx(a.x, a.y, a.z, W, H);
|
||||
if (!p) continue;
|
||||
const cx = rect.ox + (p.u / W) * rect.cw;
|
||||
const cy = rect.oy + (p.v / H) * rect.ch;
|
||||
// Skip anchors that fall well outside the visible content rect.
|
||||
const m = 24;
|
||||
if (cx < rect.ox - m || cx > rect.ox + rect.cw + m) continue;
|
||||
if (cy < rect.oy - m || cy > rect.oy + rect.ch + m) continue;
|
||||
drawMarker(ctx, cx, cy, a);
|
||||
}
|
||||
}
|
||||
export {};
|
||||
|
||||
@ -1,125 +0,0 @@
|
||||
// Server-side camera paths UI (spec M16, lane G).
|
||||
//
|
||||
// Pre-wired by foundation2: the hidden controls in index.html — #btn-path-save (💾),
|
||||
// #path-load-select (dropdown), #btn-path-del (🗑) — this module's import + init() call in
|
||||
// main.js, and the API: GET /api/paths (list of {id,name,created_at}),
|
||||
// GET /api/paths/{id} ({..., json: <camPath JSON>}), POST /api/paths {name, json:<text>}
|
||||
// (422 on invalid), DELETE /api/paths/{id}.
|
||||
//
|
||||
// Save = prompt for a name, POST camPath.toJSON(); load = fetch the selected path and
|
||||
// camPath.fromJSON(json); delete = DELETE the selected id; the dropdown refreshes after
|
||||
// every mutation. #btn-path-save / #btn-path-del carry `write-ui`, so capsule mode
|
||||
// (contract #5) hides them for free; the load dropdown stays — the capsule bakes
|
||||
// api/paths + api/paths/{id}, so loading keeps working zero-backend. If the listing
|
||||
// fetch fails in a capsule (older bundle without baked paths), the dropdown hides too.
|
||||
|
||||
import { state } from "./state.js";
|
||||
import { camPath } from "./camPath.js";
|
||||
|
||||
const PLACEHOLDER = "— saved paths —";
|
||||
|
||||
function el(id) {
|
||||
return document.getElementById(id);
|
||||
}
|
||||
|
||||
async function api(path, options) {
|
||||
const resp = await fetch(state.apiBase + path, options);
|
||||
if (!resp.ok) throw new Error(`${path} -> HTTP ${resp.status}`);
|
||||
return resp.json();
|
||||
}
|
||||
|
||||
export const pathsStore = {
|
||||
ready: false,
|
||||
|
||||
init() {
|
||||
const saveBtn = el("btn-path-save");
|
||||
const select = el("path-load-select");
|
||||
const delBtn = el("btn-path-del");
|
||||
if (!saveBtn || !select || !delBtn) return;
|
||||
|
||||
saveBtn.addEventListener("click", () => this.save());
|
||||
delBtn.addEventListener("click", () => this.remove());
|
||||
select.addEventListener("change", () => this.load());
|
||||
|
||||
saveBtn.style.display = "";
|
||||
select.style.display = "";
|
||||
delBtn.style.display = "";
|
||||
this.ready = true;
|
||||
this.refresh(); // async initial fill of the dropdown
|
||||
},
|
||||
|
||||
/** Rebuild the dropdown from GET /api/paths, keeping the selection when it survives. */
|
||||
async refresh(selectId = null) {
|
||||
const select = el("path-load-select");
|
||||
const keep = selectId != null ? String(selectId) : select.value;
|
||||
let paths;
|
||||
try {
|
||||
paths = await api("/api/paths");
|
||||
} catch (err) {
|
||||
console.warn("[festival4d] saved-paths listing unavailable", err);
|
||||
if (state.capsule) select.style.display = "none"; // baked bundle without paths
|
||||
return;
|
||||
}
|
||||
select.innerHTML = "";
|
||||
const ph = document.createElement("option");
|
||||
ph.value = "";
|
||||
ph.textContent = PLACEHOLDER;
|
||||
select.appendChild(ph);
|
||||
for (const p of paths) {
|
||||
const opt = document.createElement("option");
|
||||
opt.value = String(p.id);
|
||||
opt.textContent = p.name;
|
||||
opt.title = p.created_at ?? "";
|
||||
select.appendChild(opt);
|
||||
}
|
||||
select.value = [...select.options].some((o) => o.value === keep) ? keep : "";
|
||||
},
|
||||
|
||||
/** 💾 Save the current camera path under a prompted name. */
|
||||
async save() {
|
||||
if (!camPath.keyframes.length) {
|
||||
window.alert("No keyframes to save — press K (or ⏺+) to add some first.");
|
||||
return;
|
||||
}
|
||||
const name = window.prompt("Name this camera path:", `path ${new Date().toLocaleString()}`);
|
||||
if (!name || !name.trim()) return;
|
||||
try {
|
||||
const created = await api("/api/paths", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ name: name.trim(), json: camPath.toJSON() }),
|
||||
});
|
||||
await this.refresh(created.id); // leave the new path selected
|
||||
} catch (err) {
|
||||
console.error("[festival4d] saving camera path failed", err);
|
||||
}
|
||||
},
|
||||
|
||||
/** Load the selected saved path into camPath. */
|
||||
async load() {
|
||||
const id = el("path-load-select").value;
|
||||
if (!id) return;
|
||||
try {
|
||||
const data = await api(`/api/paths/${id}`);
|
||||
camPath.fromJSON(data.json);
|
||||
} catch (err) {
|
||||
console.error("[festival4d] loading camera path failed", err);
|
||||
this.refresh(); // it may have been deleted elsewhere — resync the dropdown
|
||||
}
|
||||
},
|
||||
|
||||
/** 🗑 Delete the selected saved path. */
|
||||
async remove() {
|
||||
const select = el("path-load-select");
|
||||
const id = select.value;
|
||||
if (!id) return;
|
||||
const name = select.options[select.selectedIndex]?.textContent ?? id;
|
||||
if (!window.confirm(`Delete saved path "${name}"?`)) return;
|
||||
try {
|
||||
await api(`/api/paths/${id}`, { method: "DELETE" });
|
||||
} catch (err) {
|
||||
console.error("[festival4d] deleting camera path failed", err);
|
||||
}
|
||||
await this.refresh();
|
||||
},
|
||||
};
|
||||
@ -1,117 +0,0 @@
|
||||
// Photo mode (spec M14, lane F).
|
||||
//
|
||||
// 📷 / "P": pause the transport if playing, hide every helper (grid/axes/frusta/gizmos/anchor
|
||||
// labels — scene3d.setHelpersVisible(false), one call), render ONE frame at 3840 px wide
|
||||
// (preserving the pane's aspect) by temporarily resizing scene3d.renderer with pixelRatio 1,
|
||||
// snapshot it via canvas.toBlob (the copy is taken synchronously at call time, so it is safe
|
||||
// to restore the renderer immediately after — encoding continues async), download
|
||||
// festival4d-photo.png, then restore helpers / size / pixelRatio / camera aspect / play state.
|
||||
//
|
||||
// Splat note (pitfall #5): the DropInViewer is a plain child of scene3d.scene and renders with
|
||||
// the same renderer.render(scene, camera) call, so no splat-specific branch exists here; live
|
||||
// verification on a trained splat is deferred to the coordinator.
|
||||
//
|
||||
// No write-ui class: photo mode is read-only and stays available in capsule bundles.
|
||||
|
||||
import { state } from "./state.js";
|
||||
import { transport } from "./transport.js";
|
||||
import { scene3d } from "./scene3d.js";
|
||||
|
||||
const PHOTO_WIDTH = 3840;
|
||||
|
||||
export const photoMode = {
|
||||
ready: false,
|
||||
_busy: false,
|
||||
|
||||
init() {
|
||||
const btn = document.getElementById("btn-photo");
|
||||
if (!btn) return; // pre-wired DOM missing — degrade to hidden
|
||||
btn.addEventListener("click", () => this.shoot());
|
||||
window.addEventListener("keydown", (e) => {
|
||||
if (e.target && /^(INPUT|SELECT|TEXTAREA)$/.test(e.target.tagName)) return;
|
||||
if (e.code === "KeyP" && !e.metaKey && !e.ctrlKey && !e.altKey) {
|
||||
e.preventDefault();
|
||||
this.shoot();
|
||||
}
|
||||
});
|
||||
btn.style.display = ""; // unhide: the module is live
|
||||
this.ready = true;
|
||||
},
|
||||
|
||||
/** Take the hi-res still. Synchronous through the render + snapshot so no animation frame
|
||||
* (main.js loop) can interleave between resize, render, and capture. */
|
||||
shoot() {
|
||||
if (this._busy) return;
|
||||
const renderer = scene3d.renderer;
|
||||
const camera = scene3d.camera;
|
||||
if (!renderer || !camera || !scene3d.scene) return; // scene not booted yet
|
||||
this._busy = true;
|
||||
|
||||
// --- save everything we are about to touch ---
|
||||
const wasPlaying = state.playing;
|
||||
const prevHelpers = scene3d.helpersVisible;
|
||||
const prevRatio = renderer.getPixelRatio();
|
||||
const prevSize = { w: 0, h: 0 };
|
||||
{
|
||||
// getSize wants a Vector2; a duck-typed target keeps this module three-free.
|
||||
const v = { x: 0, y: 0, set(x, y) { this.x = x; this.y = y; return this; } };
|
||||
renderer.getSize(v);
|
||||
prevSize.w = v.x;
|
||||
prevSize.h = v.y;
|
||||
}
|
||||
const prevAspect = camera.aspect;
|
||||
|
||||
try {
|
||||
if (wasPlaying) transport.pause();
|
||||
scene3d.setHelpersVisible(false);
|
||||
// setHelpersVisible hides grid/axes/gizmos/label-sprites immediately, but the per-video
|
||||
// rig groups (frusta/markers/path lines) get their visibility recomputed inside
|
||||
// scene3d.update(). Run one update at the CURRENT size so the rigs honor the flag
|
||||
// before the hi-res render below.
|
||||
scene3d.update();
|
||||
|
||||
const w = Math.max(1, prevSize.w);
|
||||
const h = Math.max(1, prevSize.h);
|
||||
const outW = PHOTO_WIDTH;
|
||||
const outH = Math.max(1, Math.round((PHOTO_WIDTH * h) / w));
|
||||
|
||||
renderer.setPixelRatio(1);
|
||||
renderer.setSize(outW, outH, false);
|
||||
camera.aspect = outW / outH;
|
||||
camera.updateProjectionMatrix();
|
||||
renderer.render(scene3d.scene, camera);
|
||||
|
||||
// Snapshot NOW (same task as the render — the WebGL buffer is still valid).
|
||||
renderer.domElement.toBlob((blob) => {
|
||||
try {
|
||||
if (!blob) {
|
||||
console.warn("[photoMode] toBlob returned null (buffer too large for this GPU?)");
|
||||
return;
|
||||
}
|
||||
const a = document.createElement("a");
|
||||
a.href = URL.createObjectURL(blob);
|
||||
a.download = "festival4d-photo.png";
|
||||
a.click();
|
||||
setTimeout(() => URL.revokeObjectURL(a.href), 10_000);
|
||||
} finally {
|
||||
this._busy = false;
|
||||
}
|
||||
}, "image/png");
|
||||
} catch (err) {
|
||||
console.warn("[photoMode] capture failed", err);
|
||||
this._busy = false;
|
||||
} finally {
|
||||
// --- restore ALL of it (snapshot already copied; safe immediately) ---
|
||||
try {
|
||||
renderer.setPixelRatio(prevRatio);
|
||||
renderer.setSize(prevSize.w, prevSize.h, false);
|
||||
camera.aspect = prevAspect;
|
||||
camera.updateProjectionMatrix();
|
||||
scene3d.setHelpersVisible(prevHelpers);
|
||||
if (wasPlaying) transport.play();
|
||||
} catch (err) {
|
||||
console.warn("[photoMode] restore failed", err);
|
||||
}
|
||||
}
|
||||
},
|
||||
};
|
||||
@ -1,497 +1,6 @@
|
||||
// Three.js scene (spec M5): point cloud (/api/pointcloud via PLYLoader), each video's camera
|
||||
// path as a line + current-pose frustum wireframe (colored per video), OrbitControls free roam,
|
||||
// and snap-to-camera. ALL COLMAP->Three.js conversion goes through the FROZEN lib/pose.js
|
||||
// helper (colmapToThreejs) — the diag(1,-1,-1) math is never reimplemented here.
|
||||
// Three.js scene: point cloud (/api/pointcloud via PLYLoader), per-video camera paths +
|
||||
// current-pose frusta, OrbitControls, snap-to-camera (spec M5).
|
||||
// STUB — lane C fills this in. Use frontend/src/lib/pose.js (frozen contract) for ALL
|
||||
// COLMAP→Three.js conversion — do not reimplement the diag(1,-1,-1) math.
|
||||
|
||||
import * as THREE from "three";
|
||||
import { OrbitControls } from "three/addons/controls/OrbitControls.js";
|
||||
import { PLYLoader } from "three/addons/loaders/PLYLoader.js";
|
||||
import { colmapToThreejs } from "./lib/pose.js";
|
||||
import { poseAt } from "./lib/poseTrack.js";
|
||||
import { tVideoFromGlobal } from "./lib/timebase.js";
|
||||
import { state, emit } from "./state.js";
|
||||
import { transport } from "./transport.js";
|
||||
|
||||
// Per-video colors (match the frustum, path line, and grid accents).
|
||||
export const VIDEO_COLORS = ["#ff5d73", "#4dd0ff", "#c9a2ff", "#ffcf5d", "#7dffb0", "#ff9d5d"];
|
||||
export function videoColor(index) {
|
||||
return VIDEO_COLORS[index % VIDEO_COLORS.length];
|
||||
}
|
||||
|
||||
const FRUSTUM_DEPTH = 1.2;
|
||||
const DEFAULT_FOV = 50;
|
||||
const STAGE_TARGET = new THREE.Vector3(0, 0.5, 0);
|
||||
|
||||
const _m4 = new THREE.Matrix4();
|
||||
const _pos = new THREE.Vector3();
|
||||
const _quat = new THREE.Quaternion();
|
||||
const _scale = new THREE.Vector3();
|
||||
const _fwd = new THREE.Vector3();
|
||||
const _up = new THREE.Vector3();
|
||||
|
||||
/** Build a camera-facing text label sprite (used for 3D anchor labels). */
|
||||
function makeLabelSprite(text) {
|
||||
const font = 48;
|
||||
const pad = 10;
|
||||
const c = document.createElement("canvas");
|
||||
let ctx = c.getContext("2d");
|
||||
ctx.font = `600 ${font}px -apple-system, system-ui, sans-serif`;
|
||||
const w = Math.ceil(ctx.measureText(text).width) + pad * 2;
|
||||
const h = font + pad * 2;
|
||||
c.width = w;
|
||||
c.height = h;
|
||||
ctx = c.getContext("2d");
|
||||
ctx.font = `600 ${font}px -apple-system, system-ui, sans-serif`;
|
||||
ctx.fillStyle = "rgba(6,8,14,0.74)";
|
||||
ctx.fillRect(0, 0, w, h);
|
||||
ctx.fillStyle = "#e6e8ee";
|
||||
ctx.textBaseline = "middle";
|
||||
ctx.fillText(text, pad, h / 2 + 1);
|
||||
const tex = new THREE.CanvasTexture(c);
|
||||
tex.minFilter = THREE.LinearFilter;
|
||||
const sprite = new THREE.Sprite(
|
||||
new THREE.SpriteMaterial({ map: tex, depthTest: false, transparent: true })
|
||||
);
|
||||
const s = 0.006;
|
||||
sprite.scale.set(w * s, h * s, 1);
|
||||
return sprite;
|
||||
}
|
||||
|
||||
export class Scene3D {
|
||||
constructor() {
|
||||
this.videoIndex = {}; // id -> index (for stable colors)
|
||||
this.rigs = {}; // id -> { frustum, marker, pick, path }
|
||||
this._tween = null;
|
||||
this._raf = null;
|
||||
this.anchorGroup = null; // 3D anchor spheres + labels (M6 anchors + M8 resolved)
|
||||
this.pathMode = false; // when true, camPath (M9) drives the camera; we don't touch it
|
||||
this.helpersVisible = true; // M14 hook: grid/axes/frusta/gizmos/labels toggle
|
||||
this._helperObjs = []; // extra helper objects registered by other modules (camPath gizmos)
|
||||
}
|
||||
|
||||
init(canvas) {
|
||||
this.canvas = canvas;
|
||||
const parent = canvas.parentElement;
|
||||
const w = parent.clientWidth || 1;
|
||||
const h = parent.clientHeight || 1;
|
||||
|
||||
this.renderer = new THREE.WebGLRenderer({ canvas, antialias: true, alpha: false });
|
||||
this.renderer.setPixelRatio(Math.min(window.devicePixelRatio || 1, 2));
|
||||
this.renderer.setSize(w, h, false);
|
||||
this.renderer.setClearColor(0x0a0c11, 1);
|
||||
|
||||
this.scene = new THREE.Scene();
|
||||
this.camera = new THREE.PerspectiveCamera(DEFAULT_FOV, w / h, 0.05, 3000);
|
||||
this.camera.position.set(12, 9, 15);
|
||||
this.camera.lookAt(STAGE_TARGET);
|
||||
|
||||
this.controls = new OrbitControls(this.camera, this.renderer.domElement);
|
||||
this.controls.enableDamping = true;
|
||||
this.controls.dampingFactor = 0.08;
|
||||
this.controls.target.copy(STAGE_TARGET);
|
||||
this.controls.update();
|
||||
|
||||
// Lights (points ignore lighting, but frusta/markers use MeshBasic so this is mostly cosmetic).
|
||||
this.scene.add(new THREE.AmbientLight(0xffffff, 0.9));
|
||||
const grid = new THREE.GridHelper(40, 40, 0x22384a, 0x161c26);
|
||||
grid.position.y = 0;
|
||||
this.scene.add(grid);
|
||||
const axes = new THREE.AxesHelper(1.5);
|
||||
this.scene.add(axes);
|
||||
this._grid = grid;
|
||||
this._axes = axes;
|
||||
|
||||
this.anchorGroup = new THREE.Group();
|
||||
this.scene.add(this.anchorGroup);
|
||||
|
||||
this._buildRigs();
|
||||
this._loadPointCloud();
|
||||
this.refreshAnchors();
|
||||
this._wirePicking();
|
||||
this._wireResize(parent);
|
||||
}
|
||||
|
||||
/** Rebuild the 3D anchor markers from state.anchors (M6 seeded + M8 resolved). */
|
||||
refreshAnchors() {
|
||||
if (!this.anchorGroup) return;
|
||||
for (const child of [...this.anchorGroup.children]) {
|
||||
this.anchorGroup.remove(child);
|
||||
child.geometry?.dispose?.();
|
||||
if (child.material) {
|
||||
child.material.map?.dispose?.();
|
||||
child.material.dispose();
|
||||
}
|
||||
}
|
||||
for (const a of state.anchors) {
|
||||
const color = new THREE.Color(a.color || "#59d499");
|
||||
const sphere = new THREE.Mesh(
|
||||
new THREE.SphereGeometry(0.13, 16, 16),
|
||||
new THREE.MeshBasicMaterial({ color })
|
||||
);
|
||||
sphere.position.set(a.x, a.y, a.z);
|
||||
this.anchorGroup.add(sphere);
|
||||
if (a.label) {
|
||||
const label = makeLabelSprite(a.label);
|
||||
label.position.set(a.x, a.y + 0.3, a.z);
|
||||
label.visible = this.helpersVisible; // photo mode (M14) may be hiding labels right now
|
||||
this.anchorGroup.add(label);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Expose the scene graph for auxiliary overlays (e.g. camPath gizmos M9, moment FX M15). */
|
||||
addObject(obj) {
|
||||
this.scene?.add(obj);
|
||||
}
|
||||
removeObject(obj) {
|
||||
this.scene?.remove(obj);
|
||||
}
|
||||
|
||||
/** Register an added object as a "helper" so setHelpersVisible governs it (camPath gizmos). */
|
||||
registerHelper(obj) {
|
||||
this._helperObjs.push(obj);
|
||||
obj.visible = this.helpersVisible;
|
||||
}
|
||||
|
||||
/**
|
||||
* M14 hook (phase-5 contract #6): show/hide every non-content visual in one call — grid,
|
||||
* axes, camera frusta + markers + path lines, registered gizmo groups, and anchor LABEL
|
||||
* sprites (anchor spheres stay: friend tags belong in photos). update() keeps enforcing
|
||||
* the flag on the per-video rigs, whose visibility it recomputes every frame.
|
||||
*/
|
||||
setHelpersVisible(visible) {
|
||||
this.helpersVisible = !!visible;
|
||||
if (this._grid) this._grid.visible = this.helpersVisible;
|
||||
if (this._axes) this._axes.visible = this.helpersVisible;
|
||||
for (const obj of this._helperObjs) obj.visible = this.helpersVisible;
|
||||
if (this.anchorGroup)
|
||||
for (const child of this.anchorGroup.children)
|
||||
if (child.isSprite) child.visible = this.helpersVisible;
|
||||
}
|
||||
|
||||
/**
|
||||
* M13 hook (phase-5 contract #6): fly the free-roam camera's attention to a world point
|
||||
* (anchor jump-to). Detaches follow-cam / path mode, retargets OrbitControls, and backs the
|
||||
* camera off along its current viewing direction.
|
||||
*/
|
||||
focusOn(x, y, z, dist = 6) {
|
||||
if (this.pathMode) this.exitPathMode();
|
||||
if (state.followCameraId != null) {
|
||||
state.followCameraId = null;
|
||||
this._tween = null;
|
||||
emit("follow", null);
|
||||
}
|
||||
this.camera.fov = DEFAULT_FOV;
|
||||
this.camera.updateProjectionMatrix();
|
||||
const dir = this.camera.position.clone().sub(this.controls.target);
|
||||
if (dir.lengthSq() < 1e-6) dir.set(0, 0.5, 1);
|
||||
dir.normalize().multiplyScalar(dist);
|
||||
this.controls.target.set(x, y, z);
|
||||
this.camera.position.set(x + dir.x, y + dir.y, z + dir.z);
|
||||
this.controls.enabled = true;
|
||||
this.controls.update();
|
||||
}
|
||||
|
||||
/** M9: hand camera control to camPath. update() then leaves the camera alone. */
|
||||
enterPathMode() {
|
||||
this.pathMode = true;
|
||||
this.controls.enabled = false;
|
||||
state.followCameraId = null;
|
||||
this._tween = null;
|
||||
emit("follow", null);
|
||||
}
|
||||
exitPathMode() {
|
||||
if (!this.pathMode) return;
|
||||
this.pathMode = false;
|
||||
this.controls.target.copy(STAGE_TARGET);
|
||||
this.controls.enabled = true;
|
||||
this.controls.update();
|
||||
}
|
||||
/** M9: set the camera pose directly (called by camPath each frame while path-playing). */
|
||||
applyCameraPose(pos, quat, fov) {
|
||||
this.camera.position.copy(pos);
|
||||
this.camera.quaternion.copy(quat);
|
||||
if (fov != null && Math.abs(this.camera.fov - fov) > 1e-3) {
|
||||
this.camera.fov = fov;
|
||||
this.camera.updateProjectionMatrix();
|
||||
}
|
||||
}
|
||||
|
||||
_buildRigs() {
|
||||
state.videos.forEach((v, i) => {
|
||||
this.videoIndex[v.id] = i;
|
||||
const color = new THREE.Color(videoColor(i));
|
||||
const group = new THREE.Group();
|
||||
|
||||
// Frustum wireframe (built in Three.js camera-local space: looks down -z, +y up).
|
||||
const poses = state.poses[v.id] || [];
|
||||
const intr = poses[0]?.intrinsics;
|
||||
const W = v.width;
|
||||
const H = v.height;
|
||||
const hx = intr ? (FRUSTUM_DEPTH * W) / (2 * intr.fx) : FRUSTUM_DEPTH * 0.6;
|
||||
const hy = intr ? (FRUSTUM_DEPTH * H) / (2 * intr.fy) : FRUSTUM_DEPTH * 0.34;
|
||||
const d = FRUSTUM_DEPTH;
|
||||
const c = [
|
||||
[-hx, -hy, -d],
|
||||
[hx, -hy, -d],
|
||||
[hx, hy, -d],
|
||||
[-hx, hy, -d],
|
||||
];
|
||||
const apex = [0, 0, 0];
|
||||
const segs = [
|
||||
apex, c[0], apex, c[1], apex, c[2], apex, c[3],
|
||||
c[0], c[1], c[1], c[2], c[2], c[3], c[3], c[0],
|
||||
];
|
||||
const fg = new THREE.BufferGeometry();
|
||||
fg.setAttribute("position", new THREE.Float32BufferAttribute(segs.flat(), 3));
|
||||
const frustum = new THREE.LineSegments(
|
||||
fg,
|
||||
new THREE.LineBasicMaterial({ color })
|
||||
);
|
||||
frustum.matrixAutoUpdate = true;
|
||||
group.add(frustum);
|
||||
|
||||
// Camera-center marker (visible) + an invisible larger sphere for click picking.
|
||||
const marker = new THREE.Mesh(
|
||||
new THREE.SphereGeometry(0.12, 12, 12),
|
||||
new THREE.MeshBasicMaterial({ color })
|
||||
);
|
||||
group.add(marker);
|
||||
this.scene.add(group);
|
||||
|
||||
const pick = new THREE.Mesh(
|
||||
new THREE.SphereGeometry(0.55, 8, 8),
|
||||
new THREE.MeshBasicMaterial({ visible: false })
|
||||
);
|
||||
pick.userData.videoId = v.id;
|
||||
group.add(pick);
|
||||
|
||||
// Camera path line (one vertex per stored pose center).
|
||||
let path = null;
|
||||
if (poses.length > 1) {
|
||||
const pts = [];
|
||||
for (const p of poses) {
|
||||
const { position } = colmapToThreejs(p.q, p.t);
|
||||
pts.push(position[0], position[1], position[2]);
|
||||
}
|
||||
const pg = new THREE.BufferGeometry();
|
||||
pg.setAttribute("position", new THREE.Float32BufferAttribute(pts, 3));
|
||||
path = new THREE.Line(
|
||||
pg,
|
||||
new THREE.LineBasicMaterial({ color, transparent: true, opacity: 0.5 })
|
||||
);
|
||||
this.scene.add(path);
|
||||
}
|
||||
|
||||
this.rigs[v.id] = { group, frustum, marker, pick, path };
|
||||
});
|
||||
}
|
||||
|
||||
_loadPointCloud() {
|
||||
// Prefer a trained 3DGS splat when the backend has one (docs/modelbeast-crossover.md);
|
||||
// fall back to the sparse COLMAP point cloud. The splat library is heavy, so it's
|
||||
// lazy-imported only when actually needed.
|
||||
if (state.hasSplat) {
|
||||
this._loadSplat();
|
||||
return;
|
||||
}
|
||||
const loader = new PLYLoader();
|
||||
loader.load(
|
||||
state.apiBase + "/api/pointcloud",
|
||||
(geometry) => {
|
||||
geometry.computeBoundingBox();
|
||||
const hasColor = !!geometry.getAttribute("color");
|
||||
const material = new THREE.PointsMaterial({
|
||||
size: 0.05,
|
||||
sizeAttenuation: true,
|
||||
vertexColors: hasColor,
|
||||
color: hasColor ? 0xffffff : 0x88ccff,
|
||||
});
|
||||
this.points = new THREE.Points(geometry, material);
|
||||
this.scene.add(this.points);
|
||||
emit("pointcloud-loaded", geometry.getAttribute("position")?.count ?? 0);
|
||||
},
|
||||
undefined,
|
||||
(err) => {
|
||||
console.warn("[scene3d] point cloud load failed", err);
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
async _loadSplat() {
|
||||
try {
|
||||
const GS = await import("@mkkellogg/gaussian-splats-3d");
|
||||
// DropInViewer is a THREE.Object3D — it joins the existing scene and
|
||||
// renders with our camera/controls, so rigs/anchors/paths overlay as usual.
|
||||
const splatViewer = new GS.DropInViewer({ sharedMemoryForWorkers: false });
|
||||
await splatViewer.addSplatScene(state.apiBase + "/api/splat", {
|
||||
showLoadingUI: false,
|
||||
progressiveLoad: true,
|
||||
format: GS.SceneFormat.Ply,
|
||||
});
|
||||
this.splat = splatViewer;
|
||||
this.scene.add(splatViewer);
|
||||
emit("pointcloud-loaded", -1); // -1 = splat (no discrete point count)
|
||||
} catch (err) {
|
||||
console.warn("[scene3d] splat load failed — falling back to point cloud", err);
|
||||
state.hasSplat = false;
|
||||
this._loadPointCloud();
|
||||
}
|
||||
}
|
||||
|
||||
// Live pose (COLMAP convention) for a video at the current master time, or null.
|
||||
_livePose(videoId) {
|
||||
const poses = state.poses[videoId];
|
||||
if (!poses || poses.length === 0) return null;
|
||||
const meta = state.videos.find((v) => v.id === videoId);
|
||||
const tv = tVideoFromGlobal(state.tGlobal, meta.offset_ms || 0, meta.drift_ppm || 0);
|
||||
return poseAt(poses, tv);
|
||||
}
|
||||
|
||||
/** Snap the viewer camera to a video's pose and follow it while playing. */
|
||||
snapTo(videoId) {
|
||||
const pose = this._livePose(videoId);
|
||||
if (!pose) return;
|
||||
if (this.pathMode) this.exitPathMode();
|
||||
state.followCameraId = videoId;
|
||||
this.controls.enabled = false;
|
||||
const targetFov = (2 * Math.atan(state.videos.find((v) => v.id === videoId).height /
|
||||
(2 * pose.intrinsics.fy)) * 180) / Math.PI;
|
||||
this._tween = {
|
||||
start: performance.now(),
|
||||
dur: 600,
|
||||
fromPos: this.camera.position.clone(),
|
||||
fromQuat: this.camera.quaternion.clone(),
|
||||
fromFov: this.camera.fov,
|
||||
toFov: targetFov,
|
||||
};
|
||||
emit("follow", videoId);
|
||||
}
|
||||
|
||||
/** Detach to free roam (OrbitControls). */
|
||||
freeRoam() {
|
||||
if (state.followCameraId == null) return;
|
||||
state.followCameraId = null;
|
||||
this._tween = null;
|
||||
this.camera.fov = DEFAULT_FOV;
|
||||
this.camera.updateProjectionMatrix();
|
||||
// Re-anchor OrbitControls to the stage, orbiting from the current position.
|
||||
this.controls.target.copy(STAGE_TARGET);
|
||||
this.controls.enabled = true;
|
||||
this.controls.update();
|
||||
emit("follow", null);
|
||||
}
|
||||
|
||||
_applyPoseToCamera(pose, targetFov) {
|
||||
const { matrixWorld } = colmapToThreejs(pose.q, pose.t);
|
||||
_m4.fromArray(matrixWorld);
|
||||
_m4.decompose(_pos, _quat, _scale);
|
||||
const t = this._tween;
|
||||
if (t) {
|
||||
const k = Math.min(1, (performance.now() - t.start) / t.dur);
|
||||
const e = k * k * (3 - 2 * k); // smoothstep
|
||||
this.camera.position.lerpVectors(t.fromPos, _pos, e);
|
||||
this.camera.quaternion.copy(t.fromQuat).slerp(_quat, e);
|
||||
this.camera.fov = t.fromFov + (t.toFov - t.fromFov) * e;
|
||||
this.camera.updateProjectionMatrix();
|
||||
if (k >= 1) this._tween = null;
|
||||
} else {
|
||||
this.camera.position.copy(_pos);
|
||||
this.camera.quaternion.copy(_quat);
|
||||
if (Math.abs(this.camera.fov - targetFov) > 1e-3) {
|
||||
this.camera.fov = targetFov;
|
||||
this.camera.updateProjectionMatrix();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Per-frame update: move frusta to current poses, drive follow-cam, render. */
|
||||
update() {
|
||||
for (const v of state.videos) {
|
||||
const rig = this.rigs[v.id];
|
||||
if (!rig) continue;
|
||||
const pose = this._livePose(v.id);
|
||||
const on = pose && state.enabled[v.id] !== false;
|
||||
rig.group.visible = !!on && this.helpersVisible;
|
||||
if (rig.path) rig.path.visible = state.enabled[v.id] !== false && this.helpersVisible;
|
||||
if (!on) continue;
|
||||
const { matrixWorld } = colmapToThreejs(pose.q, pose.t);
|
||||
_m4.fromArray(matrixWorld);
|
||||
_m4.decompose(rig.group.position, rig.group.quaternion, rig.group.scale);
|
||||
}
|
||||
|
||||
if (this.pathMode) {
|
||||
// camPath (M9) already set the camera pose this frame; don't fight it.
|
||||
} else if (state.followCameraId != null) {
|
||||
const pose = this._livePose(state.followCameraId);
|
||||
if (pose) {
|
||||
const meta = state.videos.find((v) => v.id === state.followCameraId);
|
||||
const targetFov =
|
||||
(2 * Math.atan(meta.height / (2 * pose.intrinsics.fy)) * 180) / Math.PI;
|
||||
this._applyPoseToCamera(pose, targetFov);
|
||||
}
|
||||
} else {
|
||||
this.controls.update();
|
||||
}
|
||||
|
||||
this._updateAudio();
|
||||
this.renderer.render(this.scene, this.camera);
|
||||
}
|
||||
|
||||
// 🎧 Feed the transport's spatial audio: listener = viewer camera, emitters = camera rigs
|
||||
// (rig.group already carries each camera's live pose from the loop above).
|
||||
_updateAudio() {
|
||||
if (!state.spatialAudio) return;
|
||||
this.camera.getWorldPosition(_pos);
|
||||
_fwd.set(0, 0, -1).applyQuaternion(this.camera.quaternion);
|
||||
_up.set(0, 1, 0).applyQuaternion(this.camera.quaternion);
|
||||
const positions = {};
|
||||
for (const v of state.videos) {
|
||||
const rig = this.rigs[v.id];
|
||||
if (rig?.group.visible) positions[v.id] = rig.group.position;
|
||||
}
|
||||
transport.updateSpatial(
|
||||
{ px: _pos.x, py: _pos.y, pz: _pos.z, fx: _fwd.x, fy: _fwd.y, fz: _fwd.z, ux: _up.x, uy: _up.y, uz: _up.z },
|
||||
positions
|
||||
);
|
||||
}
|
||||
|
||||
_wirePicking() {
|
||||
const el = this.renderer.domElement;
|
||||
const ray = new THREE.Raycaster();
|
||||
const ndc = new THREE.Vector2();
|
||||
let downX = 0;
|
||||
let downY = 0;
|
||||
el.addEventListener("pointerdown", (e) => {
|
||||
downX = e.clientX;
|
||||
downY = e.clientY;
|
||||
});
|
||||
el.addEventListener("pointerup", (e) => {
|
||||
if (Math.hypot(e.clientX - downX, e.clientY - downY) > 5) return; // was a drag, not a click
|
||||
const rect = el.getBoundingClientRect();
|
||||
ndc.x = ((e.clientX - rect.left) / rect.width) * 2 - 1;
|
||||
ndc.y = -((e.clientY - rect.top) / rect.height) * 2 + 1;
|
||||
ray.setFromCamera(ndc, this.camera);
|
||||
const picks = Object.values(this.rigs)
|
||||
.map((r) => r.pick)
|
||||
.filter((m) => m.parent && m.parent.visible);
|
||||
const hits = ray.intersectObjects(picks, false);
|
||||
if (hits.length) this.snapTo(hits[0].object.userData.videoId);
|
||||
});
|
||||
}
|
||||
|
||||
_wireResize(parent) {
|
||||
const ro = new ResizeObserver(() => {
|
||||
const w = parent.clientWidth || 1;
|
||||
const h = parent.clientHeight || 1;
|
||||
this.renderer.setSize(w, h, false);
|
||||
// Aspect always follows the 3D canvas (never the video) so nothing stretches; snap-to-
|
||||
// camera only overrides the vertical fov + pose, matching the video's vertical framing.
|
||||
this.camera.aspect = w / h;
|
||||
this.camera.updateProjectionMatrix();
|
||||
});
|
||||
ro.observe(parent);
|
||||
}
|
||||
}
|
||||
|
||||
export const scene3d = new Scene3D();
|
||||
export {};
|
||||
|
||||
@ -1,66 +1,9 @@
|
||||
// Global store: manifest, playhead (t_global), playing, rate, selected camera, audio source,
|
||||
// plus the loaded poses / anchors / events and per-video live playback status. Single source
|
||||
// of truth: the transport writes `tGlobal` every animation frame; every module reads it.
|
||||
// Discrete changes (play/pause/seek/selection) go through the tiny pub/sub below.
|
||||
// Global store: manifest, playhead (t_global), playing, rate, selected camera, audio source.
|
||||
// STUB — lane C (M4) fills this in. Kept as an empty module so the file exists per the
|
||||
// spec's repo layout; main.js does not import it yet.
|
||||
|
||||
// API origin/prefix. Local dev talks to the uvicorn server cross-origin on :8000 (default).
|
||||
// A hosted build sets VITE_API_BASE to a same-origin path prefix (e.g. "/festifun") so every
|
||||
// request — fetch, <video> src, PLY loader — goes through the reverse proxy under that path.
|
||||
// Empty string = same-origin at root. See deploy/DEPLOY.md.
|
||||
// Phase-5 contract #5: a RUNTIME override wins over the build-time value — the memory capsule
|
||||
// (M17) injects `window.__F4D_API_BASE__ = ""` into its baked index.html so the same dist
|
||||
// bundle turns zero-backend. `??` (not `||`): empty string is a valid value (same-origin).
|
||||
export const API_BASE =
|
||||
window.__F4D_API_BASE__ ?? import.meta.env.VITE_API_BASE ?? "http://localhost:8000";
|
||||
// Example shape lane C will build toward:
|
||||
// export const state = { manifest: null, tGlobal: 0, playing: false, rate: 1,
|
||||
// selectedCamera: null, audioSource: null };
|
||||
|
||||
export const state = {
|
||||
apiBase: API_BASE,
|
||||
|
||||
// loaded from the API (see main.js boot)
|
||||
manifest: null,
|
||||
videos: [], // [{id, filename, url, duration_s, fps, width, height, offset_ms, drift_ppm}]
|
||||
tGlobalMax: 0,
|
||||
hasPoses: false,
|
||||
hasSplat: false, // optional 3DGS splat at /api/splat (preferred over the point cloud)
|
||||
capsule: false, // manifest.capsule (M17): static baked bundle => hide ALL write UI
|
||||
poses: {}, // id -> [{frame_idx, t_video_s, q:[w,x,y,z], t:[x,y,z], intrinsics, registered}]
|
||||
anchors: [], // [{id, label, x, y, z, color}]
|
||||
events: [], // [{id, t_global_s, duration_s, event_type, confidence, description, source}]
|
||||
hasTracks: false, // manifest.has_tracks (M18): any solved friend track => load + render tracks
|
||||
tracks: [], // [{id, marker_key, label, color, points:[{t_global_s,x,y,z,quality,views}]}] (M18)
|
||||
|
||||
// transport / playback
|
||||
tGlobal: 0,
|
||||
playing: false,
|
||||
rate: 1,
|
||||
audioSourceId: null, // video id whose audio you hear (WebAudio soundtrack; fallback: unmuted <video>)
|
||||
spatialAudio: false, // 🎧 every camera is a positional emitter at its 3D pose; listener = viewer cam
|
||||
enabled: {}, // id -> bool (per-video enable; disabled videos pause + dim + drop from sync)
|
||||
inRange: {}, // id -> bool (t_video within [0, duration] and enabled)
|
||||
syncErrorMs: {}, // id -> number | null (currentTime - target, ms; null when out of range)
|
||||
|
||||
// 3D viewer
|
||||
followCameraId: null, // video id the viewer camera is snapped to, or null for free roam
|
||||
};
|
||||
|
||||
// --- tiny pub/sub for discrete events -------------------------------------------------------
|
||||
const listeners = new Map();
|
||||
|
||||
/** Subscribe to an event; returns an unsubscribe fn. */
|
||||
export function on(evt, fn) {
|
||||
if (!listeners.has(evt)) listeners.set(evt, new Set());
|
||||
listeners.get(evt).add(fn);
|
||||
return () => listeners.get(evt)?.delete(fn);
|
||||
}
|
||||
|
||||
/** Emit an event to all subscribers. */
|
||||
export function emit(evt, payload) {
|
||||
const set = listeners.get(evt);
|
||||
if (set) for (const fn of [...set]) fn(payload);
|
||||
}
|
||||
|
||||
/** The reference video (offset 0) if present, else the first video. Used as default audio source. */
|
||||
export function referenceVideoId() {
|
||||
const ref = state.videos.find((v) => (v.offset_ms ?? 0) === 0);
|
||||
return ref ? ref.id : state.videos[0]?.id ?? null;
|
||||
}
|
||||
export {};
|
||||
|
||||
@ -1,161 +1,6 @@
|
||||
// Timeline scrubber + event markers (spec M4 transport + M7 markers).
|
||||
//
|
||||
// Renders a draggable scrubber over [0, tGlobalMax] with the playhead, plus GET /api/events as
|
||||
// colored ticks (color by event_type, legend in the corner). Hover a marker -> description
|
||||
// tooltip; click a marker -> jump the playhead there. Event data is already seeded
|
||||
// synthetically; classification QUALITY is lane D's concern, not this renderer's.
|
||||
// Scrubber + event markers (spec M4 transport + M7 timeline markers).
|
||||
// STUB — lane C fills this in. Render GET /api/events as colored markers (color by
|
||||
// event_type, legend in a corner); hover → description tooltip; click → jump the playhead.
|
||||
// The event data is already seeded synthetically; classification quality is lane D's job.
|
||||
|
||||
import { state, emit } from "./state.js";
|
||||
|
||||
export const EVENT_COLORS = {
|
||||
bass_drop: "#ff3b6b",
|
||||
pyro: "#ff8c1a",
|
||||
confetti: "#ffd23b",
|
||||
crowd_wave: "#3bc9ff",
|
||||
artist_moment: "#c77dff",
|
||||
light_show: "#59d499",
|
||||
quiet_moment: "#8a92a6",
|
||||
candidate: "#c0c0c0",
|
||||
other: "#9aa3b2",
|
||||
};
|
||||
export function eventColor(type) {
|
||||
return EVENT_COLORS[type] || EVENT_COLORS.other;
|
||||
}
|
||||
|
||||
function fmtTime(s) {
|
||||
s = Math.max(0, s);
|
||||
const m = Math.floor(s / 60);
|
||||
const sec = s - m * 60;
|
||||
return `${m}:${sec.toFixed(1).padStart(4, "0")}`;
|
||||
}
|
||||
|
||||
export class Timeline {
|
||||
constructor() {
|
||||
this.max = 1;
|
||||
}
|
||||
|
||||
init(root, transport) {
|
||||
this.transport = transport;
|
||||
this.max = state.tGlobalMax || 1;
|
||||
root.innerHTML = "";
|
||||
|
||||
const track = document.createElement("div");
|
||||
track.className = "tl-track";
|
||||
|
||||
const fill = document.createElement("div");
|
||||
fill.className = "tl-fill";
|
||||
|
||||
const markers = document.createElement("div");
|
||||
markers.className = "tl-markers";
|
||||
|
||||
const playhead = document.createElement("div");
|
||||
playhead.className = "tl-playhead";
|
||||
|
||||
track.append(fill, markers, playhead);
|
||||
|
||||
const tip = document.createElement("div");
|
||||
tip.className = "tl-tooltip";
|
||||
tip.style.display = "none";
|
||||
|
||||
root.append(track, tip);
|
||||
this.root = root;
|
||||
this.track = track;
|
||||
this.fill = fill;
|
||||
this.markers = markers;
|
||||
this.playhead = playhead;
|
||||
this.tip = tip;
|
||||
|
||||
this._buildMarkers(markers, tip);
|
||||
this._buildLegend(root);
|
||||
this._wireScrub(track);
|
||||
}
|
||||
|
||||
/** Rebuild markers + legend after events change (e.g. an M8 type correction). */
|
||||
refresh() {
|
||||
if (!this.markers) return;
|
||||
this.markers.innerHTML = "";
|
||||
this._buildMarkers(this.markers, this.tip);
|
||||
this.root.querySelector(".tl-legend")?.remove();
|
||||
this._buildLegend(this.root);
|
||||
}
|
||||
|
||||
_buildMarkers(container, tip) {
|
||||
for (const ev of state.events) {
|
||||
const m = document.createElement("div");
|
||||
m.className = "tl-marker";
|
||||
m.style.left = `${(ev.t_global_s / this.max) * 100}%`;
|
||||
m.style.background = eventColor(ev.event_type);
|
||||
m.addEventListener("click", (e) => {
|
||||
e.stopPropagation();
|
||||
this.transport.seek(ev.t_global_s);
|
||||
emit("event-selected", ev); // open the M8 correction panel
|
||||
});
|
||||
const show = (e) => {
|
||||
tip.innerHTML =
|
||||
`<strong>${ev.event_type.replace(/_/g, " ")}</strong> · ${fmtTime(ev.t_global_s)}` +
|
||||
`<br>${ev.description || ""}` +
|
||||
(ev.confidence != null
|
||||
? `<br><span class="tl-conf">confidence ${(ev.confidence * 100).toFixed(0)}% · ${ev.source}</span>`
|
||||
: "");
|
||||
tip.style.display = "block";
|
||||
const rect = this.track.getBoundingClientRect();
|
||||
const x = (ev.t_global_s / this.max) * rect.width;
|
||||
tip.style.left = `${Math.max(4, Math.min(rect.width - 4, x))}px`;
|
||||
};
|
||||
m.addEventListener("mouseenter", show);
|
||||
m.addEventListener("mousemove", show);
|
||||
m.addEventListener("mouseleave", () => (tip.style.display = "none"));
|
||||
container.append(m);
|
||||
}
|
||||
}
|
||||
|
||||
_buildLegend(root) {
|
||||
const types = [...new Set(state.events.map((e) => e.event_type))];
|
||||
if (types.length === 0) return;
|
||||
const legend = document.createElement("div");
|
||||
legend.className = "tl-legend";
|
||||
for (const t of types) {
|
||||
const item = document.createElement("span");
|
||||
item.className = "tl-legend-item";
|
||||
item.innerHTML = `<i style="background:${eventColor(t)}"></i>${t.replace(/_/g, " ")}`;
|
||||
legend.append(item);
|
||||
}
|
||||
root.append(legend);
|
||||
}
|
||||
|
||||
_wireScrub(track) {
|
||||
let dragging = false;
|
||||
const seekTo = (clientX) => {
|
||||
const rect = track.getBoundingClientRect();
|
||||
const f = Math.max(0, Math.min(1, (clientX - rect.left) / rect.width));
|
||||
this.transport.seek(f * this.max);
|
||||
};
|
||||
track.addEventListener("pointerdown", (e) => {
|
||||
dragging = true;
|
||||
track.setPointerCapture(e.pointerId);
|
||||
seekTo(e.clientX);
|
||||
});
|
||||
track.addEventListener("pointermove", (e) => {
|
||||
if (dragging) seekTo(e.clientX);
|
||||
});
|
||||
const end = (e) => {
|
||||
dragging = false;
|
||||
try {
|
||||
track.releasePointerCapture(e.pointerId);
|
||||
} catch {
|
||||
/* ignore */
|
||||
}
|
||||
};
|
||||
track.addEventListener("pointerup", end);
|
||||
track.addEventListener("pointercancel", end);
|
||||
}
|
||||
|
||||
/** Move the playhead + fill to the current master time. */
|
||||
update(tGlobal) {
|
||||
const f = this.max > 0 ? Math.max(0, Math.min(1, tGlobal / this.max)) : 0;
|
||||
this.playhead.style.left = `${f * 100}%`;
|
||||
this.fill.style.width = `${f * 100}%`;
|
||||
}
|
||||
}
|
||||
|
||||
export const timeline = new Timeline();
|
||||
export {};
|
||||
|
||||
@ -1,525 +1,8 @@
|
||||
// Master clock + per-video synchronization (spec M4 — the heart of the app).
|
||||
//
|
||||
// AUDIO IS THE MASTER CLOCK when possible: the selected audio source's soundtrack (fetched from
|
||||
// /api/audio/{id}, decoded once) plays through WebAudio, and t_global is derived from
|
||||
// AudioContext.currentTime — sample-accurate, immune to rAF throttling, and NEVER corrected.
|
||||
// Every <video> element is muted picture-only and is seeked / rate-trimmed freely against that
|
||||
// clock with zero audible artifacts (previously the unmuted video's own corrections warbled).
|
||||
// If the track can't be fetched/decoded, we fall back per-session to the legacy behavior:
|
||||
// performance.now() clock + the selected video's element audio unmuted.
|
||||
// We NEVER trust a <video> element as the clock (HTML5 video is not frame-accurate —
|
||||
// pitfall #2). Correction is continuous:
|
||||
// |err| > 150 ms -> hard seek (fastSeek where available)
|
||||
// 20 ms < |err| <= 150 ms -> nudge playbackRate within master rate x[0.95, 1.05]
|
||||
// |err| <= 20 ms -> locked: playbackRate = master rate
|
||||
//
|
||||
// While PLAYING we measure error per presented frame via requestVideoFrameCallback: `mediaTime`
|
||||
// is the exact timestamp of the frame on screen and `expectedDisplayTime` is when it shows, so
|
||||
// err = mediaTime - target(clock @ expectedDisplayTime) is the true on-screen sync error (far
|
||||
// better than sampling the frame-quantized currentTime in rAF). Browsers without rVFC fall back
|
||||
// to a currentTime measurement inside the rAF tick(). While PAUSED / seeking, videos are parked
|
||||
// on their target frame via currentTime. Videos whose target t_video falls outside [0, duration]
|
||||
// pause and are marked out-of-range (the grid dims them).
|
||||
// Master clock + per-video sync (spec M4 — the heart of the app).
|
||||
// STUB — lane C fills this in. Build and verify the master-clock transport before the 3D
|
||||
// work. Use the timebase from config.py (mirrored here):
|
||||
// t_video = (t_global - offset_ms/1000) * (1 + drift_ppm * 1e-6)
|
||||
// Never trust a single <video> element as the clock; drive t_global from performance.now()
|
||||
// and continuously correct each video (hard-seek >150ms, nudge playbackRate 20–150ms).
|
||||
|
||||
import { state, emit, on } from "./state.js";
|
||||
import { tVideoFromGlobal } from "./lib/timebase.js";
|
||||
|
||||
const HARD_SEEK_S = 0.15; // > this error => hard seek
|
||||
const NUDGE_MIN_S = 0.02; // > this (and <= HARD_SEEK) => trim playbackRate
|
||||
const RATE_TRIM = 0.05; // max ±5% playbackRate trim while nudging
|
||||
|
||||
export class Transport {
|
||||
constructor() {
|
||||
this.cells = []; // [{ id, meta, video, ... }] from videoGrid
|
||||
this._t0 = 0; // master-timeline anchor (seconds)
|
||||
this._wall0 = performance.now(); // wall-clock anchor (ms) matching _t0
|
||||
this._visWired = false;
|
||||
|
||||
// WebAudio master clock (see header). All lazy: nothing is created until needed.
|
||||
this._actx = null; // AudioContext
|
||||
this._ctx0 = 0; // AudioContext.currentTime anchor (s) matching _t0
|
||||
this._audioActive = false; // audio clock is currently driving t_global
|
||||
this._buffers = new Map(); // video id -> AudioBuffer (decoded) | null (unavailable)
|
||||
this._decoding = new Set(); // video ids with a fetch/decode in flight
|
||||
this._tracks = new Map(); // playing tracks: video id -> { src, gain, panner|null }
|
||||
this._captureDest = null; // MediaStreamAudioDestinationNode while an export taps the mix (M12)
|
||||
}
|
||||
|
||||
/** Register the video cells the transport drives and start their frame loops. */
|
||||
register(cells) {
|
||||
this.cells = cells;
|
||||
this._applyAudio();
|
||||
this._loadTrack(state.audioSourceId); // start fetching the soundtrack immediately
|
||||
for (const cell of this.cells) this._startFrameLoop(cell);
|
||||
if (!this._enabledWired) {
|
||||
// In spatial mode a toggled video's emitter must join/leave the running mix.
|
||||
on("enabled", () => {
|
||||
if (state.spatialAudio) this._restartIfPlaying();
|
||||
});
|
||||
this._enabledWired = true;
|
||||
}
|
||||
if (!this._visWired) {
|
||||
// A throttled/backgrounded tab stalls rAF while decoders free-run; re-lock on return.
|
||||
document.addEventListener("visibilitychange", () => {
|
||||
if (!document.hidden) this._resync();
|
||||
});
|
||||
this._visWired = true;
|
||||
}
|
||||
}
|
||||
|
||||
/** Master-timeline seconds at a given wall-clock (ms) timestamp, clamped to [0, tGlobalMax]. */
|
||||
_clockAt(wallMs) {
|
||||
if (!state.playing) return this._t0;
|
||||
let elapsed;
|
||||
if (this._audioActive) {
|
||||
// Audio clock: map the wall timestamp onto the AudioContext clock (the two clocks tick
|
||||
// together; only their epochs differ, and this mapping is sub-ms), measure from anchor.
|
||||
const ctxNow = this._actx.currentTime + (wallMs - performance.now()) / 1000;
|
||||
elapsed = ctxNow - this._ctx0;
|
||||
} else {
|
||||
elapsed = (wallMs - this._wall0) / 1000;
|
||||
}
|
||||
const t = this._t0 + elapsed * state.rate;
|
||||
if (t <= 0) return 0;
|
||||
if (t >= state.tGlobalMax) return state.tGlobalMax;
|
||||
return t;
|
||||
}
|
||||
|
||||
computeTGlobal() {
|
||||
return this._clockAt(performance.now());
|
||||
}
|
||||
|
||||
_targetFor(meta, tGlobal) {
|
||||
return tVideoFromGlobal(tGlobal, meta.offset_ms || 0, meta.drift_ppm || 0);
|
||||
}
|
||||
|
||||
play() {
|
||||
if (state.playing) return;
|
||||
let t = state.tGlobal;
|
||||
if (t >= state.tGlobalMax - 1e-3) t = 0; // restart from the top if parked at the end
|
||||
this._t0 = t;
|
||||
this._wall0 = performance.now();
|
||||
state.tGlobal = t;
|
||||
state.playing = true;
|
||||
this._startAudio(t); // play() is a user gesture -> AudioContext may resume / videos unmute
|
||||
emit("play");
|
||||
}
|
||||
|
||||
pause() {
|
||||
if (!state.playing) return;
|
||||
state.tGlobal = this.computeTGlobal(); // BEFORE stopping the audio clock's source node
|
||||
this._t0 = state.tGlobal;
|
||||
state.playing = false;
|
||||
this._stopTracks();
|
||||
this._audioActive = false;
|
||||
for (const { video } of this.cells) this._safePause(video);
|
||||
emit("pause");
|
||||
}
|
||||
|
||||
toggle() {
|
||||
if (state.playing) this.pause();
|
||||
else this.play();
|
||||
}
|
||||
|
||||
/** Jump the master timeline; parks every in-range video on its target frame immediately. */
|
||||
seek(t) {
|
||||
t = Math.max(0, Math.min(state.tGlobalMax, t));
|
||||
this._t0 = t;
|
||||
this._wall0 = performance.now();
|
||||
state.tGlobal = t;
|
||||
if (state.playing) this._startAudio(t); // restart the soundtrack at the new position
|
||||
for (const cell of this.cells) {
|
||||
const target = this._targetFor(cell.meta, t);
|
||||
const inRange =
|
||||
state.enabled[cell.id] !== false && target >= 0 && target <= cell.meta.duration_s;
|
||||
state.inRange[cell.id] = inRange;
|
||||
if (inRange) {
|
||||
this._hardSeek(cell.video, target);
|
||||
cell.video.playbackRate = state.rate;
|
||||
state.syncErrorMs[cell.id] = 0;
|
||||
} else {
|
||||
state.syncErrorMs[cell.id] = null;
|
||||
this._safePause(cell.video);
|
||||
}
|
||||
}
|
||||
emit("seek", t);
|
||||
}
|
||||
|
||||
/** Nudge the playhead by a delta (keyboard ±1 s). */
|
||||
seekBy(dt) {
|
||||
this.seek((state.playing ? this.computeTGlobal() : this._t0) + dt);
|
||||
}
|
||||
|
||||
/** Set playback speed, rebasing the clock so t_global stays continuous. */
|
||||
setRate(r) {
|
||||
state.tGlobal = this.computeTGlobal();
|
||||
this._t0 = state.tGlobal;
|
||||
this._wall0 = performance.now();
|
||||
state.rate = r;
|
||||
for (const { video } of this.cells) video.playbackRate = r; // re-trimmed by correction
|
||||
if (state.playing) this._startAudio(state.tGlobal); // restart the track at the new rate
|
||||
emit("rate", r);
|
||||
}
|
||||
|
||||
/** Choose whose audio you hear (the WebAudio track when decoded, else that video unmuted). */
|
||||
setAudioSource(id) {
|
||||
state.audioSourceId = id;
|
||||
this._loadTrack(id);
|
||||
this._restartIfPlaying();
|
||||
emit("audiosource", id);
|
||||
}
|
||||
|
||||
/** 🎧 Toggle spatial audio: every camera becomes a positional emitter at its 3D pose. */
|
||||
setSpatial(enabled) {
|
||||
state.spatialAudio = !!enabled;
|
||||
if (state.spatialAudio) for (const { id } of this.cells) this._loadTrack(id);
|
||||
this._restartIfPlaying();
|
||||
emit("spatial", state.spatialAudio);
|
||||
}
|
||||
|
||||
_restartIfPlaying() {
|
||||
if (state.playing) this._startAudio(this.computeTGlobal());
|
||||
else this._applyAudio();
|
||||
}
|
||||
|
||||
_applyAudio() {
|
||||
// With a decoded soundtrack the WebAudio track is the only audible thing: every video is
|
||||
// muted picture-only. Fallback (no track): the selected video's element audio, as before.
|
||||
const trackReady = state.spatialAudio
|
||||
? [...this._buffers.values()].some((b) => b instanceof AudioBuffer)
|
||||
: this._buffers.get(state.audioSourceId) instanceof AudioBuffer;
|
||||
for (const { id, video } of this.cells) {
|
||||
const isSrc = !trackReady && id === state.audioSourceId;
|
||||
video.muted = !isSrc;
|
||||
if (isSrc) video.volume = 1;
|
||||
}
|
||||
}
|
||||
|
||||
// --- WebAudio master clock ----------------------------------------------------------------
|
||||
|
||||
_ensureCtx() {
|
||||
if (!this._actx) this._actx = new (window.AudioContext || window.webkitAudioContext)();
|
||||
return this._actx;
|
||||
}
|
||||
|
||||
/** Fetch + decode a video's soundtrack once; upgrade the running clock when it lands. */
|
||||
async _loadTrack(id) {
|
||||
if (id == null || this._buffers.has(id) || this._decoding.has(id)) return;
|
||||
if (typeof window === "undefined" || !(window.AudioContext || window.webkitAudioContext)) return;
|
||||
this._decoding.add(id);
|
||||
try {
|
||||
const resp = await fetch(`${state.apiBase}/api/audio/${id}`);
|
||||
if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
|
||||
const buf = await this._ensureCtx().decodeAudioData(await resp.arrayBuffer());
|
||||
// Decoded PCM is big (~23 MB/min stereo 48k): keep only a few tracks around
|
||||
// (spatial mode needs one per camera, so the cap widens to the video count).
|
||||
const cap = state.spatialAudio ? Math.max(this.cells.length, 3) : 3;
|
||||
for (const key of this._buffers.keys()) {
|
||||
if (this._buffers.size < cap) break;
|
||||
if (key !== state.audioSourceId) this._buffers.delete(key);
|
||||
}
|
||||
this._buffers.set(id, buf);
|
||||
} catch (err) {
|
||||
console.warn(`[transport] soundtrack for video ${id} unavailable — using <video> audio`, err);
|
||||
this._buffers.set(id, null);
|
||||
} finally {
|
||||
this._decoding.delete(id);
|
||||
}
|
||||
// A track landing mid-play joins in place: in spatial mode it enters the running mix; as
|
||||
// the selected source it upgrades the fallback clock (both clocks agree on t_global).
|
||||
const decoded = this._buffers.get(id) instanceof AudioBuffer;
|
||||
if (
|
||||
state.playing &&
|
||||
decoded &&
|
||||
(state.spatialAudio || (id === state.audioSourceId && !this._audioActive))
|
||||
) {
|
||||
this._startAudio(this.computeTGlobal());
|
||||
} else if (!state.playing) {
|
||||
this._applyAudio();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* (Re)start the soundtrack at t_global and re-anchor the master clock. Single-track mode
|
||||
* plays the selected source; spatial mode plays every enabled camera's track through a
|
||||
* PannerNode at its 3D pose. Falls back to the performance.now() clock (selected video
|
||||
* unmuted) when no decoded track is available.
|
||||
*/
|
||||
_startAudio(tGlobal) {
|
||||
this._stopTracks();
|
||||
this._t0 = tGlobal;
|
||||
this._wall0 = performance.now();
|
||||
const ready = (id) => this._buffers.get(id) instanceof AudioBuffer;
|
||||
const ids = state.spatialAudio
|
||||
? this.cells.filter((c) => state.enabled[c.id] !== false && ready(c.id)).map((c) => c.id)
|
||||
: ready(state.audioSourceId)
|
||||
? [state.audioSourceId]
|
||||
: [];
|
||||
if (ids.length === 0) {
|
||||
this._audioActive = false;
|
||||
this._applyAudio(); // fallback: unmute the selected video
|
||||
return;
|
||||
}
|
||||
const ctx = this._ensureCtx();
|
||||
if (ctx.state === "suspended") ctx.resume(); // play()/click gestures allow this
|
||||
// N time-aligned copies of the same sound sum loud — normalize by 1/sqrt(N).
|
||||
const gainValue = 1 / Math.sqrt(ids.length);
|
||||
for (const id of ids) this._startTrack(ctx, id, tGlobal, state.spatialAudio, gainValue);
|
||||
// Even with no live node (source out of range => silence), the audio clock drives time.
|
||||
this._ctx0 = ctx.currentTime;
|
||||
this._audioActive = true;
|
||||
this._applyAudio();
|
||||
}
|
||||
|
||||
_startTrack(ctx, id, tGlobal, spatial, gainValue) {
|
||||
const buf = this._buffers.get(id);
|
||||
const meta =
|
||||
this.cells.find((c) => c.id === id)?.meta ?? state.videos.find((v) => v.id === id);
|
||||
const drift = 1 + (meta?.drift_ppm || 0) * 1e-6;
|
||||
const tv = tVideoFromGlobal(tGlobal, meta?.offset_ms || 0, meta?.drift_ppm || 0);
|
||||
if (tv >= buf.duration) return; // this camera's audio is already over
|
||||
const src = ctx.createBufferSource();
|
||||
src.buffer = buf;
|
||||
// The track's local time must advance at rate·(1+drift) per global second.
|
||||
src.playbackRate.value = state.rate * drift;
|
||||
const gain = ctx.createGain();
|
||||
gain.gain.value = gainValue;
|
||||
src.connect(gain);
|
||||
let panner = null;
|
||||
if (spatial) {
|
||||
panner = ctx.createPanner();
|
||||
panner.panningModel = "HRTF";
|
||||
panner.distanceModel = "inverse";
|
||||
panner.refDistance = 2; // scene units — cameras sit ~5–15 from the stage at COLMAP scale
|
||||
panner.rolloffFactor = 0.6;
|
||||
gain.connect(panner);
|
||||
panner.connect(ctx.destination);
|
||||
} else {
|
||||
gain.connect(ctx.destination);
|
||||
}
|
||||
if (this._captureDest) (panner ?? gain).connect(this._captureDest); // live export keeps hearing us
|
||||
if (tv >= 0) src.start(0, tv);
|
||||
else src.start(ctx.currentTime + -tv / state.rate, 0); // camera starts later than t_global
|
||||
this._tracks.set(id, { src, gain, panner });
|
||||
}
|
||||
|
||||
/**
|
||||
* M12 hook (phase-5 contract #6): a MediaStream carrying the live WebAudio mix — tapped
|
||||
* post-gain, after the spatial panner when 🎧 is on — for MediaRecorder muxing. Tracks that
|
||||
* (re)start while the tap is live (seeks, source switches) join it automatically
|
||||
* (_startTrack above). With no decoded track (fallback <video> audio) the stream is silent —
|
||||
* degrade, don't block (pitfall #5). Call releaseAudioStream() when the export ends.
|
||||
*/
|
||||
captureAudioStream() {
|
||||
const ctx = this._ensureCtx();
|
||||
if (!this._captureDest) {
|
||||
this._captureDest = ctx.createMediaStreamDestination();
|
||||
for (const { gain, panner } of this._tracks.values())
|
||||
(panner ?? gain).connect(this._captureDest);
|
||||
}
|
||||
return this._captureDest.stream;
|
||||
}
|
||||
|
||||
/** Drop the M12 export tap (the matching half of captureAudioStream()). */
|
||||
releaseAudioStream() {
|
||||
if (!this._captureDest) return;
|
||||
for (const { gain, panner } of this._tracks.values()) {
|
||||
try {
|
||||
(panner ?? gain).disconnect(this._captureDest);
|
||||
} catch {
|
||||
/* already disconnected */
|
||||
}
|
||||
}
|
||||
this._captureDest = null;
|
||||
}
|
||||
|
||||
/**
|
||||
* 🎧 Per-frame 3D audio update (called by scene3d): the listener follows the viewer camera
|
||||
* and each playing track's panner sits at its camera's current pose. Takes plain numbers in
|
||||
* scene (Three.js) space so the transport stays THREE-free.
|
||||
*/
|
||||
updateSpatial(listener, positions) {
|
||||
if (!state.spatialAudio || !this._audioActive || !this._actx) return;
|
||||
const L = this._actx.listener;
|
||||
if (L.positionX) {
|
||||
L.positionX.value = listener.px;
|
||||
L.positionY.value = listener.py;
|
||||
L.positionZ.value = listener.pz;
|
||||
L.forwardX.value = listener.fx;
|
||||
L.forwardY.value = listener.fy;
|
||||
L.forwardZ.value = listener.fz;
|
||||
L.upX.value = listener.ux;
|
||||
L.upY.value = listener.uy;
|
||||
L.upZ.value = listener.uz;
|
||||
} else {
|
||||
// Safari: AudioListener still uses the legacy setters.
|
||||
L.setPosition(listener.px, listener.py, listener.pz);
|
||||
L.setOrientation(listener.fx, listener.fy, listener.fz, listener.ux, listener.uy, listener.uz);
|
||||
}
|
||||
for (const [id, { panner }] of this._tracks) {
|
||||
const pos = positions[id];
|
||||
if (!panner || !pos) continue;
|
||||
if (panner.positionX) {
|
||||
panner.positionX.value = pos.x;
|
||||
panner.positionY.value = pos.y;
|
||||
panner.positionZ.value = pos.z;
|
||||
} else {
|
||||
panner.setPosition(pos.x, pos.y, pos.z);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
_stopTracks() {
|
||||
for (const { src, gain, panner } of this._tracks.values()) {
|
||||
try {
|
||||
src.stop();
|
||||
} catch {
|
||||
/* never started or already ended */
|
||||
}
|
||||
try {
|
||||
src.disconnect();
|
||||
gain.disconnect();
|
||||
panner?.disconnect();
|
||||
} catch {
|
||||
/* ignore */
|
||||
}
|
||||
}
|
||||
this._tracks.clear();
|
||||
}
|
||||
|
||||
// Per-presented-frame correction via requestVideoFrameCallback (accurate). Self-sustaining:
|
||||
// each callback re-registers the next. No-ops while paused / out of range; rAF tick() handles
|
||||
// the fallback path for browsers without rVFC.
|
||||
_startFrameLoop(cell) {
|
||||
const { video } = cell;
|
||||
if (typeof video.requestVideoFrameCallback !== "function") {
|
||||
cell._useRvfc = false;
|
||||
return;
|
||||
}
|
||||
cell._useRvfc = true;
|
||||
cell._lastRvfc = -Infinity;
|
||||
const cb = (now, meta) => {
|
||||
cell._rvfcId = video.requestVideoFrameCallback(cb);
|
||||
cell._lastRvfc = performance.now();
|
||||
if (!state.playing || !state.inRange[cell.id]) return;
|
||||
const displayWall = meta.expectedDisplayTime ?? now;
|
||||
const target = this._targetFor(cell.meta, this._clockAt(displayWall));
|
||||
// mediaTime is the exact PTS of the on-screen frame — no quantization compensation needed.
|
||||
this._correct(cell, meta.mediaTime, target, true);
|
||||
};
|
||||
cell._rvfcId = video.requestVideoFrameCallback(cb);
|
||||
}
|
||||
|
||||
_correct(cell, measured, target, allowSeek) {
|
||||
const { id, video } = cell;
|
||||
const err = measured - target; // + => video is ahead of where it should be
|
||||
state.syncErrorMs[id] = err * 1000;
|
||||
if (allowSeek && Math.abs(err) > HARD_SEEK_S) {
|
||||
this._hardSeek(video, target);
|
||||
video.playbackRate = state.rate;
|
||||
} else if (Math.abs(err) > NUDGE_MIN_S) {
|
||||
// Ahead (err>0) => slow down; behind (err<0) => speed up. Proportional, clamped ±5%.
|
||||
const trim = Math.max(-1, Math.min(1, err / HARD_SEEK_S)) * RATE_TRIM;
|
||||
video.playbackRate = state.rate * (1 - trim);
|
||||
} else {
|
||||
video.playbackRate = state.rate;
|
||||
}
|
||||
}
|
||||
|
||||
_hardSeek(video, target) {
|
||||
if (typeof video.fastSeek === "function") {
|
||||
try {
|
||||
video.fastSeek(target);
|
||||
return;
|
||||
} catch {
|
||||
/* fall through */
|
||||
}
|
||||
}
|
||||
video.currentTime = target;
|
||||
}
|
||||
|
||||
/** Re-lock every in-range video to the current master time (after a tab-visibility stall). */
|
||||
_resync() {
|
||||
// Compute fresh: with the audio clock the soundtrack kept playing while rAF (and therefore
|
||||
// state.tGlobal) was frozen, so the live clock may be well past the last stored value.
|
||||
const t = this.computeTGlobal();
|
||||
state.tGlobal = t;
|
||||
for (const cell of this.cells) {
|
||||
const target = this._targetFor(cell.meta, t);
|
||||
if (state.enabled[cell.id] !== false && target >= 0 && target <= cell.meta.duration_s) {
|
||||
this._hardSeek(cell.video, target);
|
||||
cell.video.playbackRate = state.rate;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Drive one animation-frame tick: advance the clock and manage each video. */
|
||||
tick() {
|
||||
const t = this.computeTGlobal();
|
||||
state.tGlobal = t;
|
||||
if (state.playing && t >= state.tGlobalMax) {
|
||||
this.pause();
|
||||
return;
|
||||
}
|
||||
for (const cell of this.cells) this._manage(cell, t);
|
||||
}
|
||||
|
||||
// Play/pause + range management every rAF. For non-rVFC browsers this also runs correction
|
||||
// from the (frame-quantized) currentTime; with rVFC, correction happens in the frame loop.
|
||||
_manage(cell, tGlobal) {
|
||||
const { id, meta, video } = cell;
|
||||
const target = this._targetFor(meta, tGlobal);
|
||||
const enabled = state.enabled[id] !== false;
|
||||
const inRange = enabled && target >= 0 && target <= meta.duration_s;
|
||||
state.inRange[id] = inRange;
|
||||
|
||||
if (!inRange) {
|
||||
state.syncErrorMs[id] = null;
|
||||
this._safePause(video);
|
||||
return;
|
||||
}
|
||||
|
||||
if (state.playing) {
|
||||
this._ensurePlaying(video);
|
||||
// rVFC drives correction when it's firing. Fall back to a currentTime measurement when
|
||||
// rVFC is unavailable OR stale (>100 ms) — e.g. a tab where rVFC is throttled but rAF runs.
|
||||
const rvfcStale = !cell._useRvfc || performance.now() - (cell._lastRvfc ?? -Infinity) > 100;
|
||||
if (rvfcStale) this._correct(cell, this._measureFallback(cell), target, true);
|
||||
} else {
|
||||
this._safePause(video);
|
||||
state.syncErrorMs[id] = (this._measureFallback(cell) - target) * 1000;
|
||||
}
|
||||
}
|
||||
|
||||
// Estimate a playing video's true position from currentTime. currentTime is quantized to the
|
||||
// displayed frame (whose PTS is <= the true position), so add half a frame to de-bias it. The
|
||||
// rVFC path uses exact mediaTime instead and needs no such correction.
|
||||
_measureFallback(cell) {
|
||||
const fps = cell.meta.fps || 30;
|
||||
const halfFrame = state.playing ? (0.5 / fps) * state.rate : 0;
|
||||
return cell.video.currentTime + halfFrame;
|
||||
}
|
||||
|
||||
_ensurePlaying(video) {
|
||||
if (video.paused && !video._playPending && video.readyState >= 2) {
|
||||
video._playPending = true;
|
||||
video.play().then(
|
||||
() => (video._playPending = false),
|
||||
() => (video._playPending = false)
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
_safePause(video) {
|
||||
if (video._playPending) return; // let the pending play() settle; next tick re-evaluates
|
||||
if (!video.paused) {
|
||||
try {
|
||||
video.pause();
|
||||
} catch {
|
||||
/* ignore */
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export const transport = new Transport();
|
||||
export {};
|
||||
|
||||
@ -1,114 +1,5 @@
|
||||
// <video> grid + per-video transparent overlay canvases (spec M4 / M6).
|
||||
//
|
||||
// Each cell stacks a <video> (object-fit: contain) under a transparent <canvas> that exactly
|
||||
// covers the element box. Because object-fit letterboxes, the video's CONTENT rect is smaller
|
||||
// than the element (pitfall #4) — contentRect() computes it so overlays.js can map projected
|
||||
// video-pixel coordinates to the right place on the canvas.
|
||||
// <video> grid + per-video transparent overlay canvases (spec M4/M6).
|
||||
// STUB — lane C fills this in. Mind spec pitfall #4: object-fit letterboxing means canvas
|
||||
// px != video px; compute each video's content rect for the overlay canvas.
|
||||
|
||||
import { state, emit } from "./state.js";
|
||||
|
||||
/**
|
||||
* Build the grid inside `container`. Returns cell descriptors:
|
||||
* { id, meta, el, video, canvas, statusEl, audioBtn, snapBtn }
|
||||
*/
|
||||
export function createVideoGrid(container, videos, apiBase, transport) {
|
||||
container.innerHTML = "";
|
||||
const cells = [];
|
||||
|
||||
for (const meta of videos) {
|
||||
const el = document.createElement("div");
|
||||
el.className = "cell";
|
||||
el.dataset.id = String(meta.id);
|
||||
|
||||
const video = document.createElement("video");
|
||||
video.className = "cell-video";
|
||||
video.src = apiBase + meta.url;
|
||||
video.muted = true;
|
||||
video.loop = false;
|
||||
video.playsInline = true;
|
||||
video.setAttribute("playsinline", "");
|
||||
video.setAttribute("webkit-playsinline", "");
|
||||
video.preload = "auto";
|
||||
// No crossOrigin: we never read video pixels (overlays are a separate canvas), so we avoid
|
||||
// making playback depend on CORS headers for the /media mount.
|
||||
|
||||
const canvas = document.createElement("canvas");
|
||||
canvas.className = "cell-overlay";
|
||||
|
||||
const bar = document.createElement("div");
|
||||
bar.className = "cell-bar";
|
||||
|
||||
const label = document.createElement("span");
|
||||
label.className = "cell-label";
|
||||
label.textContent = meta.filename;
|
||||
|
||||
const audioBtn = document.createElement("button");
|
||||
audioBtn.className = "cell-btn audio-btn";
|
||||
audioBtn.title = "Use this video's audio";
|
||||
audioBtn.textContent = "🔈";
|
||||
audioBtn.addEventListener("click", (e) => {
|
||||
e.stopPropagation();
|
||||
transport.setAudioSource(meta.id);
|
||||
});
|
||||
|
||||
const snapBtn = document.createElement("button");
|
||||
snapBtn.className = "cell-btn snap-btn";
|
||||
snapBtn.title = "Snap 3D view to this camera";
|
||||
snapBtn.textContent = "⛶";
|
||||
// wired in main.js (needs scene3d)
|
||||
|
||||
const enableBtn = document.createElement("button");
|
||||
enableBtn.className = "cell-btn enable-btn";
|
||||
enableBtn.title = "Enable / disable this video";
|
||||
enableBtn.textContent = "◉";
|
||||
enableBtn.addEventListener("click", (e) => {
|
||||
e.stopPropagation();
|
||||
const now = state.enabled[meta.id] !== false;
|
||||
state.enabled[meta.id] = !now;
|
||||
emit("enabled", { id: meta.id, enabled: !now });
|
||||
});
|
||||
|
||||
const status = document.createElement("span");
|
||||
status.className = "cell-status";
|
||||
|
||||
bar.append(label, status, enableBtn, snapBtn, audioBtn);
|
||||
el.append(video, canvas, bar);
|
||||
container.append(el);
|
||||
|
||||
// Clicking the video frame selects its audio (natural "listen to this one" gesture).
|
||||
video.addEventListener("click", () => transport.setAudioSource(meta.id));
|
||||
|
||||
cells.push({ id: meta.id, meta, el, video, canvas, statusEl: status, audioBtn, snapBtn });
|
||||
}
|
||||
|
||||
return cells;
|
||||
}
|
||||
|
||||
/**
|
||||
* The video's displayed content rectangle inside its element, in CSS pixels, accounting for
|
||||
* object-fit: contain letterboxing. Maps native video pixels -> element pixels via:
|
||||
* x = ox + (u / vw) * cw , y = oy + (v / vh) * ch
|
||||
*/
|
||||
export function contentRect(video) {
|
||||
const elW = video.clientWidth;
|
||||
const elH = video.clientHeight;
|
||||
const vw = video.videoWidth || 1;
|
||||
const vh = video.videoHeight || 1;
|
||||
const elAspect = elW / elH;
|
||||
const vidAspect = vw / vh;
|
||||
let cw, ch, ox, oy;
|
||||
if (vidAspect > elAspect) {
|
||||
// wider than the box -> pillarbox: full width, bars top/bottom
|
||||
cw = elW;
|
||||
ch = elW / vidAspect;
|
||||
ox = 0;
|
||||
oy = (elH - ch) / 2;
|
||||
} else {
|
||||
// taller than the box -> letterbox: full height, bars left/right
|
||||
ch = elH;
|
||||
cw = elH * vidAspect;
|
||||
oy = 0;
|
||||
ox = (elW - cw) / 2;
|
||||
}
|
||||
return { ox, oy, cw, ch, elW, elH, vw, vh };
|
||||
}
|
||||
export {};
|
||||
|
||||
@ -1,9 +0,0 @@
|
||||
import { defineConfig } from "vite";
|
||||
|
||||
// `base` is the public path the built assets are served from.
|
||||
// - local dev / root hosting: "/" (default)
|
||||
// - hosted under a path prefix: "/festifun/" via FESTIVAL4D_BASE
|
||||
// Pair with VITE_API_BASE (see src/state.js) so API + media requests use the same prefix.
|
||||
export default defineConfig({
|
||||
base: process.env.FESTIVAL4D_BASE || "/",
|
||||
});
|
||||
@ -1,198 +0,0 @@
|
||||
# Phase 5 — Friends, Director's Cut & Capsules (M10–M17)
|
||||
|
||||
Canonical spec for the phase-5 milestones. **M0–M9 live in [`../OPUS_BUILD_INSTRUCTIONS.md`](../OPUS_BUILD_INSTRUCTIONS.md)
|
||||
and every contract frozen there still applies** (timebase, DB schema, pose convention,
|
||||
classifier contract, synthetic fixture). Standing protocol (status files, directives loop,
|
||||
evidence discipline, worktrees) is unchanged from [`README.md`](README.md); the newest round of
|
||||
[`DIRECTIVES.md`](DIRECTIVES.md) always wins.
|
||||
|
||||
## Context — what shipped since v0.1.0 (read before planning anything)
|
||||
|
||||
- **Splat rendering:** `GET /api/splat` + viewer `DropInViewer` with point-cloud fallback
|
||||
(`docs/modelbeast-crossover.md`).
|
||||
- **Live capture:** `FESTIVAL4D_CAPTURE=1` mounts `/capture` (README).
|
||||
- **WebAudio master clock:** `t_global` derives from `AudioContext.currentTime`;
|
||||
`GET /api/audio/{id}` serves a lazily-extracted stereo AAC per video; all `<video>` elements
|
||||
are muted picture-only, with graceful fallback to the old `performance.now()` clock
|
||||
(`frontend/src/transport.js` header comment is the reference).
|
||||
- **Spatial audio (🎧):** per-camera HRTF `PannerNode`s at live poses; listener follows the
|
||||
viewer camera (`transport.updateSpatial`, fed by `scene3d._updateAudio`).
|
||||
- **Sync solver:** pairwise GCC-PHAT now reuses one rFFT per signal (`audio_sync.pairwise_offsets`).
|
||||
- **Suite floor is 121 passed.** It only goes up.
|
||||
|
||||
## Phase map
|
||||
|
||||
```
|
||||
Phase 5a (serial, ONE agent) Phase 5b (parallel, one agent per lane) Phase 5c (serial, ONE agent)
|
||||
┌───────────────────────┐ ┌ lane/e-director ── M10 M11 M12 ─┐ ┌───────────────────────────┐
|
||||
│ foundation2 │ ├ lane/f-social ──── M13 M14 M15 ─┤ │ integration2 │
|
||||
│ contracts + stubs + │──merge──┤ ├──merge──│ full-system pass + │
|
||||
│ UI wiring points │ └ lane/g-capsule ─── M16 M17 ─────┘ │ real-footage field test │
|
||||
│ branch: foundation2 │ (independent — merge in any order) │ branch: integration2 │
|
||||
└───────────────────────┘ └─────────────────────────────────── └───────────────────────────┘
|
||||
```
|
||||
|
||||
Lanes depend **only** on foundation2, never on each other. Everything is developed and verified
|
||||
against the synthetic fixture first; real footage is integration2's job.
|
||||
|
||||
## Milestones
|
||||
|
||||
### M10 — Audio features: beats & onsets (lane E, backend)
|
||||
`backend/festival4d/audio_features.py`, entrypoint `run_features()`, CLI `python -m festival4d features`.
|
||||
Load the **reference** video's ingest WAV (16 kHz mono; the reference timeline *is* `t_global`).
|
||||
Beat/tempo tracking + onset-strength detection (librosa is already a dependency). Write
|
||||
`data/work/beats.json` (shape frozen below); `GET /api/beats` serves it, 404 when absent.
|
||||
Quiet/short/beatless audio must produce a *valid, empty* result — never a crash.
|
||||
**Accept:** synthetic master audio's known pulse grid recovered (beats within ±50 ms, tempo ±2 BPM);
|
||||
silence → empty-but-valid JSON; tests in `backend/tests/test_audio_features.py`.
|
||||
|
||||
### M11 — Auto-director (lane E, backend + thin UI)
|
||||
`backend/festival4d/director.py`, entrypoint `generate_path(top_n=8, lead_s=2.0)`, CLI
|
||||
`python -m festival4d direct`, `POST /api/director` (stubbed by foundation2) returning a
|
||||
**camPath-compatible JSON** (contract below). Deterministic v1 rules — no ML:
|
||||
pick the top-N events by confidence (ties → earlier); for each, choose the registered camera
|
||||
with a pose nearest the event time (via `db` + `geometry.colmap_to_threejs`); keyframes at
|
||||
`t_event − lead_s` and `t_event + duration`; when `beats.json` exists, snap keyframe times to
|
||||
the nearest beat; FOV from that camera's intrinsics (same formula as `scene3d.snapTo`).
|
||||
Frontend: fill the stub `frontend/src/director.js` — the hidden **🎬 Auto-path** button fetches
|
||||
`/api/director` and loads it via `camPath.fromJSON` (do not edit camPath.js).
|
||||
**Accept:** output round-trips through `camPath.fromJSON` (structure-locked test against the
|
||||
frozen shape); covers the top-N events; keyframe times land on beats when beats exist; live
|
||||
browser evidence of the path playing.
|
||||
|
||||
### M12 — Cinematic export (lane E, frontend)
|
||||
Fill the stub `frontend/src/exportVideo.js`. The hidden **⏺ Export** button: seek to the first
|
||||
keyframe, play the path, capture `scene3d` canvas via `captureStream(30)` + the soundtrack via
|
||||
the `transport.captureAudioStream()` hook (foundation2 provides it — taps the WebAudio graph
|
||||
post-gain, spatial mix included), mux with `MediaRecorder` (vp9/opus webm), stop at the last
|
||||
keyframe, download `festival4d-cut.webm`. Restore all prior UI/transport state afterwards.
|
||||
**Accept:** exported file duration within ±5% of the path span, contains an audio track, plays
|
||||
in Chrome — browser evidence (file size, duration probe via ffprobe) in the status file.
|
||||
|
||||
### M13 — Anchor manager & friend tags (lane F, frontend)
|
||||
Fill the stub `frontend/src/anchorPanel.js`: an always-available collapsible panel listing every
|
||||
anchor (color dot, label, jump-to, delete, rename/recolor via the new `PATCH /api/anchors/{id}`).
|
||||
Jump-to uses the foundation2 hook `scene3d.focusOn(x, y, z)`. Add the "friend tag" flow: a
|
||||
**＋ Tag** action that runs the existing M8 bbox-annotation flow *without* an event and names the
|
||||
resulting anchor (friends become labeled, colored anchors that x-ray onto every video for free).
|
||||
**Accept:** create → rename → recolor → jump → delete round-trip live in the browser; labels
|
||||
visible in the 3D scene and video overlays; anchors survive reload.
|
||||
|
||||
### M14 — Photo mode (lane F, frontend)
|
||||
Fill the stub `frontend/src/photoMode.js`. **📷 / `P`**: pause, hide helpers + gizmos + anchor
|
||||
labels (foundation2 hook `scene3d.setHelpersVisible(false)`), render one frame at 3840-wide
|
||||
(preserving aspect) to an offscreen target, download PNG, restore everything.
|
||||
**Accept:** PNG ≥ 3840 px wide with no grid/axes/frusta/gizmos; works in free-roam and
|
||||
follow-cam; with a splat present it renders the splat (verify on the synthetic point cloud +
|
||||
note the splat path).
|
||||
|
||||
### M15 — Moment FX (lane F, frontend)
|
||||
Fill the stub `frontend/src/fx.js`. When the playhead crosses a `pyro`/`confetti` event, fire a
|
||||
short particle burst at the event's resolved anchor (fall back to the stage centroid); on
|
||||
`bass_drop`, pulse the point-cloud/splat scale for ~a beat. Toggle via the hidden **✨** button;
|
||||
uses only `scene3d.addObject/removeObject`. Must fire when crossing markers in play *and* scrub.
|
||||
**Accept:** browser evidence of both effect types; frame rate stays ≥ 30 fps on the synthetic
|
||||
scene with FX enabled.
|
||||
|
||||
### M16 — Server-side camera paths (lane G, backend + thin UI)
|
||||
Implement the foundation2-stubbed routes over the new `paths` table:
|
||||
`GET /api/paths` (list), `GET /api/paths/{id}`, `POST /api/paths {name, json}`,
|
||||
`DELETE /api/paths/{id}` — `json` is the frozen camPath JSON as text, validated on POST.
|
||||
Fill the stub `frontend/src/pathsStore.js`: save-current-path (name prompt), load dropdown,
|
||||
delete — wired to the hidden path-save controls.
|
||||
**Accept:** endpoint tests (roundtrip deep-equal, 404s, invalid-JSON 422) in
|
||||
`backend/tests/test_paths.py`; live save → reload page → load reproduces keyframes exactly.
|
||||
|
||||
### M17 — Memory capsule: static shareable export (lane G, backend)
|
||||
`backend/festival4d/capsule.py`, CLI `python -m festival4d capsule [--out DIR]` (default
|
||||
`data/capsule/`). Build a **zero-backend** bundle:
|
||||
copy `frontend/dist` (error clearly if missing — tell the user to `npm run build`); copy media;
|
||||
bake every read-only API response to files at the *same relative paths* (`api/manifest`,
|
||||
`api/events`, `api/anchors`, `api/beats`, `api/videos/{id}/poses`, `api/pointcloud`, `api/splat`,
|
||||
`api/audio/{id}` — extracting audio if not yet cached); inject
|
||||
`<script>window.__F4D_API_BASE__=""</script>` into the copied `index.html`; set
|
||||
`"capsule": true` in the baked manifest (the frontend hides all write UI — foundation2 wires
|
||||
the flag); ship a stdlib **`serve.py`** in the bundle root that serves with HTTP Range support
|
||||
(video seeking needs 206s; plain `http.server` can't).
|
||||
**Accept:** `python -m festival4d capsule` on the synthetic project, then `python serve.py` in
|
||||
the bundle: app loads with **zero** requests to :8000, videos play *and seek*, 3D scene +
|
||||
timeline + spatial audio work, write UI absent; `backend/tests/test_capsule.py` covers bundle
|
||||
structure, baked-JSON validity, and serve.py Range (206) behavior.
|
||||
|
||||
## Contracts (frozen at foundation2 merge)
|
||||
|
||||
1. **beats.json** (and the `GET /api/beats` body):
|
||||
`{"tempo_bpm": float|null, "beats_s": [float], "onsets": [{"t_global_s": float, "strength": float 0..1}], "generated_by": "festival4d.audio_features"}`
|
||||
— all times are `t_global` seconds.
|
||||
2. **camPath JSON** (already shipped in `camPath.js`; now frozen):
|
||||
`{"version": 1, "keyframes": [{"t_global": float, "pos": [x,y,z], "quat": [x,y,z,w], "fov": float}]}`
|
||||
— **Three.js scene space**, quat order **[x,y,z,w]** (NOT COLMAP's [w,x,y,z]; convert via
|
||||
`geometry.colmap_to_threejs` on the backend, never inline).
|
||||
3. **paths table:** `paths(id INTEGER PK, name TEXT NOT NULL, created_at TEXT, path_json TEXT NOT NULL)`
|
||||
+ the four routes in M16.
|
||||
4. **`PATCH /api/anchors/{id}`** `{label?, color?}` → updated anchor dict (404 on missing).
|
||||
5. **Runtime API base override:** `state.js` resolves `window.__F4D_API_BASE__` first, then
|
||||
`VITE_API_BASE`, then the localhost default. **`manifest.capsule: true`** ⇒ read-only UI.
|
||||
6. **`transport.captureAudioStream()`** → `MediaStream` tapping the live WebAudio mix (and a
|
||||
matching `releaseAudioStream()`); `scene3d.focusOn(x,y,z)` and `scene3d.setHelpersVisible(bool)`.
|
||||
7. **Frozen files during 5b:** everything frozen in phase 1 **plus**
|
||||
`frontend/src/{transport,scene3d,state,main,camPath,annotate}.js` and `frontend/index.html`
|
||||
(foundation2 pre-wires every button — hidden — and every module import; lanes fill their own
|
||||
stub modules only). Believe a frozen file must change → `plan/CHANGE_REQUESTS.md`, work
|
||||
around locally, integration2 adjudicates.
|
||||
|
||||
## foundation2 scope (Phase 5a, one agent, branch `foundation2`)
|
||||
|
||||
- DB: `paths` table + `db.py` helpers. API: `/api/beats`, `/api/director`, `/api/paths*`,
|
||||
`PATCH /api/anchors/{id}` — implemented against synthetic data where cheap, stubbed with the
|
||||
spec's graceful-degradation pattern where a lane owns the logic. CLI: `features`, `direct`,
|
||||
`capsule` subcommands dispatching into stubs.
|
||||
- Backend stubs: `audio_features.py`, `director.py`, `capsule.py` (raise `NotImplementedError`;
|
||||
API/CLI catch and degrade, exactly like phase 1).
|
||||
- Frontend: create empty stub modules `director.js`, `exportVideo.js`, `anchorPanel.js`,
|
||||
`photoMode.js`, `fx.js`, `pathsStore.js`; import them from `main.js`; add ALL new buttons to
|
||||
`index.html` (hidden until their module reports ready); add the transport/scene3d hooks and
|
||||
the `__F4D_API_BASE__` + `manifest.capsule` wiring from the contracts above.
|
||||
- Tests for every new route shape (mirroring `test_api.py`'s exact-set style — note it locks
|
||||
the manifest key set; add `capsule` consciously).
|
||||
- Update the ownership table below if anything moved. Suite ≥ 121 + your new tests. Merge fast.
|
||||
|
||||
## integration2 scope (Phase 5c, one agent, branch `integration2`)
|
||||
|
||||
- Merge order irrelevant; resolve CHANGE_REQUESTS; full synthetic pass of every milestone
|
||||
(evidence per feature); raise the suite floor to the new count.
|
||||
- **Real-footage field test** (the actual milestone): 2–4 real overlapping clips through
|
||||
`ingest → sync → reconstruct → events → features → direct → serve`, then capsule the result.
|
||||
Log every failure as a `plan/ISSUES.md` entry (ISSUE-1 is the format) — fix nothing mid-test.
|
||||
- Tag `v0.2.0` when accepted by the coordinator.
|
||||
|
||||
## File ownership (Phase 5b)
|
||||
|
||||
| Owner | Files |
|
||||
|---|---|
|
||||
| Lane E | `backend/festival4d/audio_features.py`, `director.py`, `backend/tests/test_audio_features.py`, `test_director.py`, `frontend/src/director.js`, `exportVideo.js` |
|
||||
| Lane F | `frontend/src/anchorPanel.js`, `photoMode.js`, `fx.js` |
|
||||
| Lane G | `backend/festival4d/capsule.py`, `backend/tests/test_paths.py`, `test_capsule.py`, `frontend/src/pathsStore.js` — the paths-route bodies were CRUD one-liners over the db helpers, so foundation2 implemented them (basic tests in `test_phase5_api.py`); lane G owns deepening `test_paths.py` + the UI |
|
||||
| Every agent | its own `plan/status/<lane>.md` |
|
||||
| **Coordinator only** | `plan/DIRECTIVES.md` |
|
||||
| **Frozen during 5b** | contract item 7 above |
|
||||
|
||||
## Phase-5 pitfalls
|
||||
|
||||
1. **Quaternion order.** camPath is `[x,y,z,w]` (Three.js); COLMAP storage is `[w,x,y,z]`.
|
||||
Every past pose bug in this repo was a convention slip — go through `geometry.colmap_to_threejs`
|
||||
/ `lib/pose.js`, never inline the math.
|
||||
2. **The pane you verify in throttles rAF** (backgrounded tab). Drive the loop with the
|
||||
`window.__f4d` pump (`localStorage.setItem("f4dDebug","1")` in production builds); don't
|
||||
file "playback frozen" issues against your own hidden tab. MediaRecorder export (M12) needs
|
||||
a *focused* tab to capture real frames — say so in your evidence if you couldn't.
|
||||
3. **`test_api.py` locks response key-sets on purpose.** Adding a manifest key or route means
|
||||
consciously updating the exact-set assertion in the same commit — a failing set-lock is a
|
||||
feature, not an obstacle.
|
||||
4. **Range serving.** Any static path that serves video must answer 206 (spec M3 pitfall #3);
|
||||
`python -m http.server` does not — that's why M17 ships `serve.py`.
|
||||
5. **Never block on a missing optional input.** No beats.json → director still works (no beat
|
||||
snapping). No splat → photo mode shoots the point cloud. No audio track → transport already
|
||||
falls back. Follow the degradation pattern everywhere; it's the house style.
|
||||
6. **Decoded audio is big** (~23 MB/min stereo). M12/M17 must not decode all tracks just
|
||||
because they can — reuse what the transport already has.
|
||||
@ -1,214 +0,0 @@
|
||||
# Phase 6 — Friend Tracks (M18–M24): markers → moving people in the 4D scene
|
||||
|
||||
Canonical spec for phase 6. **Everything frozen in `../OPUS_BUILD_INSTRUCTIONS.md` (M0–M9)
|
||||
and `20-phase5.md` (M10–M17 contracts) still applies.** Standing protocol (status files,
|
||||
directives loop, evidence discipline, per-agent worktrees) is unchanged from
|
||||
[`README.md`](README.md); the newest round of [`DIRECTIVES.md`](DIRECTIVES.md) always wins.
|
||||
|
||||
## The idea
|
||||
|
||||
A friend tag today is a *static* anchor. Phase 6 makes friends **move**: wearable markers
|
||||
(distinct-hue patches or blinking LED badges) are detected in every camera, associated across
|
||||
synced views, triangulated per timestep, and become **tracks** — `t_global → (x,y,z)` paths
|
||||
rendered as ribbons in the 3D scene with moving labels, a follow-a-friend camera mode, and
|
||||
capsule support. Same machinery as annotation resolution (M8), run automatically over time.
|
||||
|
||||
## Context — what exists that you build on (read, don't re-invent)
|
||||
|
||||
- `geometry.py` — COLMAP↔Three.js conversion, ray casting, triangulation primitives
|
||||
(`resolve.py` shows exactly how a bbox becomes a 3D point; your solver is that, per-timestep).
|
||||
- `frames.py` — frame sampling via ffmpeg (SfM uses it; detection re-uses it).
|
||||
- `config.t_video_from_global` / `t_global_from_video` — the FROZEN timebase; detections are
|
||||
found at `t_video`, tracks live at `t_global`.
|
||||
- The synthetic fixture (`synthetic.py`) — every acceptance below is against it first.
|
||||
- Suite floor is **189 passed**. It only goes up.
|
||||
|
||||
## Phase map
|
||||
|
||||
```
|
||||
Phase 6a (serial, ONE agent) Phase 6b (parallel, one agent per lane) Phase 6c (serial)
|
||||
┌───────────────────────┐ ┌ lane/h-tracker ─── M19 M20 ──────┐ ┌─────────────────┐
|
||||
│ foundation3 │ ├ lane/j-ribbons ─── M21 M22 ──────┤ │ integration3 │
|
||||
│ fixture ground truth +│──merge──┤ ├──merge─│ full pass + │
|
||||
│ contracts + stubs │ └ lane/k-badges ──── M23 M24 ──────┘ │ tracked field │
|
||||
│ branch: foundation3 │ (independent — merge in any order) │ test → v0.3.0 │
|
||||
└───────────────────────┘ └────────────────────────────────── └─────────────────┘
|
||||
```
|
||||
|
||||
Lanes depend **only** on foundation3, never on each other.
|
||||
|
||||
## Milestones
|
||||
|
||||
### M18 — foundation3: fixture ground truth, contracts, stubs (serial, ONE agent)
|
||||
1. **Synthetic fixture upgrade** (`synthetic.py` — coordinator-sanctioned edit): render **two
|
||||
moving markers** into every camera's video via the existing camera projection:
|
||||
- marker A: a solid bright disc (frozen hue, e.g. pure magenta #ff00ff) following a known
|
||||
smooth trajectory across the stage (e.g. a slow circle, radius ~2, y≈1.5);
|
||||
- marker B: a **blinking** white/bright disc following a different known path, blinking the
|
||||
frozen protocol below with ID=5.
|
||||
Markers must be small (≤2 % of frame area) and rendered *after* the existing content so the
|
||||
full existing suite stays green (offsets, events, poses, SfM tests untouched). Write ground
|
||||
truth to `data/work/track_truth.json`: `{"markers": [{"marker_key": ..., "points":
|
||||
[{"t_global_s", "x", "y", "z"}]}]}` (Three.js scene space).
|
||||
2. **DB:** `tracks` + `track_points` tables (contract #2) + `db.py` helpers
|
||||
(`add_track`, `get_tracks`, `set_track_points`, `patch_track`, `delete_track`).
|
||||
3. **API:** `GET /api/tracks`, `PATCH /api/tracks/{id}` `{label?, color?}`,
|
||||
`DELETE /api/tracks/{id}`, `POST /api/tracks/solve` (dispatches into the lane-H stub with
|
||||
the house degradation: valid empty result + note while stubbed). Manifest gains
|
||||
`"has_tracks": bool` — update `test_api.py`'s key-set lock consciously in the same commit.
|
||||
4. **CLI:** `python -m festival4d track [--detect-only|--solve-only]` → lane-H stubs.
|
||||
5. **Backend stubs:** `tracker_detect.py` (`run_detect()`), `tracker_solve.py`
|
||||
(`run_solve()`) — `NotImplementedError` with the full contract in the docstring.
|
||||
6. **Frontend:** stub `frontend/src/friendTracks.js` imported from `main.js`, `init()` +
|
||||
per-frame `friendTracks.update(tGlobal)` wired into the master loop (fx.js pattern);
|
||||
hidden **👣 Friends** button `#btn-friends` + empty `#friends-panel` container in
|
||||
`index.html`; `state.tracks = []` loaded at boot when `manifest.has_tracks`.
|
||||
7. **Capsule:** add `api/tracks` to `capsule.py`'s bake list (additive; update lane G's
|
||||
bundle-structure test consciously in the same commit, CR-style note in the commit message).
|
||||
8. Tests for every new route shape; suite ≥ 189 + new tests. Merge fast — three lanes wait.
|
||||
|
||||
### M19 — Marker detection (lane H, backend)
|
||||
Fill `tracker_detect.py`. `run_detect()`: sample frames (~8 fps via `frames.py` machinery) from
|
||||
every video; detect markers two ways: **color mode** (HSV blob of the frozen marker hues;
|
||||
`marker_key = "hue:<bucket>"`) and **blink mode** (per-blob brightness over a sliding window,
|
||||
decode the frozen OOK protocol; `marker_key = "code:<id>"`). Write
|
||||
`data/work/detections.json` (contract #3). OpenCV (`opencv-python-headless`) is already a
|
||||
dependency. No markers → valid-empty detections, never a crash.
|
||||
**Accept:** on the synthetic fixture, ≥90 % of sampled frames where a marker is visible yield
|
||||
a detection within 3 px (normalized ≤0.008) of the projected ground-truth center; blink ID 5
|
||||
decoded; tests in `test_tracker_detect.py`.
|
||||
|
||||
### M20 — Track solving (lane H, backend)
|
||||
Fill `tracker_solve.py`. `run_solve()`: load detections; group by `marker_key`; per time bucket
|
||||
(≈0.25 s) collect same-key detections across cameras at matching `t_global`; **2+ views →
|
||||
triangulate** through `geometry` primitives (the `resolve.py` pattern — never inline pose
|
||||
math); **1 view → ray ∩ ground plane** (contract #5); smooth each track (moving average or
|
||||
Catmull-Rom, your call — document it); write `tracks`/`track_points` through `db` helpers;
|
||||
solid color for hue markers, palette color for code markers. `POST /api/tracks/solve` and the
|
||||
CLI now run detect→solve end-to-end.
|
||||
**Accept:** synthetic recovery — median 3D error of both tracks < 0.3 scene units against
|
||||
`track_truth.json`, ≥80 % temporal coverage of each marker's visible span; single-view
|
||||
fallback exercised by a test that hides all-but-one camera's detections; degradation (no
|
||||
detections → zero tracks + note); tests in `test_tracker_solve.py`.
|
||||
|
||||
### M21 — Ribbons, moving labels, friends panel (lane J, frontend)
|
||||
Fill `friendTracks.js`. On init (when `manifest.has_tracks`): unhide 👣; fetch
|
||||
`/api/tracks` into `state.tracks`. Toggle renders per-track: a **ribbon** polyline through the
|
||||
points (track color) + a **moving marker** (sphere + label sprite, position interpolated at
|
||||
the current `t_global`, hidden outside the track's time span) — objects via
|
||||
`scene3d.addObject/removeObject` only, registered with `scene3d.registerHelper` so photo mode
|
||||
hides them. Panel in `#friends-panel` (anchorPanel.js is the style/UX reference — read it,
|
||||
don't edit it): list tracks (dot, label, time span), rename/recolor via
|
||||
`PATCH /api/tracks/{id}` (controls carry `write-ui`), delete, and a **Follow** button per row.
|
||||
**Accept:** live-browser evidence — ribbons + moving labels track the synthetic markers in
|
||||
play and scrub; rename/recolor/delete round-trip; photo mode hides ribbons; capsule mode
|
||||
hides write controls but keeps ribbons/follow.
|
||||
|
||||
### M22 — Follow-a-friend camera (lane J, frontend)
|
||||
Chase cam on any track: `scene3d.enterPathMode()` hands you the camera (the camPath.js
|
||||
pattern — read it); every `update(tGlobal)` compute the track position, place the camera at a
|
||||
smoothed offset (behind/above, ~4 units, low-pass the position so it glides), look at the
|
||||
friend (`applyCameraPose`). Esc / Free-roam / snap-to-camera exits (pathMode already handles
|
||||
contention — mirror camPath's "someone else took the camera" check).
|
||||
**Accept:** live-browser evidence — camera glides after the moving marker in play and scrub;
|
||||
clean handoff to/from free-roam, snap-to-cam, and camPath playback; no fighting.
|
||||
|
||||
### M23 — LED badge firmware + hardware kit (lane K, hardware/docs)
|
||||
`hardware/badge/badge.ino` — ESP32/ATtiny-compatible Arduino sketch blinking the frozen OOK
|
||||
protocol (contract #4): configurable 6-bit ID, bright single LED or WS2812; plus
|
||||
`hardware/beacon/beacon.ino` — the sync/calibration beacon (distinct reserved ID 63,
|
||||
double-brightness). `docs/hardware.md`: parts list (boards, LEDs, battery, diffuser), build
|
||||
photos placeholders, wearing guidance (chest/hat height, diffusion, spacing), and the
|
||||
camera-side truth: what ranges/framerates the protocol survives (≥24 fps, exposure caveats).
|
||||
**Accept:** sketches compile (`arduino-cli compile` for esp32 + attiny if available — else
|
||||
document exact compile commands and mark unverified); protocol constants byte-identical to
|
||||
contract #4 (import/duplicate them in ONE header with a test-facing copy check documented).
|
||||
|
||||
### M24 — Bench self-test tool (lane K, backend script)
|
||||
`scripts/badge_selftest.py`: given a short phone clip of a badge on a desk, run the SAME
|
||||
decoder path as lane H (`tracker_detect` blink mode) and print the decoded ID + confidence —
|
||||
John's hardware smoke test before a shoot. Works on a synthetic self-test too:
|
||||
`--synthetic` renders a blinking clip via the fixture helpers and asserts round-trip.
|
||||
**Accept:** `uv run python scripts/badge_selftest.py --synthetic` decodes the rendered ID;
|
||||
clear failure text for: no blob, ID mismatch, too-short clip. Small test in
|
||||
`test_badge_selftest.py` (synthetic path only).
|
||||
|
||||
## Contracts (frozen at foundation3 merge)
|
||||
|
||||
1. **Marker hues (color mode):** magenta `#ff00ff` and cyan `#00ffff` are the two supported
|
||||
fixture hues; real shoots may configure others via `FESTIVAL4D_MARKER_HUES` (comma-sep hex).
|
||||
2. **Schema:**
|
||||
`tracks(id INTEGER PK, marker_key TEXT NOT NULL, label TEXT, color TEXT, created_at TEXT)`
|
||||
`track_points(id INTEGER PK, track_id FK, t_global_s FLOAT, x FLOAT, y FLOAT, z FLOAT,
|
||||
quality FLOAT, views INTEGER)` — points ordered by `t_global_s`; `views` = cameras used
|
||||
(1 = ground-plane fallback); coordinates in **Three.js scene space**.
|
||||
`GET /api/tracks` → `[{id, marker_key, label, color, points: [{t_global_s,x,y,z,quality,views}]}]`.
|
||||
3. **detections.json:** `{"videos": {"<video_id>": [{"t_video_s": float, "marker_key": str,
|
||||
"cx": float, "cy": float, "area": float, "conf": float 0..1}]}, "generated_by":
|
||||
"festival4d.tracker_detect"}` — `cx`/`cy` normalized 0..1 (bbox-annotation convention).
|
||||
4. **Blink protocol (OOK, camera-decodable):** bit period **133 ms** (4 frames @ 30 fps);
|
||||
word = preamble `11100` + 6-bit ID (MSB first) + even parity = 12 bits, repeated
|
||||
continuously (~1.6 s/word). ID 63 reserved for the sync beacon. Decoder needs ≥24 fps
|
||||
video and ≥1 full word of visibility.
|
||||
5. **Ground plane:** the synthetic fixture's ground is `y = 0` (scene space). For real
|
||||
projects the solver estimates it as the 5th-percentile-y horizontal plane of the point
|
||||
cloud and records it in its result dict. Single-view fallback intersects the pixel ray
|
||||
with this plane; such points get `views = 1` and `quality ≤ 0.5`.
|
||||
6. **Frozen during 6b:** everything frozen in phases 1–5 **plus** all phase-5 lane modules
|
||||
(`director.js`, `exportVideo.js`, `anchorPanel.js`, `photoMode.js`, `fx.js`,
|
||||
`pathsStore.js`, `audio_features.py`, `director.py`, `capsule.py`) and foundation3's edits
|
||||
(`synthetic.py`, `main.js`, `index.html`, api/cli/db). Lanes fill their own new files only.
|
||||
Believe a frozen file must change → `plan/CHANGE_REQUESTS.md`, work around, integration3
|
||||
adjudicates. Stub-era test assertions superseded by your milestone landing follow the
|
||||
CR-1/-4/-5 pattern: minimal edit + CR entry.
|
||||
|
||||
## integration3 scope (Phase 6c)
|
||||
Merge order irrelevant; resolve CRs; full synthetic pass (evidence per milestone); raise the
|
||||
suite floor; **tracked field test** — real clips with at least one badge/marker in frame
|
||||
(combine with the still-open phase-5 field test); failures → `plan/ISSUES.md`; propose
|
||||
`v0.3.0` (with `v0.2.0` if the phase-5 field test passes in the same session).
|
||||
|
||||
## File ownership (Phase 6b)
|
||||
|
||||
| Owner | Files |
|
||||
|---|---|
|
||||
| Lane H | `backend/festival4d/tracker_detect.py`, `tracker_solve.py`, `backend/tests/test_tracker_detect.py`, `test_tracker_solve.py` |
|
||||
| Lane J | `frontend/src/friendTracks.js` |
|
||||
| Lane K | `hardware/**`, `docs/hardware.md`, `scripts/badge_selftest.py`, `backend/tests/test_badge_selftest.py` |
|
||||
| Every agent | its own `plan/status/<lane>.md` |
|
||||
| **Coordinator only** | `plan/DIRECTIVES.md` |
|
||||
| **Frozen during 6b** | contract #6 above |
|
||||
|
||||
## Phase-6 pitfalls
|
||||
|
||||
1. **Time alignment is the whole game.** Detections happen at `t_video`; association happens
|
||||
at `t_global`. One `t_video_from_global` inversion mistake silently shears every track.
|
||||
Use the frozen helpers; test association against ground truth, not against itself.
|
||||
2. **Stage lighting lies about color.** The color detector must gate on saturation+value, not
|
||||
hue alone, and the acceptance is on the *synthetic* fixture — note in your status file that
|
||||
real-world hue robustness is a known field-test risk (that's why blink mode exists).
|
||||
3. **Blink decoding vs. video fps:** cameras at 24–60 fps sample the 133 ms bit period
|
||||
differently. Decode by integrating brightness per bit window using the video's actual fps
|
||||
(`meta.fps`), never by counting frames.
|
||||
4. **Don't fight the camera owners.** Follow-mode (M22) shares `pathMode` with camPath —
|
||||
mirror camPath's contention checks exactly, or two drivers will interleave writes.
|
||||
5. **rAF throttle (standing):** verify with the `__f4d` pump in hidden panes
|
||||
(`localStorage f4dDebug=1`); the WebAudio timer exemption gives you ~50 Hz `setTimeout`.
|
||||
6. **Fixture compatibility:** foundation3's synthetic edit runs UNDER the existing 189 tests —
|
||||
markers must not move offsets, events, poses, or break SfM-adjacent tests.
|
||||
|
||||
## Kickoff prompts (copy-paste one per Opus 4.8 session, in run order)
|
||||
|
||||
**foundation3 (first, alone):**
|
||||
> Clone ssh://git@100.71.119.27:222/monster/festifun.git. Read OPUS_BUILD_INSTRUCTIONS.md,
|
||||
> plan/20-phase5.md, plan/30-phase6.md (M18 is yours), and the latest round of
|
||||
> plan/DIRECTIVES.md. Execute foundation3 on branch `foundation3`; maintain
|
||||
> plan/status/foundation3.md per plan/status/TEMPLATE.md; suite ≥ 189 + your tests; push the
|
||||
> branch and report — the coordinator reviews and merges.
|
||||
|
||||
**lane H / lane J / lane K (parallel, after foundation3 merges):**
|
||||
> Clone ssh://git@100.71.119.27:222/monster/festifun.git and create your own git worktree
|
||||
> before any edit. Read OPUS_BUILD_INSTRUCTIONS.md, plan/30-phase6.md, plan/lane-<X>.md, and
|
||||
> the latest round of plan/DIRECTIVES.md. Execute your lane on branch `lane/<x>-<name>`;
|
||||
> maintain plan/status/lane-<X>.md; obey the ownership table and frozen-files contract; push
|
||||
> your branch and report — the coordinator reviews and merges. Do not merge to main.
|
||||
@ -31,105 +31,5 @@ Instead append an entry here and work around it locally; the integration agent a
|
||||
- **Workaround in place:** applied the minimal edit directly (test file, not a frozen contract
|
||||
file). The frozen `api.py` app is **unchanged**. Only lane D's owned modules + this obsolete
|
||||
stub assertion were touched. Full suite green (89 passed).
|
||||
- **Decision:** **APPLIED** (integration, 2026-07-16). Confirmed against the frozen api.py
|
||||
docstring: `POST /api/events/detect` returns `{result, events}` (spec M3). The renamed
|
||||
`test_detect_events_endpoint` asserts exactly that shape and is idempotent. No frozen file
|
||||
changed; approval also recorded in DIRECTIVES round 1.
|
||||
|
||||
### CR-2 — integration — 2026-07-16 — M8 needs annotation↔event link + event-type correction
|
||||
- **File:** `db.py` (`Annotation` schema), `api.py` (`AnnotationIn`, events routes)
|
||||
- **Problem:** Spec M8 requires (a) triangulating "another annotation **for the same event**
|
||||
from a different video" — but the frozen `annotations` schema has no event reference, and
|
||||
`AnnotationIn` carries no event id, so annotations can't be grouped per event; and (b) a
|
||||
correction dropdown that "writes `source='user'`" to an event's type — but there is no
|
||||
event-update helper or endpoint (only add/clear).
|
||||
- **Proposed change (additive only — no existing shape altered):**
|
||||
1. `annotations.event_id` — new **nullable** `ForeignKey("events.id")` column. Nullable ⇒
|
||||
existing rows/inserts unaffected; the synthetic fixture creates annotations = none, so no
|
||||
migration. Serves the two-view grouping.
|
||||
2. `AnnotationIn.event_id: int | None = None` — additive optional request field.
|
||||
3. `db.update_event(event_id, event_type=?, source=?, description=?)` helper +
|
||||
`PATCH /api/events/{event_id}` — additive route; existing GET/POST/detect unchanged.
|
||||
- **Workaround in place:** none needed — adjudicated and applied directly in this integration
|
||||
phase (the phase where the ownership matrix ends and frozen contracts may change on an
|
||||
approved CR). All changes are strictly additive; every pre-existing route/shape/column and
|
||||
the timebase + pose contracts are byte-for-byte unchanged. Full suite stays green.
|
||||
- **Decision:** **APPLIED** (integration, 2026-07-16). Additive, backward-compatible, and
|
||||
required for M8 acceptance. Frontend and backend M8 both depend on it.
|
||||
|
||||
### CR-3 — phase4 — 2026-07-16 — Anchor lifecycle: supersede + delete
|
||||
- **File:** `api.py` (`create_annotation`, new `DELETE /api/anchors/{id}`), `db.py`
|
||||
(`get_anchor`/`update_anchor`/`delete_anchor`, `get_annotations` event filter)
|
||||
- **Problem:** Round-3 review found each annotation created a NEW anchor, so the two-view
|
||||
"refine" flow left two identically-labeled anchors (a rough fallback + the triangulation), and
|
||||
there was no way to remove an anchor at all.
|
||||
- **Change (additive — no existing shape altered):** one anchor per event — a later view that
|
||||
triangulates supersedes the fallback anchor **in place** (same id; `create_annotation` response
|
||||
gains a `superseded` bool); a later fallback links to the existing anchor without moving it.
|
||||
New `DELETE /api/anchors/{id}` (unlinks annotations first, 404 on unknown). `db` gains
|
||||
`get_anchor`/`update_anchor`/`delete_anchor` and an `event_id` filter on `get_annotations`.
|
||||
- **Workaround in place:** none — applied directly (Phase 4, ownership matrix ended). Every
|
||||
pre-existing route/shape/column and the timebase + pose contracts are unchanged; the annotation
|
||||
response only gained a field. Suite 101 → **103** (added supersede + delete tests).
|
||||
- **Decision:** **APPLIED** (phase4, 2026-07-16). Fixes the sole Round-3 wart; strictly additive.
|
||||
|
||||
### CR-4 — lane G — 2026-07-17 — Drop obsolete `capsule` stub assertion in test_phase5_api.py
|
||||
- **File:** `backend/tests/test_phase5_api.py::test_cli_dispatches_stubs_gracefully`
|
||||
- **Problem:** The foundation2 test asserts `cli.main(["capsule"]) == 2`, i.e. the *stub*
|
||||
contract (`build_capsule` raises `NotImplementedError`). With M17 landed the entrypoint is
|
||||
real, so the CLI no longer exits 2 — the assertion fails by design, exactly like CR-1 when
|
||||
lane D landed M7.
|
||||
- **Proposed change:** remove only `["capsule"]` from the loop (features/direct stay — lane E
|
||||
still stubbed) with a comment pointing at the landed coverage. The capsule CLI dispatch is
|
||||
now covered for real in `backend/tests/test_capsule.py` (builds a bundle via
|
||||
`cli.main(["capsule", "--out", ...])`, and asserts the clear no-dist error path).
|
||||
- **Workaround in place:** none possible without perverting `build_capsule` semantics (the
|
||||
spec requires a clear "run npm run build" error, not `NotImplementedError`); applied the
|
||||
minimal edit directly per the CR-1 precedent, flagged here for integration2 to ratify.
|
||||
- **Decision:** **APPLIED — RATIFIED by coordinator at merge** (2026-07-17). Exact CR-1
|
||||
situation: a stub-contract assertion superseded by the milestone landing; the real CLI
|
||||
dispatch is covered in test_capsule.py. Suite 127 → **169** on the merged tree.
|
||||
|
||||
### CR-5 — lane E — 2026-07-17 — Retire remaining stub-era assertions in test_phase5_api.py (M10/M11 landed)
|
||||
- **File:** `backend/tests/test_phase5_api.py::test_cli_dispatches_stubs_gracefully` and
|
||||
`::test_director_degrades_to_valid_empty_campath`
|
||||
- **Problem:** Both assert lane E's *stub* contracts, correct only while M10/M11 were
|
||||
unimplemented. With `run_features`/`generate_path` landed: (a) `cli.main(["features"])` /
|
||||
`["direct"]` no longer raise `NotImplementedError`, so they exit 0, not 2; (b)
|
||||
`POST /api/director` on the synthetic fixture (events + poses present) returns a real
|
||||
non-empty camPath, so `keyframes == []` / `"note" in data` fail. Identical in kind to
|
||||
**CR-1** and to **CR-4** (lane G, coordinator-ratified the same day for `capsule`).
|
||||
- **Proposed change:** (a) with CR-4 already applied, lane E's landing empties the CLI stub
|
||||
loop entirely — retire `test_cli_dispatches_stubs_gracefully` (a comment points at the real
|
||||
dispatch coverage: test_capsule.py / test_audio_features.py / test_director.py); (b) rename
|
||||
the director test to `test_director_returns_valid_campath`, asserting the frozen camPath
|
||||
*shape* (version 1, exact keyframe key-set — holds stubbed or implemented); the empty+note
|
||||
degradation is now covered by lane E's own `test_director.py` (no-events / no-poses cases).
|
||||
- **Workaround in place:** applied the minimal edit directly per the CR-1/CR-4 precedent (test
|
||||
file, not a frozen contract file; `api.py`/`cli.py` byte-for-byte unchanged), including the
|
||||
merge reconciliation with CR-4's side of the same loop. Suite green on the merged tree.
|
||||
- **Decision:** **APPLIED — RATIFIED by coordinator at merge** (2026-07-17). Same
|
||||
supersession pattern as CR-1/CR-4; the shape-locked director test is stronger than the
|
||||
stub assertion it replaces. Suite 169 → **189** on the merged tree.
|
||||
|
||||
### CR-6 — foundation3 — 2026-07-17 — Conscious test-lock updates for the two additive M18 contract fields
|
||||
- **File:** `backend/tests/test_api.py::test_manifest_shape` (manifest exact-set lock) and
|
||||
`backend/tests/test_capsule.py::test_bundle_structure` (baked-paths list).
|
||||
- **Problem:** Not a supersession — these are the two *deliberate* lock edits M18 mandates
|
||||
(spec `plan/30-phase6.md` steps 3 & 7). foundation3 adds an additive manifest field
|
||||
`has_tracks` (drives the frontend loading friend tracks) and an additive capsule bake
|
||||
`api/tracks` (so ribbons + follow work zero-backend). The manifest key-set lock is
|
||||
intentionally exact ("has already caught two silent drifts"), and the capsule
|
||||
bundle-structure test enumerates baked paths, so both must be updated **consciously in the
|
||||
same commit** as the source change — the spec calls for exactly this + a CR note.
|
||||
- **Proposed change:** (a) add `"has_tracks"` to the `test_manifest_shape` exact-set and assert
|
||||
it is `False` on the live synthetic build (no solver has run — solving is lane H); (b) add
|
||||
`"api/tracks"` to the `test_bundle_structure` baked-paths tuple so the test now asserts the
|
||||
new read-only bake exists. No behavior removed; both are stronger post-edit.
|
||||
- **Workaround in place:** N/A — these are foundation-sanctioned edits to test files (not to a
|
||||
frozen *contract* file: `db.py`/`api.py`/`capsule.py` were extended additively per M18, and
|
||||
`synthetic.py`'s marker code is a coordinator-sanctioned M18 edit). Applied directly in the
|
||||
foundation3 commit per the house pattern; flagged here for integration3 to ratify.
|
||||
- **Decision:** **APPLIED — RATIFIED by coordinator at merge** (2026-07-17). Both edits are
|
||||
the spec-mandated conscious lock updates (30-phase6.md steps 3 & 7), additive and stronger
|
||||
post-edit. Suite 189 → **209** on the merged tree.
|
||||
- **Decision:** (integration) — informational; the change is a direct, unavoidable consequence
|
||||
of lane D landing. Confirm the assertion matches the intended M3 detect-response shape.
|
||||
|
||||
@ -4,213 +4,6 @@ Written **only** by the coordinator (the human's planning session). Agents: read
|
||||
|
||||
---
|
||||
|
||||
## Round 6 — 2026-07-17 — PHASE 6 planned: Friend Tracks (M18–M24)
|
||||
|
||||
**Spec:** `plan/30-phase6.md` (canonical). Lane briefs: `plan/lane-H-tracker.md`,
|
||||
`plan/lane-J-ribbons.md`, `plan/lane-K-badges.md`. Kickoff prompts are at the bottom of the
|
||||
spec — the human launches each Opus session with one.
|
||||
|
||||
**Directives:**
|
||||
1. **Now: `foundation3` only** (one agent, branch `foundation3`, M18). Contracts correctness
|
||||
beats speed — three lanes hard-depend on your fixture ground truth, schema, detections
|
||||
shape, and blink protocol. The synthetic-fixture edit must keep all 189 existing tests
|
||||
green (spec pitfall #6). Maintain `plan/status/foundation3.md`. **Push your branch; do NOT
|
||||
merge to main** — this phase the coordinator merges after review (see #4).
|
||||
2. **After foundation3 merges — lanes in priority order** (all three concurrently if capacity
|
||||
allows): **H** (tracker — longest, validates every contract), **J** (ribbons/follow — the
|
||||
visible payoff), **K** (badges — independent, hardware-facing). Per-agent worktrees remain
|
||||
MANDATORY.
|
||||
3. **Merge policy (changed from Round 4):** agents do NOT merge to main and do NOT tag.
|
||||
Push your lane branch + status file, report completion to the human; the human pings the
|
||||
coordinator session, which reviews (independent suite re-run, ownership check, CR
|
||||
adjudication) and merges in completion order. Everything else from Round 0 protocol
|
||||
(status cadence, blocker rule, evidence discipline) unchanged.
|
||||
4. **Review loop:** coordinator reads `plan/status/<lane>.md` + the branch diff on each ping;
|
||||
findings come back as a new DIRECTIVES round or direct merge. Stub-era assertions
|
||||
superseded by a landing milestone: CR entry + minimal edit (CR-1/-4/-5 house pattern).
|
||||
5. **integration3** (after all three merge): full synthetic pass, floor raise, tracked field
|
||||
test (fold in the still-open phase-5 field test), `v0.3.0` proposal. `v0.2.0` remains held
|
||||
per Round 5.
|
||||
|
||||
**For the human:** (a) launch foundation3 first, alone, with its kickoff prompt; tell the
|
||||
coordinator when its status file says ready; (b) then launch H, J, K in parallel; (c) the
|
||||
field-test gate is unchanged — real clips in `data/raw/`, now ideally with a marker or badge
|
||||
in frame; (d) if you order badge parts, `docs/hardware.md` (lane K) will have the list.
|
||||
|
||||
## Round 5 — 2026-07-17 — PHASE 5 COMPLETE on synthetic; floor 189; v0.2.0 waits on the field test
|
||||
|
||||
**Execution report (all same-day):** foundation2 → lanes E/G/F in parallel worktrees (per
|
||||
directive 3) → integration2. Every lane merged after independent coordinator verification
|
||||
(suite re-run in the lane's worktree, then on the merged tree, live-browser checks per
|
||||
feature). Full evidence: plan/status/{foundation2,lane-E,lane-F,lane-G,integration2}.md.
|
||||
|
||||
**Accepted:** M10–M17 all pass acceptance on the synthetic fixture. Highlights: beats recover
|
||||
the 120 BPM ground truth (worst beat 30 ms); director output is beat-snapped and shape-locked
|
||||
through frozen geometry helpers; export produced a 13.98 s vp9/opus webm (±5% ✓, audio track
|
||||
live) driven headless via the `__f4d` pump; the capsule boots with ZERO off-origin requests,
|
||||
seeks over Range, hides all write UI. Lane F self-verified live in isolated servers — model
|
||||
evidence discipline, worth repeating.
|
||||
|
||||
**Ratified:** CR-4 (lane G) and CR-5 (lane E) — stub-era test assertions superseded by
|
||||
milestones landing, CR-1 precedent. **The test floor is now 189.** Build stays clean.
|
||||
|
||||
**Standing decisions:**
|
||||
1. **v0.2.0 tag is HELD** until the real-footage field test runs (it is integration2's actual
|
||||
milestone; synthetic-only doesn't earn the tag). Failures go to plan/ISSUES.md, not fixes.
|
||||
2. Stub-era assertions in foundation tests are now an acknowledged lifecycle: when a milestone
|
||||
lands, retiring its stub assertion via a CR entry + pointer to the real coverage is the
|
||||
house pattern (CR-1/-4/-5). Foundation agents: prefer writing stub tests that survive the
|
||||
landing (assert shape, not stub-ness) so future lanes don't need CRs.
|
||||
3. `.gitignore` gained `data/capsule/` and `.claude/worktrees/`.
|
||||
|
||||
**For the human (the entire remaining critical path):**
|
||||
(a) **Drop 2–4 real overlapping clips into `data/raw/`** — the field test and v0.2.0 gate on
|
||||
this and only this.
|
||||
(b) Focused-window eyeball pass (10 min, headphones): sync panel stays green during playback;
|
||||
🎧 spatial audio while flying around; 🎬 auto-path ride; ⏺ export and CHECK THE FRAMES
|
||||
(headless verified duration/audio only — rendered frame quality needs human eyes);
|
||||
📷 photo PNG; ✨ FX bursts at full rAF rate; anchor panel roundtrip.
|
||||
(c) Optional but high-impact: train a splat for the real project
|
||||
(`scripts/splat_via_modelbeast.sh`) before capsuling it — photo mode + capsule get
|
||||
dramatically better.
|
||||
|
||||
## Round 4 — 2026-07-17 — Fresh-eyes upgrades landed; PHASE 5 begins (M10–M17)
|
||||
|
||||
**Coordinator work landed directly on `main` (self-reviewed + live-verified):**
|
||||
- `95b7178` — sync solver caches one rFFT per signal across the pairwise GCC-PHAT stage
|
||||
(1.8× on 6 cams × 10 min; equivalence locked by a new unequal-lengths test).
|
||||
- `c24af80` — **audio is now the master clock** (`GET /api/audio/{id}` AAC + WebAudio;
|
||||
`t_global` from `AudioContext.currentTime`; all `<video>` muted picture-only; graceful
|
||||
fallback) **+ spatial audio** (🎧 per-camera HRTF panners at live poses, listener = viewer
|
||||
camera, gains 1/√N). Verified live: clock drives playback, mid-play source switching is
|
||||
continuous, panners sit at the correct camera positions, toggle on/off mid-play is clean.
|
||||
- `db42c42` — chore: `python-multipart` locked (declared-but-unlocked drift), launch configs.
|
||||
- Suite is **121 passed**; frontend build clean. **The test floor is now 121.**
|
||||
|
||||
**Contract amendments (coordinator-ratified, additive only):** new route `GET /api/audio/{id}`;
|
||||
`transport.js` is now the WebAudio clock owner (its header comment is the reference);
|
||||
the `window.__f4d` debug hook is available in production builds via `localStorage f4dDebug=1`.
|
||||
`test_api.py`'s manifest key-set lock is unchanged.
|
||||
|
||||
**Directives — Phase 5 (spec: `plan/20-phase5.md`; lane briefs: `plan/lane-E/F/G-*.md`):**
|
||||
1. **Now: `foundation2` only** (one agent, branch `foundation2`, scope in 20-phase5.md).
|
||||
Same rule as phase 1: contracts correctness beats speed — three lanes will hard-depend on
|
||||
your route shapes, stubs, hooks, and UI wiring points. Maintain `plan/status/foundation2.md`.
|
||||
2. **After foundation2 merges — lanes in priority order** (all three concurrently if capacity
|
||||
allows): **E** (director — longest, earliest contract validation), **G** (capsule — the
|
||||
shareable payoff), **F** (social UX — smallest, pure frontend).
|
||||
3. **Worktree rule is MANDATORY this time:** every parallel agent creates its own `git worktree`
|
||||
before its first edit (Round 1 standing lesson; the phase-2 shared-tree collision cost us a
|
||||
stash cleanup). Single-agent phases (5a, 5c) may use the main working directory.
|
||||
4. **Merge policy unchanged:** merge to `main` per-milestone where the lane brief allows it;
|
||||
never wait on another lane. Then `integration2` (branch `integration2`), which owns the
|
||||
real-footage field test and the `v0.2.0` tag proposal — failures go to `plan/ISSUES.md`,
|
||||
not into hasty fixes.
|
||||
5. Standing protocol (status files, evidence discipline, blocker rule, DIRECTIVES re-read after
|
||||
every milestone) is unchanged from Round 0.
|
||||
|
||||
**For the human:** (a) the Round 3 focused-browser sync check is still open — now also put on
|
||||
headphones, enable 🎧 3D audio, and fly around; (b) real overlapping clips for the field test
|
||||
remain the gating item for integration2; (c) optional: a trained splat for a real project makes
|
||||
photo mode (M14) and the capsule (M17) dramatically more impressive —
|
||||
`scripts/splat_via_modelbeast.sh`.
|
||||
|
||||
## Round 3 — 2026-07-16 — Integration review: v0.1.0 ACCEPTED; polish + field test next
|
||||
|
||||
**Coordinator review of the integration work (independently verified):** `main` == `origin/main`
|
||||
(d313c8d), tag `v0.1.0` on e5d6b2c. Re-ran the suite: **101 passed**; frontend build clean. The
|
||||
M8 acceptance test is faithful (forward-projects ground truth, no self-confirmation); `resolve.py`
|
||||
is clean orchestration over lane B's frozen primitives; CR-1/CR-2 correctly adjudicated and
|
||||
strictly additive; camPath's Catmull-Rom parameterization maps exactly to three.js segment
|
||||
lookup (checked). The live-browser M8/M9 verification methodology was sound given the
|
||||
hidden-tab rAF throttle. **v0.1.0 accepted.**
|
||||
|
||||
**Ratification:** Round 2 was written by the integration agent; DIRECTIVES.md is coordinator-only.
|
||||
Content verified accurate — ratified as written. Rule stands for future rounds.
|
||||
|
||||
**Review findings (all polish; none block the tag):**
|
||||
1. **Anchor stacking — the one real wart.** Each annotation creates a NEW anchor, so the
|
||||
documented two-view "refine" flow leaves BOTH the view-1 fallback anchor and the view-2
|
||||
triangulated anchor, identically labeled (reproduced: 2 anchors for 1 object). On real
|
||||
footage the fallback can be far off and there is no way to remove it — no
|
||||
`DELETE /api/anchors/{id}`, no UI delete.
|
||||
2. `docs/ideas.md` (spec §5 future-extensions note) was never created.
|
||||
3. Human focused-browser M4 sync eyeball still open (carried from Round 2).
|
||||
|
||||
**Directives — Phase 4 (one agent, branch `phase4/polish-field-test`):**
|
||||
1. **Polish pass (small, do first):**
|
||||
- When an annotation resolves for an event that already has a resolved anchor from a prior
|
||||
annotation of the same event, **update that anchor in place** (supersede the fallback with
|
||||
the triangulation) instead of creating a sibling. Keep per-annotation `resolved_anchor_id`
|
||||
history consistent.
|
||||
- Add `DELETE /api/anchors/{id}` + a small delete affordance in the UI (e.g. in the
|
||||
annotation panel or anchor list). Additive API only; file a CR entry noting it.
|
||||
- Create `docs/ideas.md` (gaussian splatting, MP4 fly-through export, realtime ingest,
|
||||
multi-modal audio analysis — from the spec's future-extensions list).
|
||||
- Suite must stay ≥ 101; add tests for supersede + delete.
|
||||
2. **Field test protocol (the real milestone):** when the user drops 2–4 real overlapping clips
|
||||
into `data/raw/`, run `ingest → sync → reconstruct → events → serve`; record in
|
||||
`plan/status/phase4.md`: per-pair sync confidence, offsets, COLMAP registration rate,
|
||||
which milestones survived contact with real footage. **File failures as entries in a new
|
||||
`plan/ISSUES.md` rather than rushing fixes** (per plan/10-integration.md item 4).
|
||||
3. Maintain `plan/status/phase4.md`; same evidence discipline.
|
||||
|
||||
**For the human (unchanged + one new):** (a) open the app in a normal focused browser window
|
||||
once and confirm the sync panel stays green during continuous playback; (b) shoot or collect
|
||||
2–4 overlapping clips of the same performance for the field test; (c) ~~optionally set
|
||||
`GEMINI_API_KEY`~~ done — see addendum.
|
||||
|
||||
**Round 3 addendum (coordinator, same day) — real API keys wired + first field finding:**
|
||||
- A local **`.env`** now exists at the repo root (gitignored, chmod 600, **never commit**) with
|
||||
`GEMINI_API_KEY` (default provider) and OpenRouter creds mapped to the `local` provider
|
||||
(`FESTIVAL4D_OPENAI_*`, model `google/gemini-2.5-flash`). Load with
|
||||
`set -a; . ./.env; set +a` before `python -m festival4d events`. Phase-4 agent: the file is
|
||||
machine-local — if it's missing in your worktree, ask the coordinator, don't recreate keys.
|
||||
- **Field finding #1 (fixed by coordinator hotfix):** Google retired `gemini-2.5-flash` for new
|
||||
API keys (404 "no longer available to new users"). The Gemini model is now env-overridable
|
||||
(`FESTIVAL4D_GEMINI_MODEL`) with default `gemini-3.1-flash-lite` — verified live: inline
|
||||
video + JSON-schema structured output OK; **3/3 synthetic candidates classified correctly**
|
||||
(`light_show`/`bass_drop`, coherent descriptions). Suite still 101 passed. The degradation
|
||||
path also behaved exactly as designed during the failure — good lane D work.
|
||||
- Phase-4 agent: consider a `plan/ISSUES.md` entry template seeded from this finding's shape
|
||||
(symptom → root cause → fix → evidence).
|
||||
|
||||
## Round 2 — 2026-07-16 — Integration complete; v0.1.0 shipped
|
||||
|
||||
**Integration merged to `main` (e5d6b2c) and tagged `v0.1.0`.** M8 (annotation→3D correction +
|
||||
triangulation) and M9 (keyframed camera paths) are done; README rewritten for v0.1.0. Both
|
||||
change requests adjudicated and applied (CR-1, CR-2 — additive only; timebase/schema-columns/
|
||||
pose/API shapes otherwise unchanged).
|
||||
|
||||
**Verification (coordinator-run on merged tree):** 101 backend tests pass (up from the 96
|
||||
floor); two-view stage-corner recovery err **0.00000** (< 0.2); frontend build clean; live
|
||||
in-browser M8 triangulation (gap 0.21) and M9 path (camera lands exactly on keyframe at t);
|
||||
M4 sync ~16 ms via deterministic pump. Worktrees removed, debris stash dropped, `frontend/dist`
|
||||
gitignored.
|
||||
|
||||
**One open item for the human (not code-blocking):** the automated browser pane runs
|
||||
`document.hidden=true`, which throttles `requestAnimationFrame`, so continuous multi-cam
|
||||
playback sync could only be verified deterministically, not by eye. **Open the app in a normal
|
||||
focused browser window once** (`python -m festival4d serve` + `cd frontend && npm run dev`) and
|
||||
confirm the top-right sync panel stays green (<50 ms) during continuous playback.
|
||||
|
||||
**Next:** the pipeline is ready for real footage — drop 2–4 overlapping concert clips into
|
||||
`data/raw/` and run `ingest → sync → reconstruct → events → serve` (see README).
|
||||
|
||||
## Round 1 — 2026-07-16 — All lanes merged & verified; integration begins
|
||||
|
||||
**Coordinator review result:** foundation + all four lanes are merged to `main` (merge commits `e28dc78` A, `2cad2f7` B, `9d17b47` C; D fast-forwarded earlier at `5c2d7c6`). Independently verified on the merged tree: **96 backend tests pass**, synthetic `ingest → sync` recovers ground-truth offsets exactly (+0 / +1370 / −842 ms, drift 0), `events` finds 3 candidates and degrades gracefully unconfigured, frontend production build clean. Excellent evidence discipline across all lanes — keep it.
|
||||
|
||||
**Directives:**
|
||||
|
||||
1. **Integration phase starts now** — one agent, branch `integration` from `main`, per `plan/10-integration.md`. Maintain `plan/status/integration.md`.
|
||||
2. **CR-1 (lane D, test_api.py stub assertion): APPROVED by coordinator.** The edit was a direct consequence of M7 landing and touched no frozen contract file. Integration agent: mark the Decision line in `plan/CHANGE_REQUESTS.md` and confirm the new assertion matches the spec M3 `POST /api/events/detect` response shape (`{result, events}`).
|
||||
3. **Integration additions to the plan/10-integration.md scope** (small, discovered in review):
|
||||
- The shared-worktree collision (see lane A/C status round 0) left a coordinator-stashed debris stash in the main working dir (`git stash list` — "stale lane A/B debris"). After confirming nothing of value remains, drop it, and remove the now-merged worktrees `../festifun-laneA` and `../festifun-laneB` (`git worktree remove`).
|
||||
- Lane C's env note: continuous-playback sync was verified deterministically because the automated browser tab was backgrounded. During your full-system pass, verify the M4 sync overlay (<50 ms) once in a real focused browser window.
|
||||
- Test count sanity: suite is 96 passed on merge day; it must never drop below that.
|
||||
4. **Worktree protocol lesson (standing):** integration runs single-agent, so use the main working directory directly on branch `integration`. If parallel agents are ever launched again, each MUST create its own `git worktree` before its first edit — never share a working directory.
|
||||
|
||||
## Round 0 — 2026-07-16 — Run order & protocol
|
||||
|
||||
**Execution order:**
|
||||
|
||||
@ -1,27 +0,0 @@
|
||||
# Field Issues
|
||||
|
||||
Problems found when the pipeline meets **real footage** (or real APIs), logged here instead of
|
||||
being rushed into fixes mid-field-test (per plan/10-integration.md item 4). Coordinator triages;
|
||||
polish/fixes land in a later phase.
|
||||
|
||||
**Format** (newest at the bottom):
|
||||
|
||||
```
|
||||
### ISSUE-<n> — <area> — <date> — <one-line title>
|
||||
- **Symptom:** what was observed (exact error / measurement)
|
||||
- **Root cause:** why (if known; else "unknown — needs investigation")
|
||||
- **Fix / workaround:** what was done, or "OPEN"
|
||||
- **Evidence:** command output, test name, or measurement proving symptom/fix
|
||||
- **Status:** OPEN | FIXED | WONTFIX
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
<!-- entries below -->
|
||||
|
||||
### ISSUE-1 — events (AI classifier) — 2026-07-16 — gemini-2.5-flash 404 for new API keys
|
||||
- **Symptom:** `events` with a valid `GEMINI_API_KEY` returned `404 NOT_FOUND … models/gemini-2.5-flash is no longer available to new users`; 0/3 candidates classified (degraded to candidates-only).
|
||||
- **Root cause:** Google retired `gemini-2.5-flash` for newly-provisioned API keys; the model id was hardcoded in `GeminiClassifier`.
|
||||
- **Fix / workaround:** made the Gemini model env-overridable (`FESTIVAL4D_GEMINI_MODEL`), default `gemini-3.1-flash-lite` (verified to accept inline video + JSON-schema structured output). Coordinator hotfix, commit `8dca2f5`.
|
||||
- **Evidence:** post-fix run `classified=3 candidates_only=0`; labels `light_show` / `bass_drop` with coherent descriptions; suite 101 passed.
|
||||
- **Status:** FIXED
|
||||
@ -2,13 +2,6 @@
|
||||
|
||||
**Read [`../OPUS_BUILD_INSTRUCTIONS.md`](../OPUS_BUILD_INSTRUCTIONS.md) first — it is the canonical spec** (architecture, repo layout, data model, milestone details M0–M9, pitfalls). This directory only organizes *who builds what, in what order, without colliding*. Lane briefs reference spec milestones by number; they do not restate them.
|
||||
|
||||
> **Current phase: 6** (M18–M24, Friend Tracks) — spec and organization in
|
||||
> [`30-phase6.md`](30-phase6.md), lane briefs `lane-H/J/K-*.md`, run order in DIRECTIVES
|
||||
> Round 6 (note: this phase the **coordinator merges**, agents only push branches).
|
||||
> Phases 1–5 below are complete (v0.1.0 shipped; phase 5 M10–M17 accepted on synthetic,
|
||||
> suite floor 189, v0.2.0 held for the field test); their protocol sections (git, status,
|
||||
> directives loop, evidence discipline) remain the standing rules.
|
||||
|
||||
## Phase map
|
||||
|
||||
```
|
||||
|
||||
@ -1,18 +0,0 @@
|
||||
# Lane E — Director & export (branch `lane/e-director`)
|
||||
|
||||
**Scope: phase-5 milestones M10 + M11 + M12** (beats/onsets, auto-director, cinematic export) — see [`20-phase5.md`](20-phase5.md).
|
||||
|
||||
- Owned files: `backend/festival4d/audio_features.py`, `backend/festival4d/director.py`,
|
||||
`backend/tests/test_audio_features.py`, `backend/tests/test_director.py`,
|
||||
`frontend/src/director.js`, `frontend/src/exportVideo.js`. Nothing else.
|
||||
- Fill the stub bodies foundation2 left; signatures, routes, and CLI dispatch already exist.
|
||||
- Contracts you consume (frozen): `beats.json` shape, camPath JSON (quat `[x,y,z,w]`, Three.js
|
||||
space — convert COLMAP poses via `geometry.colmap_to_threejs`, never inline),
|
||||
`transport.captureAudioStream()`.
|
||||
- Build order M10 → M11 → M12: the director consumes beats; export consumes the director's path.
|
||||
Each milestone is independently mergeable — don't hold M10/M11 hostage to M12.
|
||||
- Degradation: no beats → director skips beat-snapping; no poses → director returns a clear
|
||||
error body, never a crash; export with no path → button stays disabled.
|
||||
|
||||
**Done when:** M10–M12 acceptance passes (browser evidence for M11/M12 in your status file),
|
||||
`pytest` ≥ 121 + your tests, merged to `main` with synthetic end-to-end intact.
|
||||
@ -1,17 +0,0 @@
|
||||
# Lane F — Social & fun UX (branch `lane/f-social`)
|
||||
|
||||
**Scope: phase-5 milestones M13 + M14 + M15** (anchor manager + friend tags, photo mode, moment FX) — see [`20-phase5.md`](20-phase5.md).
|
||||
|
||||
- Owned files: `frontend/src/anchorPanel.js`, `frontend/src/photoMode.js`, `frontend/src/fx.js`.
|
||||
Nothing else — buttons, imports, and hooks are pre-wired by foundation2.
|
||||
- Contracts you consume (frozen): `PATCH /api/anchors/{id}` + existing anchor GET/POST/DELETE,
|
||||
`scene3d.focusOn(x,y,z)`, `scene3d.setHelpersVisible(bool)`, `scene3d.addObject/removeObject`,
|
||||
the M8 annotation flow (drive it, don't modify `annotate.js`).
|
||||
- This is the "friends connect & remember" lane — labels, colors, and jump-to should feel like
|
||||
tagging people, not editing database rows. Keep interactions one-click where possible.
|
||||
- All verification is in-browser: use the `__f4d` debug pump (phase-5 pitfall #2) and put
|
||||
screenshots/measurements in your status file. No backend tests to write; the suite must
|
||||
simply stay green.
|
||||
|
||||
**Done when:** M13–M15 acceptance passes with browser evidence, frontend `npm run build` clean,
|
||||
merged to `main` with synthetic end-to-end intact.
|
||||
@ -1,17 +0,0 @@
|
||||
# Lane G — Persistence & capsule (branch `lane/g-capsule`)
|
||||
|
||||
**Scope: phase-5 milestones M16 + M17** (server-side camera paths, static memory-capsule export) — see [`20-phase5.md`](20-phase5.md).
|
||||
|
||||
- Owned files: `backend/festival4d/capsule.py`, `backend/tests/test_paths.py`,
|
||||
`backend/tests/test_capsule.py`, `frontend/src/pathsStore.js`, plus the paths-route bodies in
|
||||
the region foundation2 marks for lane G. Nothing else.
|
||||
- Contracts you consume (frozen): `paths` table + route shapes, camPath JSON (validate on POST),
|
||||
`window.__F4D_API_BASE__` override, `manifest.capsule` read-only flag.
|
||||
- M17 discipline: the capsule bakes API responses to files at the **same relative paths** the
|
||||
live API serves — the frontend must not know the difference. Ship `serve.py` (stdlib,
|
||||
Range-capable) in the bundle root; acceptance explicitly includes video *seeking* under it.
|
||||
- The capsule is the shareable artifact of the whole project — treat "zero requests to :8000"
|
||||
and "write UI absent" as hard acceptance, not nice-to-haves.
|
||||
|
||||
**Done when:** M16–M17 acceptance passes (endpoint tests + a served-capsule browser pass with
|
||||
evidence), `pytest` ≥ 121 + your tests, merged to `main` with synthetic end-to-end intact.
|
||||
@ -1,37 +0,0 @@
|
||||
# Lane H — Tracker (M19 detection, M20 solving)
|
||||
|
||||
You turn wearable markers in the videos into 3D friend tracks. Backend only.
|
||||
Spec: `30-phase6.md` M19/M20 + contracts #1–#5. Branch `lane/h-tracker`, own worktree.
|
||||
|
||||
## What foundation3 gives you
|
||||
- Stubs `tracker_detect.py` / `tracker_solve.py` with the contracts in their docstrings;
|
||||
API (`POST /api/tracks/solve` etc.), CLI (`track`), DB helpers, and the synthetic fixture's
|
||||
two moving markers + `data/work/track_truth.json` ground truth — all already wired.
|
||||
- You fill the two stub bodies and write the two test files. Nothing else.
|
||||
|
||||
## M19 — detection, the parts that bite
|
||||
- Sample frames with the `frames.py` machinery (~8 fps is plenty; document your rate).
|
||||
- Color mode: convert to HSV, gate on **saturation AND value** before hue (pitfall #2);
|
||||
connected components; centroid → normalized cx/cy. Hue buckets from contract #1 +
|
||||
`FESTIVAL4D_MARKER_HUES`.
|
||||
- Blink mode: track candidate bright blobs across the sampled window, integrate brightness
|
||||
per 133 ms bit window **using the video's real fps** (pitfall #3), correlate against the
|
||||
preamble, check parity, emit `code:<id>`.
|
||||
- Empty/dark/markerless video → valid-empty detections. Never crash.
|
||||
|
||||
## M20 — solving, the parts that bite
|
||||
- `t_video → t_global` through the frozen helpers ONLY (pitfall #1).
|
||||
- Triangulation through `geometry` primitives exactly as `resolve.py` does — read it first.
|
||||
- Bucket ≈0.25 s; 2+ cameras → triangulate (record `views`, `quality` from residual);
|
||||
1 camera → ray ∩ ground plane per contract #5 (`views=1`, `quality ≤ 0.5`).
|
||||
- Smooth per track; write through `db` helpers; re-solve replaces that marker's tracks
|
||||
(idempotent — don't stack duplicates on every run).
|
||||
|
||||
## Acceptance (evidence in plan/status/lane-H.md)
|
||||
- [ ] Detection: ≥90 % recall on visible synthetic markers, ≤0.008 normalized center error;
|
||||
blink ID 5 decoded (`test_tracker_detect.py`)
|
||||
- [ ] Solve: median 3D error < 0.3 vs `track_truth.json`, ≥80 % coverage; single-view
|
||||
fallback test; idempotent re-solve; degradations (`test_tracker_solve.py`)
|
||||
- [ ] CLI + `POST /api/tracks/solve` run detect→solve end-to-end on the fixture
|
||||
- [ ] Suite ≥ foundation3's count + yours, green; stub-era assertions superseded by your
|
||||
landing → CR entry (CR-1/-4/-5 pattern)
|
||||
@ -1,38 +0,0 @@
|
||||
# Lane J — Ribbons & follow-a-friend (M21, M22)
|
||||
|
||||
You make tracks visible and rideable. Pure frontend — ONE file: `frontend/src/friendTracks.js`.
|
||||
Spec: `30-phase6.md` M21/M22 + contract #2. Branch `lane/j-ribbons`, own worktree.
|
||||
|
||||
## What foundation3 gives you
|
||||
- Your stub module imported from main.js with `init()` + per-frame `update(tGlobal)` already
|
||||
called; hidden `#btn-friends` + `#friends-panel` in index.html; `state.tracks` loaded at
|
||||
boot when `manifest.has_tracks`.
|
||||
- Read-only building blocks you may CALL but never edit: `scene3d.addObject/removeObject`,
|
||||
`registerHelper` (photo mode hides you for free), `enterPathMode`/`exitPathMode`/
|
||||
`applyCameraPose` (the camPath.js pattern — read that file first), `focusOn`;
|
||||
`PATCH/DELETE /api/tracks/{id}`. anchorPanel.js is your UX/style reference (inject your own
|
||||
<style> tag; reuse the CSS vars). Mark mutating controls `write-ui`.
|
||||
|
||||
## M21 — ribbons + panel, the parts that bite
|
||||
- Interpolate the moving marker between track points at the current tGlobal (linear is fine);
|
||||
HIDE it outside the track's span — don't park it at the ends.
|
||||
- Rebuild ribbons only on tracks-changed, not per frame; per-frame work = one position lerp
|
||||
per visible track (fx.js shows the hot-path discipline).
|
||||
- Empty tracks (solver not run) → panel says so, no errors; capsule mode: ribbons + follow
|
||||
stay, write controls hidden.
|
||||
|
||||
## M22 — follow cam, the parts that bite
|
||||
- Mirror camPath's contention handling EXACTLY: if `scene3d.pathMode` goes false under you
|
||||
(snap-to-cam, Esc, free-roam), stop following without a fight (pitfall #4).
|
||||
- Low-pass the camera position (e.g. lerp factor ~0.08/frame) so it glides; look at the
|
||||
friend, offset ~4 units behind/above; scrubbing far must not whiplash (clamp the lerp on
|
||||
big jumps: teleport if the target moved > ~5 units).
|
||||
- Starting a camPath playback or clicking 🎬 must cleanly cancel follow mode (and vice versa).
|
||||
|
||||
## Acceptance (evidence in plan/status/lane-J.md — you likely can't drive a browser;
|
||||
write defensively, list exactly what the coordinator must verify live)
|
||||
- [ ] Ribbons + moving labels track the synthetic markers in play AND scrub
|
||||
- [ ] Panel: rename/recolor (PATCH) + delete round-trip; `write-ui` on mutators
|
||||
- [ ] Follow glides after a friend; clean handoffs with free-roam / snap / camPath / 🎬
|
||||
- [ ] Photo mode hides ribbons (registerHelper); capsule keeps read-only features
|
||||
- [ ] `npm run build` clean; backend suite unchanged and green
|
||||
@ -1,40 +0,0 @@
|
||||
# Lane K — LED badges & bench kit (M23 firmware+docs, M24 self-test)
|
||||
|
||||
You build the wearable side: badge firmware speaking the frozen blink protocol, the hardware
|
||||
guide, and the bench self-test John runs before a shoot. Spec: `30-phase6.md` M23/M24 +
|
||||
contract #4. Branch `lane/k-badges`, own worktree.
|
||||
|
||||
## M23 — firmware + docs, the parts that bite
|
||||
- Contract #4 is FROZEN: 133 ms bit, `11100` preamble + 6-bit ID + even parity, continuous
|
||||
repeat, ID 63 = sync beacon. Keep the constants in ONE header (`hardware/badge/protocol.h`)
|
||||
shared by badge.ino and beacon.ino; document the manual-sync rule with
|
||||
`tracker_detect.py`'s decoder constants (can't import across languages — state both
|
||||
locations in the header comment and in docs/hardware.md).
|
||||
- badge.ino: plain Arduino APIs (millis()-paced, no delay()-drift), single bright LED on a
|
||||
PWM pin OR a WS2812 (compile-time #define), ID set by #define (document per-badge flashing).
|
||||
Hardware is never ideal on paper: expose a `BIT_TRIM_US` calibration knob for clock drift.
|
||||
- beacon.ino: ID 63, double LED brightness/size guidance.
|
||||
- docs/hardware.md: parts list with rough prices (ESP32 dev board / ATtiny85, LED + resistor
|
||||
or WS2812, LiPo/coin cell, diffuser ping-pong ball), wiring, flashing (`arduino-cli`
|
||||
commands), wearing guidance (chest/hat, diffusion, one badge per person, ≥24 fps cameras,
|
||||
exposure caveat: blown-out video kills OOK contrast).
|
||||
- Compile check: `arduino-cli compile --fqbn esp32:esp32:esp32` (and attiny if the core is
|
||||
installable). If arduino-cli isn't available in your environment, document the exact
|
||||
commands and mark the box unverified — do NOT claim compiled without output.
|
||||
|
||||
## M24 — bench self-test, the parts that bite
|
||||
- `scripts/badge_selftest.py` must run lane H's REAL decoder (`tracker_detect` blink path) —
|
||||
import it, don't re-implement; if lane H hasn't merged yet, code against the frozen
|
||||
contract #3/#4 shapes and mark the integration box pending (integration3 re-runs it).
|
||||
- `--synthetic` mode: render a small blinking clip using the fixture's marker-rendering
|
||||
helpers (foundation3 exposes them in synthetic.py) and assert the ID round-trips.
|
||||
- Clear, John-readable failure messages: no blob found / decoded wrong ID / clip too short
|
||||
(< 1 word). Exit codes: 0 ok, 1 decode-fail, 2 usage.
|
||||
|
||||
## Acceptance (evidence in plan/status/lane-K.md)
|
||||
- [ ] protocol.h constants byte-identical to contract #4; badge + beacon sketches compile
|
||||
(output pasted) or compile commands documented + marked unverified
|
||||
- [ ] docs/hardware.md complete enough that John can order parts from it
|
||||
- [ ] `uv run python scripts/badge_selftest.py --synthetic` decodes the rendered ID
|
||||
(`test_badge_selftest.py` covers it); failure paths print actionable text
|
||||
- [ ] Backend suite green; `npm` untouched
|
||||
@ -1,38 +0,0 @@
|
||||
# Status — deploy (digalot.fyi/festifun)
|
||||
|
||||
## Round 1 — 2026-07-16 — STATUS: app deploy-ready; BLOCKED on server access
|
||||
|
||||
**Ask:** host the app live at `digalot.fyi/festifun` on `dealgod@100.94.195.115` (tailscale).
|
||||
**Owner decision:** fully open (all features public + unauthenticated). Risk flagged + accepted.
|
||||
|
||||
### Done — the app is now deployable behind a path prefix
|
||||
- **Frontend deploy-ready.** `API_BASE` is env-driven (`VITE_API_BASE`, default `localhost:8000`
|
||||
so dev is unchanged); added `vite.config.js` with `base` from `FESTIVAL4D_BASE`. A prod build
|
||||
`FESTIVAL4D_BASE=/festifun/ VITE_API_BASE=/festifun npm run build` emits assets under
|
||||
`/festifun/` with the API base baked as `/festifun` (localhost:8000 absent from the bundle).
|
||||
- **Validated end-to-end locally.** Built the prod bundle, ran a Python reverse proxy mirroring
|
||||
the nginx config (static + prefix-stripping `/festifun/api` & `/festifun/media` with Range),
|
||||
loaded it in a browser: **app boots, manifest/pointcloud proxy OK, media Range = 206, videos
|
||||
play, 3D scene + overlays + timeline all render under the `/festifun` prefix.**
|
||||
- **Deploy artifacts written** in `deploy/`: `festifun.nginx.conf` (location blocks, prefix
|
||||
strip, Range), `festifun-api.service` (uvicorn on 127.0.0.1:8000, localhost-only, no keys),
|
||||
`DEPLOY.md` (build → rsync → service → nginx, plus Caddy note).
|
||||
- Default build unchanged (root asset paths); backend suite still **103 passed**.
|
||||
|
||||
### Deliberate deploy choice — keys OFF the public box
|
||||
`POST /api/events/detect` spends real AI credits. Rather than expose that on an unauthenticated
|
||||
public endpoint, moment labels are baked into the shipped DB by running `events` locally before
|
||||
deploy; the server carries no `GEMINI_API_KEY`/OpenRouter creds and `detect` degrades to
|
||||
candidate-only (no spend). Owner can opt into live classification, but that needs nginx
|
||||
rate-limiting first (documented in DEPLOY.md).
|
||||
|
||||
### BLOCKED — cannot finish without server access
|
||||
- My SSH key is **not authorized** on `dealgod@100.94.195.115` (rejected directly and via the
|
||||
johnking box — both `Permission denied (publickey)`). Need the key added to that host's
|
||||
`~/.ssh/authorized_keys`:
|
||||
`ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPL3EE4wE6nv0V0BNMbJ0Gl+px3wOM8K+V6LvCJZa7PM monsterrobotparty@gmail.com`
|
||||
- Also need to know the server's web stack (nginx vs caddy, where digalot.fyi's server block
|
||||
lives). Recon command in the chat; I'll adapt the config once I can see it.
|
||||
|
||||
**Next (once access granted):** recon the web stack → adapt `deploy/festifun.nginx.conf` (or write
|
||||
the Caddy equivalent) → build+rsync+service+reload per DEPLOY.md → verify `https://digalot.fyi/festifun/`.
|
||||
@ -1,55 +0,0 @@
|
||||
# Status — foundation2
|
||||
|
||||
## Round 1 — 2026-07-17 — STATUS: ready_to_merge
|
||||
|
||||
**Directives acknowledged:** Round 4 of plan/DIRECTIVES.md. Single-agent phase 5a — main
|
||||
working directory on branch `foundation2` (per directive 3).
|
||||
|
||||
**Acceptance checklist** (plan/20-phase5.md "foundation2 scope"):
|
||||
- [x] DB: `paths` table (contract #3) + helpers `add/get/get_one/delete_path`, plus
|
||||
`patch_anchor` (label/color only) — `db.py`
|
||||
- [x] API: `GET /api/beats` (serves `config.BEATS_JSON`, 404 when absent),
|
||||
`POST /api/director` (degrades to a VALID empty camPath + `note` while lane E is
|
||||
stubbed), `GET/POST/DELETE /api/paths*` (implemented — CRUD over the helpers was
|
||||
cheaper than stubbing; POST validates the camPath shape → 422),
|
||||
`PATCH /api/anchors/{id}` (contract #4) — `api.py`
|
||||
- [x] Manifest gained `"capsule": false` — `test_api.py` exact-set lock updated in the same
|
||||
commit (pitfall #3)
|
||||
- [x] CLI: `features`, `direct`, `capsule` subcommands dispatch into stubs; house
|
||||
degradation (exit 2, no traceback) — `cli.py`
|
||||
- [x] Backend stubs raise NotImplementedError with the frozen contract in the docstring:
|
||||
`audio_features.py`, `director.py`, `capsule.py`
|
||||
- [x] Frontend stub modules created + imported from `main.js`, each wiring its own hidden
|
||||
button in init(): `director.js`, `exportVideo.js`, `anchorPanel.js`, `photoMode.js`
|
||||
(+ per-frame `fx.update()` call in the master loop), `fx.js`, `pathsStore.js`
|
||||
- [x] index.html: ALL phase-5 buttons pre-wired hidden — #btn-director, #btn-path-save,
|
||||
#path-load-select, #btn-path-del, #btn-export, #btn-photo, #btn-fx, #btn-anchors,
|
||||
#anchor-panel container; `.write-ui` class + `body.capsule .write-ui{display:none}`
|
||||
- [x] Hooks (contract #6): `transport.captureAudioStream()/releaseAudioStream()` (post-gain
|
||||
tap, spatial included, tracks re-join across seeks), `scene3d.focusOn(x,y,z)`,
|
||||
`scene3d.setHelpersVisible(bool)` (+ `registerHelper`; camPath gizmos registered)
|
||||
- [x] `state.js`: `window.__F4D_API_BASE__` ?? `VITE_API_BASE` ?? localhost (contract #5);
|
||||
`state.capsule` from the manifest toggles `body.capsule`
|
||||
- [x] Suite ≥ 121 + new tests: **127 passed** (121 floor + 6 in `test_phase5_api.py`)
|
||||
- [x] Ownership table updated (paths-route bodies → foundation2; noted in 20-phase5.md)
|
||||
|
||||
**Done this round — evidence:**
|
||||
- `uv run pytest backend/tests` → **127 passed** (was 121).
|
||||
- `npm run build` → clean (pre-existing chunk-size warning only).
|
||||
- Live browser (Vite :5173 + uvicorn :8000, synthetic fixture): app boots (3 videos, sync
|
||||
+0 ms); all 9 new controls present and `display:none`; `setHelpersVisible(false)` hides
|
||||
grid (and restores); `focusOn(1,0.5,0)` retargets OrbitControls to exactly [1,0.5,0];
|
||||
`captureAudioStream()` returns a MediaStream with 1 audio track, release clean;
|
||||
`POST /api/director` → `{version:1, keyframes:[], note:…}` and round-trips
|
||||
`camPath.fromJSON` (0 keyframes, no throw); paths save→load→fromJSON reproduces 2
|
||||
keyframes, DELETE → 200 then 404; `PATCH /api/anchors/1 {color}` → updated dict;
|
||||
`GET /api/beats` → 404; `body.capsule` hides #annot-panel/#btn-path-save/#btn-path-del.
|
||||
|
||||
**Blockers / questions for coordinator:** none. One conscious deviation, self-adjudicated
|
||||
(coordinator == foundation2 agent this round): paths route bodies implemented here instead
|
||||
of stub-marked for lane G — they were one-line calls into the db helpers I had to write
|
||||
anyway; stubbing them would have been more code than implementing. Lane G scope shrinks to
|
||||
capsule + pathsStore.js + deepening test_paths.py.
|
||||
|
||||
**Next:** merge to `main`, then lanes E, G, F in parallel (each in its own worktree —
|
||||
directive 3 is mandatory for 5b).
|
||||
@ -1,57 +0,0 @@
|
||||
# Status — foundation3 (Phase 6a / M18)
|
||||
|
||||
## Round 6 — 2026-07-17 — STATUS: ready_to_merge
|
||||
|
||||
**Directives acknowledged:** round 6 of plan/DIRECTIVES.md. Directive #1: `foundation3` only,
|
||||
branch `foundation3`, push — do NOT merge. Contracts correctness beats speed; keep 189 green.
|
||||
|
||||
**Acceptance checklist** (M18, plan/30-phase6.md §M18):
|
||||
- [x] 1. Synthetic fixture: two moving markers (A magenta `hue:150`, B blink `code:5`)
|
||||
composited through each camera's real pose projection; `data/work/track_truth.json`
|
||||
built from the SAME 3D paths (`synthetic.build_track_truth`, Three.js scene space).
|
||||
`synthetic.project_point` is the exact inverse of `geometry.ray_from_pixel`. Base
|
||||
`testsrc2` is saturation/brightness-muted (`BASE_SATURATION`/`BASE_BRIGHTNESS`) so its
|
||||
own magenta/cyan colour bars don't swamp lane H's colour detector — the discs are the
|
||||
only high-sat magenta/cyan + only bright small blobs (verified by a real-render test).
|
||||
- [x] 2. DB: `Track`/`TrackPoint` tables (contract #2) + `add_track`/`get_tracks`/`get_track`/
|
||||
`get_track_points`/`set_track_points`/`patch_track`/`delete_track`/`has_tracks`.
|
||||
- [x] 3. API: GET /api/tracks, PATCH/DELETE /api/tracks/{id}, POST /api/tracks/solve (degrades
|
||||
to valid empty + note while lane H stubbed), manifest `has_tracks`. Key-set lock edit → CR-6.
|
||||
- [x] 4. CLI: `python -m festival4d track [--detect-only|--solve-only]` → lane-H stubs.
|
||||
- [x] 5. Backend stubs: `tracker_detect.py`/`tracker_solve.py` — `NotImplementedError` + full
|
||||
contracts #1–#5 in docstrings; foundation-owned frozen constants (blink OOK, hue keys)
|
||||
live in `tracker_detect` as the single source of truth (`synthetic` imports them).
|
||||
- [x] 6. Frontend: `friendTracks.js` stub (never-throws, fx.js pattern) wired into `main.js`
|
||||
loop; hidden `#btn-friends` + `#friends-panel`; `state.tracks` loaded when `has_tracks`.
|
||||
`npm run build` clean (27 modules, 851 ms).
|
||||
- [x] 7. Capsule: `capsule.py` bakes `api/tracks` (additive). Bundle-structure test edit → CR-6.
|
||||
- [x] 8. Tests: `test_tracks_foundation.py` (20) — DB helpers, routes, solve degradation,
|
||||
manifest, frozen protocol/encoders, track_truth shape, the **projection round-trip**, and
|
||||
a real-ffmpeg render test proving marker A is the dominant magenta blob after base-muting.
|
||||
|
||||
**Done this round:** all 8 steps land. Evidence:
|
||||
- Full suite **209 passed, exit 0** (`uv run pytest -q`) — the 14 pre-existing files sum to
|
||||
exactly 189 (floor held), + 20 new in `test_tracks_foundation.py` (all passed, none skipped).
|
||||
- `test_projection_round_trip_recovers_ground_truth`: projects both markers' 3D truth through
|
||||
every camera pose, back-projects via frozen `ray_from_pixel`, triangulates via
|
||||
`triangulate_rays` — **worst recovery error < 1e-3** across ≥8 samples in 2+ views. Lane H's
|
||||
M20 target (median 3D error < 0.3) is reachable by construction (~300× margin).
|
||||
- `encode_word(5) == [1,1,1,0,0,0,0,0,1,0,1,1]`; all 64 IDs preamble+parity+MSB-recover checked.
|
||||
- Frontend production build clean; manifest `has_tracks` reflects the DB live.
|
||||
- CR-6 filed (plan/CHANGE_REQUESTS.md): the two sanctioned M18 test-lock edits (manifest key-set,
|
||||
capsule bundle paths) — additive, flagged for integration3 ratification.
|
||||
|
||||
**Recovery note (2026-07-17):** a prior foundation3 attempt left M18 half-done and **uncommitted**
|
||||
in a stale locked worktree — steps 2/3/4/5/7 (DB/API/CLI/stubs/capsule) + the `synthetic.py`
|
||||
marker code present, but step 1's ground-truth never validated, steps 6/8 (frontend/tests) not
|
||||
started, suite never run. This session salvaged it: committed the WIP as checkpoint `6a703c3`,
|
||||
then completed steps 1/6/8 and verified every inherited contract choice against the spec. The
|
||||
one real bug found: the base `testsrc2` visual carried solid magenta+cyan bars (the marker hues)
|
||||
that would have swamped lane H's colour detector — fixed by muting base saturation/brightness
|
||||
and locked with a real-render regression test (marker A dominant-blob within 2.5 px, 24/24
|
||||
frames). No work was redone. Do NOT merge — coordinator reviews + merges (DIRECTIVES R6 #3).
|
||||
|
||||
**Blockers / questions for coordinator:** none. Ready for review + merge.
|
||||
|
||||
**Next:** coordinator reviews `git diff main..foundation3` + this file, merges to `main`.
|
||||
After merge, lanes H / J / K launch in parallel worktrees (DIRECTIVES R6 #2).
|
||||
@ -1,66 +0,0 @@
|
||||
# Status — integration
|
||||
|
||||
## Round 1 — 2026-07-16 — STATUS: ready_to_merge
|
||||
|
||||
**Directives acknowledged:** round 1 of plan/DIRECTIVES.md (all lanes merged & verified;
|
||||
integration begins; CR-1 approved; drop debris stash + remove laneA/B worktrees; real-browser
|
||||
sync check; test floor 96).
|
||||
|
||||
**Scope done: M8 + M9 + README + cross-lane wiring + cleanup.**
|
||||
|
||||
### Change requests adjudicated (plan/CHANGE_REQUESTS.md)
|
||||
- **CR-1 (lane D): APPLIED** — confirmed `POST /api/events/detect` returns `{result, events}`;
|
||||
renamed test asserts it.
|
||||
- **CR-2 (integration, filed + APPLIED):** M8 needs an annotation↔event link and event-type
|
||||
correction, neither of which the frozen contract had. Added — strictly additive:
|
||||
- `annotations.event_id` (nullable FK) + `AnnotationIn.event_id` — two-view grouping.
|
||||
- `db.update_event` + `db.get_event` + `PATCH /api/events/{id}` — the correction dropdown.
|
||||
Every pre-existing route/shape/column and the timebase + pose contracts are unchanged.
|
||||
- A second foundation stub assertion (`test_annotation_stored`: `anchor_id is None`) was, like
|
||||
CR-1, only valid pre-M8; renamed to `test_annotation_resolves_to_anchor` for the landed shape.
|
||||
|
||||
### M8 — correction + annotation → 3D
|
||||
- New `backend/festival4d/resolve.py`: ray from bbox center (pose interpolated at t_video via
|
||||
`geometry.slerp_pose`), then triangulate across two same-event views (reject near-parallel /
|
||||
gap > 0.5), else nearest point-cloud point (0.3 cyl), else ray at centroid depth. Pure
|
||||
orchestration over lane B's frozen primitives.
|
||||
- `POST /api/annotations` now resolves + creates+links an anchor (label = event description).
|
||||
- Frontend `annotate.js`: click a timeline marker → correction panel (type dropdown → PATCH,
|
||||
source→user; live marker recolor); **Annotate location** → drag a letterbox-correct box on
|
||||
any video → POST. Resolved anchors pushed live into overlays **and** the 3D scene (new anchor
|
||||
spheres + labels in `scene3d.js`).
|
||||
- **Acceptance (spec M8): PASS.** `tests/test_resolve.py::test_two_view_triangulation_recovers_corner`
|
||||
— two-view stage-corner annotation lands **0.00000 units** from ground truth (< 0.2). Verified
|
||||
live in-browser too: two-view annotation triangulated with gap 0.21 through the full
|
||||
frontend→API→geometry path; correction changed pyro→confetti with source=user and recolored
|
||||
the marker.
|
||||
|
||||
### M9 — keyframed camera paths
|
||||
- New `camPath.js`: **＋ Key**/`K` captures the free-roam pose at the current t_global; gizmos +
|
||||
Catmull-Rom preview line; **▶ Path** follows position (Catmull-Rom) + orientation (slerp) by
|
||||
t_global while the transport plays; ⤓/⤒ export/import JSON. `scene3d` gained
|
||||
`enterPathMode`/`exitPathMode`/`applyCameraPose`; snap-to-camera and Free-roam/`Esc` cancel a
|
||||
playing path.
|
||||
- **Verified live:** 3 keyframes → path mode on → at t=6.0 s the viewer camera sat exactly on
|
||||
keyframe #2 `[-10,6,12]` (Catmull-Rom driven).
|
||||
- Polish present from lanes + this phase: loading state, per-video enable/disable, keyboard
|
||||
(`Space`,`←/→`,`Shift`,`1–9`,`Esc/0`,`K`).
|
||||
|
||||
### Full-system pass
|
||||
- **Backend: 101 passed** (`../.venv/bin/python -m pytest`) — up from the 96 merge-day floor
|
||||
(+5: 4 resolve + 1 event-patch; annotation test renamed).
|
||||
- **Frontend: `npm run build` clean** (18 modules).
|
||||
- End-to-end on the synthetic fixture: `synthetic → ingest → sync → events → serve` + frontend
|
||||
loads with 0 console errors; M4 sync error ~16 ms at t=8 s (deterministic pump; half-frame
|
||||
residual, < 50 ms).
|
||||
- Cleanup: dropped the shared-worktree debris stash; removed `../festifun-laneA` and
|
||||
`../festifun-laneB` worktrees; `frontend/dist/` gitignored; pristine demo DB regenerated.
|
||||
|
||||
**Blockers / notes for coordinator:**
|
||||
- **Real-focused-browser M4 check is the one item I could not fully close from here.** The
|
||||
automated browser pane runs `document.hidden=true`, so rAF is throttled and continuous
|
||||
playback can't be observed (same limitation lane C hit). I verified sync deterministically
|
||||
(pump → ~16 ms) and every interaction via real handlers, but a human should open the app in a
|
||||
normal focused window once to eyeball continuous multi-cam sync. Not code-blocking.
|
||||
|
||||
**Next:** merge `integration` → `main`; tag `v0.1.0`.
|
||||
@ -1,49 +0,0 @@
|
||||
# Status — integration2
|
||||
|
||||
## Round 1 — 2026-07-17 — STATUS: merged (synthetic pass complete; field test GATED on real clips)
|
||||
|
||||
**Directives acknowledged:** Round 4. Single-agent phase 5c, main working directory on
|
||||
branch `integration2`. Coordinator and integration agent are the same session this round.
|
||||
|
||||
**Merge state:** foundation2, lane/g-capsule, lane/f-social, lane/e-director2 all merged to
|
||||
`main` in completion order; every merge independently re-verified (suite re-run in the lane
|
||||
worktree before merging, then on the merged tree). CR-4 and CR-5 ratified at merge (CR-1
|
||||
precedent; decisions recorded in plan/CHANGE_REQUESTS.md). Suite: 127 → 169 → **189 passed**;
|
||||
`npm run build` clean at every step.
|
||||
|
||||
**Full-system synthetic pass — evidence per milestone:**
|
||||
- **M10 (beats):** `ingest → features` on the repo project: 29 beats, tempo **120.0 BPM**
|
||||
(ground truth 120), beat grid at 0.5 s spacing (~+20 ms bias, well inside ±50 ms), 40
|
||||
onsets → `data/work/beats.json` matches contract #1.
|
||||
- **M11 (director):** CLI `direct --top-n 4` emits frozen-shape camPath JSON, keyframe times
|
||||
ON the beat grid (3.02, 4.02, …). Live browser: 🎬 visible, click → **10 keyframes loaded,
|
||||
all beat-snapped (≤11 ms), path auto-plays** (⏹ Path state).
|
||||
- **M12 (export):** live browser via the `__f4d` pump at 50 Hz (hidden pane, rAF=0 — pitfall
|
||||
#2 workaround; WebAudio grants the timer exemption): full-length export auto-stopped at the
|
||||
last keyframe → `video/webm;codecs=vp9,opus`, **duration 13.98 s vs 14.0 expected (±5% ✓)**,
|
||||
audio track present and decoding (webkitAudioDecodedByteCount 28 770 after 1.5 s), state
|
||||
restored, ⏺ re-armed. NOT verifiable headless: rendered-frame quality (needs a focused
|
||||
window — flagged for the human).
|
||||
- **M13–M15 (lane F):** lane self-verified live in isolated servers (see plan/status/lane-F.md
|
||||
— full create→rename→recolor→jump→delete roundtrip, 3840×3093 helper-free PNG, both FX
|
||||
types, 0.101 ms/frame worst case). Coordinator re-check on merged main: all modules
|
||||
ready=true, buttons visible, anchor panel lists 4 anchors with tag/jump/edit/delete, FX
|
||||
fires on a scrub crossing of bass_drop, **zero console errors**.
|
||||
- **M16 (paths):** live: save (via module's own prompt flow) → dropdown refreshes → load
|
||||
reproduces keyframes **exactly** (deep-equal on toJSON) → delete refreshes. 30 lane tests.
|
||||
- **M17 (capsule):** baked from the real dist (19 files, 6.8 MB), served by bundle
|
||||
`serve.py`: app boots in the browser with **zero off-origin requests**
|
||||
(performance.getEntriesByType — nothing to :8000), `capsule: true`, all `.write-ui` hidden,
|
||||
**video seek to t=10 exact** (Range/206 over serve.py), `/api/audio/1` → 200 (WebAudio
|
||||
clock live), 4 anchors + 7 events + point cloud + timeline all present. Screenshot on file.
|
||||
|
||||
**Housekeeping:** `.gitignore` now covers `data/capsule/` and `.claude/worktrees/`
|
||||
(lane G's note).
|
||||
|
||||
**Field test:** NOT RUN — gated on 2–4 real overlapping clips in `data/raw/` (the standing
|
||||
human item since Round 2). Per the adjusted plan (DIRECTIVES Round 5) the **v0.2.0 tag waits**
|
||||
for the field test; the synthetic integration pass is complete and accepted.
|
||||
|
||||
**Blockers:** none for code. Human items listed in DIRECTIVES Round 5.
|
||||
|
||||
**Next:** field test on real footage when clips land → ISSUES.md entries → v0.2.0.
|
||||
@ -1,24 +0,0 @@
|
||||
# Status — lane-A
|
||||
|
||||
## Round 0 — 2026-07-16 — STATUS: ready_to_merge (committed on isolated worktree)
|
||||
|
||||
**Directives acknowledged:** round 0 of plan/DIRECTIVES.md (run order & protocol; Lane A = spec M1).
|
||||
|
||||
**Acceptance checklist** (spec M1 + lane brief) — ALL PASSING:
|
||||
- [x] `ingest`: ffprobe each file in `data/raw`, register in `videos`, extract mono 16 kHz WAV into `data/work/audio/`.
|
||||
- [x] `sync`: PHAT-whitened GCC-PHAT pairwise offsets, scored by peak-to-second-peak ratio; sub-sample parabolic peak refinement.
|
||||
- [x] Global confidence-weighted least-squares solve; cycle-consistency rejection (>50 ms residual).
|
||||
- [x] Drift: windowed GCC-PHAT + line fit → `drift_ppm`; store 0 if `|drift| < 5 ppm`.
|
||||
- [x] Disconnected components → `offset_ms = None` (spec pitfall #5).
|
||||
- [x] Persist via `db.update_video_sync`; export `config.SYNC_JSON`.
|
||||
- [x] `synthetic → ingest → sync` recovers GT offsets (err < 0.001 ms) and drift 0 ppm — inside ±10 ms / ±3 ppm.
|
||||
- [x] `pytest` green in isolation: **43 passed** (24 foundation + 19 new) on pristine foundation + Lane A only.
|
||||
|
||||
**Done this round:** implemented `ingest.py` + `audio_sync.py`; wrote `test_ingest.py` (6) + `test_audio_sync.py` (13). Committed as `157c783` on `lane/a-media`, based directly on `main`/foundation (`5fa7301`) — exactly the 5 owned files, no other lane's changes. Evidence (run in the isolated `festifun-laneA` worktree):
|
||||
- `pytest` → `43 passed, 4 warnings` (warnings are librosa/audioread deprecations only).
|
||||
- `synthetic → ingest → sync` → DB cam0/1/2 = +0.000 / +1370.000 / −842.000 ms, drift 0, err ≤ 0.0002 ms vs `ground_truth.json`.
|
||||
- `sync` exported `data/work/sync.json` (reference id, per-video offset/drift/confidence, raw edges, components).
|
||||
|
||||
**Coordination note (was a blocker; resolved for Lane A):** Lanes A/C/D were sharing one working directory (`/Users/m3ultra/Documents/festifun`) and colliding — HEAD there was moved to `lane/d-events`, and the shared tree held four lanes' uncommitted changes intermixed. Per coordinator sign-off, Lane A was landed into its own `git worktree` (`festifun-laneA`, branch `lane/a-media` reset to `main`) and committed there, disturbing no other worktree. **Still open for coordinator:** lanes C and D also need their own worktrees (lane B already has one); the shared dir still contains their intermixed uncommitted work.
|
||||
|
||||
**Next:** awaiting coordinator go-ahead to `git push origin lane/a-media` and merge to `main` (merge is coordination-sensitive given the collision; branch is ready).
|
||||
@ -1,59 +0,0 @@
|
||||
# Status — lane-B
|
||||
|
||||
## Round 0 — 2026-07-16 — STATUS: ready_to_merge
|
||||
|
||||
**Directives acknowledged:** round 0 of plan/DIRECTIVES.md (run order C→A→D→B; merge when
|
||||
acceptance passes, don't wait on siblings; maintain this status file; evidence discipline).
|
||||
|
||||
**Environment:** COLMAP 4.1.0 (CPU-only) + ffmpeg both installed, so the pipeline was
|
||||
exercised against **real COLMAP end-to-end**, not just the synthetic export path.
|
||||
|
||||
**Isolation note (see blocker):** worked in a dedicated git worktree
|
||||
(`/Users/m3ultra/Documents/festifun-laneB` on `lane/b-recon`) because all lane agents were
|
||||
sharing one working directory. Only the 6 lane-B-owned files were touched.
|
||||
|
||||
**Acceptance checklist** (spec M2 + M8 geometry + lane brief) — all pass:
|
||||
- [x] `geometry.py` M8 stubs implemented: `slerp_pose` (shortest-arc, double-cover), `ray_from_pixel`
|
||||
(COLMAP +y-down back-projection), `triangulate_rays` (closest-point + parallel guard),
|
||||
`nearest_point_on_ray` (in-front radius cylinder). Frozen `colmap_to_threejs`/`quat_to_mat`/
|
||||
`mat_to_quat` and the 3 frozen POSE_TEST_VECTORS untouched (`git diff 5fa7301` shows 0 removed
|
||||
lines in test_geometry.py; the frozen funcs are unchanged).
|
||||
- [x] `frames.py`: `sharpness` (variance of Laplacian) + windowed sharpest-frame `sample_frames`.
|
||||
- [x] COLMAP TXT parsers (images/cameras/points3D) with hand-written-snippet tests + a truncated-PARAMS
|
||||
guard (clean ValueError).
|
||||
- [x] `normalize_scene`: centroid≈0, camera-sphere radius≈10, up≈+Y — verified by `test_normalize_scene_invariants`
|
||||
and a projection-invariance test (`test_normalize_scene_preserves_projection`).
|
||||
- [x] pose interpolation (slerp+lerp, `registered=False`, NO extrapolation past first/last registered) —
|
||||
both guard directions tested.
|
||||
- [x] PLY export via frozen `synthetic.write_ply`; poses written via atomic `db.set_poses`.
|
||||
- [x] graceful degradation: COLMAP absent / <60% frames / <2 videos / no frames / missing files →
|
||||
diagnostic, existing poses UNTOUCHED, DB uncorrupted. Mixed-subset case (some videos registered,
|
||||
others kept) tested for the DB-safety invariant.
|
||||
- [x] real end-to-end: `synthetic` → `reconstruct` ran the full COLMAP CLI (feature_extractor →
|
||||
exhaustive_matcher → mapper → image_undistorter → model_converter → my parsers), produced 6
|
||||
components (2/15/11/16/17/37 imgs), correctly selected the largest by `images.bin` header count,
|
||||
judged 37/120 (31%) weak, degraded gracefully, **left all 41 synthetic poses/video untouched, exit 0.**
|
||||
- [x] `pytest` green — **61 passed** (24 foundation + 37 lane-B; test_geometry.py + test_sfm.py).
|
||||
- [x] no edits outside owned files (`git diff 5fa7301 --name-only` = frames/geometry/sfm/test_geometry only;
|
||||
untracked: test_sfm.py, this status file); no new deps.
|
||||
|
||||
**Bugs found & fixed while validating on real COLMAP:**
|
||||
- COLMAP 4.x renamed `SiftExtraction`/`SiftMatching` → `FeatureExtraction`/`FeatureMatching`; option
|
||||
detected from `--help` (which COLMAP prints to **stderr**) so the pipeline runs on 3.x and 4.x.
|
||||
- `_largest_model_dir` ranked binary models by file size (tracks keypoints, not image count) → now reads
|
||||
the exact registered-image count from the `images.bin` uint64 header (validated on the real 6-component run).
|
||||
- workspace cleared before each run so a stale `database.db` can't fail reruns.
|
||||
|
||||
**Adversarial review:** 5-lens review + verify pass (23 agents) → 18 raw findings, 5 CONFIRMED (2 code, 3
|
||||
test-coverage), all applied; 13 refuted (incl. triangulate/nearest-point "behind origin" — correct-as-written).
|
||||
|
||||
**Blockers / questions for coordinator:**
|
||||
- **Shared working tree collision.** All four lane agents are operating in the *same* working directory
|
||||
(`/Users/m3ultra/Documents/festifun`), not separate clones/worktrees as `plan/README.md` requires.
|
||||
Mid-session the shared branch was switched to `lane/d-events` and the tree accumulated uncommitted edits
|
||||
from lanes A/B/C/D at once. I isolated lane B in its own worktree so nothing is clobbered. **Recommend the
|
||||
coordinator have each lane use `git worktree`/separate clones, and adjudicate merges centrally**, since a
|
||||
naive `git branch -f main` from any lane could silently drop a sibling's merge. I did NOT auto-merge to
|
||||
`main` for this reason — lane B is ready and awaiting a safe merge.
|
||||
|
||||
**Next:** commit lane B to `lane/b-recon`; coordinator to merge to `main` (safely, given the shared-tree issue).
|
||||
@ -1,53 +1,37 @@
|
||||
# Status — lane-C
|
||||
|
||||
## Round 1 — 2026-07-16 — STATUS: ready_to_merge
|
||||
## Round 1 — 2026-07-16 — STATUS: in_progress
|
||||
|
||||
**Directives acknowledged:** round 0 of plan/DIRECTIVES.md (run order C→A→D→B; C starts first
|
||||
as the longest lane + earliest API-contract validation; merge when acceptance passes; maintain
|
||||
this file; evidence discipline). No new rounds present on origin at time of writing.
|
||||
**Directives acknowledged:** round 0 of plan/DIRECTIVES.md (run order C→A→D→B; C starts
|
||||
first as the longest lane + earliest API-contract validation; merge when acceptance passes,
|
||||
don't wait on siblings; maintain this status file; evidence discipline — a checked box names
|
||||
the test/command+output that proves it).
|
||||
|
||||
**Acceptance checklist (lane brief M4/M5/M6 + timeline markers) — all pass:**
|
||||
- [x] **M4 synchronized playback.** Master clock from `performance.now()` (never a `<video>`).
|
||||
Timebase mapping proven exact by seek: at t_global=8.0 → cam0=8.000, cam1=6.630,
|
||||
cam2=8.842 s, err=0 ms each. Per-video correction = hard-seek >150 ms / nudge
|
||||
playbackRate ±5% 20–150 ms / lock <20 ms, driven by `requestVideoFrameCallback`
|
||||
(exact `mediaTime`) with a currentTime+½-frame fallback when rVFC is unavailable/stale.
|
||||
Out-of-range videos pause+dim (cam1 at t=0, target −1.37 s). One audio source; dev
|
||||
overlay shows per-video sync ms. Measured continuous-playback **inter-video desync =
|
||||
13 ms mean / 46 ms max** over 6 s (< 50 ms), driven by a setTimeout pump (see env note).
|
||||
- [x] **M5 3D viewer.** PLY point cloud (3106 pts, vertex colors); per-video camera-path lines
|
||||
+ current-pose frusta (colored); OrbitControls; snap-to-camera tweens viewer to the pose
|
||||
(following cam1: camPos≈(1.78,2.63,9.25)≈true center (0,2.4,9.4); fov=36°≈2·atan(H/2fy);
|
||||
orbit disabled) and free-roam restores (fov→50°, orbit on). All COLMAP→Three via frozen
|
||||
`lib/pose.js` — never reimplemented.
|
||||
- [x] **M6 anchor overlays.** Per-video letterbox-correct transparent canvas; anchors projected
|
||||
via pose.js camera. Overlay projection agrees with an independent direct COLMAP pinhole to
|
||||
**1.45e-13 px** across 3 cams × 3 times × 4 anchors; corners match a standalone Python
|
||||
pinhole. Behind-camera cull (z_cam<0). Stage corners render/track on all in-range videos.
|
||||
- [x] **Timeline event markers.** 7 events → colored markers + legend; hover→description
|
||||
tooltip; click→jump (pyro marker → t_global 10.00). Scrubber seeks/parks all videos.
|
||||
- [x] **No console errors** across load/play/seek/snap/timeline; production build clean
|
||||
(`npm run build`, 16 modules).
|
||||
**Acceptance checklist** (lane brief M4/M5/M6 + timeline markers):
|
||||
- [ ] M4 — synchronized playback: master clock; per-video correction (hard-seek >150ms,
|
||||
nudge playbackRate 20–150ms, lock <20ms); out-of-range videos pause+dim; one audio
|
||||
source; dev overlay per-video sync error stays <50ms during continuous playback.
|
||||
- [ ] M5 — 3D viewer: PLY point cloud; per-video camera-path lines + current-pose frusta
|
||||
(colored); OrbitControls; snap-to-camera (intrinsics→PerspectiveCamera) matches the
|
||||
video; free-roam detaches. All COLMAP→Three via frozen `lib/pose.js`.
|
||||
- [ ] M6 — anchor overlays: per-video transparent canvas, letterbox-correct; project each
|
||||
anchor via pose.js camera; skip if behind (z_cam<0); corners track the stage box in
|
||||
all 3 videos while playing.
|
||||
- [ ] Timeline event markers: `GET /api/events` as colored markers (color by event_type,
|
||||
legend); hover tooltip; click-to-jump.
|
||||
- [ ] App runs against synthetic data with **no console errors**; merged to `main`.
|
||||
|
||||
**Env note (not a blocker):** the automated browser pane reports `document.hidden=true`, so rAF
|
||||
and rVFC only fire during screenshots — continuous playback can't be observed by polling here.
|
||||
Verified deterministically instead (seek exactness to 0 ms; projection to 1e-13 px; a
|
||||
setTimeout-driven run of the real correction code → 13 ms mean inter-video desync). A real
|
||||
focused browser runs the rAF loop at 60 fps and rVFC drives even tighter absolute sync.
|
||||
**Done this round (session start):** verified environment and the frozen contracts against the
|
||||
live server (`python -m festival4d serve`): manifest (3 videos, offsets 0/+1370/−842 ms,
|
||||
t_global_max 21.37), 41 registered poses/video keyed by frame_idx (centered intrinsics
|
||||
fx=fy=554.26, cx=320, cy=180), 4 stage-corner anchors, 7 typed events, Range→206, colored
|
||||
PLY (3106 pts). Independently verified stage-corner projection lands all 4 corners in-frame,
|
||||
in front of the camera, for all 3 videos (cam0 left, cam1 front-center, cam2 right; centers on
|
||||
an arc at radius ~9.5) — so M5/M6 geometry is sound before writing render code.
|
||||
|
||||
**Owned-files discipline:** committed only `frontend/**` + this status file. Left untouched the
|
||||
other lanes' in-tree changes (A: audio_sync/ingest; B: frames/geometry/sfm; D: events_ai).
|
||||
Added `frontend/src/lib/timebase.js` + `poseTrack.js` (new files inside lane C's ownership).
|
||||
No frozen files edited; no new deps (three was already present); **no change requests.**
|
||||
**Blockers / questions for coordinator:** none. No change requests — the frozen API/pose/
|
||||
timebase contracts are sufficient for the whole lane as-is.
|
||||
|
||||
**Repo note for coordinator:** this working directory is shared/contended — `lane/c-viewer`
|
||||
was reset to foundation (5fa7301) and HEAD moved to `lane/d-events` between my session-start
|
||||
commit and now, and `main` has since advanced to lane D's self-merge (6ec55c7). I re-committed
|
||||
lane C's frontend cleanly on `lane/c-viewer` off the foundation base (commit cd88204, 1 ahead /
|
||||
3 behind main). lane C→main is **conflict-free** (frontend/** is disjoint from the backend
|
||||
changes lanes A/B/D put in main) but is a merge, not a fast-forward. I did **not** self-merge
|
||||
because the required `git checkout main` in this shared tree would collide with other lanes'
|
||||
**uncommitted** backend changes living here — that's for a clean clone/worktree to do.
|
||||
|
||||
**Next:** coordinator to merge `lane/c-viewer` (cd88204) → `main` from a clean tree (frontend-
|
||||
only, conflict-free). Phase-3 seams left as clean stubs: `annotate.js` (M8), `camPath.js` (M9).
|
||||
**Next:** build in dependency order — state → transport (verify M4 + dev sync overlay first,
|
||||
per brief) → videoGrid → timeline → scene3d → overlays; verify live in-browser; adversarial
|
||||
review pass; merge.
|
||||
|
||||
@ -1,66 +0,0 @@
|
||||
# Status — lane-E (director & export, phase 5b — M10/M11/M12)
|
||||
|
||||
## Round 1 — 2026-07-17 — STATUS: ready_to_merge
|
||||
|
||||
**Directives acknowledged:** Round 4 of plan/DIRECTIVES.md (phase 5 begins; foundation2 merged;
|
||||
suite floor 121 — was 127 at lane start after foundation2's tests).
|
||||
|
||||
**Branch:** `lane/e-director2` (own worktree per Round-4 rule). Commits: `0f8685b` M10,
|
||||
`52e8912` M11, `ddb8dea` M12. Not merged, not pushed — coordinator merges.
|
||||
|
||||
**Acceptance checklist**
|
||||
- [x] **M10** beats/onsets: synthetic 120 BPM pulse grid recovered — tempo **120.000** (±2 req),
|
||||
worst beat **30 ms** off grid (±50 req) — `test_run_features_recovers_synthetic_pulse_grid`;
|
||||
the 3 ground-truth bangs found as strong onsets — `test_onsets_catch_the_ground_truth_bangs`;
|
||||
silence/short → valid-empty JSON, missing WAV → note + nothing written, beatless noise →
|
||||
valid typed result, CLI end-to-end — 7 tests in `test_audio_features.py`. Key impl detail:
|
||||
10 ms analysis hop (librosa's default 32 ms grid at 16 kHz misses the ±50 ms bound; measured
|
||||
60 ms → 30 ms), tempo from median inter-beat interval (117.19 → 120.000).
|
||||
- [x] **M11** auto-director: structure-locked against the frozen camPath shape (exact key sets,
|
||||
unit quats, monotone times) — `test_output_matches_frozen_campath_shape`; top-3 grid ==
|
||||
`[1,4,8,11,14]` incl. shared-keyframe dedupe; **quat order [x,y,z,w] locked** via
|
||||
round-trip against frozen `geometry.colmap_to_threejs` + `quat_to_mat`
|
||||
(`test_keyframe_pose_matches_frozen_conversion`); beat snap puts every key on the grid, no
|
||||
beats.json / corrupt beats.json → unsnapped (no block); no events / no poses → valid empty
|
||||
path + note; confidence ties → earlier; lead clamps at 0; unregistered poses excluded; API
|
||||
route + CLI dispatch real output — 14 tests in `test_director.py`. `director.js` wires 🎬:
|
||||
POST → `camPath.fromJSON` → auto-play via the existing #btn-path flow; empty path shows
|
||||
"no events yet" on the button.
|
||||
- [x] **M12** export (code-complete; needs live browser, below): canvas captureStream(30) +
|
||||
`transport.captureAudioStream()` tap → MediaRecorder vp9/opus webm (vp8/webm fallbacks),
|
||||
first→last keyframe at forced rate 1, downloads `festival4d-cut.webm`, restores playhead/
|
||||
rate/play/path/follow state, releases tap + tracks. Button disabled < 2 keyframes (chained
|
||||
onto main.js's `camPath.onChange` — no frozen edits).
|
||||
- [x] Suite on the merged tree (main incl. lane G merged into this branch):
|
||||
`uv run pytest backend/tests -p no:warnings` → **189 passed, 0 failed** (169 on main after
|
||||
lane G + 21 lane-E − 1 retired stub test). Pre-merge lane-only figure was 148 (127 + 21).
|
||||
`cd frontend && npm run build` → clean (pre-existing chunk-size warning only).
|
||||
|
||||
**Change request filed:** **CR-5** (plan/CHANGE_REQUESTS.md) — two stub-era assertions in
|
||||
`test_phase5_api.py` asserted lane E's *unimplemented* behavior (CLI exit 2; empty director
|
||||
path) and fail by design once M10/M11 land. Applied the minimal test edit directly per the
|
||||
**CR-1 precedent** — and, mid-lane, main moved under me with lane G's **coordinator-ratified
|
||||
CR-4** doing the identical thing for `capsule`, which confirms the pattern. Merged main into
|
||||
`lane/e-director2` and reconciled: the CLI stub loop is now empty (all three subcommands
|
||||
real) so `test_cli_dispatches_stubs_gracefully` is retired with a pointer comment; the
|
||||
director test is renamed `test_director_returns_valid_campath` asserting the frozen camPath
|
||||
shape. api.py/cli.py byte-for-byte untouched. If the coordinator prefers the stricter
|
||||
reading, revert `backend/tests/test_phase5_api.py` and re-apply at integration2; the suite
|
||||
will then show those stub-era failures until adjudication.
|
||||
|
||||
**Coordinator must verify live in a FOCUSED browser** (this session had no browser; rAF/
|
||||
captureStream need focus — pitfall #2):
|
||||
1. M11: `python -m festival4d synthetic && ingest && features && serve` + vite dev → 🎬 visible;
|
||||
click → path loads (＋ Key counter jumps), camera flies the cut, #btn-path shows ⏹.
|
||||
With `data/work/beats.json` deleted, 🎬 still works (unsnapped). With events cleared,
|
||||
button flashes "🎬 no events yet", no console error.
|
||||
2. M12: with a path loaded, ⏺ → records the ride, auto-stops at the last keyframe, downloads
|
||||
`festival4d-cut.webm`; `ffprobe` it: duration within ±5% of the path span (top-8 synthetic:
|
||||
1.0→20.0 s ⇒ 19 s ±0.95), has an opus audio track; plays in Chrome. Then confirm prior
|
||||
state restored (playhead back, rate back, path button reset). Clicking ⏺ mid-ride stops
|
||||
early and still yields a valid file. Note: hidden/backgrounded pane records frozen frames.
|
||||
3. M12 + 🎧: enable spatial audio before ⏺ — exported track carries the spatial mix.
|
||||
|
||||
**Blockers / questions:** none beyond CR-4 adjudication.
|
||||
|
||||
**Next:** coordinator merge + live checks above; integration2 owns the real-footage pass.
|
||||
@ -1,34 +0,0 @@
|
||||
# Status — lane-F (social & fun UX: M13 anchor manager + friend tags, M14 photo mode, M15 moment FX)
|
||||
|
||||
## Round 1 — 2026-07-17 — STATUS: ready_to_merge
|
||||
|
||||
**Directives acknowledged:** Round 4 of plan/DIRECTIVES.md (phase 5b begins; foundation2 merged; worktree rule followed — this lane ran in its own worktree on branch `lane/f-social`).
|
||||
|
||||
**Ownership kept:** only `frontend/src/anchorPanel.js`, `frontend/src/photoMode.js`, `frontend/src/fx.js`, and this file were touched. No frozen file edited; no CHANGE_REQUESTS entry needed.
|
||||
|
||||
**Friend-tag design choice (M13):** the ＋ Tag flow DRIVES the frozen M8 annotator without an event — no workaround needed. Rationale: `annotate.js` is null-event safe by construction (`startAnnotateMode()` only needs the registered cells; `_submit` POSTs `event_id: ev ? ev.id : null`; every panel-content access is optional-chained), and the backend explicitly supports it (`api.py`: "Annotations with no event stay independent (each makes its own anchor)"). Flow: prompt for the friend's name → `annotator.close()` (clears any currentEvent) → `annotator.startAnnotateMode()` → user drags a bbox over the friend on ANY video → the resolved anchor arrives via `emit("anchors-changed", anchor)` → anchorPanel PATCHes the pending name + a palette color onto it and stops annotate mode. Esc or ＋ Tag again cancels. This reuses the real M8 ray-cast/nearest-point resolution (friends land ON the point cloud), keeps annotate.js untouched, and is one drag for the user.
|
||||
|
||||
**Acceptance checklist — ALL verified live** (worktree servers: backend :8010 with an isolated copy of the synthetic data — the main project DB was never touched — + Vite :5175 with `VITE_API_BASE`; browser = Claude Code Browser pane, loop driven with the `__f4d` pump per pitfall #2 where noted):
|
||||
- [x] M13 create → rename → recolor → jump → delete roundtrip live: ＋ Tag with prompt "Zoe" + real 35×30 px drag on cam0 → anchor id 5 `{label:"Zoe", color:"#7dffb0"}` created via the frozen annotator; ✎ rename → `"Zoe B"`; color input change → `#ff00aa`; label-click jump → `scene3d.controls.target == (1.60, 0.00, -6.50) ==` anchor pos exactly, follow-cam detached; ✕ delete → state.anchors 5→4, row removed.
|
||||
- [x] M13 labels visible in 3D + video overlays: screenshot shows the pink "Zoe B" sphere+label centered in the 3D scene after jump AND "Zoe B" x-ray markers on cam0 + cam2 overlays.
|
||||
- [x] M13 anchors survive reload: full page reload → anchor id 5 still `{label:"Zoe B", color:"#ff00aa"}` from GET /api/anchors.
|
||||
- [x] M13 capsule mode: with `body.capsule`, tag/rename/delete/recolor controls hidden (`offsetParent === null`), panel + list + jump-to remain visible.
|
||||
- [x] M14 PNG ≥3840 wide, aspect preserved, no helpers: intercepted the real toBlob → PNG 3840×3093 px (351 KB) from a 704×567 pane (aspect 1.2416 preserved); rendered the captured blob full-screen and screenshotted it: point cloud + anchor SPHERES only (spheres stay by scene3d design — "friend tags belong in photos"), zero grid/axes/frusta/path-lines/label-sprites.
|
||||
- [x] M14 restore: pixelRatio 1→1, size 704×567, camera.aspect 1.2416, helpersVisible true all restored; shot MID-PLAY → transport paused for the shot and `state.playing === true` after (play-state restored). Works in free-roam; follow-cam uses the same render path (aspect saved/restored around the shot).
|
||||
- [x] M15 pyro fires on PLAY crossing: seek 9.5 → play → burst spawned exactly at t=10.00, opacity 1→0.01 over its 1.0 s life across 53 pumped frames, then removed from the scene (`allRemoved: true`).
|
||||
- [x] M15 confetti fires on SCRUB crossing: paused discrete seeks 16.4→16.6→16.8→16.95→17.1 → burst fired at the 17.1 step (crossing 17.0). Backward jump to 15.0 fired nothing (tracker reset).
|
||||
- [x] M15 bass_drop pulse: play across t=3 → `scene3d.points.scale` swelled to 1.1599 and restored to exactly 1.0 after ~one beat (0.5 s default; tempo from GET /api/beats when it exists).
|
||||
- [x] M15 performance: worst case (all 6 burst slots active + pulse) full frame body (`transport.tick` + `fx.update` + `scene3d.update`) = **0.101 ms/frame CPU** over 300 iterations — ~330× inside the 33 ms/30 fps budget. Pool preallocated once; `update()` allocates nothing; toggle-off removes all objects and restores scale (verified: 0 active, 0 in scene, scale 1).
|
||||
- [x] fx.update can never kill the master loop: whole body in try/catch → on error logs once, disables itself, cleans up (breaker `_dead` verified false throughout).
|
||||
- [x] Zero console errors across the entire live session.
|
||||
- [x] Regression: `uv run pytest backend/tests -p no:warnings` → **127 passed** (floor met; backend untouched). `cd frontend && npm run build` clean (pre-existing >500 kB chunk warning only).
|
||||
|
||||
**Notes / deferred to coordinator (integration2):**
|
||||
- M14 splat path: the DropInViewer is a scene child and renders with the same `renderer.render(scene, camera)` call, so photo mode has no splat branch — but live verification ON A TRAINED SPLAT is deferred (synthetic project has point cloud only).
|
||||
- M15 event→anchor mapping: events carry no anchor id in state, so fx maps event→anchor by the backend's labeling convention (resolved anchor label = `description or event_type`), falling back to the stage centroid (0, 0.5, 0). Both pyro/confetti fired at the centroid in this test (no event-resolved anchors existed); please verify a burst lands ON an anchor after annotating a pyro/confetti event.
|
||||
- M12 interaction: fx pulse writes `points/splat.scale` between fx.update and scene3d.update — same-frame, restored exactly; no interaction with export expected, but a combined FX+export pass is worth one look.
|
||||
- Real-focused-browser eyeball (bursts/pulse at full rAF rate) — the pane verifications above used the pump; a human glance in a focused window is the usual final check.
|
||||
|
||||
**Blockers:** none.
|
||||
|
||||
**Next:** merge `lane/f-social` → `main` (coordinator/integration2 discretion).
|
||||
@ -1,101 +0,0 @@
|
||||
# Status — lane G (persistence & capsule, branch `lane/g-capsule`)
|
||||
|
||||
## Round 1 — 2026-07-17 — STATUS: ready_to_merge
|
||||
|
||||
**Directives acknowledged:** Round 4 of plan/DIRECTIVES.md (phase 5 begins; foundation2 merged;
|
||||
worktree rule followed — this lane runs in its own git worktree).
|
||||
|
||||
**Acceptance checklist**
|
||||
|
||||
M16 — server-side camera paths:
|
||||
- [x] Routes implemented — by foundation2 (per the phase-5 ownership table); lane G deepened
|
||||
coverage only, per scope.
|
||||
- [x] `backend/tests/test_paths.py` — 30 tests: roundtrip deep-equal incl. float precision +
|
||||
a 500-keyframe path, multiple paths coexist / id-ordered listing / summary-only shape,
|
||||
duplicate + unicode + whitespace + 300-char names verbatim, ISO `created_at`, empty-keyframes
|
||||
path valid (director degrade must be saveable), extra JSON keys ride along, get/delete 404s,
|
||||
delete idempotence, an 18-case parametrized 422 matrix over the frozen camPath contract,
|
||||
422 creates no row, missing/aliased request fields 422. Cleans up its rows (test_phase5_api
|
||||
runs later and asserts an empty table).
|
||||
- [x] `frontend/src/pathsStore.js` filled: 💾 save = name prompt + POST `camPath.toJSON()`;
|
||||
dropdown load = GET `/api/paths/{id}` + `camPath.fromJSON(data.json)`; 🗑 delete with
|
||||
confirm; dropdown refreshed after every mutation (new save left selected); controls
|
||||
unhidden + `ready = true`. Capsule mode: save/del hide via `write-ui`; the load dropdown
|
||||
stays (paths are baked read-only — below) and hides itself if the listing fetch fails.
|
||||
- [ ] Live save → reload page → load reproduces keyframes exactly — NEEDS LIVE BROWSER (list
|
||||
for coordinator below). The exact-roundtrip property is locked server-side by
|
||||
`test_roundtrip_preserves_float_precision` / `test_roundtrip_large_keyframe_list`.
|
||||
|
||||
M17 — memory capsule:
|
||||
- [x] `backend/festival4d/capsule.py::build_capsule` per the frozen contract: copies
|
||||
`frontend/dist` (clear "run `cd frontend && npm run build`" error if missing), copies the
|
||||
manifest-referenced media to `media/`, bakes every read-only API response extension-less at
|
||||
the live API's relative paths (`api/manifest` with `capsule: true` + `has_capture: false`,
|
||||
`api/events`, `api/anchors`, `api/videos/{id}/poses`, `api/beats`/`api/pointcloud`/
|
||||
`api/splat` when present, `api/audio/{id}` via the same lazy `ingest.extract_audio_hq`
|
||||
cache the live route uses), injects `<script>window.__F4D_API_BASE__=""</script>` ahead of
|
||||
the app `<script>`, ships stdlib `serve.py` (HTTP/1.1, 206 Range slicing incl. `bytes=N-`
|
||||
and suffix `bytes=-N`, 416 past-EOF, `Accept-Ranges`, JSON content types for the
|
||||
extension-less `api/*` files). Optional artifacts degrade by omission; refuses to clobber
|
||||
a non-capsule output dir; rebuild over a previous capsule is idempotent (no double-inject).
|
||||
Bonus within scope: M16 saved paths are baked read-only (`api/paths/{id}`, listing at
|
||||
`api/paths/index.html` — a filesystem can't hold `paths` as both file and dir; the stock
|
||||
301 → `/api/paths/` → index.html keeps `fetch("/api/paths")` working unchanged).
|
||||
- [x] `backend/tests/test_capsule.py` — 12 tests: bundle structure; baked JSON deep-equal vs the
|
||||
live route functions (shape drift impossible — capsule bakes *through* `api.py`); manifest
|
||||
key-set == live route, urls resolve inside the bundle; injection present exactly once and
|
||||
before the module script; degrade-when-absent; no-dist error; overwrite guard; CLI dispatch
|
||||
(`cli.main(["capsule","--out",…]) == 0`, hermetic via monkeypatched REPO_ROOT); serve.py
|
||||
exercised over real HTTP in a subprocess — 200 + Accept-Ranges, `bytes=0-100` → 206 with
|
||||
`Content-Range: bytes 0-100/{size}` and 101 body bytes (mp4 `ftyp` magic verified),
|
||||
open-ended + suffix ranges, 416, audio 200 `audio/mp4` + ranged, 404s.
|
||||
- [x] End-to-end proof WITHOUT a browser (evidence below).
|
||||
|
||||
**Quality bar**
|
||||
- `uv run pytest backend/tests -p no:warnings` → **169 passed** (floor 127 + 30 paths + 12
|
||||
capsule), zero failures.
|
||||
- `cd frontend && npm run build` → clean, only the pre-existing >500 kB chunk-size warning.
|
||||
|
||||
**E2E capsule evidence (exact commands, run in the worktree):**
|
||||
```
|
||||
$ cd frontend && npm run build # ✓ built in 860ms
|
||||
$ uv run python -m festival4d synthetic # 3 videos, 3106 points, 7 events, 4 anchors
|
||||
$ uv run python -m festival4d capsule
|
||||
capsule: baked 18 files (6826705 bytes) at <worktree>/data/capsule
|
||||
$ cd data/capsule && python3 serve.py --port 8901
|
||||
$ curl -s http://127.0.0.1:8901/ | grep -o 'window.__F4D_API_BASE__=""'
|
||||
window.__F4D_API_BASE__=""
|
||||
$ curl -si http://127.0.0.1:8901/api/manifest
|
||||
HTTP/1.1 200 OK / Content-Type: application/json
|
||||
capsule: True | videos: 3 | t_global_max: 21.37 | has_poses: True
|
||||
$ curl -si -H "Range: bytes=0-100" http://127.0.0.1:8901/media/cam0.mp4
|
||||
HTTP/1.1 206 Partial Content / Accept-Ranges: bytes / Content-Length: 101
|
||||
Content-Range: bytes 0-100/1533609
|
||||
$ curl -s -H "Range: bytes=500000-500099" .../media/cam0.mp4 # mid-file seek
|
||||
206, received 100B
|
||||
$ curl -s http://127.0.0.1:8901/api/audio/1
|
||||
200, audio/mp4, 440201 bytes, magic `ftypM4A`
|
||||
$ curl -sL http://127.0.0.1:8901/api/paths # [] final:200 application/json
|
||||
$ curl .../api/pointcloud → 200 46800B; .../api/beats → 404 (degrade: no analysis run)
|
||||
```
|
||||
|
||||
**Blockers / change requests:** none blocking. **CR-4 filed** (plan/CHANGE_REQUESTS.md):
|
||||
`test_phase5_api.py::test_cli_dispatches_stubs_gracefully` asserted the *stub* exit-2 contract
|
||||
for `capsule`; with M17 landed that assertion fails by design (CR-1 precedent). Applied the
|
||||
minimal edit (removed only `["capsule"]`; features/direct untouched for lane E) — integration2
|
||||
to ratify. Capsule CLI dispatch is now covered for real in test_capsule.py.
|
||||
|
||||
Small note for integration2 (frozen file, not edited): `.gitignore` covers `data/raw/` +
|
||||
`data/work/` but not `data/capsule/` or `data/project.db`, so a default capsule build leaves a
|
||||
~7 MB untracked tree; consider ignoring `data/` wholesale (or adding the two entries).
|
||||
|
||||
**Needs live-browser verification (for coordinator / integration2):**
|
||||
1. Capsule acceptance proper: `python serve.py` in the bundle, open it focused — app loads with
|
||||
**zero requests to :8000** (network tab), videos play *and seek*, 3D scene + timeline work,
|
||||
🎧 spatial audio works, all `.write-ui` controls absent (body.capsule), saved-path dropdown
|
||||
loads a baked path.
|
||||
2. M16 live flow: save a path (💾, name prompt) → reload page → load from dropdown → camera
|
||||
lands on the exact keyframes; delete removes it from the dropdown.
|
||||
3. pathsStore dropdown behavior with an *older* capsule (no baked paths): dropdown hides itself.
|
||||
|
||||
**Next:** hand to coordinator for merge; not merging to main per lane instructions this round.
|
||||
@ -1,41 +0,0 @@
|
||||
# Status — phase4 (polish + field test)
|
||||
|
||||
## Round 1 — 2026-07-16 — STATUS: polish ready_to_merge; field test AWAITING FOOTAGE
|
||||
|
||||
**Directives acknowledged:** round 3 + addendum of plan/DIRECTIVES.md (anchor supersede + delete,
|
||||
docs/ideas.md, suite ≥101, field-test protocol + ISSUES.md; local `.env` is machine-local).
|
||||
|
||||
### Polish pass — DONE
|
||||
- [x] **Anchor supersede (fixes the Round-3 wart).** One anchor per event: first annotation
|
||||
creates it (single-view fallback); a later view that triangulates **supersedes it in place**
|
||||
(same anchor id, response gains `superseded: true`); a later fallback links without moving.
|
||||
Annotations with no event stay independent.
|
||||
- Evidence: reproduced the old "2 anchors for 1 object" scenario → now **1 anchor**, both
|
||||
annotations link to the same id, superseded position err **0.0000** from ground truth
|
||||
(`tests/test_resolve.py::test_two_view_annotation_supersedes_to_single_anchor`).
|
||||
- [x] **`DELETE /api/anchors/{id}`** (additive) — unlinks annotations (NULLs `resolved_anchor_id`)
|
||||
then deletes; 404 on unknown. `db.get_anchor`/`update_anchor`/`delete_anchor` +
|
||||
`get_annotations(event_id=)` added.
|
||||
- Evidence: `tests/test_resolve.py::test_delete_anchor_unlinks_annotations_and_404`.
|
||||
- [x] **Frontend delete affordance** — the correction panel now has a scrollable **anchors list**
|
||||
(color dot + label + ✕). Delete → `DELETE` → splice `state.anchors` → `anchors-changed`
|
||||
(scene3d refresh; overlays read state each frame). `_submit` upserts by id so a supersede
|
||||
replaces (not duplicates) the anchor in state. `npm run build` clean (18 modules).
|
||||
- [x] **`docs/ideas.md`** created (gaussian splatting, MP4 path export, realtime ingest,
|
||||
multi-modal audio, object tracking, QoL).
|
||||
- [x] **CR-3** filed + APPLIED in plan/CHANGE_REQUESTS.md (strictly additive).
|
||||
- [x] **Suite: 101 → 103 passed** (`../.venv/bin/python -m pytest`), frontend build clean.
|
||||
|
||||
### Field test — AWAITING FOOTAGE (protocol ready)
|
||||
Scaffolding in place: this file + `plan/ISSUES.md` (seeded with ISSUE-1, the gemini-2.5-flash
|
||||
retirement, FIXED). When 2–4 real overlapping clips land in `data/raw/`, run
|
||||
`ingest → sync → reconstruct → events → serve` and record here: per-pair sync confidence +
|
||||
offsets, COLMAP registration rate, and which milestones survived real footage. File failures as
|
||||
`plan/ISSUES.md` entries, don't rush fixes.
|
||||
|
||||
**Blockers / notes for coordinator:**
|
||||
- Field test cannot start until the human provides real footage (carried item).
|
||||
- Human focused-browser M4 sync eyeball still open (carried from Round 2/3).
|
||||
- `.env` (Gemini + OpenRouter creds) is machine-local + gitignored; present on this machine.
|
||||
|
||||
**Next:** merge `phase4/polish-field-test` → `main`; run the field test when footage arrives.
|
||||
@ -14,7 +14,6 @@ dependencies = [
|
||||
"sqlalchemy>=2.0",
|
||||
"pydantic>=2.6",
|
||||
"opencv-python-headless>=4.9",
|
||||
"python-multipart>=0.0.9", # multipart uploads for live capture (backend/capture.py)
|
||||
# classifier providers (lane D) — imported lazily, never at module load
|
||||
"google-genai>=0.3",
|
||||
"anthropic>=0.40",
|
||||
|
||||
@ -1,40 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Train a 3DGS splat of this project's scene on the MODELBEAST fleet, and drop
|
||||
# it where the viewer looks for it (data/work/splat.ply).
|
||||
#
|
||||
# Uses the sampled SfM frames (data/work/frames) as input: uploads them to
|
||||
# MODELBEAST, runs colmap_poses + brush_train on the render farm's queue, and
|
||||
# downloads the trained splat back. Requires the `mb` CLI env:
|
||||
# export MB_HOST=http://100.89.131.57:8777 # M3 primary
|
||||
# export MB_TOKEN=... # from MODELBEAST Settings → Users
|
||||
#
|
||||
# (v1 re-runs COLMAP on the farm rather than shipping our local reconstruction —
|
||||
# simpler and uses the farm's known-good chain; direct dataset handoff is a
|
||||
# future optimization. See docs/modelbeast-crossover.md.)
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
MB=${MB:-/Users/m3ultra/Documents/MODELBEAST/mb}
|
||||
FRAMES_DIR=${FESTIVAL4D_DATA_DIR:-data}/work/frames
|
||||
OUT=${FESTIVAL4D_DATA_DIR:-data}/work
|
||||
|
||||
[ -d "$FRAMES_DIR" ] || { echo "no frames at $FRAMES_DIR — run the reconstruct step first"; exit 1; }
|
||||
command -v "$MB" >/dev/null || { echo "mb CLI not found (set MB=/path/to/mb)"; exit 1; }
|
||||
|
||||
echo "[splat] uploading frames..."
|
||||
mapfile -t frames < <(find "$FRAMES_DIR" -name '*.jpg' -o -name '*.png' | sort)
|
||||
echo "[splat] ${#frames[@]} frames"
|
||||
ids=$("$MB" upload "${frames[@]}" | awk '{print $1}')
|
||||
first=$(echo "$ids" | head -1)
|
||||
|
||||
echo "[splat] running colmap_poses on the farm..."
|
||||
cj=$("$MB" run colmap_poses --asset "$first" $(echo "$ids" | tail -n +2 | sed 's/^/--asset /') | grep -oE '[0-9a-f]{10,}' | head -1)
|
||||
"$MB" wait "$cj"
|
||||
ds=$("$MB" assets | awk '/colmap_dataset/ {print $1; exit}')
|
||||
echo "[splat] dataset: $ds — training splat (brush)..."
|
||||
bj=$("$MB" run brush_train --asset "$ds" | grep -oE '[0-9a-f]{10,}' | head -1)
|
||||
"$MB" wait "$bj" --download "$OUT/_splat_dl"
|
||||
ply=$(find "$OUT/_splat_dl" -name '*.ply' | head -1)
|
||||
[ -n "$ply" ] || { echo "no .ply came back"; exit 1; }
|
||||
mv "$ply" "$OUT/splat.ply" && rm -rf "$OUT/_splat_dl"
|
||||
echo "[splat] done → $OUT/splat.ply (restart/reload the app; the 3D view now renders the splat)"
|
||||
11
uv.lock
generated
11
uv.lock
generated
@ -441,7 +441,6 @@ dependencies = [
|
||||
{ name = "openai" },
|
||||
{ name = "opencv-python-headless" },
|
||||
{ name = "pydantic" },
|
||||
{ name = "python-multipart" },
|
||||
{ name = "scipy", version = "1.17.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" },
|
||||
{ name = "scipy", version = "1.18.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" },
|
||||
{ name = "soundfile" },
|
||||
@ -465,7 +464,6 @@ requires-dist = [
|
||||
{ name = "opencv-python-headless", specifier = ">=4.9" },
|
||||
{ name = "pydantic", specifier = ">=2.6" },
|
||||
{ name = "pytest", marker = "extra == 'dev'", specifier = ">=8.0" },
|
||||
{ name = "python-multipart", specifier = ">=0.0.9" },
|
||||
{ name = "scipy", specifier = ">=1.11" },
|
||||
{ name = "soundfile", specifier = ">=0.12" },
|
||||
{ name = "sqlalchemy", specifier = ">=2.0" },
|
||||
@ -1278,15 +1276,6 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/0b/d7/1959b9648791274998a9c3526f6d0ec8fd2233e4d4acce81bbae76b44b2a/python_dotenv-1.2.2-py3-none-any.whl", hash = "sha256:1d8214789a24de455a8b8bd8ae6fe3c6b69a5e3d64aa8a8e5d68e694bbcb285a", size = 22101, upload-time = "2026-03-01T16:00:25.09Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "python-multipart"
|
||||
version = "0.0.32"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/5b/42/55c32bb9b12693c092ad250a0e82edb5b31ddeda6eb772de5f308b3804ad/python_multipart-0.0.32.tar.gz", hash = "sha256:be54b7f3fa167bb83e4fcd936b887b708f4e57fe75911c02aebf53efaf8d938e", size = 46881, upload-time = "2026-06-04T16:18:58.647Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/e1/04/e8135ebd1ad02c56ec633277529b2602ff99ff634be76cdba5744cf554fd/python_multipart-0.0.32-py3-none-any.whl", hash = "sha256:ff6d3f776f16878c894e52e107296ffc890e913c611b1a4ec6c44e2821fe2e23", size = 30042, upload-time = "2026-06-04T16:18:57.319Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pyyaml"
|
||||
version = "6.0.3"
|
||||
|
||||
Loading…
Reference in New Issue
Block a user