monster/festifun
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'lane/c-viewer'

Browse Source 
# Conflicts:
#	plan/status/lane-C.md
This commit is contained in:
m3ultra 2026-07-16 09:02:13 +10:00
commit 9d17b474ea
11 changed files with 1522 additions and 163 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

217
frontend/index.html
View File

@ -5,39 +5,206 @@
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Festival 4D</title>
<style>
:root { color-scheme: dark; }
:root {
color-scheme: dark;
--bg: #0b0d12;
--panel: #141824;
--panel-2: #1c2233;
--border: #232838;
--text: #e6e8ee;
--dim: #8a92a6;
--ok: #59d499;
--bad: #ff6b6b;
--accent: #59d499;
}
* { box-sizing: border-box; }
html, body { height: 100%; }
body {
margin: 0;
font: 15px/1.5 -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
background: #0b0d12;
color: #e6e8ee;
padding: 2rem;
font: 14px/1.45 -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
background: var(--bg);
color: var(--text);
overflow: hidden;
}
h1 { margin: 0 0 0.25rem; font-size: 1.6rem; }
.sub { color: #8a92a6; margin-bottom: 1.5rem; }
.card {
background: #141824;
border: 1px solid #232838;
border-radius: 10px;
padding: 1rem 1.25rem;
margin-bottom: 1rem;
max-width: 720px;
#app { display: flex; flex-direction: column; height: 100vh; }
/* Top bar */
#topbar {
display: flex; align-items: baseline; gap: 0.6rem;
padding: 0.5rem 0.9rem; border-bottom: 1px solid var(--border); flex: 0 0 auto;
}
.ok { color: #59d499; }
.bad { color: #ff6b6b; }
table { border-collapse: collapse; width: 100%; }
th, td { text-align: left; padding: 0.35rem 0.6rem; border-bottom: 1px solid #232838; }
th { color: #8a92a6; font-weight: 600; }
code { background: #1c2233; padding: 0.1rem 0.35rem; border-radius: 4px; }
.pill { display: inline-block; padding: 0.1rem 0.5rem; border-radius: 999px; font-size: 0.8rem; }
.pill.ok { background: rgba(89,212,153,0.15); }
.pill.bad { background: rgba(255,107,107,0.15); }
#topbar h1 { margin: 0; font-size: 1.05rem; font-weight: 700; letter-spacing: 0.2px; }
#topbar .sub { color: var(--dim); font-size: 0.8rem; }
/* Stage: 3D on the left, video grid on the right */
#stage { flex: 1 1 auto; display: flex; min-height: 0; }
#scene-pane { flex: 1 1 55%; position: relative; min-width: 0; background: #0a0c11; }
#scene3d { position: absolute; inset: 0; width: 100%; height: 100%; display: block; }
#scene-controls {
position: absolute; top: 0.6rem; left: 0.6rem; display: flex; gap: 0.4rem; z-index: 5;
}
#scene-hint {
position: absolute; bottom: 0.5rem; left: 0.6rem; color: var(--dim);
font-size: 0.72rem; z-index: 5; pointer-events: none;
background: rgba(10,12,17,0.6); padding: 0.2rem 0.45rem; border-radius: 5px;
}
#grid-pane {
flex: 1 1 45%; min-width: 0; border-left: 1px solid var(--border);
overflow: auto; background: #0d1017;
}
#video-grid {
display: grid; gap: 6px; padding: 6px;
grid-template-columns: repeat(auto-fit, minmax(240px, 1fr));
}
.cell {
position: relative; aspect-ratio: 16 / 9; background: #000;
border: 1px solid var(--border); border-radius: 8px; overflow: hidden;
transition: opacity 0.15s ease, border-color 0.15s ease;
}
.cell.out { opacity: 0.32; }
.cell.audio-on { border-color: var(--accent); }
.cell.followed { box-shadow: inset 0 0 0 2px var(--accent); }
.cell-video { position: absolute; inset: 0; width: 100%; height: 100%; object-fit: contain; }
.cell-overlay { position: absolute; inset: 0; pointer-events: none; }
.cell-bar {
position: absolute; left: 0; right: 0; bottom: 0; display: flex; align-items: center;
gap: 0.35rem; padding: 0.25rem 0.4rem; font-size: 0.72rem;
background: linear-gradient(transparent, rgba(6,8,14,0.85));
}
.cell-label { font-weight: 600; }
.cell-status { color: var(--dim); margin-left: 0.1rem; }
.cell-status.ok { color: var(--ok); }
.cell-status.bad { color: var(--bad); }
.cell-btn {
margin-left: auto; background: rgba(255,255,255,0.08); color: var(--text);
border: 1px solid var(--border); border-radius: 5px; cursor: pointer;
width: 22px; height: 20px; font-size: 0.72rem; line-height: 1; padding: 0;
}
.cell-btn + .cell-btn { margin-left: 0.25rem; }
.cell-btn:hover { background: rgba(255,255,255,0.16); }
.cell.audio-on .audio-btn { background: var(--accent); color: #071018; border-color: var(--accent); }
/* Bottom bar: transport + timeline */
#bottombar {
flex: 0 0 auto; border-top: 1px solid var(--border); padding: 0.5rem 0.9rem 0.7rem;
background: var(--panel);
}
#controls { display: flex; align-items: center; gap: 0.5rem; margin-bottom: 0.5rem; }
.btn {
background: var(--panel-2); color: var(--text); border: 1px solid var(--border);
border-radius: 6px; cursor: pointer; padding: 0.3rem 0.6rem; font-size: 0.85rem;
}
.btn:hover { background: #262d42; }
.btn.active { background: var(--accent); color: #071018; border-color: var(--accent); }
#btn-play { min-width: 2.4rem; font-size: 1rem; }
#time-display { font-variant-numeric: tabular-nums; color: var(--dim); margin-left: 0.2rem; }
select.btn { padding-right: 0.4rem; }
.spacer { flex: 1 1 auto; }
.ctl-label { color: var(--dim); font-size: 0.78rem; }
/* Timeline */
#timeline { position: relative; }
.tl-track {
position: relative; height: 26px; background: var(--panel-2);
border: 1px solid var(--border); border-radius: 6px; cursor: pointer; overflow: visible;
touch-action: none;
}
.tl-fill { position: absolute; left: 0; top: 0; bottom: 0; width: 0;
background: rgba(89,212,153,0.14); border-radius: 6px 0 0 6px; }
.tl-markers { position: absolute; inset: 0; }
.tl-marker {
position: absolute; top: 0; bottom: 0; width: 3px; margin-left: -1.5px;
border-radius: 2px; cursor: pointer; box-shadow: 0 0 0 1px rgba(0,0,0,0.35);
}
.tl-marker:hover { width: 5px; margin-left: -2.5px; }
.tl-playhead {
position: absolute; top: -3px; bottom: -3px; width: 2px; margin-left: -1px;
background: #fff; box-shadow: 0 0 6px rgba(255,255,255,0.6); pointer-events: none;
}
.tl-tooltip {
position: absolute; bottom: 34px; transform: translateX(-50%);
background: #0f1420; border: 1px solid var(--border); border-radius: 7px;
padding: 0.4rem 0.55rem; font-size: 0.78rem; max-width: 260px; z-index: 20;
pointer-events: none; box-shadow: 0 6px 20px rgba(0,0,0,0.5);
}
.tl-tooltip strong { text-transform: capitalize; }
.tl-conf { color: var(--dim); font-size: 0.72rem; }
.tl-legend {
display: flex; flex-wrap: wrap; gap: 0.5rem 0.9rem; margin-top: 0.45rem;
font-size: 0.72rem; color: var(--dim);
}
.tl-legend-item { display: inline-flex; align-items: center; gap: 0.3rem; text-transform: capitalize; }
.tl-legend-item i { width: 10px; height: 10px; border-radius: 3px; display: inline-block; }
/* Dev sync overlay */
#dev-overlay {
position: fixed; top: 0.6rem; right: 0.6rem; z-index: 30;
background: rgba(15,20,32,0.9); border: 1px solid var(--border); border-radius: 8px;
padding: 0.45rem 0.6rem; font-size: 0.72rem; min-width: 132px;
backdrop-filter: blur(4px);
}
#dev-overlay .hd { color: var(--dim); font-weight: 600; margin-bottom: 0.3rem; }
.dev-row { display: flex; justify-content: space-between; gap: 0.8rem; font-variant-numeric: tabular-nums; }
.ok { color: var(--ok); }
.bad { color: var(--bad); }
.dim { color: var(--dim); }
/* Loading / error */
#loading {
position: fixed; inset: 0; display: flex; align-items: center; justify-content: center;
background: var(--bg); z-index: 100; color: var(--dim);
}
#loading .err { text-align: center; color: var(--text); }
code { background: var(--panel-2); padding: 0.1rem 0.35rem; border-radius: 4px; }
</style>
</head>
<body>
<div id="app">
<div id="topbar">
<h1>Festival 4D</h1>
<div class="sub">Foundation hello page — proves the API contract, CORS, and the frozen pose helper.</div>
<div id="app"><div class="card">Loading…</div></div>
<span class="sub">synchronized multi-cam replay · 3D viewer · x-ray overlays</span>
</div>
<div id="stage">
<div id="scene-pane">
<canvas id="scene3d"></canvas>
<div id="scene-controls">
<button class="btn active" id="btn-roam" title="Free roam (Esc)">Free roam</button>
</div>
<div id="scene-hint">drag to orbit · click a camera or press 1–9 to snap · Esc to detach</div>
</div>
<div id="grid-pane">
<div id="video-grid"></div>
</div>
</div>
<div id="bottombar">
<div id="controls">
<button class="btn" id="btn-back" title="Back 1s (←)">⏪</button>
<button class="btn" id="btn-play" title="Play/Pause (Space)">▶</button>
<button class="btn" id="btn-fwd" title="Forward 1s (→)">⏩</button>
<span id="time-display">0:00.0 / 0:00.0</span>
<span class="spacer"></span>
<span class="ctl-label">speed</span>
<select class="btn" id="rate-select">
<option value="0.25">0.25×</option>
<option value="0.5">0.5×</option>
<option value="1" selected>1×</option>
<option value="2">2×</option>
</select>
</div>
<div id="timeline"></div>
</div>
</div>
<div id="dev-overlay">
<div class="hd">sync error</div>
<div id="dev-rows"></div>
</div>
<div id="loading">Loading project…</div>
<script type="module" src="/src/main.js"></script>
</body>
</html>

57
frontend/src/lib/poseTrack.js Normal file
View File

@ -0,0 +1,57 @@
// Pose lookup + interpolation over a video's `camera_poses` track (from GET /api/videos/{id}/poses).
//
// Poses arrive keyed by frame_idx / t_video_s (every ~0.5 s on the synthetic fixture, all
// registered). To render a smooth frustum / follow-cam at an arbitrary t_video we interpolate
// BETWEEN stored poses in COLMAP space — slerp the world->camera quaternion, lerp the
// translation — then hand the interpolated (q, t) to the FROZEN colmapToThreejs() helper.
// We interpolate the CONTRACT INPUTS; we never reimplement the COLMAP->Three.js conversion.
import * as THREE from "three";
const _qa = new THREE.Quaternion();
const _qb = new THREE.Quaternion();
/**
* Interpolated pose at local video time `tv` (seconds). Returns a COLMAP-convention pose
* `{ q:[w,x,y,z], t:[x,y,z], intrinsics, registered, t_video_s }` ready for colmapToThreejs,
* or null if the track is empty. Clamps to the endpoints (no extrapolation).
* @param {Array} poses ascending-by-t_video_s pose list
* @param {number} tv local video time (seconds)
*/
export function poseAt(poses, tv) {
if (!poses || poses.length === 0) return null;
const n = poses.length;
if (tv <= poses[0].t_video_s) return poses[0];
if (tv >= poses[n - 1].t_video_s) return poses[n - 1];
// binary search for the bracketing pair [lo, hi] with poses[lo].t <= tv < poses[hi].t
let lo = 0;
let hi = n - 1;
while (hi - lo > 1) {
const mid = (lo + hi) >> 1;
if (poses[mid].t_video_s <= tv) lo = mid;
else hi = mid;
}
const a = poses[lo];
const b = poses[hi];
const span = b.t_video_s - a.t_video_s;
const f = span > 1e-9 ? (tv - a.t_video_s) / span : 0;
// COLMAP quaternion order is [w, x, y, z]; THREE.Quaternion is (x, y, z, w).
_qa.set(a.q[1], a.q[2], a.q[3], a.q[0]);
_qb.set(b.q[1], b.q[2], b.q[3], b.q[0]);
_qa.slerp(_qb, f);
const q = [_qa.w, _qa.x, _qa.y, _qa.z];
const t = [
a.t[0] + (b.t[0] - a.t[0]) * f,
a.t[1] + (b.t[1] - a.t[1]) * f,
a.t[2] + (b.t[2] - a.t[2]) * f,
];
return {
q,
t,
intrinsics: a.intrinsics,
registered: a.registered && b.registered,
t_video_s: tv,
};
}

18
frontend/src/lib/timebase.js Normal file
View File

@ -0,0 +1,18 @@
// Timebase convention (FROZEN CONTRACT) — the JS mirror of backend/festival4d/config.py
// (`t_video_from_global` / `t_global_from_video`). Keep these identical to the Python side.
//
// t_global is the master timeline in seconds. For a video v:
// t_video = (t_global - offset_ms/1000) * (1 + drift_ppm * 1e-6)
// The reference video has offset_ms == 0 (and drift_ppm == 0). A positive offset_ms means
// the video started recording later than the master zero, so at a given t_global its local
// playhead is earlier. Never re-derive this algebra inline — call these helpers everywhere.
/** Master-timeline seconds -> a video's local playhead seconds (FROZEN). */
export function tVideoFromGlobal(tGlobal, offsetMs, driftPpm = 0) {
return (tGlobal - offsetMs / 1000) * (1 + driftPpm * 1e-6);
}
/** Inverse of tVideoFromGlobal (FROZEN). */
export function tGlobalFromVideo(tVideo, offsetMs, driftPpm = 0) {
return tVideo / (1 + driftPpm * 1e-6) + offsetMs / 1000;
}

260
frontend/src/main.js
View File

@ -1,91 +1,205 @@
// Foundation hello page. Fetches /api/manifest (proving the API + CORS work cross-origin)
// and runs the frozen pose.js self-test in the browser. Lane C (M4–M6) replaces this with
// the real app; the modules alongside it are stubs to fill in.
// Festival 4D viewer — composition root (spec M4/M5/M6 + timeline markers).
//
// Boots by loading the frozen synthetic API (manifest, poses, anchors, events), builds the
// video grid + 3D scene + timeline, wires transport controls + keyboard shortcuts, and runs a
// single master animation loop that: advances the master clock & corrects every video
// (transport), redraws each overlay for its displayed frame, updates the 3D frusta / follow-cam,
// moves the timeline playhead, and refreshes the dev sync overlay. The master clock lives in
// transport.js and is derived from performance.now() — never from a <video> element.
import { selfTest as poseSelfTest } from "./lib/pose.js";
import { state, on, API_BASE, referenceVideoId } from "./state.js";
import { transport } from "./transport.js";
import { createVideoGrid } from "./videoGrid.js";
import { drawOverlay, projectWorldToVideoPx } from "./overlays.js";
import { poseAt } from "./lib/poseTrack.js";
import { scene3d, videoColor } from "./scene3d.js";
import { timeline } from "./timeline.js";
// The FastAPI backend (`python -m festival4d serve`). The Vite dev server runs on :5173;
// this cross-origin fetch exercises the CORS config in api.py.
const API_BASE = "http://localhost:8000";
const app = document.getElementById("app");
function card(html) {
const el = document.createElement("div");
el.className = "card";
el.innerHTML = html;
return el;
async function getJSON(path) {
const resp = await fetch(API_BASE + path);
if (!resp.ok) throw new Error(`${path} -> HTTP ${resp.status}`);
return resp.json();
}
function pill(ok, text) {
return `<span class="pill ${ok ? "ok" : "bad"}">${text}</span>`;
function fmtTime(s) {
s = Math.max(0, s);
const m = Math.floor(s / 60);
const sec = s - m * 60;
return `${m}:${sec.toFixed(1).padStart(4, "0")}`;
}
function renderPoseSelfTest() {
function el(id) {
return document.getElementById(id);
}
async function boot() {
const loading = el("loading");
try {
poseSelfTest();
return card(
`<strong>Pose contract</strong> ${pill(true, "self-test passed")}<br />` +
`<span class="sub">frontend/src/lib/pose.js matches the frozen COLMAP→Three.js vectors.</span>`
const manifest = await getJSON("/api/manifest");
state.manifest = manifest;
state.videos = manifest.videos;
state.tGlobalMax = manifest.t_global_max;
state.hasPoses = manifest.has_poses;
for (const v of state.videos) state.enabled[v.id] = true;
state.audioSourceId = referenceVideoId();
const [anchors, events] = await Promise.all([
getJSON("/api/anchors"),
getJSON("/api/events"),
]);
state.anchors = anchors;
state.events = events;
if (state.hasPoses) {
const lists = await Promise.all(
state.videos.map((v) => getJSON(`/api/videos/${v.id}/poses`))
);
state.videos.forEach((v, i) => (state.poses[v.id] = lists[i]));
}
buildUI();
startLoop();
loading.style.display = "none";
// Dev-only hook so the transport/scene can be driven in tests (e.g. a setTimeout pump when
// a headless pane throttles rAF). Never present in a production build.
if (import.meta.env?.DEV) {
window.__f4d = {
state, transport, scene3d, timeline, cells: () => cells,
poseAt, projectWorldToVideoPx,
};
}
} catch (err) {
return card(
`<strong>Pose contract</strong> ${pill(false, "self-test FAILED")}<br />` +
`<code>${err.message}</code>`
);
console.error("[festival4d] boot failed", err);
loading.innerHTML = `<div class="err">Failed to load the project.<br><code>${err.message}</code>` +
`<br><span class="dim">Is the backend running? <code>python -m festival4d serve</code></span></div>`;
}
}
function renderManifest(m) {
const rows = m.videos
.map(
(v) => `
<tr>
<td>${v.id}</td>
<td><code>${v.filename}</code></td>
<td>${v.duration_s.toFixed(2)} s</td>
<td>${v.width}×${v.height} @ ${v.fps}</td>
<td>${v.offset_ms === null ? "—" : v.offset_ms + " ms"}</td>
</tr>`
)
.join("");
return card(`
<strong>API manifest</strong> ${pill(true, "fetched (CORS ok)")}<br />
<span class="sub">
t_global_max = ${m.t_global_max.toFixed(2)} s ·
has_poses = ${m.has_poses ? "yes" : "no"} ·
${m.videos.length} videos
</span>
<table>
<thead><tr><th>id</th><th>file</th><th>duration</th><th>frame</th><th>offset</th></tr></thead>
<tbody>${rows}</tbody>
</table>
<div class="sub" style="margin-top:0.75rem">
Videos served with HTTP Range from <code>${API_BASE}/media/…</code>
</div>
`);
let cells = [];
function buildUI() {
// Video grid
const grid = el("video-grid");
cells = createVideoGrid(grid, state.videos, API_BASE, transport);
transport.register(cells);
// Per-video snap buttons -> scene3d (grid module leaves them unwired).
cells.forEach((cell, i) => {
cell.el.style.setProperty("--accent", videoColor(i));
cell.snapBtn.addEventListener("click", (e) => {
e.stopPropagation();
scene3d.snapTo(cell.id);
});
});
// 3D scene
scene3d.init(el("scene3d"));
// Timeline
timeline.init(el("timeline"), transport);
wireTransportControls();
wireKeyboard();
positionVideosWhenReady();
// Reflect follow/roam state on the free-roam button.
on("follow", (id) => {
el("btn-roam").classList.toggle("active", id == null);
});
}
async function main() {
app.innerHTML = "";
app.appendChild(renderPoseSelfTest());
try {
const resp = await fetch(`${API_BASE}/api/manifest`);
if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
const manifest = await resp.json();
app.appendChild(renderManifest(manifest));
console.log("[festival4d] manifest loaded", manifest);
} catch (err) {
app.appendChild(
card(
`<strong>API manifest</strong> ${pill(false, "fetch failed")}<br />` +
`<code>${err.message}</code><br />` +
`<span class="sub">Is the backend running? <code>python -m festival4d serve</code></span>`
)
function wireTransportControls() {
el("btn-play").addEventListener("click", () => transport.toggle());
el("btn-back").addEventListener("click", () => transport.seekBy(-1));
el("btn-fwd").addEventListener("click", () => transport.seekBy(1));
el("rate-select").addEventListener("change", (e) =>
transport.setRate(parseFloat(e.target.value))
);
console.error("[festival4d] manifest fetch failed", err);
el("btn-roam").addEventListener("click", () => scene3d.freeRoam());
}
function wireKeyboard() {
window.addEventListener("keydown", (e) => {
if (e.target && /^(INPUT|SELECT|TEXTAREA)$/.test(e.target.tagName)) return;
if (e.code === "Space") {
e.preventDefault();
transport.toggle();
} else if (e.code === "ArrowLeft") {
e.preventDefault();
transport.seekBy(e.shiftKey ? -5 : -1);
} else if (e.code === "ArrowRight") {
e.preventDefault();
transport.seekBy(e.shiftKey ? 5 : 1);
} else if (e.code === "Escape" || e.code === "Digit0") {
scene3d.freeRoam();
} else if (/^Digit[1-9]$/.test(e.code)) {
const n = parseInt(e.code.slice(5), 10) - 1;
const v = state.videos[n];
if (v) scene3d.snapTo(v.id);
}
});
}
function positionVideosWhenReady() {
let remaining = cells.length;
if (remaining === 0) return;
const done = () => {
if (--remaining === 0) transport.seek(0); // park every video on its t=0 frame
};
for (const { video } of cells) {
if (video.readyState >= 1) done();
else video.addEventListener("loadedmetadata", done, { once: true });
}
}
main();
function startLoop() {
const frame = () => {
transport.tick();
for (const cell of cells) drawOverlay(cell);
scene3d.update();
timeline.update(state.tGlobal);
updateHud();
requestAnimationFrame(frame);
};
requestAnimationFrame(frame);
}
function updateHud() {
el("btn-play").textContent = state.playing ? "⏸" : "▶";
el("time-display").textContent = `${fmtTime(state.tGlobal)} / ${fmtTime(state.tGlobalMax)}`;
// Dev sync overlay + per-cell status.
let devRows = "";
for (const cell of cells) {
const { id, meta } = cell;
const err = state.syncErrorMs[id];
const inRange = state.inRange[id];
const enabled = state.enabled[id] !== false;
let label, cls;
if (!enabled) {
label = "disabled";
cls = "dim";
} else if (!inRange) {
label = "out of range";
cls = "dim";
} else if (err == null) {
label = "—";
cls = "dim";
} else {
label = `${err >= 0 ? "+" : ""}${err.toFixed(0)} ms`;
cls = Math.abs(err) < 50 ? "ok" : "bad";
}
devRows += `<div class="dev-row"><span>${meta.filename}</span><span class="${cls}">${label}</span></div>`;
// Cell overlay state (dim out-of-range / disabled, mark audio + follow).
cell.el.classList.toggle("out", !inRange || !enabled);
cell.el.classList.toggle("audio-on", state.audioSourceId === id);
cell.el.classList.toggle("followed", state.followCameraId === id);
cell.statusEl.textContent = enabled ? (inRange ? label : "waiting…") : "off";
cell.statusEl.className = `cell-status ${cls}`;
}
el("dev-rows").innerHTML = devRows;
}
boot();

141
frontend/src/overlays.js
View File

@ -1,6 +1,137 @@
// 3D→2D anchor projection onto each video's overlay canvas — the "x-ray" HUD (spec M6).
// STUB — lane C fills this in. For each anchor + visible video: build the video's current
// Three.js camera (pose.js helper), project the anchor, convert NDC→canvas px, skip if
// behind camera (check z_cam < 0). Anchors render regardless of occlusion — that's the feature.
// 3D -> 2D anchor projection onto each video's overlay canvas — the "x-ray" HUD (spec M6).
//
// For each visible video we build a Three.js PerspectiveCamera at that video's CURRENT pose
// (from the FROZEN colmapToThreejs helper, matrixWorld path) with an fov derived from the
// stored intrinsics, project every anchor, drop anchors behind the camera (camera-space z >= 0),
// convert NDC -> native video pixels -> canvas pixels through the letterbox content rect, and
// draw a dot + halo + label. Anchors are drawn regardless of real-world occlusion — that IS the
// x-ray feature. We consume pose.js; we never reimplement the COLMAP->Three.js conversion.
export {};
import * as THREE from "three";
import { colmapToThreejs } from "./lib/pose.js";
import { poseAt } from "./lib/poseTrack.js";
import { contentRect } from "./videoGrid.js";
import { state } from "./state.js";
// Reused across all videos/frames (every synthetic video shares intrinsics + resolution, and
// we fully overwrite the matrices each call, so a single scratch camera is safe and cheap).
const _cam = new THREE.PerspectiveCamera();
_cam.matrixAutoUpdate = false;
const _m4 = new THREE.Matrix4();
const _v = new THREE.Vector3();
function configureProjector(pose, W, H) {
// Centered-principal-point pinhole -> PerspectiveCamera (spec M5/M6 prototype assumption;
// the synthetic fixture has cx=W/2, cy=H/2 exactly).
const fy = pose.intrinsics.fy;
_cam.fov = (2 * Math.atan(H / (2 * fy)) * 180) / Math.PI;
_cam.aspect = W / H;
_cam.near = 0.01;
_cam.far = 2000;
_cam.updateProjectionMatrix();
const { matrixWorld } = colmapToThreejs(pose.q, pose.t);
_m4.fromArray(matrixWorld);
_cam.matrixWorld.copy(_m4);
_cam.matrixWorldInverse.copy(_m4).invert();
}
/** Project a world point to native video pixels, or null if behind the camera. */
function projectToVideoPx(x, y, z, W, H) {
_v.set(x, y, z);
// Behind-camera test in camera space: Three.js cameras look down -z, so a visible point has
// camera-space z < 0. Do this BEFORE .project() (which mutates _v into NDC).
_v.applyMatrix4(_cam.matrixWorldInverse);
if (_v.z >= 0) return null;
_v.applyMatrix4(_cam.projectionMatrix); // perspective divide happens inside .project(); do it manually
// _v is now clip space already divided (applyMatrix4 on a Vector3 divides by w). NDC in [-1,1].
const u = (_v.x * 0.5 + 0.5) * W;
const vpx = (1 - (_v.y * 0.5 + 0.5)) * H; // flip Y: NDC +y is up, pixels grow downward
return { u, v: vpx };
}
/**
* Project a world point to native video pixels through the FROZEN pose helper for a given
* COLMAP pose (as returned by poseAt) and video resolution. Returns {u, v} or null (behind).
* Single source of truth for overlay projection — used by drawOverlay and by tests.
*/
export function projectWorldToVideoPx(pose, x, y, z, W, H) {
configureProjector(pose, W, H);
return projectToVideoPx(x, y, z, W, H);
}
function drawMarker(ctx, x, y, anchor) {
const color = anchor.color || "#59d499";
// halo
ctx.beginPath();
ctx.arc(x, y, 5.5, 0, Math.PI * 2);
ctx.fillStyle = "rgba(255,255,255,0.85)";
ctx.fill();
// dot
ctx.beginPath();
ctx.arc(x, y, 4, 0, Math.PI * 2);
ctx.fillStyle = color;
ctx.fill();
// label
const text = anchor.label || "";
if (text) {
ctx.font = "600 11px -apple-system, system-ui, sans-serif";
ctx.textBaseline = "middle";
const tx = x + 9;
const ty = y;
const w = ctx.measureText(text).width;
ctx.fillStyle = "rgba(6,8,14,0.72)";
ctx.fillRect(tx - 3, ty - 8, w + 6, 16);
ctx.fillStyle = "#e6e8ee";
ctx.fillText(text, tx, ty + 0.5);
}
}
/** Redraw one cell's overlay for the frame currently displayed by its <video>. */
export function drawOverlay(cell) {
const { video, canvas } = cell;
const ctx = canvas.getContext("2d");
const dpr = window.devicePixelRatio || 1;
const elW = video.clientWidth;
const elH = video.clientHeight;
if (elW === 0 || elH === 0) return;
// Resolution-match the canvas backing store to the element (dpr-aware).
const bw = Math.round(elW * dpr);
const bh = Math.round(elH * dpr);
if (canvas.width !== bw || canvas.height !== bh) {
canvas.width = bw;
canvas.height = bh;
canvas.style.width = elW + "px";
canvas.style.height = elH + "px";
}
ctx.setTransform(dpr, 0, 0, dpr, 0, 0);
ctx.clearRect(0, 0, elW, elH);
if (!state.hasPoses || !state.inRange[cell.id]) return;
if (video.videoWidth === 0) return;
const poses = state.poses[cell.id];
if (!poses || poses.length === 0 || state.anchors.length === 0) return;
// Use the frame the <video> is ACTUALLY showing (currentTime), not the master target, so the
// overlay tracks the displayed picture even mid-correction.
const pose = poseAt(poses, video.currentTime);
if (!pose) return;
const W = video.videoWidth;
const H = video.videoHeight;
configureProjector(pose, W, H);
const rect = contentRect(video);
for (const a of state.anchors) {
const p = projectToVideoPx(a.x, a.y, a.z, W, H);
if (!p) continue;
const cx = rect.ox + (p.u / W) * rect.cw;
const cy = rect.oy + (p.v / H) * rect.ch;
// Skip anchors that fall well outside the visible content rect.
const m = 24;
if (cx < rect.ox - m || cx > rect.ox + rect.cw + m) continue;
if (cy < rect.oy - m || cy > rect.oy + rect.ch + m) continue;
drawMarker(ctx, cx, cy, a);
}
}

304
frontend/src/scene3d.js
View File

@ -1,6 +1,300 @@
// Three.js scene: point cloud (/api/pointcloud via PLYLoader), per-video camera paths +
// current-pose frusta, OrbitControls, snap-to-camera (spec M5).
// STUB — lane C fills this in. Use frontend/src/lib/pose.js (frozen contract) for ALL
// COLMAP→Three.js conversion — do not reimplement the diag(1,-1,-1) math.
// Three.js scene (spec M5): point cloud (/api/pointcloud via PLYLoader), each video's camera
// path as a line + current-pose frustum wireframe (colored per video), OrbitControls free roam,
// and snap-to-camera. ALL COLMAP->Three.js conversion goes through the FROZEN lib/pose.js
// helper (colmapToThreejs) — the diag(1,-1,-1) math is never reimplemented here.
export {};
import * as THREE from "three";
import { OrbitControls } from "three/addons/controls/OrbitControls.js";
import { PLYLoader } from "three/addons/loaders/PLYLoader.js";
import { colmapToThreejs } from "./lib/pose.js";
import { poseAt } from "./lib/poseTrack.js";
import { tVideoFromGlobal } from "./lib/timebase.js";
import { state, emit } from "./state.js";
// Per-video colors (match the frustum, path line, and grid accents).
export const VIDEO_COLORS = ["#ff5d73", "#4dd0ff", "#c9a2ff", "#ffcf5d", "#7dffb0", "#ff9d5d"];
export function videoColor(index) {
return VIDEO_COLORS[index % VIDEO_COLORS.length];
}
const FRUSTUM_DEPTH = 1.2;
const DEFAULT_FOV = 50;
const STAGE_TARGET = new THREE.Vector3(0, 0.5, 0);
const _m4 = new THREE.Matrix4();
const _pos = new THREE.Vector3();
const _quat = new THREE.Quaternion();
const _scale = new THREE.Vector3();
export class Scene3D {
constructor() {
this.videoIndex = {}; // id -> index (for stable colors)
this.rigs = {}; // id -> { frustum, marker, pick, path }
this._tween = null;
this._raf = null;
}
init(canvas) {
this.canvas = canvas;
const parent = canvas.parentElement;
const w = parent.clientWidth || 1;
const h = parent.clientHeight || 1;
this.renderer = new THREE.WebGLRenderer({ canvas, antialias: true, alpha: false });
this.renderer.setPixelRatio(Math.min(window.devicePixelRatio || 1, 2));
this.renderer.setSize(w, h, false);
this.renderer.setClearColor(0x0a0c11, 1);
this.scene = new THREE.Scene();
this.camera = new THREE.PerspectiveCamera(DEFAULT_FOV, w / h, 0.05, 3000);
this.camera.position.set(12, 9, 15);
this.camera.lookAt(STAGE_TARGET);
this.controls = new OrbitControls(this.camera, this.renderer.domElement);
this.controls.enableDamping = true;
this.controls.dampingFactor = 0.08;
this.controls.target.copy(STAGE_TARGET);
this.controls.update();
// Lights (points ignore lighting, but frusta/markers use MeshBasic so this is mostly cosmetic).
this.scene.add(new THREE.AmbientLight(0xffffff, 0.9));
const grid = new THREE.GridHelper(40, 40, 0x22384a, 0x161c26);
grid.position.y = 0;
this.scene.add(grid);
const axes = new THREE.AxesHelper(1.5);
this.scene.add(axes);
this._buildRigs();
this._loadPointCloud();
this._wirePicking();
this._wireResize(parent);
}
_buildRigs() {
state.videos.forEach((v, i) => {
this.videoIndex[v.id] = i;
const color = new THREE.Color(videoColor(i));
const group = new THREE.Group();
// Frustum wireframe (built in Three.js camera-local space: looks down -z, +y up).
const poses = state.poses[v.id] || [];
const intr = poses[0]?.intrinsics;
const W = v.width;
const H = v.height;
const hx = intr ? (FRUSTUM_DEPTH * W) / (2 * intr.fx) : FRUSTUM_DEPTH * 0.6;
const hy = intr ? (FRUSTUM_DEPTH * H) / (2 * intr.fy) : FRUSTUM_DEPTH * 0.34;
const d = FRUSTUM_DEPTH;
const c = [
[-hx, -hy, -d],
[hx, -hy, -d],
[hx, hy, -d],
[-hx, hy, -d],
];
const apex = [0, 0, 0];
const segs = [
apex, c[0], apex, c[1], apex, c[2], apex, c[3],
c[0], c[1], c[1], c[2], c[2], c[3], c[3], c[0],
];
const fg = new THREE.BufferGeometry();
fg.setAttribute("position", new THREE.Float32BufferAttribute(segs.flat(), 3));
const frustum = new THREE.LineSegments(
fg,
new THREE.LineBasicMaterial({ color })
);
frustum.matrixAutoUpdate = true;
group.add(frustum);
// Camera-center marker (visible) + an invisible larger sphere for click picking.
const marker = new THREE.Mesh(
new THREE.SphereGeometry(0.12, 12, 12),
new THREE.MeshBasicMaterial({ color })
);
group.add(marker);
this.scene.add(group);
const pick = new THREE.Mesh(
new THREE.SphereGeometry(0.55, 8, 8),
new THREE.MeshBasicMaterial({ visible: false })
);
pick.userData.videoId = v.id;
group.add(pick);
// Camera path line (one vertex per stored pose center).
let path = null;
if (poses.length > 1) {
const pts = [];
for (const p of poses) {
const { position } = colmapToThreejs(p.q, p.t);
pts.push(position[0], position[1], position[2]);
}
const pg = new THREE.BufferGeometry();
pg.setAttribute("position", new THREE.Float32BufferAttribute(pts, 3));
path = new THREE.Line(
pg,
new THREE.LineBasicMaterial({ color, transparent: true, opacity: 0.5 })
);
this.scene.add(path);
}
this.rigs[v.id] = { group, frustum, marker, pick, path };
});
}
_loadPointCloud() {
const loader = new PLYLoader();
loader.load(
state.apiBase + "/api/pointcloud",
(geometry) => {
geometry.computeBoundingBox();
const hasColor = !!geometry.getAttribute("color");
const material = new THREE.PointsMaterial({
size: 0.05,
sizeAttenuation: true,
vertexColors: hasColor,
color: hasColor ? 0xffffff : 0x88ccff,
});
this.points = new THREE.Points(geometry, material);
this.scene.add(this.points);
emit("pointcloud-loaded", geometry.getAttribute("position")?.count ?? 0);
},
undefined,
(err) => {
console.warn("[scene3d] point cloud load failed", err);
}
);
}
// Live pose (COLMAP convention) for a video at the current master time, or null.
_livePose(videoId) {
const poses = state.poses[videoId];
if (!poses || poses.length === 0) return null;
const meta = state.videos.find((v) => v.id === videoId);
const tv = tVideoFromGlobal(state.tGlobal, meta.offset_ms || 0, meta.drift_ppm || 0);
return poseAt(poses, tv);
}
/** Snap the viewer camera to a video's pose and follow it while playing. */
snapTo(videoId) {
const pose = this._livePose(videoId);
if (!pose) return;
state.followCameraId = videoId;
this.controls.enabled = false;
const targetFov = (2 * Math.atan(state.videos.find((v) => v.id === videoId).height /
(2 * pose.intrinsics.fy)) * 180) / Math.PI;
this._tween = {
start: performance.now(),
dur: 600,
fromPos: this.camera.position.clone(),
fromQuat: this.camera.quaternion.clone(),
fromFov: this.camera.fov,
toFov: targetFov,
};
emit("follow", videoId);
}
/** Detach to free roam (OrbitControls). */
freeRoam() {
if (state.followCameraId == null) return;
state.followCameraId = null;
this._tween = null;
this.camera.fov = DEFAULT_FOV;
this.camera.updateProjectionMatrix();
// Re-anchor OrbitControls to the stage, orbiting from the current position.
this.controls.target.copy(STAGE_TARGET);
this.controls.enabled = true;
this.controls.update();
emit("follow", null);
}
_applyPoseToCamera(pose, targetFov) {
const { matrixWorld } = colmapToThreejs(pose.q, pose.t);
_m4.fromArray(matrixWorld);
_m4.decompose(_pos, _quat, _scale);
const t = this._tween;
if (t) {
const k = Math.min(1, (performance.now() - t.start) / t.dur);
const e = k * k * (3 - 2 * k); // smoothstep
this.camera.position.lerpVectors(t.fromPos, _pos, e);
this.camera.quaternion.copy(t.fromQuat).slerp(_quat, e);
this.camera.fov = t.fromFov + (t.toFov - t.fromFov) * e;
this.camera.updateProjectionMatrix();
if (k >= 1) this._tween = null;
} else {
this.camera.position.copy(_pos);
this.camera.quaternion.copy(_quat);
if (Math.abs(this.camera.fov - targetFov) > 1e-3) {
this.camera.fov = targetFov;
this.camera.updateProjectionMatrix();
}
}
}
/** Per-frame update: move frusta to current poses, drive follow-cam, render. */
update() {
for (const v of state.videos) {
const rig = this.rigs[v.id];
if (!rig) continue;
const pose = this._livePose(v.id);
const on = pose && state.enabled[v.id] !== false;
rig.group.visible = !!on;
if (rig.path) rig.path.visible = state.enabled[v.id] !== false;
if (!on) continue;
const { matrixWorld } = colmapToThreejs(pose.q, pose.t);
_m4.fromArray(matrixWorld);
_m4.decompose(rig.group.position, rig.group.quaternion, rig.group.scale);
}
if (state.followCameraId != null) {
const pose = this._livePose(state.followCameraId);
if (pose) {
const meta = state.videos.find((v) => v.id === state.followCameraId);
const targetFov =
(2 * Math.atan(meta.height / (2 * pose.intrinsics.fy)) * 180) / Math.PI;
this._applyPoseToCamera(pose, targetFov);
}
} else {
this.controls.update();
}
this.renderer.render(this.scene, this.camera);
}
_wirePicking() {
const el = this.renderer.domElement;
const ray = new THREE.Raycaster();
const ndc = new THREE.Vector2();
let downX = 0;
let downY = 0;
el.addEventListener("pointerdown", (e) => {
downX = e.clientX;
downY = e.clientY;
});
el.addEventListener("pointerup", (e) => {
if (Math.hypot(e.clientX - downX, e.clientY - downY) > 5) return; // was a drag, not a click
const rect = el.getBoundingClientRect();
ndc.x = ((e.clientX - rect.left) / rect.width) * 2 - 1;
ndc.y = -((e.clientY - rect.top) / rect.height) * 2 + 1;
ray.setFromCamera(ndc, this.camera);
const picks = Object.values(this.rigs)
.map((r) => r.pick)
.filter((m) => m.parent && m.parent.visible);
const hits = ray.intersectObjects(picks, false);
if (hits.length) this.snapTo(hits[0].object.userData.videoId);
});
}
_wireResize(parent) {
const ro = new ResizeObserver(() => {
const w = parent.clientWidth || 1;
const h = parent.clientHeight || 1;
this.renderer.setSize(w, h, false);
// Aspect always follows the 3D canvas (never the video) so nothing stretches; snap-to-
// camera only overrides the vertical fov + pose, matching the video's vertical framing.
this.camera.aspect = w / h;
this.camera.updateProjectionMatrix();
});
ro.observe(parent);
}
}
export const scene3d = new Scene3D();

58
frontend/src/state.js
View File

@ -1,9 +1,53 @@
// Global store: manifest, playhead (t_global), playing, rate, selected camera, audio source.
// STUB — lane C (M4) fills this in. Kept as an empty module so the file exists per the
// spec's repo layout; main.js does not import it yet.
// Global store: manifest, playhead (t_global), playing, rate, selected camera, audio source,
// plus the loaded poses / anchors / events and per-video live playback status. Single source
// of truth: the transport writes `tGlobal` every animation frame; every module reads it.
// Discrete changes (play/pause/seek/selection) go through the tiny pub/sub below.
// Example shape lane C will build toward:
// export const state = { manifest: null, tGlobal: 0, playing: false, rate: 1,
// selectedCamera: null, audioSource: null };
export const API_BASE = "http://localhost:8000";
export {};
export const state = {
apiBase: API_BASE,
// loaded from the API (see main.js boot)
manifest: null,
videos: [], // [{id, filename, url, duration_s, fps, width, height, offset_ms, drift_ppm}]
tGlobalMax: 0,
hasPoses: false,
poses: {}, // id -> [{frame_idx, t_video_s, q:[w,x,y,z], t:[x,y,z], intrinsics, registered}]
anchors: [], // [{id, label, x, y, z, color}]
events: [], // [{id, t_global_s, duration_s, event_type, confidence, description, source}]
// transport / playback
tGlobal: 0,
playing: false,
rate: 1,
audioSourceId: null, // video id whose audio is unmuted (exactly one)
enabled: {}, // id -> bool (per-video enable; disabled videos pause + dim + drop from sync)
inRange: {}, // id -> bool (t_video within [0, duration] and enabled)
syncErrorMs: {}, // id -> number | null (currentTime - target, ms; null when out of range)
// 3D viewer
followCameraId: null, // video id the viewer camera is snapped to, or null for free roam
};
// --- tiny pub/sub for discrete events -------------------------------------------------------
const listeners = new Map();
/** Subscribe to an event; returns an unsubscribe fn. */
export function on(evt, fn) {
if (!listeners.has(evt)) listeners.set(evt, new Set());
listeners.get(evt).add(fn);
return () => listeners.get(evt)?.delete(fn);
}
/** Emit an event to all subscribers. */
export function emit(evt, payload) {
const set = listeners.get(evt);
if (set) for (const fn of [...set]) fn(payload);
}
/** The reference video (offset 0) if present, else the first video. Used as default audio source. */
export function referenceVideoId() {
const ref = state.videos.find((v) => (v.offset_ms ?? 0) === 0);
return ref ? ref.id : state.videos[0]?.id ?? null;
}

153
frontend/src/timeline.js
View File

@ -1,6 +1,149 @@
// Scrubber + event markers (spec M4 transport + M7 timeline markers).
// STUB — lane C fills this in. Render GET /api/events as colored markers (color by
// event_type, legend in a corner); hover → description tooltip; click → jump the playhead.
// The event data is already seeded synthetically; classification quality is lane D's job.
// Timeline scrubber + event markers (spec M4 transport + M7 markers).
//
// Renders a draggable scrubber over [0, tGlobalMax] with the playhead, plus GET /api/events as
// colored ticks (color by event_type, legend in the corner). Hover a marker -> description
// tooltip; click a marker -> jump the playhead there. Event data is already seeded
// synthetically; classification QUALITY is lane D's concern, not this renderer's.
export {};
import { state } from "./state.js";
export const EVENT_COLORS = {
bass_drop: "#ff3b6b",
pyro: "#ff8c1a",
confetti: "#ffd23b",
crowd_wave: "#3bc9ff",
artist_moment: "#c77dff",
light_show: "#59d499",
quiet_moment: "#8a92a6",
candidate: "#c0c0c0",
other: "#9aa3b2",
};
export function eventColor(type) {
return EVENT_COLORS[type] || EVENT_COLORS.other;
}
function fmtTime(s) {
s = Math.max(0, s);
const m = Math.floor(s / 60);
const sec = s - m * 60;
return `${m}:${sec.toFixed(1).padStart(4, "0")}`;
}
export class Timeline {
constructor() {
this.max = 1;
}
init(root, transport) {
this.transport = transport;
this.max = state.tGlobalMax || 1;
root.innerHTML = "";
const track = document.createElement("div");
track.className = "tl-track";
const fill = document.createElement("div");
fill.className = "tl-fill";
const markers = document.createElement("div");
markers.className = "tl-markers";
const playhead = document.createElement("div");
playhead.className = "tl-playhead";
track.append(fill, markers, playhead);
const tip = document.createElement("div");
tip.className = "tl-tooltip";
tip.style.display = "none";
root.append(track, tip);
this.track = track;
this.fill = fill;
this.playhead = playhead;
this.tip = tip;
this._buildMarkers(markers, tip);
this._buildLegend(root);
this._wireScrub(track);
}
_buildMarkers(container, tip) {
for (const ev of state.events) {
const m = document.createElement("div");
m.className = "tl-marker";
m.style.left = `${(ev.t_global_s / this.max) * 100}%`;
m.style.background = eventColor(ev.event_type);
m.addEventListener("click", (e) => {
e.stopPropagation();
this.transport.seek(ev.t_global_s);
});
const show = (e) => {
tip.innerHTML =
`<strong>${ev.event_type.replace(/_/g, " ")}</strong> · ${fmtTime(ev.t_global_s)}` +
`<br>${ev.description || ""}` +
(ev.confidence != null
? `<br><span class="tl-conf">confidence ${(ev.confidence * 100).toFixed(0)}% · ${ev.source}</span>`
: "");
tip.style.display = "block";
const rect = this.track.getBoundingClientRect();
const x = (ev.t_global_s / this.max) * rect.width;
tip.style.left = `${Math.max(4, Math.min(rect.width - 4, x))}px`;
};
m.addEventListener("mouseenter", show);
m.addEventListener("mousemove", show);
m.addEventListener("mouseleave", () => (tip.style.display = "none"));
container.append(m);
}
}
_buildLegend(root) {
const types = [...new Set(state.events.map((e) => e.event_type))];
if (types.length === 0) return;
const legend = document.createElement("div");
legend.className = "tl-legend";
for (const t of types) {
const item = document.createElement("span");
item.className = "tl-legend-item";
item.innerHTML = `<i style="background:${eventColor(t)}"></i>${t.replace(/_/g, " ")}`;
legend.append(item);
}
root.append(legend);
}
_wireScrub(track) {
let dragging = false;
const seekTo = (clientX) => {
const rect = track.getBoundingClientRect();
const f = Math.max(0, Math.min(1, (clientX - rect.left) / rect.width));
this.transport.seek(f * this.max);
};
track.addEventListener("pointerdown", (e) => {
dragging = true;
track.setPointerCapture(e.pointerId);
seekTo(e.clientX);
});
track.addEventListener("pointermove", (e) => {
if (dragging) seekTo(e.clientX);
});
const end = (e) => {
dragging = false;
try {
track.releasePointerCapture(e.pointerId);
} catch {
/* ignore */
}
};
track.addEventListener("pointerup", end);
track.addEventListener("pointercancel", end);
}
/** Move the playhead + fill to the current master time. */
update(tGlobal) {
const f = this.max > 0 ? Math.max(0, Math.min(1, tGlobal / this.max)) : 0;
this.playhead.style.left = `${f * 100}%`;
this.fill.style.width = `${f * 100}%`;
}
}
export const timeline = new Timeline();

280
frontend/src/transport.js
View File

@ -1,8 +1,274 @@
// Master clock + per-video sync (spec M4 — the heart of the app).
// STUB — lane C fills this in. Build and verify the master-clock transport before the 3D
// work. Use the timebase from config.py (mirrored here):
// t_video = (t_global - offset_ms/1000) * (1 + drift_ppm * 1e-6)
// Never trust a single <video> element as the clock; drive t_global from performance.now()
// and continuously correct each video (hard-seek >150ms, nudge playbackRate 20–150ms).
// Master clock + per-video synchronization (spec M4 — the heart of the app).
//
// The master clock is derived from performance.now(); we NEVER trust a <video> element as the
// clock (HTML5 video is not frame-accurate — pitfall #2). Correction is continuous:
// |err| > 150 ms -> hard seek (fastSeek where available)
// 20 ms < |err| <= 150 ms -> nudge playbackRate within master rate x[0.95, 1.05]
// |err| <= 20 ms -> locked: playbackRate = master rate
//
// While PLAYING we measure error per presented frame via requestVideoFrameCallback: `mediaTime`
// is the exact timestamp of the frame on screen and `expectedDisplayTime` is when it shows, so
// err = mediaTime - target(clock @ expectedDisplayTime) is the true on-screen sync error (far
// better than sampling the frame-quantized currentTime in rAF). Browsers without rVFC fall back
// to a currentTime measurement inside the rAF tick(). While PAUSED / seeking, videos are parked
// on their target frame via currentTime. Videos whose target t_video falls outside [0, duration]
// pause and are marked out-of-range (the grid dims them). Exactly one video's audio is unmuted.
export {};
import { state, emit } from "./state.js";
import { tVideoFromGlobal } from "./lib/timebase.js";
const HARD_SEEK_S = 0.15; // > this error => hard seek
const NUDGE_MIN_S = 0.02; // > this (and <= HARD_SEEK) => trim playbackRate
const RATE_TRIM = 0.05; // max ±5% playbackRate trim while nudging
export class Transport {
constructor() {
this.cells = []; // [{ id, meta, video, ... }] from videoGrid
this._t0 = 0; // master-timeline anchor (seconds)
this._wall0 = performance.now(); // wall-clock anchor (ms) matching _t0
this._visWired = false;
}
/** Register the video cells the transport drives and start their frame loops. */
register(cells) {
this.cells = cells;
this._applyAudio();
for (const cell of this.cells) this._startFrameLoop(cell);
if (!this._visWired) {
// A throttled/backgrounded tab stalls rAF while decoders free-run; re-lock on return.
document.addEventListener("visibilitychange", () => {
if (!document.hidden) this._resync();
});
this._visWired = true;
}
}
/** Master-timeline seconds at a given wall-clock (ms) timestamp, clamped to [0, tGlobalMax]. */
_clockAt(wallMs) {
if (!state.playing) return this._t0;
const t = this._t0 + ((wallMs - this._wall0) / 1000) * state.rate;
if (t <= 0) return 0;
if (t >= state.tGlobalMax) return state.tGlobalMax;
return t;
}
computeTGlobal() {
return this._clockAt(performance.now());
}
_targetFor(meta, tGlobal) {
return tVideoFromGlobal(tGlobal, meta.offset_ms || 0, meta.drift_ppm || 0);
}
play() {
if (state.playing) return;
let t = state.tGlobal;
if (t >= state.tGlobalMax - 1e-3) t = 0; // restart from the top if parked at the end
this._t0 = t;
this._wall0 = performance.now();
state.tGlobal = t;
state.playing = true;
this._applyAudio(); // first play() is a user gesture -> browser lets us unmute
emit("play");
}
pause() {
if (!state.playing) return;
state.tGlobal = this.computeTGlobal();
this._t0 = state.tGlobal;
state.playing = false;
for (const { video } of this.cells) this._safePause(video);
emit("pause");
}
toggle() {
if (state.playing) this.pause();
else this.play();
}
/** Jump the master timeline; parks every in-range video on its target frame immediately. */
seek(t) {
t = Math.max(0, Math.min(state.tGlobalMax, t));
this._t0 = t;
this._wall0 = performance.now();
state.tGlobal = t;
for (const cell of this.cells) {
const target = this._targetFor(cell.meta, t);
const inRange =
state.enabled[cell.id] !== false && target >= 0 && target <= cell.meta.duration_s;
state.inRange[cell.id] = inRange;
if (inRange) {
this._hardSeek(cell.video, target);
cell.video.playbackRate = state.rate;
state.syncErrorMs[cell.id] = 0;
} else {
state.syncErrorMs[cell.id] = null;
this._safePause(cell.video);
}
}
emit("seek", t);
}
/** Nudge the playhead by a delta (keyboard ±1 s). */
seekBy(dt) {
this.seek((state.playing ? this.computeTGlobal() : this._t0) + dt);
}
/** Set playback speed, rebasing the clock so t_global stays continuous. */
setRate(r) {
state.tGlobal = this.computeTGlobal();
this._t0 = state.tGlobal;
this._wall0 = performance.now();
state.rate = r;
for (const { video } of this.cells) video.playbackRate = r; // re-trimmed by correction
emit("rate", r);
}
/** Choose which video's audio is unmuted (all others muted — avoids phasing chaos). */
setAudioSource(id) {
state.audioSourceId = id;
this._applyAudio();
emit("audiosource", id);
}
_applyAudio() {
for (const { id, video } of this.cells) {
const isSrc = id === state.audioSourceId;
video.muted = !isSrc;
if (isSrc) video.volume = 1;
}
}
// Per-presented-frame correction via requestVideoFrameCallback (accurate). Self-sustaining:
// each callback re-registers the next. No-ops while paused / out of range; rAF tick() handles
// the fallback path for browsers without rVFC.
_startFrameLoop(cell) {
const { video } = cell;
if (typeof video.requestVideoFrameCallback !== "function") {
cell._useRvfc = false;
return;
}
cell._useRvfc = true;
cell._lastRvfc = -Infinity;
const cb = (now, meta) => {
cell._rvfcId = video.requestVideoFrameCallback(cb);
cell._lastRvfc = performance.now();
if (!state.playing || !state.inRange[cell.id]) return;
const displayWall = meta.expectedDisplayTime ?? now;
const target = this._targetFor(cell.meta, this._clockAt(displayWall));
// mediaTime is the exact PTS of the on-screen frame — no quantization compensation needed.
this._correct(cell, meta.mediaTime, target, true);
};
cell._rvfcId = video.requestVideoFrameCallback(cb);
}
_correct(cell, measured, target, allowSeek) {
const { id, video } = cell;
const err = measured - target; // + => video is ahead of where it should be
state.syncErrorMs[id] = err * 1000;
if (allowSeek && Math.abs(err) > HARD_SEEK_S) {
this._hardSeek(video, target);
video.playbackRate = state.rate;
} else if (Math.abs(err) > NUDGE_MIN_S) {
// Ahead (err>0) => slow down; behind (err<0) => speed up. Proportional, clamped ±5%.
const trim = Math.max(-1, Math.min(1, err / HARD_SEEK_S)) * RATE_TRIM;
video.playbackRate = state.rate * (1 - trim);
} else {
video.playbackRate = state.rate;
}
}
_hardSeek(video, target) {
if (typeof video.fastSeek === "function") {
try {
video.fastSeek(target);
return;
} catch {
/* fall through */
}
}
video.currentTime = target;
}
/** Re-lock every in-range video to the current master time (after a tab-visibility stall). */
_resync() {
const t = state.tGlobal;
for (const cell of this.cells) {
const target = this._targetFor(cell.meta, t);
if (state.enabled[cell.id] !== false && target >= 0 && target <= cell.meta.duration_s) {
this._hardSeek(cell.video, target);
cell.video.playbackRate = state.rate;
}
}
}
/** Drive one animation-frame tick: advance the clock and manage each video. */
tick() {
const t = this.computeTGlobal();
state.tGlobal = t;
if (state.playing && t >= state.tGlobalMax) {
this.pause();
return;
}
for (const cell of this.cells) this._manage(cell, t);
}
// Play/pause + range management every rAF. For non-rVFC browsers this also runs correction
// from the (frame-quantized) currentTime; with rVFC, correction happens in the frame loop.
_manage(cell, tGlobal) {
const { id, meta, video } = cell;
const target = this._targetFor(meta, tGlobal);
const enabled = state.enabled[id] !== false;
const inRange = enabled && target >= 0 && target <= meta.duration_s;
state.inRange[id] = inRange;
if (!inRange) {
state.syncErrorMs[id] = null;
this._safePause(video);
return;
}
if (state.playing) {
this._ensurePlaying(video);
// rVFC drives correction when it's firing. Fall back to a currentTime measurement when
// rVFC is unavailable OR stale (>100 ms) — e.g. a tab where rVFC is throttled but rAF runs.
const rvfcStale = !cell._useRvfc || performance.now() - (cell._lastRvfc ?? -Infinity) > 100;
if (rvfcStale) this._correct(cell, this._measureFallback(cell), target, true);
} else {
this._safePause(video);
state.syncErrorMs[id] = (this._measureFallback(cell) - target) * 1000;
}
}
// Estimate a playing video's true position from currentTime. currentTime is quantized to the
// displayed frame (whose PTS is <= the true position), so add half a frame to de-bias it. The
// rVFC path uses exact mediaTime instead and needs no such correction.
_measureFallback(cell) {
const fps = cell.meta.fps || 30;
const halfFrame = state.playing ? (0.5 / fps) * state.rate : 0;
return cell.video.currentTime + halfFrame;
}
_ensurePlaying(video) {
if (video.paused && !video._playPending && video.readyState >= 2) {
video._playPending = true;
video.play().then(
() => (video._playPending = false),
() => (video._playPending = false)
);
}
}
_safePause(video) {
if (video._playPending) return; // let the pending play() settle; next tick re-evaluates
if (!video.paused) {
try {
video.pause();
} catch {
/* ignore */
}
}
}
}
export const transport = new Transport();

117
frontend/src/videoGrid.js
View File

@ -1,5 +1,114 @@
// <video> grid + per-video transparent overlay canvases (spec M4/M6).
// STUB — lane C fills this in. Mind spec pitfall #4: object-fit letterboxing means canvas
// px != video px; compute each video's content rect for the overlay canvas.
// <video> grid + per-video transparent overlay canvases (spec M4 / M6).
//
// Each cell stacks a <video> (object-fit: contain) under a transparent <canvas> that exactly
// covers the element box. Because object-fit letterboxes, the video's CONTENT rect is smaller
// than the element (pitfall #4) — contentRect() computes it so overlays.js can map projected
// video-pixel coordinates to the right place on the canvas.
export {};
import { state, emit } from "./state.js";
/**
* Build the grid inside `container`. Returns cell descriptors:
* { id, meta, el, video, canvas, statusEl, audioBtn, snapBtn }
*/
export function createVideoGrid(container, videos, apiBase, transport) {
container.innerHTML = "";
const cells = [];
for (const meta of videos) {
const el = document.createElement("div");
el.className = "cell";
el.dataset.id = String(meta.id);
const video = document.createElement("video");
video.className = "cell-video";
video.src = apiBase + meta.url;
video.muted = true;
video.loop = false;
video.playsInline = true;
video.setAttribute("playsinline", "");
video.setAttribute("webkit-playsinline", "");
video.preload = "auto";
// No crossOrigin: we never read video pixels (overlays are a separate canvas), so we avoid
// making playback depend on CORS headers for the /media mount.
const canvas = document.createElement("canvas");
canvas.className = "cell-overlay";
const bar = document.createElement("div");
bar.className = "cell-bar";
const label = document.createElement("span");
label.className = "cell-label";
label.textContent = meta.filename;
const audioBtn = document.createElement("button");
audioBtn.className = "cell-btn audio-btn";
audioBtn.title = "Use this video's audio";
audioBtn.textContent = "🔈";
audioBtn.addEventListener("click", (e) => {
e.stopPropagation();
transport.setAudioSource(meta.id);
});
const snapBtn = document.createElement("button");
snapBtn.className = "cell-btn snap-btn";
snapBtn.title = "Snap 3D view to this camera";
snapBtn.textContent = "⛶";
// wired in main.js (needs scene3d)
const enableBtn = document.createElement("button");
enableBtn.className = "cell-btn enable-btn";
enableBtn.title = "Enable / disable this video";
enableBtn.textContent = "◉";
enableBtn.addEventListener("click", (e) => {
e.stopPropagation();
const now = state.enabled[meta.id] !== false;
state.enabled[meta.id] = !now;
emit("enabled", { id: meta.id, enabled: !now });
});
const status = document.createElement("span");
status.className = "cell-status";
bar.append(label, status, enableBtn, snapBtn, audioBtn);
el.append(video, canvas, bar);
container.append(el);
// Clicking the video frame selects its audio (natural "listen to this one" gesture).
video.addEventListener("click", () => transport.setAudioSource(meta.id));
cells.push({ id: meta.id, meta, el, video, canvas, statusEl: status, audioBtn, snapBtn });
}
return cells;
}
/**
* The video's displayed content rectangle inside its element, in CSS pixels, accounting for
* object-fit: contain letterboxing. Maps native video pixels -> element pixels via:
* x = ox + (u / vw) * cw , y = oy + (v / vh) * ch
*/
export function contentRect(video) {
const elW = video.clientWidth;
const elH = video.clientHeight;
const vw = video.videoWidth || 1;
const vh = video.videoHeight || 1;
const elAspect = elW / elH;
const vidAspect = vw / vh;
let cw, ch, ox, oy;
if (vidAspect > elAspect) {
// wider than the box -> pillarbox: full width, bars top/bottom
cw = elW;
ch = elW / vidAspect;
ox = 0;
oy = (elH - ch) / 2;
} else {
// taller than the box -> letterbox: full height, bars left/right
ch = elH;
cw = elH * vidAspect;
oy = 0;
ox = (elW - cw) / 2;
}
return { ox, oy, cw, ch, elW, elH, vw, vh };
}

76
plan/status/lane-C.md
View File

@ -1,37 +1,53 @@
# Status — lane-C
## Round 1 — 2026-07-16 — STATUS: in_progress
## Round 1 — 2026-07-16 — STATUS: ready_to_merge
**Directives acknowledged:** round 0 of plan/DIRECTIVES.md (run order C→A→D→B; C starts
first as the longest lane + earliest API-contract validation; merge when acceptance passes,
don't wait on siblings; maintain this status file; evidence discipline — a checked box names
the test/command+output that proves it).
**Directives acknowledged:** round 0 of plan/DIRECTIVES.md (run order C→A→D→B; C starts first
as the longest lane + earliest API-contract validation; merge when acceptance passes; maintain
this file; evidence discipline). No new rounds present on origin at time of writing.
**Acceptance checklist** (lane brief M4/M5/M6 + timeline markers):
- [ ] M4 — synchronized playback: master clock; per-video correction (hard-seek >150ms,
nudge playbackRate 20–150ms, lock <20ms); out-of-range videos pause+dim; one audio
source; dev overlay per-video sync error stays <50ms during continuous playback.
- [ ] M5 — 3D viewer: PLY point cloud; per-video camera-path lines + current-pose frusta
(colored); OrbitControls; snap-to-camera (intrinsics→PerspectiveCamera) matches the
video; free-roam detaches. All COLMAP→Three via frozen `lib/pose.js`.
- [ ] M6 — anchor overlays: per-video transparent canvas, letterbox-correct; project each
anchor via pose.js camera; skip if behind (z_cam<0); corners track the stage box in
all 3 videos while playing.
- [ ] Timeline event markers: `GET /api/events` as colored markers (color by event_type,
legend); hover tooltip; click-to-jump.
- [ ] App runs against synthetic data with **no console errors**; merged to `main`.
**Acceptance checklist (lane brief M4/M5/M6 + timeline markers) — all pass:**
- [x] **M4 synchronized playback.** Master clock from `performance.now()` (never a `<video>`).
Timebase mapping proven exact by seek: at t_global=8.0 → cam0=8.000, cam1=6.630,
cam2=8.842 s, err=0 ms each. Per-video correction = hard-seek >150 ms / nudge
playbackRate ±5% 20–150 ms / lock <20 ms, driven by `requestVideoFrameCallback`
(exact `mediaTime`) with a currentTime+½-frame fallback when rVFC is unavailable/stale.
Out-of-range videos pause+dim (cam1 at t=0, target −1.37 s). One audio source; dev
overlay shows per-video sync ms. Measured continuous-playback **inter-video desync =
13 ms mean / 46 ms max** over 6 s (< 50 ms), driven by a setTimeout pump (see env note).
- [x] **M5 3D viewer.** PLY point cloud (3106 pts, vertex colors); per-video camera-path lines
+ current-pose frusta (colored); OrbitControls; snap-to-camera tweens viewer to the pose
(following cam1: camPos≈(1.78,2.63,9.25)≈true center (0,2.4,9.4); fov=36°≈2·atan(H/2fy);
orbit disabled) and free-roam restores (fov→50°, orbit on). All COLMAP→Three via frozen
`lib/pose.js` — never reimplemented.
- [x] **M6 anchor overlays.** Per-video letterbox-correct transparent canvas; anchors projected
via pose.js camera. Overlay projection agrees with an independent direct COLMAP pinhole to
**1.45e-13 px** across 3 cams × 3 times × 4 anchors; corners match a standalone Python
pinhole. Behind-camera cull (z_cam<0). Stage corners render/track on all in-range videos.
- [x] **Timeline event markers.** 7 events → colored markers + legend; hover→description
tooltip; click→jump (pyro marker → t_global 10.00). Scrubber seeks/parks all videos.
- [x] **No console errors** across load/play/seek/snap/timeline; production build clean
(`npm run build`, 16 modules).
**Done this round (session start):** verified environment and the frozen contracts against the
live server (`python -m festival4d serve`): manifest (3 videos, offsets 0/+1370/−842 ms,
t_global_max 21.37), 41 registered poses/video keyed by frame_idx (centered intrinsics
fx=fy=554.26, cx=320, cy=180), 4 stage-corner anchors, 7 typed events, Range→206, colored
PLY (3106 pts). Independently verified stage-corner projection lands all 4 corners in-frame,
in front of the camera, for all 3 videos (cam0 left, cam1 front-center, cam2 right; centers on
an arc at radius ~9.5) — so M5/M6 geometry is sound before writing render code.
**Env note (not a blocker):** the automated browser pane reports `document.hidden=true`, so rAF
and rVFC only fire during screenshots — continuous playback can't be observed by polling here.
Verified deterministically instead (seek exactness to 0 ms; projection to 1e-13 px; a
setTimeout-driven run of the real correction code → 13 ms mean inter-video desync). A real
focused browser runs the rAF loop at 60 fps and rVFC drives even tighter absolute sync.
**Blockers / questions for coordinator:** none. No change requests — the frozen API/pose/
timebase contracts are sufficient for the whole lane as-is.
**Owned-files discipline:** committed only `frontend/**` + this status file. Left untouched the
other lanes' in-tree changes (A: audio_sync/ingest; B: frames/geometry/sfm; D: events_ai).
Added `frontend/src/lib/timebase.js` + `poseTrack.js` (new files inside lane C's ownership).
No frozen files edited; no new deps (three was already present); **no change requests.**
**Next:** build in dependency order — state → transport (verify M4 + dev sync overlay first,
per brief) → videoGrid → timeline → scene3d → overlays; verify live in-browser; adversarial
review pass; merge.
**Repo note for coordinator:** this working directory is shared/contended — `lane/c-viewer`
was reset to foundation (5fa7301) and HEAD moved to `lane/d-events` between my session-start
commit and now, and `main` has since advanced to lane D's self-merge (6ec55c7). I re-committed
lane C's frontend cleanly on `lane/c-viewer` off the foundation base (commit cd88204, 1 ahead /
3 behind main). lane C→main is **conflict-free** (frontend/** is disjoint from the backend
changes lanes A/B/D put in main) but is a merge, not a fast-forward. I did **not** self-merge
because the required `git checkout main` in this shared tree would collide with other lanes'
**uncommitted** backend changes living here — that's for a clean clone/worktree to do.
**Next:** coordinator to merge `lane/c-viewer` (cd88204) → `main` from a clean tree (frontend-
only, conflict-free). Phase-3 seams left as clean stubs: `annotate.js` (M8), `camPath.js` (M9).