Live capture: USB webcams, laptop cams, and phones -> data/raw
Records rather than streams: each device records a clip WITH AUDIO and uploads
it; the normal offline pipeline takes over. Fits the design almost for free —
MediaRecorder's webm/mp4 are already ingestible, and audio-based sync means
devices need no clock sync and needn't start together.
- capture.py: POST /api/capture/upload (streamed, sanitized name, size cap,
partial cleanup), heartbeat/devices registry, GET /capture page.
- static/capture.html: standalone page (no build step) — device picker, live
preview, record->upload, 1fps snapshot heartbeat, 'who's shooting what'
monitor, and secure-context detection that tells you to use
(getUserMedia needs HTTPS — the #1 gotcha for phones).
- OPT-IN via FESTIVAL4D_CAPTURE=1; manifest reports has_capture and the viewer
only then shows a Capture link. Public deploy leaves it off (open upload
endpoint would be unsafe) — noted in DEPLOY.md.
- Adds python-multipart dep. Suite 103 -> 119.
Also fixes test_manifest_shape, which was already RED on main: f12b6e0 added
has_splat to the manifest without updating the exact-set lock. Updated to the
true set (now also has_capture); the lock is what caught both drifts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
m3ultra
parent
f12b6e0847
commit
36bacf67f0
48
README.md
48
README.md
@ -67,6 +67,54 @@ moment markers on the timeline.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Capture: record straight from webcams & phones
|
||||||
|
|
||||||
|
Don't have footage yet? Record it with whatever cameras you have — USB webcams, the laptop cam,
|
||||||
|
and any phone with a browser. **No app install.** Cameras *record* clips that feed the normal
|
||||||
|
pipeline (this isn't live streaming — see `docs/ideas.md` for that).
|
||||||
|
|
||||||
|
This works because of two properties of the design: browser recordings (`.webm`, or `.mp4` on
|
||||||
|
iOS) are already ingestible formats, and **sync is audio-based** — so the devices need **no clock
|
||||||
|
sync and don't even have to start together**. They only have to record the same moment and hear
|
||||||
|
the same sound.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# capture is OPT-IN (it accepts uploads) — enable it explicitly:
|
||||||
|
FESTIVAL4D_CAPTURE=1 uv run python -m festival4d serve
|
||||||
|
```
|
||||||
|
|
||||||
|
Then open **`/capture`** on each device (a 📹 Capture link also appears in the viewer): name the
|
||||||
|
device, pick a camera + mic, hit record, hit stop — the clip uploads into `data/raw/`. The page
|
||||||
|
shows a live "who's shooting what" monitor of every connected camera. When you're done:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python -m festival4d ingest && uv run python -m festival4d sync # then reconstruct / events
|
||||||
|
```
|
||||||
|
|
||||||
|
### Getting a phone connected (the HTTPS bit)
|
||||||
|
|
||||||
|
Browsers **only grant camera access on a secure origin**. A phone opening `http://<lan-ip>:8000`
|
||||||
|
gets *no camera*, silently. The easiest fix if you use Tailscale — it issues a real cert, so
|
||||||
|
there's nothing to trust manually:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tailscale serve --bg 8000 # prints https://<machine>.<tailnet>.ts.net
|
||||||
|
```
|
||||||
|
|
||||||
|
Join the phone to the tailnet and open `https://<machine>.<tailnet>.ts.net/capture`. (Without
|
||||||
|
Tailscale: any HTTPS reverse proxy works; `localhost` is also treated as secure, which is why the
|
||||||
|
laptop's own cameras work with no setup.)
|
||||||
|
|
||||||
|
### Capture tips
|
||||||
|
|
||||||
|
- **Every clip needs audio**, and all cameras must hear the same sound — that's the only thing
|
||||||
|
aligning them. Testing indoors? Play music.
|
||||||
|
- **USB bandwidth is the real limit** on multiple webcams: two 1080p cams on one controller often
|
||||||
|
fails. Drop to 720p or use separate ports/hubs.
|
||||||
|
- Follow the *Shooting tips* below for reconstruction-friendly angles.
|
||||||
|
- Capture routes stay **unmounted** unless `FESTIVAL4D_CAPTURE=1` — a public deployment must
|
||||||
|
never expose an open upload endpoint.
|
||||||
|
|
||||||
## Real-footage workflow
|
## Real-footage workflow
|
||||||
|
|
||||||
1. **Shoot / gather** 2–4 videos of the same performance from different positions (see
|
1. **Shoot / gather** 2–4 videos of the same performance from different positions (see
|
||||||
|
|||||||
@ -52,6 +52,14 @@ config.RAW_DIR.mkdir(parents=True, exist_ok=True)
|
|||||||
# HTTP Range-capable static serving of source videos (spec M3, pitfall #3).
|
# HTTP Range-capable static serving of source videos (spec M3, pitfall #3).
|
||||||
app.mount("/media", StaticFiles(directory=config.RAW_DIR), name="media")
|
app.mount("/media", StaticFiles(directory=config.RAW_DIR), name="media")
|
||||||
|
|
||||||
|
# Live capture (webcams / phones -> data/raw). OPT-IN: these routes accept file uploads, so they
|
||||||
|
# stay unmounted unless FESTIVAL4D_CAPTURE=1 — a public deployment must never expose them.
|
||||||
|
from festival4d import capture # noqa: E402 (import after app/config are ready)
|
||||||
|
|
||||||
|
if capture.capture_enabled():
|
||||||
|
app.include_router(capture.router)
|
||||||
|
log.info("capture routes ENABLED (/capture, /api/capture/*) — FESTIVAL4D_CAPTURE is set")
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Request models
|
# Request models
|
||||||
@ -145,6 +153,7 @@ def manifest() -> dict:
|
|||||||
return {
|
return {
|
||||||
"videos": [_video_dict(v) for v in videos],
|
"videos": [_video_dict(v) for v in videos],
|
||||||
"t_global_max": t_global_max,
|
"t_global_max": t_global_max,
|
||||||
|
"has_capture": capture.capture_enabled(),
|
||||||
"has_poses": db.has_poses(),
|
"has_poses": db.has_poses(),
|
||||||
"has_splat": config.SPLAT_PLY.exists(),
|
"has_splat": config.SPLAT_PLY.exists(),
|
||||||
}
|
}
|
||||||
|
|||||||
166
backend/festival4d/capture.py
Normal file
166
backend/festival4d/capture.py
Normal file
@ -0,0 +1,166 @@
|
|||||||
|
"""Live capture — USB webcams, laptop cams, and phones -> ``data/raw`` (Phase 5).
|
||||||
|
|
||||||
|
**Records, it doesn't stream.** Each device records a clip *with audio* and uploads it; the normal
|
||||||
|
offline pipeline (``ingest -> sync -> reconstruct -> events``) takes it from there. This fits the
|
||||||
|
app's design almost for free:
|
||||||
|
|
||||||
|
- Browser ``MediaRecorder`` emits WebM (Safari/iOS: MP4) — both are already in ingest's
|
||||||
|
``_VIDEO_EXTS``, so no format work is needed.
|
||||||
|
- Sync is **audio-based** (GCC-PHAT), so capture devices need **no clock sync and no
|
||||||
|
synchronized start** — they only have to record the same moment and hear the same sound.
|
||||||
|
Every recording MUST carry an audio track or ``sync`` has nothing to align.
|
||||||
|
|
||||||
|
A lightweight device registry powers the "who's shooting what" monitor: each device posts a
|
||||||
|
heartbeat with its status and a small JPEG snapshot (~1 fps), so the capture page can show every
|
||||||
|
camera without a WebRTC stack.
|
||||||
|
|
||||||
|
**Disabled by default.** These routes accept file uploads; they must never be exposed on a public
|
||||||
|
deployment. Set ``FESTIVAL4D_CAPTURE=1`` to mount them (see :func:`capture_enabled`).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import time
|
||||||
|
from datetime import datetime
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from fastapi import APIRouter, Form, HTTPException, UploadFile
|
||||||
|
from fastapi.responses import HTMLResponse
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
|
from festival4d import config
|
||||||
|
|
||||||
|
log = logging.getLogger("festival4d.capture")
|
||||||
|
|
||||||
|
router = APIRouter()
|
||||||
|
|
||||||
|
STATIC_DIR = Path(__file__).resolve().parent / "static"
|
||||||
|
|
||||||
|
# Containers a browser MediaRecorder can produce, intersected with what ingest accepts.
|
||||||
|
ALLOWED_EXTS = {".webm", ".mp4", ".mov", ".mkv"}
|
||||||
|
MAX_UPLOAD_BYTES = int(os.environ.get("FESTIVAL4D_MAX_UPLOAD_MB", "2048")) * 1024 * 1024
|
||||||
|
MAX_SNAPSHOT_CHARS = 300_000 # data-URL cap for a heartbeat thumbnail
|
||||||
|
DEVICE_STALE_S = 15.0 # drop devices that stop heartbeating
|
||||||
|
|
||||||
|
|
||||||
|
def capture_enabled() -> bool:
|
||||||
|
"""Capture routes are opt-in — they accept uploads, so never on by default."""
|
||||||
|
return os.environ.get("FESTIVAL4D_CAPTURE", "").strip().lower() in {"1", "true", "yes", "on"}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Device registry (in-memory, ephemeral — it only powers the live monitor)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
_devices: dict[str, dict] = {}
|
||||||
|
|
||||||
|
|
||||||
|
class Heartbeat(BaseModel):
|
||||||
|
device_id: str = Field(min_length=1, max_length=64)
|
||||||
|
name: str = Field(default="camera", max_length=64)
|
||||||
|
status: str = Field(default="idle", max_length=16) # idle | recording
|
||||||
|
elapsed_s: float = 0.0
|
||||||
|
snapshot: str | None = None # data:image/jpeg;base64,...
|
||||||
|
|
||||||
|
|
||||||
|
def _prune() -> None:
|
||||||
|
now = time.monotonic()
|
||||||
|
for did in [d for d, v in _devices.items() if now - v["_seen"] > DEVICE_STALE_S]:
|
||||||
|
_devices.pop(did, None)
|
||||||
|
|
||||||
|
|
||||||
|
def _slug(name: str) -> str:
|
||||||
|
"""Filesystem-safe slug from a user-supplied device name."""
|
||||||
|
s = re.sub(r"[^a-zA-Z0-9]+", "-", name).strip("-").lower()
|
||||||
|
return (s or "cam")[:40]
|
||||||
|
|
||||||
|
|
||||||
|
def _safe_raw_path(device_name: str, filename: str) -> Path:
|
||||||
|
"""Build a sanitized destination inside RAW_DIR. Never trusts the client filename."""
|
||||||
|
ext = Path(filename or "").suffix.lower()
|
||||||
|
if ext not in ALLOWED_EXTS:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=400,
|
||||||
|
detail=f"unsupported container '{ext or '?'}' (allowed: {sorted(ALLOWED_EXTS)})",
|
||||||
|
)
|
||||||
|
stamp = datetime.now().strftime("%Y%m%d-%H%M%S")
|
||||||
|
dest = (config.RAW_DIR / f"{_slug(device_name)}-{stamp}{ext}").resolve()
|
||||||
|
# Defence in depth: the slug can't traverse, but verify containment anyway.
|
||||||
|
if not str(dest).startswith(str(config.RAW_DIR.resolve()) + os.sep):
|
||||||
|
raise HTTPException(status_code=400, detail="invalid destination path")
|
||||||
|
return dest
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Routes
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
@router.get("/capture", response_class=HTMLResponse)
|
||||||
|
def capture_page() -> str:
|
||||||
|
"""The standalone capture page (open this on a phone over HTTPS — see README)."""
|
||||||
|
page = STATIC_DIR / "capture.html"
|
||||||
|
if not page.is_file():
|
||||||
|
raise HTTPException(status_code=500, detail="capture.html missing from the install")
|
||||||
|
return page.read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/api/capture/upload")
|
||||||
|
async def upload(file: UploadFile, device_name: str = Form("camera")) -> dict:
|
||||||
|
"""Stream an uploaded clip into ``data/raw`` under a sanitized name."""
|
||||||
|
dest = _safe_raw_path(device_name, file.filename or "")
|
||||||
|
config.RAW_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
written = 0
|
||||||
|
try:
|
||||||
|
with open(dest, "wb") as out:
|
||||||
|
while chunk := await file.read(1 << 20): # 1 MiB
|
||||||
|
written += len(chunk)
|
||||||
|
if written > MAX_UPLOAD_BYTES:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=413,
|
||||||
|
detail=f"upload exceeds {MAX_UPLOAD_BYTES // (1024*1024)} MB",
|
||||||
|
)
|
||||||
|
out.write(chunk)
|
||||||
|
except Exception:
|
||||||
|
dest.unlink(missing_ok=True) # never leave a partial file for ingest to trip on
|
||||||
|
raise
|
||||||
|
|
||||||
|
if written == 0:
|
||||||
|
dest.unlink(missing_ok=True)
|
||||||
|
raise HTTPException(status_code=400, detail="empty upload")
|
||||||
|
|
||||||
|
log.info("capture: saved %s (%.1f MB) from %r", dest.name, written / 1e6, device_name)
|
||||||
|
return {"filename": dest.name, "bytes": written, "next": "run `python -m festival4d ingest`"}
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/api/capture/heartbeat")
|
||||||
|
def heartbeat(hb: Heartbeat) -> dict:
|
||||||
|
"""Register/refresh a capture device and its latest thumbnail (powers the monitor)."""
|
||||||
|
snap = hb.snapshot
|
||||||
|
if snap and len(snap) > MAX_SNAPSHOT_CHARS:
|
||||||
|
snap = None # oversized thumbnail: keep the device, drop the image
|
||||||
|
entry = _devices.setdefault(hb.device_id, {})
|
||||||
|
entry.update(
|
||||||
|
{
|
||||||
|
"device_id": hb.device_id,
|
||||||
|
"name": hb.name,
|
||||||
|
"status": hb.status,
|
||||||
|
"elapsed_s": hb.elapsed_s,
|
||||||
|
"_seen": time.monotonic(),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
if snap:
|
||||||
|
entry["snapshot"] = snap
|
||||||
|
_prune()
|
||||||
|
return {"ok": True, "devices": len(_devices)}
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/api/capture/devices")
|
||||||
|
def devices() -> list[dict]:
|
||||||
|
"""Live capture devices (stale ones drop off after ~15 s)."""
|
||||||
|
_prune()
|
||||||
|
return [
|
||||||
|
{k: v for k, v in d.items() if not k.startswith("_")}
|
||||||
|
for d in sorted(_devices.values(), key=lambda d: d["name"])
|
||||||
|
]
|
||||||
276
backend/festival4d/static/capture.html
Normal file
276
backend/festival4d/static/capture.html
Normal file
@ -0,0 +1,276 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover" />
|
||||||
|
<title>Festival 4D — Capture</title>
|
||||||
|
<style>
|
||||||
|
:root {
|
||||||
|
color-scheme: dark;
|
||||||
|
--bg: #0b0d12; --panel: #141824; --panel-2: #1c2233; --border: #232838;
|
||||||
|
--text: #e6e8ee; --dim: #8a92a6; --ok: #59d499; --bad: #ff6b6b; --rec: #ff3b6b;
|
||||||
|
}
|
||||||
|
* { box-sizing: border-box; }
|
||||||
|
body {
|
||||||
|
margin: 0; background: var(--bg); color: var(--text); min-height: 100vh;
|
||||||
|
font: 15px/1.45 -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
|
||||||
|
padding: env(safe-area-inset-top) 0.8rem calc(env(safe-area-inset-bottom) + 0.8rem);
|
||||||
|
}
|
||||||
|
h1 { font-size: 1.1rem; margin: 0.8rem 0 0.2rem; }
|
||||||
|
.sub { color: var(--dim); font-size: 0.8rem; margin-bottom: 0.8rem; }
|
||||||
|
.card {
|
||||||
|
background: var(--panel); border: 1px solid var(--border); border-radius: 10px;
|
||||||
|
padding: 0.7rem; margin-bottom: 0.7rem;
|
||||||
|
}
|
||||||
|
label { display: block; color: var(--dim); font-size: 0.76rem; margin-bottom: 0.2rem; }
|
||||||
|
select, input[type="text"] {
|
||||||
|
width: 100%; background: var(--panel-2); color: var(--text); font-size: 1rem;
|
||||||
|
border: 1px solid var(--border); border-radius: 7px; padding: 0.5rem;
|
||||||
|
margin-bottom: 0.55rem;
|
||||||
|
}
|
||||||
|
#preview {
|
||||||
|
width: 100%; aspect-ratio: 16/9; background: #000; border-radius: 8px;
|
||||||
|
object-fit: contain; display: block;
|
||||||
|
}
|
||||||
|
.row { display: flex; gap: 0.5rem; align-items: center; }
|
||||||
|
button {
|
||||||
|
flex: 1 1 auto; font-size: 1rem; font-weight: 600; padding: 0.75rem 0.6rem;
|
||||||
|
border-radius: 9px; border: 1px solid var(--border); background: var(--panel-2);
|
||||||
|
color: var(--text); cursor: pointer;
|
||||||
|
}
|
||||||
|
button:disabled { opacity: 0.45; cursor: default; }
|
||||||
|
#rec.recording { background: var(--rec); border-color: var(--rec); color: #fff; }
|
||||||
|
.pill {
|
||||||
|
display: inline-flex; align-items: center; gap: 0.3rem; font-size: 0.75rem;
|
||||||
|
color: var(--dim);
|
||||||
|
}
|
||||||
|
.dot { width: 8px; height: 8px; border-radius: 50%; background: var(--dim); }
|
||||||
|
.dot.on { background: var(--rec); animation: pulse 1s infinite; }
|
||||||
|
@keyframes pulse { 50% { opacity: 0.25; } }
|
||||||
|
.msg { font-size: 0.82rem; margin-top: 0.5rem; line-height: 1.4; }
|
||||||
|
.err { color: var(--bad); }
|
||||||
|
.ok { color: var(--ok); }
|
||||||
|
code {
|
||||||
|
background: var(--panel-2); padding: 0.1rem 0.3rem; border-radius: 4px;
|
||||||
|
font-size: 0.85em; word-break: break-all;
|
||||||
|
}
|
||||||
|
/* monitor */
|
||||||
|
#mon { display: grid; grid-template-columns: repeat(auto-fill, minmax(104px, 1fr)); gap: 0.5rem; }
|
||||||
|
.mcell { background: var(--panel-2); border: 1px solid var(--border); border-radius: 7px; overflow: hidden; }
|
||||||
|
.mcell img { width: 100%; aspect-ratio: 16/9; object-fit: cover; display: block; background: #000; }
|
||||||
|
.mcell .cap { padding: 0.25rem 0.35rem; font-size: 0.68rem; display: flex; align-items: center; gap: 0.25rem; }
|
||||||
|
.mcell .nm { flex: 1 1 auto; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
||||||
|
.mcell.rec { border-color: var(--rec); }
|
||||||
|
#gate { display: none; }
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<h1>Festival 4D — Capture</h1>
|
||||||
|
<div class="sub">Record this camera. Clips upload to the project and sync by audio.</div>
|
||||||
|
|
||||||
|
<!-- Shown when getUserMedia is unavailable (insecure origin) -->
|
||||||
|
<div class="card" id="gate">
|
||||||
|
<div class="msg err"><b>Camera blocked — this page isn't a secure context.</b></div>
|
||||||
|
<div class="msg">
|
||||||
|
Browsers only grant camera access over <b>HTTPS</b> (or localhost). On the machine running
|
||||||
|
the backend:
|
||||||
|
<div style="margin:0.4rem 0"><code>tailscale serve --bg 8000</code></div>
|
||||||
|
then open the printed <code>https://<machine>.<tailnet>.ts.net/capture</code>
|
||||||
|
URL on this device (join the tailnet first).
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div id="app">
|
||||||
|
<div class="card">
|
||||||
|
<label for="name">Device name (used in the filename)</label>
|
||||||
|
<input type="text" id="name" placeholder="e.g. phone-left / logitech-c920" />
|
||||||
|
<label for="cam">Camera</label>
|
||||||
|
<select id="cam"></select>
|
||||||
|
<label for="mic">Microphone <span style="color:var(--bad)">— required for sync</span></label>
|
||||||
|
<select id="mic"></select>
|
||||||
|
<video id="preview" playsinline muted autoplay></video>
|
||||||
|
<div class="row" style="margin-top:0.55rem">
|
||||||
|
<span class="pill"><span class="dot" id="recdot"></span><span id="status">idle</span></span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="row">
|
||||||
|
<button id="rec">● Record</button>
|
||||||
|
</div>
|
||||||
|
<div class="msg" id="msg"></div>
|
||||||
|
|
||||||
|
<div class="card" style="margin-top:0.8rem">
|
||||||
|
<label style="margin-bottom:0.4rem">Who's shooting what — live monitor</label>
|
||||||
|
<div id="mon"></div>
|
||||||
|
<div class="msg" id="monempty" style="color:var(--dim)">no other cameras yet</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="msg" style="color:var(--dim)">
|
||||||
|
All cameras must hear the <b>same sound</b> — that's how clips get aligned. No clock sync
|
||||||
|
needed; you don't even have to start together. When done, run
|
||||||
|
<code>python -m festival4d ingest && python -m festival4d sync</code>.
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<script>
|
||||||
|
const $ = (id) => document.getElementById(id);
|
||||||
|
const deviceId = (localStorage.f4dDevId ||= "dev-" + Math.random().toString(36).slice(2, 9));
|
||||||
|
let stream = null, recorder = null, chunks = [], startedAt = 0, timer = null;
|
||||||
|
|
||||||
|
const secure = window.isSecureContext && navigator.mediaDevices?.getUserMedia;
|
||||||
|
if (!secure) {
|
||||||
|
$("gate").style.display = "block";
|
||||||
|
$("app").style.display = "none";
|
||||||
|
}
|
||||||
|
|
||||||
|
function msg(html, cls = "") { $("msg").className = "msg " + cls; $("msg").innerHTML = html; }
|
||||||
|
|
||||||
|
async function listDevices() {
|
||||||
|
const devs = await navigator.mediaDevices.enumerateDevices();
|
||||||
|
const fill = (sel, kind, fallback) => {
|
||||||
|
const cur = sel.value;
|
||||||
|
sel.innerHTML = devs
|
||||||
|
.filter((d) => d.kind === kind)
|
||||||
|
.map((d, i) => `<option value="${d.deviceId}">${d.label || `${fallback} ${i + 1}`}</option>`)
|
||||||
|
.join("");
|
||||||
|
if (cur) sel.value = cur;
|
||||||
|
};
|
||||||
|
fill($("cam"), "videoinput", "Camera");
|
||||||
|
fill($("mic"), "audioinput", "Microphone");
|
||||||
|
}
|
||||||
|
|
||||||
|
async function start(camId, micId) {
|
||||||
|
stream?.getTracks().forEach((t) => t.stop());
|
||||||
|
// Audio is NOT optional: the pipeline aligns clips by sound.
|
||||||
|
stream = await navigator.mediaDevices.getUserMedia({
|
||||||
|
video: camId ? { deviceId: { exact: camId } } : true,
|
||||||
|
audio: micId ? { deviceId: { exact: micId } } : true,
|
||||||
|
});
|
||||||
|
$("preview").srcObject = stream;
|
||||||
|
await listDevices(); // labels only populate after permission is granted
|
||||||
|
}
|
||||||
|
|
||||||
|
function pickMime() {
|
||||||
|
const want = [
|
||||||
|
"video/webm;codecs=vp9,opus",
|
||||||
|
"video/webm;codecs=vp8,opus",
|
||||||
|
"video/webm",
|
||||||
|
"video/mp4", // Safari / iOS
|
||||||
|
];
|
||||||
|
return want.find((m) => MediaRecorder.isTypeSupported?.(m)) || "";
|
||||||
|
}
|
||||||
|
|
||||||
|
$("rec").addEventListener("click", () => (recorder ? stop() : record()));
|
||||||
|
|
||||||
|
function record() {
|
||||||
|
if (!stream) return msg("no camera stream yet", "err");
|
||||||
|
if (stream.getAudioTracks().length === 0)
|
||||||
|
return msg("no audio track — clips without sound can't be synced.", "err");
|
||||||
|
const mime = pickMime();
|
||||||
|
chunks = [];
|
||||||
|
recorder = new MediaRecorder(stream, mime ? { mimeType: mime } : undefined);
|
||||||
|
recorder.ondataavailable = (e) => e.data.size && chunks.push(e.data);
|
||||||
|
recorder.onstop = upload;
|
||||||
|
recorder.start(1000);
|
||||||
|
startedAt = Date.now();
|
||||||
|
$("rec").textContent = "■ Stop";
|
||||||
|
$("rec").classList.add("recording");
|
||||||
|
$("recdot").classList.add("on");
|
||||||
|
timer = setInterval(tick, 250);
|
||||||
|
msg("recording — point at the stage and keep some static structure in frame.");
|
||||||
|
}
|
||||||
|
|
||||||
|
function stop() {
|
||||||
|
clearInterval(timer);
|
||||||
|
recorder?.stop();
|
||||||
|
recorder = null;
|
||||||
|
$("rec").textContent = "● Record";
|
||||||
|
$("rec").classList.remove("recording");
|
||||||
|
$("recdot").classList.remove("on");
|
||||||
|
tick();
|
||||||
|
}
|
||||||
|
|
||||||
|
function elapsed() { return startedAt ? (Date.now() - startedAt) / 1000 : 0; }
|
||||||
|
function tick() {
|
||||||
|
$("status").textContent = recorder ? `recording ${elapsed().toFixed(0)}s` : "idle";
|
||||||
|
}
|
||||||
|
|
||||||
|
async function upload() {
|
||||||
|
const type = chunks[0]?.type || "video/webm";
|
||||||
|
const blob = new Blob(chunks, { type });
|
||||||
|
const ext = type.includes("mp4") ? "mp4" : "webm";
|
||||||
|
const name = ($("name").value || "camera").trim();
|
||||||
|
const fd = new FormData();
|
||||||
|
fd.append("device_name", name);
|
||||||
|
fd.append("file", blob, `clip.${ext}`);
|
||||||
|
msg(`uploading ${(blob.size / 1e6).toFixed(1)} MB…`);
|
||||||
|
try {
|
||||||
|
const r = await fetch("api/capture/upload", { method: "POST", body: fd });
|
||||||
|
const j = await r.json();
|
||||||
|
if (!r.ok) throw new Error(j.detail || `HTTP ${r.status}`);
|
||||||
|
msg(`saved <code>${j.filename}</code> (${(j.bytes / 1e6).toFixed(1)} MB). Record again or run the pipeline.`, "ok");
|
||||||
|
} catch (e) {
|
||||||
|
msg(`upload failed: ${e.message}`, "err");
|
||||||
|
}
|
||||||
|
startedAt = 0;
|
||||||
|
tick();
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- heartbeat + monitor -------------------------------------------------------------
|
||||||
|
const snapCanvas = document.createElement("canvas");
|
||||||
|
function snapshot() {
|
||||||
|
const v = $("preview");
|
||||||
|
if (!v.videoWidth) return null;
|
||||||
|
snapCanvas.width = 160;
|
||||||
|
snapCanvas.height = Math.round((160 * v.videoHeight) / v.videoWidth);
|
||||||
|
snapCanvas.getContext("2d").drawImage(v, 0, 0, snapCanvas.width, snapCanvas.height);
|
||||||
|
return snapCanvas.toDataURL("image/jpeg", 0.4);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function beat() {
|
||||||
|
try {
|
||||||
|
await fetch("api/capture/heartbeat", {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "Content-Type": "application/json" },
|
||||||
|
body: JSON.stringify({
|
||||||
|
device_id: deviceId,
|
||||||
|
name: ($("name").value || "camera").trim(),
|
||||||
|
status: recorder ? "recording" : "idle",
|
||||||
|
elapsed_s: elapsed(),
|
||||||
|
snapshot: snapshot(),
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
const devs = await (await fetch("api/capture/devices")).json();
|
||||||
|
renderMonitor(devs.filter((d) => d.device_id !== deviceId));
|
||||||
|
} catch { /* offline: monitor just goes stale */ }
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderMonitor(devs) {
|
||||||
|
$("monempty").style.display = devs.length ? "none" : "block";
|
||||||
|
$("mon").innerHTML = devs
|
||||||
|
.map(
|
||||||
|
(d) => `<div class="mcell ${d.status === "recording" ? "rec" : ""}">
|
||||||
|
${d.snapshot ? `<img src="${d.snapshot}" alt="">` : `<img alt="">`}
|
||||||
|
<div class="cap"><span class="dot ${d.status === "recording" ? "on" : ""}"></span>
|
||||||
|
<span class="nm">${d.name}</span>
|
||||||
|
<span>${d.status === "recording" ? d.elapsed_s.toFixed(0) + "s" : ""}</span></div>
|
||||||
|
</div>`
|
||||||
|
)
|
||||||
|
.join("");
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- boot ----------------------------------------------------------------------------
|
||||||
|
if (secure) {
|
||||||
|
$("name").value = localStorage.f4dName || "";
|
||||||
|
$("name").addEventListener("change", () => (localStorage.f4dName = $("name").value));
|
||||||
|
$("cam").addEventListener("change", () => start($("cam").value, $("mic").value).catch((e) => msg(e.message, "err")));
|
||||||
|
$("mic").addEventListener("change", () => start($("cam").value, $("mic").value).catch((e) => msg(e.message, "err")));
|
||||||
|
start().then(() => msg("ready — name this device, then record.")).catch((e) =>
|
||||||
|
msg(`camera error: ${e.message}`, "err")
|
||||||
|
);
|
||||||
|
setInterval(beat, 2000);
|
||||||
|
navigator.mediaDevices.addEventListener?.("devicechange", listDevices);
|
||||||
|
}
|
||||||
|
</script>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@ -32,8 +32,11 @@ def client():
|
|||||||
|
|
||||||
def test_manifest_shape(client):
|
def test_manifest_shape(client):
|
||||||
data = client.get("/api/manifest").json()
|
data = client.get("/api/manifest").json()
|
||||||
assert set(data) == {"videos", "t_global_max", "has_poses"}
|
# Exact-set lock: additive manifest fields are fine, but must be added here consciously
|
||||||
|
# (this assertion has already caught two silent drifts — has_splat and has_capture).
|
||||||
|
assert set(data) == {"videos", "t_global_max", "has_poses", "has_splat", "has_capture"}
|
||||||
assert data["has_poses"] is True
|
assert data["has_poses"] is True
|
||||||
|
assert data["has_capture"] is False # capture routes are opt-in (FESTIVAL4D_CAPTURE)
|
||||||
assert len(data["videos"]) == 3
|
assert len(data["videos"]) == 3
|
||||||
v = data["videos"][0]
|
v = data["videos"][0]
|
||||||
assert set(v) == {
|
assert set(v) == {
|
||||||
|
|||||||
137
backend/tests/test_capture.py
Normal file
137
backend/tests/test_capture.py
Normal file
@ -0,0 +1,137 @@
|
|||||||
|
"""Live capture tests (Phase 5) — upload safety, the env gate, and the device monitor.
|
||||||
|
|
||||||
|
The router is exercised directly on a throwaway app: ``capture_enabled()`` is read at
|
||||||
|
``api`` import time, so mounting it here is both simpler and keeps the gate test honest
|
||||||
|
(the real app must NOT expose these routes unless FESTIVAL4D_CAPTURE is set).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi import FastAPI
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from festival4d import capture, config
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture()
|
||||||
|
def client():
|
||||||
|
app = FastAPI()
|
||||||
|
app.include_router(capture.router)
|
||||||
|
config.RAW_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
with TestClient(app) as c:
|
||||||
|
yield c
|
||||||
|
|
||||||
|
|
||||||
|
def _upload(client, name: str, filename: str, data: bytes = b"fake-video-bytes"):
|
||||||
|
return client.post(
|
||||||
|
"/api/capture/upload",
|
||||||
|
data={"device_name": name},
|
||||||
|
files={"file": (filename, data, "video/webm")},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# --- the gate ---------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def test_capture_disabled_by_default():
|
||||||
|
"""FESTIVAL4D_CAPTURE is unset in tests => the real app exposes no upload surface."""
|
||||||
|
assert capture.capture_enabled() is False
|
||||||
|
from festival4d import api
|
||||||
|
|
||||||
|
paths = {r.path for r in api.app.routes}
|
||||||
|
assert "/capture" not in paths
|
||||||
|
assert not any(p.startswith("/api/capture") for p in paths)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("val,expected", [("1", True), ("true", True), ("on", True),
|
||||||
|
("0", False), ("", False), ("no", False)])
|
||||||
|
def test_capture_enabled_parsing(monkeypatch, val, expected):
|
||||||
|
monkeypatch.setenv("FESTIVAL4D_CAPTURE", val)
|
||||||
|
assert capture.capture_enabled() is expected
|
||||||
|
|
||||||
|
|
||||||
|
# --- upload -----------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def test_upload_saves_sanitized_file(client):
|
||||||
|
r = _upload(client, "Phone Left!", "clip.webm")
|
||||||
|
assert r.status_code == 200, r.text
|
||||||
|
body = r.json()
|
||||||
|
saved = config.RAW_DIR / body["filename"]
|
||||||
|
assert saved.is_file() and saved.read_bytes() == b"fake-video-bytes"
|
||||||
|
assert body["filename"].startswith("phone-left-") # slugified, timestamped
|
||||||
|
assert body["filename"].endswith(".webm")
|
||||||
|
assert body["bytes"] == len(b"fake-video-bytes")
|
||||||
|
saved.unlink()
|
||||||
|
|
||||||
|
|
||||||
|
def test_upload_name_cannot_traverse(client):
|
||||||
|
r = _upload(client, "../../etc/passwd", "clip.webm")
|
||||||
|
assert r.status_code == 200
|
||||||
|
saved = config.RAW_DIR / r.json()["filename"]
|
||||||
|
# The slug strips separators entirely; the file lands inside RAW_DIR, nowhere else.
|
||||||
|
assert saved.resolve().parent == config.RAW_DIR.resolve()
|
||||||
|
assert ".." not in r.json()["filename"]
|
||||||
|
saved.unlink()
|
||||||
|
|
||||||
|
|
||||||
|
def test_upload_rejects_bad_container(client):
|
||||||
|
r = _upload(client, "cam", "payload.exe")
|
||||||
|
assert r.status_code == 400
|
||||||
|
assert "unsupported container" in r.json()["detail"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_upload_rejects_empty(client):
|
||||||
|
r = _upload(client, "cam", "clip.webm", data=b"")
|
||||||
|
assert r.status_code == 400
|
||||||
|
assert "empty" in r.json()["detail"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_upload_rejects_oversize(client, monkeypatch):
|
||||||
|
monkeypatch.setattr(capture, "MAX_UPLOAD_BYTES", 8)
|
||||||
|
r = _upload(client, "cam", "clip.webm", data=b"x" * 64)
|
||||||
|
assert r.status_code == 413
|
||||||
|
# The partial file must not be left behind for ingest to trip on.
|
||||||
|
assert not any(p.name.startswith("cam-") for p in config.RAW_DIR.glob("cam-*.webm"))
|
||||||
|
|
||||||
|
|
||||||
|
# --- monitor ----------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def test_heartbeat_and_devices_monitor(client):
|
||||||
|
capture._devices.clear()
|
||||||
|
client.post("/api/capture/heartbeat", json={
|
||||||
|
"device_id": "d1", "name": "phone-left", "status": "recording",
|
||||||
|
"elapsed_s": 3.5, "snapshot": "data:image/jpeg;base64,AAAA",
|
||||||
|
})
|
||||||
|
client.post("/api/capture/heartbeat", json={"device_id": "d2", "name": "laptop"})
|
||||||
|
devs = client.get("/api/capture/devices").json()
|
||||||
|
assert [d["name"] for d in devs] == ["laptop", "phone-left"] # sorted by name
|
||||||
|
d1 = next(d for d in devs if d["device_id"] == "d1")
|
||||||
|
assert d1["status"] == "recording" and d1["snapshot"].startswith("data:image/jpeg")
|
||||||
|
assert "_seen" not in d1 # internals not leaked
|
||||||
|
capture._devices.clear()
|
||||||
|
|
||||||
|
|
||||||
|
def test_oversized_snapshot_dropped_but_device_kept(client):
|
||||||
|
capture._devices.clear()
|
||||||
|
client.post("/api/capture/heartbeat", json={
|
||||||
|
"device_id": "big", "name": "cam", "snapshot": "data:image/jpeg;base64," + "A" * 400_000,
|
||||||
|
})
|
||||||
|
devs = client.get("/api/capture/devices").json()
|
||||||
|
assert len(devs) == 1 and "snapshot" not in devs[0]
|
||||||
|
capture._devices.clear()
|
||||||
|
|
||||||
|
|
||||||
|
def test_stale_devices_pruned(client, monkeypatch):
|
||||||
|
capture._devices.clear()
|
||||||
|
client.post("/api/capture/heartbeat", json={"device_id": "old", "name": "cam"})
|
||||||
|
assert len(client.get("/api/capture/devices").json()) == 1
|
||||||
|
monkeypatch.setattr(capture, "DEVICE_STALE_S", -1.0) # everything is instantly stale
|
||||||
|
assert client.get("/api/capture/devices").json() == []
|
||||||
|
capture._devices.clear()
|
||||||
|
|
||||||
|
|
||||||
|
def test_capture_page_served(client):
|
||||||
|
r = client.get("/capture")
|
||||||
|
assert r.status_code == 200
|
||||||
|
assert "Festival 4D — Capture" in r.text
|
||||||
|
assert "getUserMedia" in r.text # the page really is the capture app
|
||||||
@ -106,3 +106,13 @@ backend or data changed; a frontend-only change just needs the `dist` rsync).
|
|||||||
dev-only wide CORS in `config.py` stays as-is; it doesn't affect the hosted site.
|
dev-only wide CORS in `config.py` stays as-is; it doesn't affect the hosted site.
|
||||||
- **Single project.** The app assumes one project at a time (SQLite at `data/project.db`).
|
- **Single project.** The app assumes one project at a time (SQLite at `data/project.db`).
|
||||||
- The backend binds `127.0.0.1` only; nginx is the sole public entry point.
|
- The backend binds `127.0.0.1` only; nginx is the sole public entry point.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Live capture on the public box: leave it OFF
|
||||||
|
|
||||||
|
`FESTIVAL4D_CAPTURE` is **not** set in `deploy/festifun-api.service`, so `/capture` and
|
||||||
|
`/api/capture/*` are never mounted publicly — an open upload endpoint on a public host would let
|
||||||
|
anyone write files to the server. Capture is for your local/tailnet machine (see the README's
|
||||||
|
Capture section: `FESTIVAL4D_CAPTURE=1` + `tailscale serve`). Record locally, run the pipeline,
|
||||||
|
then deploy the resulting project.
|
||||||
|
|||||||
@ -211,6 +211,10 @@
|
|||||||
<div id="topbar">
|
<div id="topbar">
|
||||||
<h1>Festival 4D</h1>
|
<h1>Festival 4D</h1>
|
||||||
<span class="sub">synchronized multi-cam replay · 3D viewer · x-ray overlays</span>
|
<span class="sub">synchronized multi-cam replay · 3D viewer · x-ray overlays</span>
|
||||||
|
<span class="spacer"></span>
|
||||||
|
<!-- shown only when the backend has capture enabled (FESTIVAL4D_CAPTURE=1) -->
|
||||||
|
<a id="capture-link" class="btn" style="display:none; text-decoration:none"
|
||||||
|
title="Record from webcams / phones into this project" target="_blank">📹 Capture</a>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div id="stage">
|
<div id="stage">
|
||||||
|
|||||||
@ -42,6 +42,12 @@ async function boot() {
|
|||||||
state.videos = manifest.videos;
|
state.videos = manifest.videos;
|
||||||
state.tGlobalMax = manifest.t_global_max;
|
state.tGlobalMax = manifest.t_global_max;
|
||||||
state.hasPoses = manifest.has_poses;
|
state.hasPoses = manifest.has_poses;
|
||||||
|
// Capture is opt-in on the backend; only surface the link when it's actually mounted.
|
||||||
|
if (manifest.has_capture) {
|
||||||
|
const link = el("capture-link");
|
||||||
|
link.href = `${API_BASE}/capture`;
|
||||||
|
link.style.display = "";
|
||||||
|
}
|
||||||
state.hasSplat = !!manifest.has_splat;
|
state.hasSplat = !!manifest.has_splat;
|
||||||
for (const v of state.videos) state.enabled[v.id] = true;
|
for (const v of state.videos) state.enabled[v.id] = true;
|
||||||
state.audioSourceId = referenceVideoId();
|
state.audioSourceId = referenceVideoId();
|
||||||
|
|||||||
@ -14,6 +14,7 @@ dependencies = [
|
|||||||
"sqlalchemy>=2.0",
|
"sqlalchemy>=2.0",
|
||||||
"pydantic>=2.6",
|
"pydantic>=2.6",
|
||||||
"opencv-python-headless>=4.9",
|
"opencv-python-headless>=4.9",
|
||||||
|
"python-multipart>=0.0.9", # multipart uploads for live capture (backend/capture.py)
|
||||||
# classifier providers (lane D) — imported lazily, never at module load
|
# classifier providers (lane D) — imported lazily, never at module load
|
||||||
"google-genai>=0.3",
|
"google-genai>=0.3",
|
||||||
"anthropic>=0.40",
|
"anthropic>=0.40",
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user