From 36bacf67f050f35cfcf30eebb712481c2ee28aac Mon Sep 17 00:00:00 2001 From: m3ultra Date: Fri, 17 Jul 2026 15:43:51 +1000 Subject: [PATCH] Live capture: USB webcams, laptop cams, and phones -> data/raw MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records rather than streams: each device records a clip WITH AUDIO and uploads it; the normal offline pipeline takes over. Fits the design almost for free — MediaRecorder's webm/mp4 are already ingestible, and audio-based sync means devices need no clock sync and needn't start together. - capture.py: POST /api/capture/upload (streamed, sanitized name, size cap, partial cleanup), heartbeat/devices registry, GET /capture page. - static/capture.html: standalone page (no build step) — device picker, live preview, record->upload, 1fps snapshot heartbeat, 'who's shooting what' monitor, and secure-context detection that tells you to use (getUserMedia needs HTTPS — the #1 gotcha for phones). - OPT-IN via FESTIVAL4D_CAPTURE=1; manifest reports has_capture and the viewer only then shows a Capture link. Public deploy leaves it off (open upload endpoint would be unsafe) — noted in DEPLOY.md. - Adds python-multipart dep. Suite 103 -> 119. Also fixes test_manifest_shape, which was already RED on main: f12b6e0 added has_splat to the manifest without updating the exact-set lock. Updated to the true set (now also has_capture); the lock is what caught both drifts. Co-Authored-By: Claude Opus 4.8 --- README.md | 48 +++++ backend/festival4d/api.py | 9 + backend/festival4d/capture.py | 166 +++++++++++++++ backend/festival4d/static/capture.html | 276 +++++++++++++++++++++++++ backend/tests/test_api.py | 5 +- backend/tests/test_capture.py | 137 ++++++++++++ deploy/DEPLOY.md | 10 + frontend/index.html | 4 + frontend/src/main.js | 6 + pyproject.toml | 1 + 10 files changed, 661 insertions(+), 1 deletion(-) create mode 100644 backend/festival4d/capture.py create mode 100644 backend/festival4d/static/capture.html create mode 100644 backend/tests/test_capture.py diff --git a/README.md b/README.md index e5b0539..cc7ae0a 100644 --- a/README.md +++ b/README.md @@ -67,6 +67,54 @@ moment markers on the timeline. --- +## Capture: record straight from webcams & phones + +Don't have footage yet? Record it with whatever cameras you have — USB webcams, the laptop cam, +and any phone with a browser. **No app install.** Cameras *record* clips that feed the normal +pipeline (this isn't live streaming — see `docs/ideas.md` for that). + +This works because of two properties of the design: browser recordings (`.webm`, or `.mp4` on +iOS) are already ingestible formats, and **sync is audio-based** — so the devices need **no clock +sync and don't even have to start together**. They only have to record the same moment and hear +the same sound. + +```bash +# capture is OPT-IN (it accepts uploads) — enable it explicitly: +FESTIVAL4D_CAPTURE=1 uv run python -m festival4d serve +``` + +Then open **`/capture`** on each device (a 📹 Capture link also appears in the viewer): name the +device, pick a camera + mic, hit record, hit stop — the clip uploads into `data/raw/`. The page +shows a live "who's shooting what" monitor of every connected camera. When you're done: + +```bash +uv run python -m festival4d ingest && uv run python -m festival4d sync # then reconstruct / events +``` + +### Getting a phone connected (the HTTPS bit) + +Browsers **only grant camera access on a secure origin**. A phone opening `http://:8000` +gets *no camera*, silently. The easiest fix if you use Tailscale — it issues a real cert, so +there's nothing to trust manually: + +```bash +tailscale serve --bg 8000 # prints https://..ts.net +``` + +Join the phone to the tailnet and open `https://..ts.net/capture`. (Without +Tailscale: any HTTPS reverse proxy works; `localhost` is also treated as secure, which is why the +laptop's own cameras work with no setup.) + +### Capture tips + +- **Every clip needs audio**, and all cameras must hear the same sound — that's the only thing + aligning them. Testing indoors? Play music. +- **USB bandwidth is the real limit** on multiple webcams: two 1080p cams on one controller often + fails. Drop to 720p or use separate ports/hubs. +- Follow the *Shooting tips* below for reconstruction-friendly angles. +- Capture routes stay **unmounted** unless `FESTIVAL4D_CAPTURE=1` — a public deployment must + never expose an open upload endpoint. + ## Real-footage workflow 1. **Shoot / gather** 2–4 videos of the same performance from different positions (see diff --git a/backend/festival4d/api.py b/backend/festival4d/api.py index 1d11158..8d64ed6 100644 --- a/backend/festival4d/api.py +++ b/backend/festival4d/api.py @@ -52,6 +52,14 @@ config.RAW_DIR.mkdir(parents=True, exist_ok=True) # HTTP Range-capable static serving of source videos (spec M3, pitfall #3). app.mount("/media", StaticFiles(directory=config.RAW_DIR), name="media") +# Live capture (webcams / phones -> data/raw). OPT-IN: these routes accept file uploads, so they +# stay unmounted unless FESTIVAL4D_CAPTURE=1 — a public deployment must never expose them. +from festival4d import capture # noqa: E402 (import after app/config are ready) + +if capture.capture_enabled(): + app.include_router(capture.router) + log.info("capture routes ENABLED (/capture, /api/capture/*) — FESTIVAL4D_CAPTURE is set") + # --------------------------------------------------------------------------- # Request models @@ -145,6 +153,7 @@ def manifest() -> dict: return { "videos": [_video_dict(v) for v in videos], "t_global_max": t_global_max, + "has_capture": capture.capture_enabled(), "has_poses": db.has_poses(), "has_splat": config.SPLAT_PLY.exists(), } diff --git a/backend/festival4d/capture.py b/backend/festival4d/capture.py new file mode 100644 index 0000000..ace7bfb --- /dev/null +++ b/backend/festival4d/capture.py @@ -0,0 +1,166 @@ +"""Live capture — USB webcams, laptop cams, and phones -> ``data/raw`` (Phase 5). + +**Records, it doesn't stream.** Each device records a clip *with audio* and uploads it; the normal +offline pipeline (``ingest -> sync -> reconstruct -> events``) takes it from there. This fits the +app's design almost for free: + +- Browser ``MediaRecorder`` emits WebM (Safari/iOS: MP4) — both are already in ingest's + ``_VIDEO_EXTS``, so no format work is needed. +- Sync is **audio-based** (GCC-PHAT), so capture devices need **no clock sync and no + synchronized start** — they only have to record the same moment and hear the same sound. + Every recording MUST carry an audio track or ``sync`` has nothing to align. + +A lightweight device registry powers the "who's shooting what" monitor: each device posts a +heartbeat with its status and a small JPEG snapshot (~1 fps), so the capture page can show every +camera without a WebRTC stack. + +**Disabled by default.** These routes accept file uploads; they must never be exposed on a public +deployment. Set ``FESTIVAL4D_CAPTURE=1`` to mount them (see :func:`capture_enabled`). +""" + +from __future__ import annotations + +import logging +import os +import re +import time +from datetime import datetime +from pathlib import Path + +from fastapi import APIRouter, Form, HTTPException, UploadFile +from fastapi.responses import HTMLResponse +from pydantic import BaseModel, Field + +from festival4d import config + +log = logging.getLogger("festival4d.capture") + +router = APIRouter() + +STATIC_DIR = Path(__file__).resolve().parent / "static" + +# Containers a browser MediaRecorder can produce, intersected with what ingest accepts. +ALLOWED_EXTS = {".webm", ".mp4", ".mov", ".mkv"} +MAX_UPLOAD_BYTES = int(os.environ.get("FESTIVAL4D_MAX_UPLOAD_MB", "2048")) * 1024 * 1024 +MAX_SNAPSHOT_CHARS = 300_000 # data-URL cap for a heartbeat thumbnail +DEVICE_STALE_S = 15.0 # drop devices that stop heartbeating + + +def capture_enabled() -> bool: + """Capture routes are opt-in — they accept uploads, so never on by default.""" + return os.environ.get("FESTIVAL4D_CAPTURE", "").strip().lower() in {"1", "true", "yes", "on"} + + +# --------------------------------------------------------------------------- +# Device registry (in-memory, ephemeral — it only powers the live monitor) +# --------------------------------------------------------------------------- +_devices: dict[str, dict] = {} + + +class Heartbeat(BaseModel): + device_id: str = Field(min_length=1, max_length=64) + name: str = Field(default="camera", max_length=64) + status: str = Field(default="idle", max_length=16) # idle | recording + elapsed_s: float = 0.0 + snapshot: str | None = None # data:image/jpeg;base64,... + + +def _prune() -> None: + now = time.monotonic() + for did in [d for d, v in _devices.items() if now - v["_seen"] > DEVICE_STALE_S]: + _devices.pop(did, None) + + +def _slug(name: str) -> str: + """Filesystem-safe slug from a user-supplied device name.""" + s = re.sub(r"[^a-zA-Z0-9]+", "-", name).strip("-").lower() + return (s or "cam")[:40] + + +def _safe_raw_path(device_name: str, filename: str) -> Path: + """Build a sanitized destination inside RAW_DIR. Never trusts the client filename.""" + ext = Path(filename or "").suffix.lower() + if ext not in ALLOWED_EXTS: + raise HTTPException( + status_code=400, + detail=f"unsupported container '{ext or '?'}' (allowed: {sorted(ALLOWED_EXTS)})", + ) + stamp = datetime.now().strftime("%Y%m%d-%H%M%S") + dest = (config.RAW_DIR / f"{_slug(device_name)}-{stamp}{ext}").resolve() + # Defence in depth: the slug can't traverse, but verify containment anyway. + if not str(dest).startswith(str(config.RAW_DIR.resolve()) + os.sep): + raise HTTPException(status_code=400, detail="invalid destination path") + return dest + + +# --------------------------------------------------------------------------- +# Routes +# --------------------------------------------------------------------------- +@router.get("/capture", response_class=HTMLResponse) +def capture_page() -> str: + """The standalone capture page (open this on a phone over HTTPS — see README).""" + page = STATIC_DIR / "capture.html" + if not page.is_file(): + raise HTTPException(status_code=500, detail="capture.html missing from the install") + return page.read_text(encoding="utf-8") + + +@router.post("/api/capture/upload") +async def upload(file: UploadFile, device_name: str = Form("camera")) -> dict: + """Stream an uploaded clip into ``data/raw`` under a sanitized name.""" + dest = _safe_raw_path(device_name, file.filename or "") + config.RAW_DIR.mkdir(parents=True, exist_ok=True) + + written = 0 + try: + with open(dest, "wb") as out: + while chunk := await file.read(1 << 20): # 1 MiB + written += len(chunk) + if written > MAX_UPLOAD_BYTES: + raise HTTPException( + status_code=413, + detail=f"upload exceeds {MAX_UPLOAD_BYTES // (1024*1024)} MB", + ) + out.write(chunk) + except Exception: + dest.unlink(missing_ok=True) # never leave a partial file for ingest to trip on + raise + + if written == 0: + dest.unlink(missing_ok=True) + raise HTTPException(status_code=400, detail="empty upload") + + log.info("capture: saved %s (%.1f MB) from %r", dest.name, written / 1e6, device_name) + return {"filename": dest.name, "bytes": written, "next": "run `python -m festival4d ingest`"} + + +@router.post("/api/capture/heartbeat") +def heartbeat(hb: Heartbeat) -> dict: + """Register/refresh a capture device and its latest thumbnail (powers the monitor).""" + snap = hb.snapshot + if snap and len(snap) > MAX_SNAPSHOT_CHARS: + snap = None # oversized thumbnail: keep the device, drop the image + entry = _devices.setdefault(hb.device_id, {}) + entry.update( + { + "device_id": hb.device_id, + "name": hb.name, + "status": hb.status, + "elapsed_s": hb.elapsed_s, + "_seen": time.monotonic(), + } + ) + if snap: + entry["snapshot"] = snap + _prune() + return {"ok": True, "devices": len(_devices)} + + +@router.get("/api/capture/devices") +def devices() -> list[dict]: + """Live capture devices (stale ones drop off after ~15 s).""" + _prune() + return [ + {k: v for k, v in d.items() if not k.startswith("_")} + for d in sorted(_devices.values(), key=lambda d: d["name"]) + ] diff --git a/backend/festival4d/static/capture.html b/backend/festival4d/static/capture.html new file mode 100644 index 0000000..00b8902 --- /dev/null +++ b/backend/festival4d/static/capture.html @@ -0,0 +1,276 @@ + + + + + + Festival 4D — Capture + + + +

Festival 4D — Capture

+
Record this camera. Clips upload to the project and sync by audio.
+ + +
+
Camera blocked — this page isn't a secure context.
+
+ Browsers only grant camera access over HTTPS (or localhost). On the machine running + the backend: +
tailscale serve --bg 8000
+ then open the printed https://<machine>.<tailnet>.ts.net/capture + URL on this device (join the tailnet first). +
+
+ +
+
+ + + + + + + +
+ idle +
+
+ +
+ +
+
+ +
+ +
+
no other cameras yet
+
+ +
+ All cameras must hear the same sound — that's how clips get aligned. No clock sync + needed; you don't even have to start together. When done, run + python -m festival4d ingest && python -m festival4d sync. +
+
+ + + + diff --git a/backend/tests/test_api.py b/backend/tests/test_api.py index 5eff676..88a1a38 100644 --- a/backend/tests/test_api.py +++ b/backend/tests/test_api.py @@ -32,8 +32,11 @@ def client(): def test_manifest_shape(client): data = client.get("/api/manifest").json() - assert set(data) == {"videos", "t_global_max", "has_poses"} + # Exact-set lock: additive manifest fields are fine, but must be added here consciously + # (this assertion has already caught two silent drifts — has_splat and has_capture). + assert set(data) == {"videos", "t_global_max", "has_poses", "has_splat", "has_capture"} assert data["has_poses"] is True + assert data["has_capture"] is False # capture routes are opt-in (FESTIVAL4D_CAPTURE) assert len(data["videos"]) == 3 v = data["videos"][0] assert set(v) == { diff --git a/backend/tests/test_capture.py b/backend/tests/test_capture.py new file mode 100644 index 0000000..4bef1de --- /dev/null +++ b/backend/tests/test_capture.py @@ -0,0 +1,137 @@ +"""Live capture tests (Phase 5) — upload safety, the env gate, and the device monitor. + +The router is exercised directly on a throwaway app: ``capture_enabled()`` is read at +``api`` import time, so mounting it here is both simpler and keeps the gate test honest +(the real app must NOT expose these routes unless FESTIVAL4D_CAPTURE is set). +""" + +from __future__ import annotations + +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient + +from festival4d import capture, config + + +@pytest.fixture() +def client(): + app = FastAPI() + app.include_router(capture.router) + config.RAW_DIR.mkdir(parents=True, exist_ok=True) + with TestClient(app) as c: + yield c + + +def _upload(client, name: str, filename: str, data: bytes = b"fake-video-bytes"): + return client.post( + "/api/capture/upload", + data={"device_name": name}, + files={"file": (filename, data, "video/webm")}, + ) + + +# --- the gate --------------------------------------------------------------------------------- + +def test_capture_disabled_by_default(): + """FESTIVAL4D_CAPTURE is unset in tests => the real app exposes no upload surface.""" + assert capture.capture_enabled() is False + from festival4d import api + + paths = {r.path for r in api.app.routes} + assert "/capture" not in paths + assert not any(p.startswith("/api/capture") for p in paths) + + +@pytest.mark.parametrize("val,expected", [("1", True), ("true", True), ("on", True), + ("0", False), ("", False), ("no", False)]) +def test_capture_enabled_parsing(monkeypatch, val, expected): + monkeypatch.setenv("FESTIVAL4D_CAPTURE", val) + assert capture.capture_enabled() is expected + + +# --- upload ----------------------------------------------------------------------------------- + +def test_upload_saves_sanitized_file(client): + r = _upload(client, "Phone Left!", "clip.webm") + assert r.status_code == 200, r.text + body = r.json() + saved = config.RAW_DIR / body["filename"] + assert saved.is_file() and saved.read_bytes() == b"fake-video-bytes" + assert body["filename"].startswith("phone-left-") # slugified, timestamped + assert body["filename"].endswith(".webm") + assert body["bytes"] == len(b"fake-video-bytes") + saved.unlink() + + +def test_upload_name_cannot_traverse(client): + r = _upload(client, "../../etc/passwd", "clip.webm") + assert r.status_code == 200 + saved = config.RAW_DIR / r.json()["filename"] + # The slug strips separators entirely; the file lands inside RAW_DIR, nowhere else. + assert saved.resolve().parent == config.RAW_DIR.resolve() + assert ".." not in r.json()["filename"] + saved.unlink() + + +def test_upload_rejects_bad_container(client): + r = _upload(client, "cam", "payload.exe") + assert r.status_code == 400 + assert "unsupported container" in r.json()["detail"] + + +def test_upload_rejects_empty(client): + r = _upload(client, "cam", "clip.webm", data=b"") + assert r.status_code == 400 + assert "empty" in r.json()["detail"] + + +def test_upload_rejects_oversize(client, monkeypatch): + monkeypatch.setattr(capture, "MAX_UPLOAD_BYTES", 8) + r = _upload(client, "cam", "clip.webm", data=b"x" * 64) + assert r.status_code == 413 + # The partial file must not be left behind for ingest to trip on. + assert not any(p.name.startswith("cam-") for p in config.RAW_DIR.glob("cam-*.webm")) + + +# --- monitor ---------------------------------------------------------------------------------- + +def test_heartbeat_and_devices_monitor(client): + capture._devices.clear() + client.post("/api/capture/heartbeat", json={ + "device_id": "d1", "name": "phone-left", "status": "recording", + "elapsed_s": 3.5, "snapshot": "data:image/jpeg;base64,AAAA", + }) + client.post("/api/capture/heartbeat", json={"device_id": "d2", "name": "laptop"}) + devs = client.get("/api/capture/devices").json() + assert [d["name"] for d in devs] == ["laptop", "phone-left"] # sorted by name + d1 = next(d for d in devs if d["device_id"] == "d1") + assert d1["status"] == "recording" and d1["snapshot"].startswith("data:image/jpeg") + assert "_seen" not in d1 # internals not leaked + capture._devices.clear() + + +def test_oversized_snapshot_dropped_but_device_kept(client): + capture._devices.clear() + client.post("/api/capture/heartbeat", json={ + "device_id": "big", "name": "cam", "snapshot": "data:image/jpeg;base64," + "A" * 400_000, + }) + devs = client.get("/api/capture/devices").json() + assert len(devs) == 1 and "snapshot" not in devs[0] + capture._devices.clear() + + +def test_stale_devices_pruned(client, monkeypatch): + capture._devices.clear() + client.post("/api/capture/heartbeat", json={"device_id": "old", "name": "cam"}) + assert len(client.get("/api/capture/devices").json()) == 1 + monkeypatch.setattr(capture, "DEVICE_STALE_S", -1.0) # everything is instantly stale + assert client.get("/api/capture/devices").json() == [] + capture._devices.clear() + + +def test_capture_page_served(client): + r = client.get("/capture") + assert r.status_code == 200 + assert "Festival 4D — Capture" in r.text + assert "getUserMedia" in r.text # the page really is the capture app diff --git a/deploy/DEPLOY.md b/deploy/DEPLOY.md index 488be8f..59364db 100644 --- a/deploy/DEPLOY.md +++ b/deploy/DEPLOY.md @@ -106,3 +106,13 @@ backend or data changed; a frontend-only change just needs the `dist` rsync). dev-only wide CORS in `config.py` stays as-is; it doesn't affect the hosted site. - **Single project.** The app assumes one project at a time (SQLite at `data/project.db`). - The backend binds `127.0.0.1` only; nginx is the sole public entry point. + +--- + +## Live capture on the public box: leave it OFF + +`FESTIVAL4D_CAPTURE` is **not** set in `deploy/festifun-api.service`, so `/capture` and +`/api/capture/*` are never mounted publicly — an open upload endpoint on a public host would let +anyone write files to the server. Capture is for your local/tailnet machine (see the README's +Capture section: `FESTIVAL4D_CAPTURE=1` + `tailscale serve`). Record locally, run the pipeline, +then deploy the resulting project. diff --git a/frontend/index.html b/frontend/index.html index 4bc19f1..fcbb76e 100644 --- a/frontend/index.html +++ b/frontend/index.html @@ -211,6 +211,10 @@

Festival 4D

synchronized multi-cam replay · 3D viewer · x-ray overlays + + +
diff --git a/frontend/src/main.js b/frontend/src/main.js index a21ba18..2595d8a 100644 --- a/frontend/src/main.js +++ b/frontend/src/main.js @@ -42,6 +42,12 @@ async function boot() { state.videos = manifest.videos; state.tGlobalMax = manifest.t_global_max; state.hasPoses = manifest.has_poses; + // Capture is opt-in on the backend; only surface the link when it's actually mounted. + if (manifest.has_capture) { + const link = el("capture-link"); + link.href = `${API_BASE}/capture`; + link.style.display = ""; + } state.hasSplat = !!manifest.has_splat; for (const v of state.videos) state.enabled[v.id] = true; state.audioSourceId = referenceVideoId(); diff --git a/pyproject.toml b/pyproject.toml index ff59b14..9a0d613 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -14,6 +14,7 @@ dependencies = [ "sqlalchemy>=2.0", "pydantic>=2.6", "opencv-python-headless>=4.9", + "python-multipart>=0.0.9", # multipart uploads for live capture (backend/capture.py) # classifier providers (lane D) — imported lazily, never at module load "google-genai>=0.3", "anthropic>=0.40",