cf6a6fb2c8
What now works: - M0 scaffold: pyproject (all spec deps), uv/py3.12 env, `python -m festival4d` CLI registering synthetic|ingest|sync|reconstruct|events|serve. Vite hello page. - Synthetic fixture (synthetic.py): 3 shifted-audio videos (offsets 0/+1370/-842 ms), camera-arc poses, stage point cloud -> points.ply, seeded events + anchors, ground_truth.json. `python -m festival4d synthetic` populates data/ + DB. - DB schema exactly per spec §2 (db.py) + CRUD helpers all lanes use. - M3 API (api.py) full against synthetic data: manifest/poses/pointcloud/anchors/ events/detect/annotations; Range-capable video serving (206 verified); CORS for any localhost origin. - Frozen geometry contract: geometry.colmap_to_threejs (M5 math) + unit test (3 known vectors, random round-trip, scipy oracle); mirrored frontend/src/lib/pose.js with identical POSE_TEST_VECTORS. Lane-B stubs: slerp_pose, ray_from_pixel, triangulate_rays, nearest_point_on_ray. - Classifier contract (events_ai.py): MomentClassification model + MomentClassifier protocol + Gemini/Claude/Local provider stubs. - Lane-owned modules stubbed with final signatures (ingest, audio_sync, frames, sfm, events_ai); cli/api catch NotImplementedError and degrade gracefully. - plan/CHANGE_REQUESTS.md created; plan/status/foundation.md updated. Acceptance: pytest 24 passed; serve endpoints verified via curl + browser (video seek, manifest fetch cross-origin, pose.js self-test, 0 console errors). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
67 lines
2.8 KiB
Python
67 lines
2.8 KiB
Python
"""GCC-PHAT pairwise audio offsets + global graph solve + drift (spec M1, lane A).
|
|
|
|
STUB — lane A fills the bodies; the public signatures below are frozen (``cli.py`` calls
|
|
``run_sync``).
|
|
|
|
Algorithm (spec M1, for lane A's reference):
|
|
- **GCC-PHAT**, not raw cross-correlation: whiten the cross-spectrum
|
|
``R = X * conj(Y); R /= |R| + eps; r = irfft(R)`` — robust to loud/clipped/reverberant
|
|
concert audio. numpy/scipy FFTs; librosa only for loading.
|
|
- All **pairwise** offsets, each scored by peak-to-second-peak ratio; then a global
|
|
least-squares solve over the offset graph, weighted by confidence, rejecting pairs that
|
|
violate cycle consistency (``off_ab + off_bc - off_ac > 50 ms``).
|
|
- **Drift**: re-run GCC-PHAT on 10 s windows every 30 s along the overlap; fit a line;
|
|
store the slope as ``drift_ppm`` (store 0 if ``|drift| < 5 ppm``).
|
|
- Disconnected sync-graph components -> ``offset_ms = None`` (never a guess; spec pitfall #5).
|
|
- Persist via ``db.update_video_sync`` and export ``config.SYNC_JSON``.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
|
|
import numpy as np
|
|
|
|
log = logging.getLogger("festival4d.audio_sync")
|
|
|
|
|
|
def gcc_phat(sig: np.ndarray, ref: np.ndarray, sr: int,
|
|
max_lag_s: float | None = None) -> tuple[float, float]:
|
|
"""GCC-PHAT time delay of ``sig`` relative to ``ref`` (spec M1, lane A).
|
|
|
|
Returns ``(offset_s, confidence)`` where confidence is the peak-to-second-peak ratio.
|
|
"""
|
|
raise NotImplementedError("lane A (M1): implement gcc_phat")
|
|
|
|
|
|
def pairwise_offsets(signals: dict[int, np.ndarray], sr: int) -> list[dict]:
|
|
"""All pairwise GCC-PHAT offsets with confidences (spec M1, lane A).
|
|
|
|
``signals`` maps ``video_id -> mono samples``. Returns a list of
|
|
``{a, b, offset_s, confidence}`` edges.
|
|
"""
|
|
raise NotImplementedError("lane A (M1): implement pairwise_offsets")
|
|
|
|
|
|
def solve_global_offsets(edges: list[dict], video_ids: list[int]) -> dict[int, float | None]:
|
|
"""Least-squares global solve over the pairwise-offset graph (spec M1, lane A).
|
|
|
|
Reference node is pinned to 0; disconnected components get ``None`` (spec pitfall #5).
|
|
Returns ``video_id -> offset_ms`` (or ``None``).
|
|
"""
|
|
raise NotImplementedError("lane A (M1): implement solve_global_offsets")
|
|
|
|
|
|
def estimate_drift(sig: np.ndarray, ref: np.ndarray, sr: int) -> float:
|
|
"""Estimate clock drift in ppm by fitting window-offsets over the overlap (spec M1)."""
|
|
raise NotImplementedError("lane A (M1): implement estimate_drift")
|
|
|
|
|
|
def run_sync() -> dict:
|
|
"""Sync every ingested video: pairwise GCC-PHAT, global solve, drift, persist.
|
|
|
|
Entrypoint for ``python -m festival4d sync``. Writes ``offset_ms``/``drift_ppm``/
|
|
``sync_confidence`` via ``db`` and exports ``config.SYNC_JSON``. Returns the solution.
|
|
"""
|
|
raise NotImplementedError("lane A (M1): implement run_sync")
|