diff --git a/README.md b/README.md index cc7ae0a..07eae80 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,9 @@ fully offline and local; the only optional cloud piece is moment classification. correct any misclassification. - **Click to locate things in 3D**: draw a box around an object in one or two videos and the app triangulates its 3D position, dropping an anchor that appears everywhere at once. +- **Hear the scene in 3D** 🎧: toggle spatial audio and every camera becomes a positional + sound source at its reconstructed pose — fly toward a camera and you hear *that* spot in + the crowd get closer. --- @@ -154,7 +157,8 @@ export GEMINI_API_KEY=... # default provider (Gemini flash, native v | Play / pause | `Space` or the ⏵ button | | Nudge time ±1 s (±5 s) | `←` / `→` (hold `Shift`) | | Scrub | drag the timeline | -| Pick the audio you hear | click a video (only one is unmuted at a time) | +| Pick the audio you hear | click a video (one soundtrack at a time) | +| Spatial audio | 🎧 **3D audio** (needs poses): every camera plays from its position in the scene; the mix follows the viewer camera as you fly | | Enable/disable a video | ◉ on the video | | Orbit the 3D scene | drag in the 3D pane | | Snap the 3D view to a camera | click its frustum, its ⛶ button, or press `1`–`9` | @@ -194,9 +198,13 @@ data/raw/ ──ingest──▶ data/work/audio ──sync────▶ videos classifiers), `resolve` (annotation→3D), `api` (FastAPI, Range-capable video serving), `db` (SQLAlchemy/SQLite), `synthetic` (the test fixture). - **Frontend** — vanilla JS + Three.js. A single master clock (`transport.js`) derives each - video's local time from the shared timeline and continuously corrects playback; `lib/pose.js` - is the one COLMAP→Three.js pose conversion, mirrored from `geometry.py` and locked by a shared - test. + video's local time from the shared timeline and continuously corrects playback. When possible + the clock is **audio**: the selected soundtrack (`GET /api/audio/{id}`, extracted+cached + server-side) plays through WebAudio and `t_global` derives from `AudioContext.currentTime` — + so every video is muted picture-only and can be seeked/rate-trimmed with zero audible + artifacts (falls back to a `performance.now()` clock + `