56e569fd7a
Markers/badges -> detection -> cross-view triangulation -> track ribbons + follow-a-friend + LED badge hardware kit. Coordinator-merges policy this phase. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
215 lines
15 KiB
Markdown
215 lines
15 KiB
Markdown
# Phase 6 — Friend Tracks (M18–M24): markers → moving people in the 4D scene
|
||
|
||
Canonical spec for phase 6. **Everything frozen in `../OPUS_BUILD_INSTRUCTIONS.md` (M0–M9)
|
||
and `20-phase5.md` (M10–M17 contracts) still applies.** Standing protocol (status files,
|
||
directives loop, evidence discipline, per-agent worktrees) is unchanged from
|
||
[`README.md`](README.md); the newest round of [`DIRECTIVES.md`](DIRECTIVES.md) always wins.
|
||
|
||
## The idea
|
||
|
||
A friend tag today is a *static* anchor. Phase 6 makes friends **move**: wearable markers
|
||
(distinct-hue patches or blinking LED badges) are detected in every camera, associated across
|
||
synced views, triangulated per timestep, and become **tracks** — `t_global → (x,y,z)` paths
|
||
rendered as ribbons in the 3D scene with moving labels, a follow-a-friend camera mode, and
|
||
capsule support. Same machinery as annotation resolution (M8), run automatically over time.
|
||
|
||
## Context — what exists that you build on (read, don't re-invent)
|
||
|
||
- `geometry.py` — COLMAP↔Three.js conversion, ray casting, triangulation primitives
|
||
(`resolve.py` shows exactly how a bbox becomes a 3D point; your solver is that, per-timestep).
|
||
- `frames.py` — frame sampling via ffmpeg (SfM uses it; detection re-uses it).
|
||
- `config.t_video_from_global` / `t_global_from_video` — the FROZEN timebase; detections are
|
||
found at `t_video`, tracks live at `t_global`.
|
||
- The synthetic fixture (`synthetic.py`) — every acceptance below is against it first.
|
||
- Suite floor is **189 passed**. It only goes up.
|
||
|
||
## Phase map
|
||
|
||
```
|
||
Phase 6a (serial, ONE agent) Phase 6b (parallel, one agent per lane) Phase 6c (serial)
|
||
┌───────────────────────┐ ┌ lane/h-tracker ─── M19 M20 ──────┐ ┌─────────────────┐
|
||
│ foundation3 │ ├ lane/j-ribbons ─── M21 M22 ──────┤ │ integration3 │
|
||
│ fixture ground truth +│──merge──┤ ├──merge─│ full pass + │
|
||
│ contracts + stubs │ └ lane/k-badges ──── M23 M24 ──────┘ │ tracked field │
|
||
│ branch: foundation3 │ (independent — merge in any order) │ test → v0.3.0 │
|
||
└───────────────────────┘ └────────────────────────────────── └─────────────────┘
|
||
```
|
||
|
||
Lanes depend **only** on foundation3, never on each other.
|
||
|
||
## Milestones
|
||
|
||
### M18 — foundation3: fixture ground truth, contracts, stubs (serial, ONE agent)
|
||
1. **Synthetic fixture upgrade** (`synthetic.py` — coordinator-sanctioned edit): render **two
|
||
moving markers** into every camera's video via the existing camera projection:
|
||
- marker A: a solid bright disc (frozen hue, e.g. pure magenta #ff00ff) following a known
|
||
smooth trajectory across the stage (e.g. a slow circle, radius ~2, y≈1.5);
|
||
- marker B: a **blinking** white/bright disc following a different known path, blinking the
|
||
frozen protocol below with ID=5.
|
||
Markers must be small (≤2 % of frame area) and rendered *after* the existing content so the
|
||
full existing suite stays green (offsets, events, poses, SfM tests untouched). Write ground
|
||
truth to `data/work/track_truth.json`: `{"markers": [{"marker_key": ..., "points":
|
||
[{"t_global_s", "x", "y", "z"}]}]}` (Three.js scene space).
|
||
2. **DB:** `tracks` + `track_points` tables (contract #2) + `db.py` helpers
|
||
(`add_track`, `get_tracks`, `set_track_points`, `patch_track`, `delete_track`).
|
||
3. **API:** `GET /api/tracks`, `PATCH /api/tracks/{id}` `{label?, color?}`,
|
||
`DELETE /api/tracks/{id}`, `POST /api/tracks/solve` (dispatches into the lane-H stub with
|
||
the house degradation: valid empty result + note while stubbed). Manifest gains
|
||
`"has_tracks": bool` — update `test_api.py`'s key-set lock consciously in the same commit.
|
||
4. **CLI:** `python -m festival4d track [--detect-only|--solve-only]` → lane-H stubs.
|
||
5. **Backend stubs:** `tracker_detect.py` (`run_detect()`), `tracker_solve.py`
|
||
(`run_solve()`) — `NotImplementedError` with the full contract in the docstring.
|
||
6. **Frontend:** stub `frontend/src/friendTracks.js` imported from `main.js`, `init()` +
|
||
per-frame `friendTracks.update(tGlobal)` wired into the master loop (fx.js pattern);
|
||
hidden **👣 Friends** button `#btn-friends` + empty `#friends-panel` container in
|
||
`index.html`; `state.tracks = []` loaded at boot when `manifest.has_tracks`.
|
||
7. **Capsule:** add `api/tracks` to `capsule.py`'s bake list (additive; update lane G's
|
||
bundle-structure test consciously in the same commit, CR-style note in the commit message).
|
||
8. Tests for every new route shape; suite ≥ 189 + new tests. Merge fast — three lanes wait.
|
||
|
||
### M19 — Marker detection (lane H, backend)
|
||
Fill `tracker_detect.py`. `run_detect()`: sample frames (~8 fps via `frames.py` machinery) from
|
||
every video; detect markers two ways: **color mode** (HSV blob of the frozen marker hues;
|
||
`marker_key = "hue:<bucket>"`) and **blink mode** (per-blob brightness over a sliding window,
|
||
decode the frozen OOK protocol; `marker_key = "code:<id>"`). Write
|
||
`data/work/detections.json` (contract #3). OpenCV (`opencv-python-headless`) is already a
|
||
dependency. No markers → valid-empty detections, never a crash.
|
||
**Accept:** on the synthetic fixture, ≥90 % of sampled frames where a marker is visible yield
|
||
a detection within 3 px (normalized ≤0.008) of the projected ground-truth center; blink ID 5
|
||
decoded; tests in `test_tracker_detect.py`.
|
||
|
||
### M20 — Track solving (lane H, backend)
|
||
Fill `tracker_solve.py`. `run_solve()`: load detections; group by `marker_key`; per time bucket
|
||
(≈0.25 s) collect same-key detections across cameras at matching `t_global`; **2+ views →
|
||
triangulate** through `geometry` primitives (the `resolve.py` pattern — never inline pose
|
||
math); **1 view → ray ∩ ground plane** (contract #5); smooth each track (moving average or
|
||
Catmull-Rom, your call — document it); write `tracks`/`track_points` through `db` helpers;
|
||
solid color for hue markers, palette color for code markers. `POST /api/tracks/solve` and the
|
||
CLI now run detect→solve end-to-end.
|
||
**Accept:** synthetic recovery — median 3D error of both tracks < 0.3 scene units against
|
||
`track_truth.json`, ≥80 % temporal coverage of each marker's visible span; single-view
|
||
fallback exercised by a test that hides all-but-one camera's detections; degradation (no
|
||
detections → zero tracks + note); tests in `test_tracker_solve.py`.
|
||
|
||
### M21 — Ribbons, moving labels, friends panel (lane J, frontend)
|
||
Fill `friendTracks.js`. On init (when `manifest.has_tracks`): unhide 👣; fetch
|
||
`/api/tracks` into `state.tracks`. Toggle renders per-track: a **ribbon** polyline through the
|
||
points (track color) + a **moving marker** (sphere + label sprite, position interpolated at
|
||
the current `t_global`, hidden outside the track's time span) — objects via
|
||
`scene3d.addObject/removeObject` only, registered with `scene3d.registerHelper` so photo mode
|
||
hides them. Panel in `#friends-panel` (anchorPanel.js is the style/UX reference — read it,
|
||
don't edit it): list tracks (dot, label, time span), rename/recolor via
|
||
`PATCH /api/tracks/{id}` (controls carry `write-ui`), delete, and a **Follow** button per row.
|
||
**Accept:** live-browser evidence — ribbons + moving labels track the synthetic markers in
|
||
play and scrub; rename/recolor/delete round-trip; photo mode hides ribbons; capsule mode
|
||
hides write controls but keeps ribbons/follow.
|
||
|
||
### M22 — Follow-a-friend camera (lane J, frontend)
|
||
Chase cam on any track: `scene3d.enterPathMode()` hands you the camera (the camPath.js
|
||
pattern — read it); every `update(tGlobal)` compute the track position, place the camera at a
|
||
smoothed offset (behind/above, ~4 units, low-pass the position so it glides), look at the
|
||
friend (`applyCameraPose`). Esc / Free-roam / snap-to-camera exits (pathMode already handles
|
||
contention — mirror camPath's "someone else took the camera" check).
|
||
**Accept:** live-browser evidence — camera glides after the moving marker in play and scrub;
|
||
clean handoff to/from free-roam, snap-to-cam, and camPath playback; no fighting.
|
||
|
||
### M23 — LED badge firmware + hardware kit (lane K, hardware/docs)
|
||
`hardware/badge/badge.ino` — ESP32/ATtiny-compatible Arduino sketch blinking the frozen OOK
|
||
protocol (contract #4): configurable 6-bit ID, bright single LED or WS2812; plus
|
||
`hardware/beacon/beacon.ino` — the sync/calibration beacon (distinct reserved ID 63,
|
||
double-brightness). `docs/hardware.md`: parts list (boards, LEDs, battery, diffuser), build
|
||
photos placeholders, wearing guidance (chest/hat height, diffusion, spacing), and the
|
||
camera-side truth: what ranges/framerates the protocol survives (≥24 fps, exposure caveats).
|
||
**Accept:** sketches compile (`arduino-cli compile` for esp32 + attiny if available — else
|
||
document exact compile commands and mark unverified); protocol constants byte-identical to
|
||
contract #4 (import/duplicate them in ONE header with a test-facing copy check documented).
|
||
|
||
### M24 — Bench self-test tool (lane K, backend script)
|
||
`scripts/badge_selftest.py`: given a short phone clip of a badge on a desk, run the SAME
|
||
decoder path as lane H (`tracker_detect` blink mode) and print the decoded ID + confidence —
|
||
John's hardware smoke test before a shoot. Works on a synthetic self-test too:
|
||
`--synthetic` renders a blinking clip via the fixture helpers and asserts round-trip.
|
||
**Accept:** `uv run python scripts/badge_selftest.py --synthetic` decodes the rendered ID;
|
||
clear failure text for: no blob, ID mismatch, too-short clip. Small test in
|
||
`test_badge_selftest.py` (synthetic path only).
|
||
|
||
## Contracts (frozen at foundation3 merge)
|
||
|
||
1. **Marker hues (color mode):** magenta `#ff00ff` and cyan `#00ffff` are the two supported
|
||
fixture hues; real shoots may configure others via `FESTIVAL4D_MARKER_HUES` (comma-sep hex).
|
||
2. **Schema:**
|
||
`tracks(id INTEGER PK, marker_key TEXT NOT NULL, label TEXT, color TEXT, created_at TEXT)`
|
||
`track_points(id INTEGER PK, track_id FK, t_global_s FLOAT, x FLOAT, y FLOAT, z FLOAT,
|
||
quality FLOAT, views INTEGER)` — points ordered by `t_global_s`; `views` = cameras used
|
||
(1 = ground-plane fallback); coordinates in **Three.js scene space**.
|
||
`GET /api/tracks` → `[{id, marker_key, label, color, points: [{t_global_s,x,y,z,quality,views}]}]`.
|
||
3. **detections.json:** `{"videos": {"<video_id>": [{"t_video_s": float, "marker_key": str,
|
||
"cx": float, "cy": float, "area": float, "conf": float 0..1}]}, "generated_by":
|
||
"festival4d.tracker_detect"}` — `cx`/`cy` normalized 0..1 (bbox-annotation convention).
|
||
4. **Blink protocol (OOK, camera-decodable):** bit period **133 ms** (4 frames @ 30 fps);
|
||
word = preamble `11100` + 6-bit ID (MSB first) + even parity = 12 bits, repeated
|
||
continuously (~1.6 s/word). ID 63 reserved for the sync beacon. Decoder needs ≥24 fps
|
||
video and ≥1 full word of visibility.
|
||
5. **Ground plane:** the synthetic fixture's ground is `y = 0` (scene space). For real
|
||
projects the solver estimates it as the 5th-percentile-y horizontal plane of the point
|
||
cloud and records it in its result dict. Single-view fallback intersects the pixel ray
|
||
with this plane; such points get `views = 1` and `quality ≤ 0.5`.
|
||
6. **Frozen during 6b:** everything frozen in phases 1–5 **plus** all phase-5 lane modules
|
||
(`director.js`, `exportVideo.js`, `anchorPanel.js`, `photoMode.js`, `fx.js`,
|
||
`pathsStore.js`, `audio_features.py`, `director.py`, `capsule.py`) and foundation3's edits
|
||
(`synthetic.py`, `main.js`, `index.html`, api/cli/db). Lanes fill their own new files only.
|
||
Believe a frozen file must change → `plan/CHANGE_REQUESTS.md`, work around, integration3
|
||
adjudicates. Stub-era test assertions superseded by your milestone landing follow the
|
||
CR-1/-4/-5 pattern: minimal edit + CR entry.
|
||
|
||
## integration3 scope (Phase 6c)
|
||
Merge order irrelevant; resolve CRs; full synthetic pass (evidence per milestone); raise the
|
||
suite floor; **tracked field test** — real clips with at least one badge/marker in frame
|
||
(combine with the still-open phase-5 field test); failures → `plan/ISSUES.md`; propose
|
||
`v0.3.0` (with `v0.2.0` if the phase-5 field test passes in the same session).
|
||
|
||
## File ownership (Phase 6b)
|
||
|
||
| Owner | Files |
|
||
|---|---|
|
||
| Lane H | `backend/festival4d/tracker_detect.py`, `tracker_solve.py`, `backend/tests/test_tracker_detect.py`, `test_tracker_solve.py` |
|
||
| Lane J | `frontend/src/friendTracks.js` |
|
||
| Lane K | `hardware/**`, `docs/hardware.md`, `scripts/badge_selftest.py`, `backend/tests/test_badge_selftest.py` |
|
||
| Every agent | its own `plan/status/<lane>.md` |
|
||
| **Coordinator only** | `plan/DIRECTIVES.md` |
|
||
| **Frozen during 6b** | contract #6 above |
|
||
|
||
## Phase-6 pitfalls
|
||
|
||
1. **Time alignment is the whole game.** Detections happen at `t_video`; association happens
|
||
at `t_global`. One `t_video_from_global` inversion mistake silently shears every track.
|
||
Use the frozen helpers; test association against ground truth, not against itself.
|
||
2. **Stage lighting lies about color.** The color detector must gate on saturation+value, not
|
||
hue alone, and the acceptance is on the *synthetic* fixture — note in your status file that
|
||
real-world hue robustness is a known field-test risk (that's why blink mode exists).
|
||
3. **Blink decoding vs. video fps:** cameras at 24–60 fps sample the 133 ms bit period
|
||
differently. Decode by integrating brightness per bit window using the video's actual fps
|
||
(`meta.fps`), never by counting frames.
|
||
4. **Don't fight the camera owners.** Follow-mode (M22) shares `pathMode` with camPath —
|
||
mirror camPath's contention checks exactly, or two drivers will interleave writes.
|
||
5. **rAF throttle (standing):** verify with the `__f4d` pump in hidden panes
|
||
(`localStorage f4dDebug=1`); the WebAudio timer exemption gives you ~50 Hz `setTimeout`.
|
||
6. **Fixture compatibility:** foundation3's synthetic edit runs UNDER the existing 189 tests —
|
||
markers must not move offsets, events, poses, or break SfM-adjacent tests.
|
||
|
||
## Kickoff prompts (copy-paste one per Opus 4.8 session, in run order)
|
||
|
||
**foundation3 (first, alone):**
|
||
> Clone ssh://git@100.71.119.27:222/monster/festifun.git. Read OPUS_BUILD_INSTRUCTIONS.md,
|
||
> plan/20-phase5.md, plan/30-phase6.md (M18 is yours), and the latest round of
|
||
> plan/DIRECTIVES.md. Execute foundation3 on branch `foundation3`; maintain
|
||
> plan/status/foundation3.md per plan/status/TEMPLATE.md; suite ≥ 189 + your tests; push the
|
||
> branch and report — the coordinator reviews and merges.
|
||
|
||
**lane H / lane J / lane K (parallel, after foundation3 merges):**
|
||
> Clone ssh://git@100.71.119.27:222/monster/festifun.git and create your own git worktree
|
||
> before any edit. Read OPUS_BUILD_INSTRUCTIONS.md, plan/30-phase6.md, plan/lane-<X>.md, and
|
||
> the latest round of plan/DIRECTIVES.md. Execute your lane on branch `lane/<x>-<name>`;
|
||
> maintain plan/status/lane-<X>.md; obey the ownership table and frozen-files contract; push
|
||
> your branch and report — the coordinator reviews and merges. Do not merge to main.
|