# Phase 6 — Friend Tracks (M18–M24): markers → moving people in the 4D scene

Canonical spec for phase 6. **Everything frozen in `../OPUS_BUILD_INSTRUCTIONS.md` (M0–M9)
and `20-phase5.md` (M10–M17 contracts) still applies.** Standing protocol (status files,
directives loop, evidence discipline, per-agent worktrees) is unchanged from
[`README.md`](README.md); the newest round of [`DIRECTIVES.md`](DIRECTIVES.md) always wins.

## The idea

A friend tag today is a *static* anchor. Phase 6 makes friends **move**: wearable markers
(distinct-hue patches or blinking LED badges) are detected in every camera, associated across
synced views, triangulated per timestep, and become **tracks** — `t_global → (x,y,z)` paths
rendered as ribbons in the 3D scene with moving labels, a follow-a-friend camera mode, and
capsule support. Same machinery as annotation resolution (M8), run automatically over time.

## Context — what exists that you build on (read, don't re-invent)

- `geometry.py` — COLMAP↔Three.js conversion, ray casting, triangulation primitives
  (`resolve.py` shows exactly how a bbox becomes a 3D point; your solver is that, per-timestep).
- `frames.py` — frame sampling via ffmpeg (SfM uses it; detection re-uses it).
- `config.t_video_from_global` / `t_global_from_video` — the FROZEN timebase; detections are
  found at `t_video`, tracks live at `t_global`.
- The synthetic fixture (`synthetic.py`) — every acceptance below is against it first.
- Suite floor is **189 passed**. It only goes up.

## Phase map

```
Phase 6a (serial, ONE agent)      Phase 6b (parallel, one agent per lane)     Phase 6c (serial)
┌───────────────────────┐         ┌ lane/h-tracker ─── M19 M20 ──────┐        ┌─────────────────┐
│ foundation3           │         ├ lane/j-ribbons ─── M21 M22 ──────┤        │ integration3    │
│ fixture ground truth +│──merge──┤                                  ├──merge─│ full pass +     │
│ contracts + stubs     │         └ lane/k-badges ──── M23 M24 ──────┘        │ tracked field   │
│ branch: foundation3   │         (independent — merge in any order)          │ test → v0.3.0   │
└───────────────────────┘         └──────────────────────────────────         └─────────────────┘
```

Lanes depend **only** on foundation3, never on each other.

## Milestones

### M18 — foundation3: fixture ground truth, contracts, stubs (serial, ONE agent)
1. **Synthetic fixture upgrade** (`synthetic.py` — coordinator-sanctioned edit): render **two
   moving markers** into every camera's video via the existing camera projection:
   - marker A: a solid bright disc (frozen hue, e.g. pure magenta #ff00ff) following a known
     smooth trajectory across the stage (e.g. a slow circle, radius ~2, y≈1.5);
   - marker B: a **blinking** white/bright disc following a different known path, blinking the
     frozen protocol below with ID=5.
   Markers must be small (≤2 % of frame area) and rendered *after* the existing content so the
   full existing suite stays green (offsets, events, poses, SfM tests untouched). Write ground
   truth to `data/work/track_truth.json`: `{"markers": [{"marker_key": ..., "points":
   [{"t_global_s", "x", "y", "z"}]}]}` (Three.js scene space).
2. **DB:** `tracks` + `track_points` tables (contract #2) + `db.py` helpers
   (`add_track`, `get_tracks`, `set_track_points`, `patch_track`, `delete_track`).
3. **API:** `GET /api/tracks`, `PATCH /api/tracks/{id}` `{label?, color?}`,
   `DELETE /api/tracks/{id}`, `POST /api/tracks/solve` (dispatches into the lane-H stub with
   the house degradation: valid empty result + note while stubbed). Manifest gains
   `"has_tracks": bool` — update `test_api.py`'s key-set lock consciously in the same commit.
4. **CLI:** `python -m festival4d track [--detect-only|--solve-only]` → lane-H stubs.
5. **Backend stubs:** `tracker_detect.py` (`run_detect()`), `tracker_solve.py`
   (`run_solve()`) — `NotImplementedError` with the full contract in the docstring.
6. **Frontend:** stub `frontend/src/friendTracks.js` imported from `main.js`, `init()` +
   per-frame `friendTracks.update(tGlobal)` wired into the master loop (fx.js pattern);
   hidden **👣 Friends** button `#btn-friends` + empty `#friends-panel` container in
   `index.html`; `state.tracks = []` loaded at boot when `manifest.has_tracks`.
7. **Capsule:** add `api/tracks` to `capsule.py`'s bake list (additive; update lane G's
   bundle-structure test consciously in the same commit, CR-style note in the commit message).
8. Tests for every new route shape; suite ≥ 189 + new tests. Merge fast — three lanes wait.

### M19 — Marker detection (lane H, backend)
Fill `tracker_detect.py`. `run_detect()`: sample frames (~8 fps via `frames.py` machinery) from
every video; detect markers two ways: **color mode** (HSV blob of the frozen marker hues;
`marker_key = "hue:<bucket>"`) and **blink mode** (per-blob brightness over a sliding window,
decode the frozen OOK protocol; `marker_key = "code:<id>"`). Write
`data/work/detections.json` (contract #3). OpenCV (`opencv-python-headless`) is already a
dependency. No markers → valid-empty detections, never a crash.
**Accept:** on the synthetic fixture, ≥90 % of sampled frames where a marker is visible yield
a detection within 3 px (normalized ≤0.008) of the projected ground-truth center; blink ID 5
decoded; tests in `test_tracker_detect.py`.

### M20 — Track solving (lane H, backend)
Fill `tracker_solve.py`. `run_solve()`: load detections; group by `marker_key`; per time bucket
(≈0.25 s) collect same-key detections across cameras at matching `t_global`; **2+ views →
triangulate** through `geometry` primitives (the `resolve.py` pattern — never inline pose
math); **1 view → ray ∩ ground plane** (contract #5); smooth each track (moving average or
Catmull-Rom, your call — document it); write `tracks`/`track_points` through `db` helpers;
solid color for hue markers, palette color for code markers. `POST /api/tracks/solve` and the
CLI now run detect→solve end-to-end.
**Accept:** synthetic recovery — median 3D error of both tracks < 0.3 scene units against
`track_truth.json`, ≥80 % temporal coverage of each marker's visible span; single-view
fallback exercised by a test that hides all-but-one camera's detections; degradation (no
detections → zero tracks + note); tests in `test_tracker_solve.py`.

### M21 — Ribbons, moving labels, friends panel (lane J, frontend)
Fill `friendTracks.js`. On init (when `manifest.has_tracks`): unhide 👣; fetch
`/api/tracks` into `state.tracks`. Toggle renders per-track: a **ribbon** polyline through the
points (track color) + a **moving marker** (sphere + label sprite, position interpolated at
the current `t_global`, hidden outside the track's time span) — objects via
`scene3d.addObject/removeObject` only, registered with `scene3d.registerHelper` so photo mode
hides them. Panel in `#friends-panel` (anchorPanel.js is the style/UX reference — read it,
don't edit it): list tracks (dot, label, time span), rename/recolor via
`PATCH /api/tracks/{id}` (controls carry `write-ui`), delete, and a **Follow** button per row.
**Accept:** live-browser evidence — ribbons + moving labels track the synthetic markers in
play and scrub; rename/recolor/delete round-trip; photo mode hides ribbons; capsule mode
hides write controls but keeps ribbons/follow.

### M22 — Follow-a-friend camera (lane J, frontend)
Chase cam on any track: `scene3d.enterPathMode()` hands you the camera (the camPath.js
pattern — read it); every `update(tGlobal)` compute the track position, place the camera at a
smoothed offset (behind/above, ~4 units, low-pass the position so it glides), look at the
friend (`applyCameraPose`). Esc / Free-roam / snap-to-camera exits (pathMode already handles
contention — mirror camPath's "someone else took the camera" check).
**Accept:** live-browser evidence — camera glides after the moving marker in play and scrub;
clean handoff to/from free-roam, snap-to-cam, and camPath playback; no fighting.

### M23 — LED badge firmware + hardware kit (lane K, hardware/docs)
`hardware/badge/badge.ino` — ESP32/ATtiny-compatible Arduino sketch blinking the frozen OOK
protocol (contract #4): configurable 6-bit ID, bright single LED or WS2812; plus
`hardware/beacon/beacon.ino` — the sync/calibration beacon (distinct reserved ID 63,
double-brightness). `docs/hardware.md`: parts list (boards, LEDs, battery, diffuser), build
photos placeholders, wearing guidance (chest/hat height, diffusion, spacing), and the
camera-side truth: what ranges/framerates the protocol survives (≥24 fps, exposure caveats).
**Accept:** sketches compile (`arduino-cli compile` for esp32 + attiny if available — else
document exact compile commands and mark unverified); protocol constants byte-identical to
contract #4 (import/duplicate them in ONE header with a test-facing copy check documented).

### M24 — Bench self-test tool (lane K, backend script)
`scripts/badge_selftest.py`: given a short phone clip of a badge on a desk, run the SAME
decoder path as lane H (`tracker_detect` blink mode) and print the decoded ID + confidence —
John's hardware smoke test before a shoot. Works on a synthetic self-test too:
`--synthetic` renders a blinking clip via the fixture helpers and asserts round-trip.
**Accept:** `uv run python scripts/badge_selftest.py --synthetic` decodes the rendered ID;
clear failure text for: no blob, ID mismatch, too-short clip. Small test in
`test_badge_selftest.py` (synthetic path only).

## Contracts (frozen at foundation3 merge)

1. **Marker hues (color mode):** magenta `#ff00ff` and cyan `#00ffff` are the two supported
   fixture hues; real shoots may configure others via `FESTIVAL4D_MARKER_HUES` (comma-sep hex).
2. **Schema:**
   `tracks(id INTEGER PK, marker_key TEXT NOT NULL, label TEXT, color TEXT, created_at TEXT)`
   `track_points(id INTEGER PK, track_id FK, t_global_s FLOAT, x FLOAT, y FLOAT, z FLOAT,
   quality FLOAT, views INTEGER)` — points ordered by `t_global_s`; `views` = cameras used
   (1 = ground-plane fallback); coordinates in **Three.js scene space**.
   `GET /api/tracks` → `[{id, marker_key, label, color, points: [{t_global_s,x,y,z,quality,views}]}]`.
3. **detections.json:** `{"videos": {"<video_id>": [{"t_video_s": float, "marker_key": str,
   "cx": float, "cy": float, "area": float, "conf": float 0..1}]}, "generated_by":
   "festival4d.tracker_detect"}` — `cx`/`cy` normalized 0..1 (bbox-annotation convention).
4. **Blink protocol (OOK, camera-decodable):** bit period **133 ms** (4 frames @ 30 fps);
   word = preamble `11100` + 6-bit ID (MSB first) + even parity = 12 bits, repeated
   continuously (~1.6 s/word). ID 63 reserved for the sync beacon. Decoder needs ≥24 fps
   video and ≥1 full word of visibility.
5. **Ground plane:** the synthetic fixture's ground is `y = 0` (scene space). For real
   projects the solver estimates it as the 5th-percentile-y horizontal plane of the point
   cloud and records it in its result dict. Single-view fallback intersects the pixel ray
   with this plane; such points get `views = 1` and `quality ≤ 0.5`.
6. **Frozen during 6b:** everything frozen in phases 1–5 **plus** all phase-5 lane modules
   (`director.js`, `exportVideo.js`, `anchorPanel.js`, `photoMode.js`, `fx.js`,
   `pathsStore.js`, `audio_features.py`, `director.py`, `capsule.py`) and foundation3's edits
   (`synthetic.py`, `main.js`, `index.html`, api/cli/db). Lanes fill their own new files only.
   Believe a frozen file must change → `plan/CHANGE_REQUESTS.md`, work around, integration3
   adjudicates. Stub-era test assertions superseded by your milestone landing follow the
   CR-1/-4/-5 pattern: minimal edit + CR entry.

## integration3 scope (Phase 6c)
Merge order irrelevant; resolve CRs; full synthetic pass (evidence per milestone); raise the
suite floor; **tracked field test** — real clips with at least one badge/marker in frame
(combine with the still-open phase-5 field test); failures → `plan/ISSUES.md`; propose
`v0.3.0` (with `v0.2.0` if the phase-5 field test passes in the same session).

## File ownership (Phase 6b)

| Owner | Files |
|---|---|
| Lane H | `backend/festival4d/tracker_detect.py`, `tracker_solve.py`, `backend/tests/test_tracker_detect.py`, `test_tracker_solve.py` |
| Lane J | `frontend/src/friendTracks.js` |
| Lane K | `hardware/**`, `docs/hardware.md`, `scripts/badge_selftest.py`, `backend/tests/test_badge_selftest.py` |
| Every agent | its own `plan/status/<lane>.md` |
| **Coordinator only** | `plan/DIRECTIVES.md` |
| **Frozen during 6b** | contract #6 above |

## Phase-6 pitfalls

1. **Time alignment is the whole game.** Detections happen at `t_video`; association happens
   at `t_global`. One `t_video_from_global` inversion mistake silently shears every track.
   Use the frozen helpers; test association against ground truth, not against itself.
2. **Stage lighting lies about color.** The color detector must gate on saturation+value, not
   hue alone, and the acceptance is on the *synthetic* fixture — note in your status file that
   real-world hue robustness is a known field-test risk (that's why blink mode exists).
3. **Blink decoding vs. video fps:** cameras at 24–60 fps sample the 133 ms bit period
   differently. Decode by integrating brightness per bit window using the video's actual fps
   (`meta.fps`), never by counting frames.
4. **Don't fight the camera owners.** Follow-mode (M22) shares `pathMode` with camPath —
   mirror camPath's contention checks exactly, or two drivers will interleave writes.
5. **rAF throttle (standing):** verify with the `__f4d` pump in hidden panes
   (`localStorage f4dDebug=1`); the WebAudio timer exemption gives you ~50 Hz `setTimeout`.
6. **Fixture compatibility:** foundation3's synthetic edit runs UNDER the existing 189 tests —
   markers must not move offsets, events, poses, or break SfM-adjacent tests.

## Kickoff prompts (copy-paste one per Opus 4.8 session, in run order)

**foundation3 (first, alone):**
> Clone ssh://git@100.71.119.27:222/monster/festifun.git. Read OPUS_BUILD_INSTRUCTIONS.md,
> plan/20-phase5.md, plan/30-phase6.md (M18 is yours), and the latest round of
> plan/DIRECTIVES.md. Execute foundation3 on branch `foundation3`; maintain
> plan/status/foundation3.md per plan/status/TEMPLATE.md; suite ≥ 189 + your tests; push the
> branch and report — the coordinator reviews and merges.

**lane H / lane J / lane K (parallel, after foundation3 merges):**
> Clone ssh://git@100.71.119.27:222/monster/festifun.git and create your own git worktree
> before any edit. Read OPUS_BUILD_INSTRUCTIONS.md, plan/30-phase6.md, plan/lane-<X>.md, and
> the latest round of plan/DIRECTIVES.md. Execute your lane on branch `lane/<x>-<name>`;
> maintain plan/status/lane-<X>.md; obey the ownership table and frozen-files contract; push
> your branch and report — the coordinator reviews and merges. Do not merge to main.