festifun/plan/30-phase6.md

Phase 6 — Friend Tracks (M18–M24): markers → moving people in the 4D scene

Canonical spec for phase 6. Everything frozen in ../OPUS_BUILD_INSTRUCTIONS.md (M0–M9) and 20-phase5.md (M10–M17 contracts) still applies. Standing protocol (status files, directives loop, evidence discipline, per-agent worktrees) is unchanged from README.md; the newest round of DIRECTIVES.md always wins.

The idea

A friend tag today is a static anchor. Phase 6 makes friends move: wearable markers (distinct-hue patches or blinking LED badges) are detected in every camera, associated across synced views, triangulated per timestep, and become tracks — t_global → (x,y,z) paths rendered as ribbons in the 3D scene with moving labels, a follow-a-friend camera mode, and capsule support. Same machinery as annotation resolution (M8), run automatically over time.

Context — what exists that you build on (read, don't re-invent)

• geometry.py — COLMAP↔Three.js conversion, ray casting, triangulation primitives (resolve.py shows exactly how a bbox becomes a 3D point; your solver is that, per-timestep).
• frames.py — frame sampling via ffmpeg (SfM uses it; detection re-uses it).
• config.t_video_from_global / t_global_from_video — the FROZEN timebase; detections are found at t_video, tracks live at t_global.
• The synthetic fixture (synthetic.py) — every acceptance below is against it first.
• Suite floor is 189 passed. It only goes up.

Phase map

Phase 6a (serial, ONE agent)      Phase 6b (parallel, one agent per lane)     Phase 6c (serial)
┌───────────────────────┐         ┌ lane/h-tracker ─── M19 M20 ──────┐        ┌─────────────────┐
│ foundation3           │         ├ lane/j-ribbons ─── M21 M22 ──────┤        │ integration3    │
│ fixture ground truth +│──merge──┤                                  ├──merge─│ full pass +     │
│ contracts + stubs     │         └ lane/k-badges ──── M23 M24 ──────┘        │ tracked field   │
│ branch: foundation3   │         (independent — merge in any order)          │ test → v0.3.0   │
└───────────────────────┘         └──────────────────────────────────         └─────────────────┘

Lanes depend only on foundation3, never on each other.

Milestones

M18 — foundation3: fixture ground truth, contracts, stubs (serial, ONE agent)

1. Synthetic fixture upgrade (synthetic.py — coordinator-sanctioned edit): render two moving markers into every camera's video via the existing camera projection:
   • marker A: a solid bright disc (frozen hue, e.g. pure magenta #ff00ff) following a known smooth trajectory across the stage (e.g. a slow circle, radius ~2, y≈1.5);
   • marker B: a blinking white/bright disc following a different known path, blinking the frozen protocol below with ID=5. Markers must be small (≤2 % of frame area) and rendered after the existing content so the full existing suite stays green (offsets, events, poses, SfM tests untouched). Write ground truth to data/work/track_truth.json: {"markers": [{"marker_key": ..., "points": [{"t_global_s", "x", "y", "z"}]}]} (Three.js scene space).
2. DB: tracks + track_points tables (contract #2) + db.py helpers (add_track, get_tracks, set_track_points, patch_track, delete_track).
3. API: GET /api/tracks, PATCH /api/tracks/{id} {label?, color?}, DELETE /api/tracks/{id}, POST /api/tracks/solve (dispatches into the lane-H stub with the house degradation: valid empty result + note while stubbed). Manifest gains "has_tracks": bool — update test_api.py's key-set lock consciously in the same commit.
4. CLI: python -m festival4d track [--detect-only|--solve-only] → lane-H stubs.
5. Backend stubs: tracker_detect.py (run_detect()), tracker_solve.py (run_solve()) — NotImplementedError with the full contract in the docstring.
6. Frontend: stub frontend/src/friendTracks.js imported from main.js, init() + per-frame friendTracks.update(tGlobal) wired into the master loop (fx.js pattern); hidden 👣 Friends button #btn-friends + empty #friends-panel container in index.html; state.tracks = [] loaded at boot when manifest.has_tracks.
7. Capsule: add api/tracks to capsule.py's bake list (additive; update lane G's bundle-structure test consciously in the same commit, CR-style note in the commit message).
8. Tests for every new route shape; suite ≥ 189 + new tests. Merge fast — three lanes wait.

M19 — Marker detection (lane H, backend)

Fill tracker_detect.py. run_detect(): sample frames (~8 fps via frames.py machinery) from every video; detect markers two ways: color mode (HSV blob of the frozen marker hues; marker_key = "hue:<bucket>") and blink mode (per-blob brightness over a sliding window, decode the frozen OOK protocol; marker_key = "code:<id>"). Write data/work/detections.json (contract #3). OpenCV (opencv-python-headless) is already a dependency. No markers → valid-empty detections, never a crash. Accept: on the synthetic fixture, ≥90 % of sampled frames where a marker is visible yield a detection within 3 px (normalized ≤0.008) of the projected ground-truth center; blink ID 5 decoded; tests in test_tracker_detect.py.

M20 — Track solving (lane H, backend)

Fill tracker_solve.py. run_solve(): load detections; group by marker_key; per time bucket (≈0.25 s) collect same-key detections across cameras at matching t_global; 2+ views → triangulate through geometry primitives (the resolve.py pattern — never inline pose math); 1 view → ray ∩ ground plane (contract #5); smooth each track (moving average or Catmull-Rom, your call — document it); write tracks/track_points through db helpers; solid color for hue markers, palette color for code markers. POST /api/tracks/solve and the CLI now run detect→solve end-to-end. Accept: synthetic recovery — median 3D error of both tracks < 0.3 scene units against track_truth.json, ≥80 % temporal coverage of each marker's visible span; single-view fallback exercised by a test that hides all-but-one camera's detections; degradation (no detections → zero tracks + note); tests in test_tracker_solve.py.

M21 — Ribbons, moving labels, friends panel (lane J, frontend)

Fill friendTracks.js. On init (when manifest.has_tracks): unhide 👣; fetch /api/tracks into state.tracks. Toggle renders per-track: a ribbon polyline through the points (track color) + a moving marker (sphere + label sprite, position interpolated at the current t_global, hidden outside the track's time span) — objects via scene3d.addObject/removeObject only, registered with scene3d.registerHelper so photo mode hides them. Panel in #friends-panel (anchorPanel.js is the style/UX reference — read it, don't edit it): list tracks (dot, label, time span), rename/recolor via PATCH /api/tracks/{id} (controls carry write-ui), delete, and a Follow button per row. Accept: live-browser evidence — ribbons + moving labels track the synthetic markers in play and scrub; rename/recolor/delete round-trip; photo mode hides ribbons; capsule mode hides write controls but keeps ribbons/follow.

M22 — Follow-a-friend camera (lane J, frontend)

Chase cam on any track: scene3d.enterPathMode() hands you the camera (the camPath.js pattern — read it); every update(tGlobal) compute the track position, place the camera at a smoothed offset (behind/above, ~4 units, low-pass the position so it glides), look at the friend (applyCameraPose). Esc / Free-roam / snap-to-camera exits (pathMode already handles contention — mirror camPath's "someone else took the camera" check). Accept: live-browser evidence — camera glides after the moving marker in play and scrub; clean handoff to/from free-roam, snap-to-cam, and camPath playback; no fighting.

M23 — LED badge firmware + hardware kit (lane K, hardware/docs)

hardware/badge/badge.ino — ESP32/ATtiny-compatible Arduino sketch blinking the frozen OOK protocol (contract #4): configurable 6-bit ID, bright single LED or WS2812; plus hardware/beacon/beacon.ino — the sync/calibration beacon (distinct reserved ID 63, double-brightness). docs/hardware.md: parts list (boards, LEDs, battery, diffuser), build photos placeholders, wearing guidance (chest/hat height, diffusion, spacing), and the camera-side truth: what ranges/framerates the protocol survives (≥24 fps, exposure caveats). Accept: sketches compile (arduino-cli compile for esp32 + attiny if available — else document exact compile commands and mark unverified); protocol constants byte-identical to contract #4 (import/duplicate them in ONE header with a test-facing copy check documented).

M24 — Bench self-test tool (lane K, backend script)

scripts/badge_selftest.py: given a short phone clip of a badge on a desk, run the SAME decoder path as lane H (tracker_detect blink mode) and print the decoded ID + confidence — John's hardware smoke test before a shoot. Works on a synthetic self-test too: --synthetic renders a blinking clip via the fixture helpers and asserts round-trip. Accept: uv run python scripts/badge_selftest.py --synthetic decodes the rendered ID; clear failure text for: no blob, ID mismatch, too-short clip. Small test in test_badge_selftest.py (synthetic path only).

Contracts (frozen at foundation3 merge)

1. Marker hues (color mode): magenta #ff00ff and cyan #00ffff are the two supported fixture hues; real shoots may configure others via FESTIVAL4D_MARKER_HUES (comma-sep hex).
2. Schema: tracks(id INTEGER PK, marker_key TEXT NOT NULL, label TEXT, color TEXT, created_at TEXT) track_points(id INTEGER PK, track_id FK, t_global_s FLOAT, x FLOAT, y FLOAT, z FLOAT, quality FLOAT, views INTEGER) — points ordered by t_global_s; views = cameras used (1 = ground-plane fallback); coordinates in Three.js scene space. GET /api/tracks → [{id, marker_key, label, color, points: [{t_global_s,x,y,z,quality,views}]}].
3. detections.json: {"videos": {"<video_id>": [{"t_video_s": float, "marker_key": str, "cx": float, "cy": float, "area": float, "conf": float 0..1}]}, "generated_by": "festival4d.tracker_detect"} — cx/cy normalized 0..1 (bbox-annotation convention).
4. Blink protocol (OOK, camera-decodable): bit period 133 ms (4 frames @ 30 fps); word = preamble 11100 + 6-bit ID (MSB first) + even parity = 12 bits, repeated continuously (~1.6 s/word). ID 63 reserved for the sync beacon. Decoder needs ≥24 fps video and ≥1 full word of visibility.
5. Ground plane: the synthetic fixture's ground is y = 0 (scene space). For real projects the solver estimates it as the 5th-percentile-y horizontal plane of the point cloud and records it in its result dict. Single-view fallback intersects the pixel ray with this plane; such points get views = 1 and quality ≤ 0.5.
6. Frozen during 6b: everything frozen in phases 1–5 plus all phase-5 lane modules (director.js, exportVideo.js, anchorPanel.js, photoMode.js, fx.js, pathsStore.js, audio_features.py, director.py, capsule.py) and foundation3's edits (synthetic.py, main.js, index.html, api/cli/db). Lanes fill their own new files only. Believe a frozen file must change → plan/CHANGE_REQUESTS.md, work around, integration3 adjudicates. Stub-era test assertions superseded by your milestone landing follow the CR-1/-4/-5 pattern: minimal edit + CR entry.

integration3 scope (Phase 6c)

Merge order irrelevant; resolve CRs; full synthetic pass (evidence per milestone); raise the suite floor; tracked field test — real clips with at least one badge/marker in frame (combine with the still-open phase-5 field test); failures → plan/ISSUES.md; propose v0.3.0 (with v0.2.0 if the phase-5 field test passes in the same session).

File ownership (Phase 6b)

Owner	Files
Lane H	backend/festival4d/tracker_detect.py, tracker_solve.py, backend/tests/test_tracker_detect.py, test_tracker_solve.py
Lane J	frontend/src/friendTracks.js
Lane K	hardware/**, docs/hardware.md, scripts/badge_selftest.py, backend/tests/test_badge_selftest.py
Every agent	its own plan/status/<lane>.md
Coordinator only	plan/DIRECTIVES.md
Frozen during 6b	contract #6 above

Phase-6 pitfalls

1. Time alignment is the whole game. Detections happen at t_video; association happens at t_global. One t_video_from_global inversion mistake silently shears every track. Use the frozen helpers; test association against ground truth, not against itself.
2. Stage lighting lies about color. The color detector must gate on saturation+value, not hue alone, and the acceptance is on the synthetic fixture — note in your status file that real-world hue robustness is a known field-test risk (that's why blink mode exists).
3. Blink decoding vs. video fps: cameras at 24–60 fps sample the 133 ms bit period differently. Decode by integrating brightness per bit window using the video's actual fps (meta.fps), never by counting frames.
4. Don't fight the camera owners. Follow-mode (M22) shares pathMode with camPath — mirror camPath's contention checks exactly, or two drivers will interleave writes.
5. rAF throttle (standing): verify with the __f4d pump in hidden panes (localStorage f4dDebug=1); the WebAudio timer exemption gives you ~50 Hz setTimeout.
6. Fixture compatibility: foundation3's synthetic edit runs UNDER the existing 189 tests — markers must not move offsets, events, poses, or break SfM-adjacent tests.

Kickoff prompts (copy-paste one per Opus 4.8 session, in run order)

foundation3 (first, alone):

Clone ssh://git@100.71.119.27:222/monster/festifun.git. Read OPUS_BUILD_INSTRUCTIONS.md, plan/20-phase5.md, plan/30-phase6.md (M18 is yours), and the latest round of plan/DIRECTIVES.md. Execute foundation3 on branch foundation3; maintain plan/status/foundation3.md per plan/status/TEMPLATE.md; suite ≥ 189 + your tests; push the branch and report — the coordinator reviews and merges.

lane H / lane J / lane K (parallel, after foundation3 merges):

Clone ssh://git@100.71.119.27:222/monster/festifun.git and create your own git worktree before any edit. Read OPUS_BUILD_INSTRUCTIONS.md, plan/30-phase6.md, plan/lane-.md, and the latest round of plan/DIRECTIVES.md. Execute your lane on branch lane/<x>-<name>; maintain plan/status/lane-.md; obey the ownership table and frozen-files contract; push your branch and report — the coordinator reviews and merges. Do not merge to main.