From 56e569fd7a2768948f32d6fe03c16e0f56496981 Mon Sep 17 00:00:00 2001 From: type-two Date: Fri, 17 Jul 2026 18:28:21 +1000 Subject: [PATCH] Plan: Phase 6 spec + directives round 6 (M18-M24 Friend Tracks, lanes H/J/K) Markers/badges -> detection -> cross-view triangulation -> track ribbons + follow-a-friend + LED badge hardware kit. Coordinator-merges policy this phase. Co-Authored-By: Claude Fable 5 --- plan/30-phase6.md | 214 +++++++++++++++++++++++++++++++++++++++++ plan/DIRECTIVES.md | 33 +++++++ plan/README.md | 9 +- plan/lane-H-tracker.md | 37 +++++++ plan/lane-J-ribbons.md | 38 ++++++++ plan/lane-K-badges.md | 40 ++++++++ 6 files changed, 368 insertions(+), 3 deletions(-) create mode 100644 plan/30-phase6.md create mode 100644 plan/lane-H-tracker.md create mode 100644 plan/lane-J-ribbons.md create mode 100644 plan/lane-K-badges.md diff --git a/plan/30-phase6.md b/plan/30-phase6.md new file mode 100644 index 0000000..ed950d3 --- /dev/null +++ b/plan/30-phase6.md @@ -0,0 +1,214 @@ +# Phase 6 — Friend Tracks (M18–M24): markers → moving people in the 4D scene + +Canonical spec for phase 6. **Everything frozen in `../OPUS_BUILD_INSTRUCTIONS.md` (M0–M9) +and `20-phase5.md` (M10–M17 contracts) still applies.** Standing protocol (status files, +directives loop, evidence discipline, per-agent worktrees) is unchanged from +[`README.md`](README.md); the newest round of [`DIRECTIVES.md`](DIRECTIVES.md) always wins. + +## The idea + +A friend tag today is a *static* anchor. Phase 6 makes friends **move**: wearable markers +(distinct-hue patches or blinking LED badges) are detected in every camera, associated across +synced views, triangulated per timestep, and become **tracks** — `t_global → (x,y,z)` paths +rendered as ribbons in the 3D scene with moving labels, a follow-a-friend camera mode, and +capsule support. Same machinery as annotation resolution (M8), run automatically over time. + +## Context — what exists that you build on (read, don't re-invent) + +- `geometry.py` — COLMAP↔Three.js conversion, ray casting, triangulation primitives + (`resolve.py` shows exactly how a bbox becomes a 3D point; your solver is that, per-timestep). +- `frames.py` — frame sampling via ffmpeg (SfM uses it; detection re-uses it). +- `config.t_video_from_global` / `t_global_from_video` — the FROZEN timebase; detections are + found at `t_video`, tracks live at `t_global`. +- The synthetic fixture (`synthetic.py`) — every acceptance below is against it first. +- Suite floor is **189 passed**. It only goes up. + +## Phase map + +``` +Phase 6a (serial, ONE agent) Phase 6b (parallel, one agent per lane) Phase 6c (serial) +┌───────────────────────┐ ┌ lane/h-tracker ─── M19 M20 ──────┐ ┌─────────────────┐ +│ foundation3 │ ├ lane/j-ribbons ─── M21 M22 ──────┤ │ integration3 │ +│ fixture ground truth +│──merge──┤ ├──merge─│ full pass + │ +│ contracts + stubs │ └ lane/k-badges ──── M23 M24 ──────┘ │ tracked field │ +│ branch: foundation3 │ (independent — merge in any order) │ test → v0.3.0 │ +└───────────────────────┘ └────────────────────────────────── └─────────────────┘ +``` + +Lanes depend **only** on foundation3, never on each other. + +## Milestones + +### M18 — foundation3: fixture ground truth, contracts, stubs (serial, ONE agent) +1. **Synthetic fixture upgrade** (`synthetic.py` — coordinator-sanctioned edit): render **two + moving markers** into every camera's video via the existing camera projection: + - marker A: a solid bright disc (frozen hue, e.g. pure magenta #ff00ff) following a known + smooth trajectory across the stage (e.g. a slow circle, radius ~2, y≈1.5); + - marker B: a **blinking** white/bright disc following a different known path, blinking the + frozen protocol below with ID=5. + Markers must be small (≤2 % of frame area) and rendered *after* the existing content so the + full existing suite stays green (offsets, events, poses, SfM tests untouched). Write ground + truth to `data/work/track_truth.json`: `{"markers": [{"marker_key": ..., "points": + [{"t_global_s", "x", "y", "z"}]}]}` (Three.js scene space). +2. **DB:** `tracks` + `track_points` tables (contract #2) + `db.py` helpers + (`add_track`, `get_tracks`, `set_track_points`, `patch_track`, `delete_track`). +3. **API:** `GET /api/tracks`, `PATCH /api/tracks/{id}` `{label?, color?}`, + `DELETE /api/tracks/{id}`, `POST /api/tracks/solve` (dispatches into the lane-H stub with + the house degradation: valid empty result + note while stubbed). Manifest gains + `"has_tracks": bool` — update `test_api.py`'s key-set lock consciously in the same commit. +4. **CLI:** `python -m festival4d track [--detect-only|--solve-only]` → lane-H stubs. +5. **Backend stubs:** `tracker_detect.py` (`run_detect()`), `tracker_solve.py` + (`run_solve()`) — `NotImplementedError` with the full contract in the docstring. +6. **Frontend:** stub `frontend/src/friendTracks.js` imported from `main.js`, `init()` + + per-frame `friendTracks.update(tGlobal)` wired into the master loop (fx.js pattern); + hidden **👣 Friends** button `#btn-friends` + empty `#friends-panel` container in + `index.html`; `state.tracks = []` loaded at boot when `manifest.has_tracks`. +7. **Capsule:** add `api/tracks` to `capsule.py`'s bake list (additive; update lane G's + bundle-structure test consciously in the same commit, CR-style note in the commit message). +8. Tests for every new route shape; suite ≥ 189 + new tests. Merge fast — three lanes wait. + +### M19 — Marker detection (lane H, backend) +Fill `tracker_detect.py`. `run_detect()`: sample frames (~8 fps via `frames.py` machinery) from +every video; detect markers two ways: **color mode** (HSV blob of the frozen marker hues; +`marker_key = "hue:"`) and **blink mode** (per-blob brightness over a sliding window, +decode the frozen OOK protocol; `marker_key = "code:"`). Write +`data/work/detections.json` (contract #3). OpenCV (`opencv-python-headless`) is already a +dependency. No markers → valid-empty detections, never a crash. +**Accept:** on the synthetic fixture, ≥90 % of sampled frames where a marker is visible yield +a detection within 3 px (normalized ≤0.008) of the projected ground-truth center; blink ID 5 +decoded; tests in `test_tracker_detect.py`. + +### M20 — Track solving (lane H, backend) +Fill `tracker_solve.py`. `run_solve()`: load detections; group by `marker_key`; per time bucket +(≈0.25 s) collect same-key detections across cameras at matching `t_global`; **2+ views → +triangulate** through `geometry` primitives (the `resolve.py` pattern — never inline pose +math); **1 view → ray ∩ ground plane** (contract #5); smooth each track (moving average or +Catmull-Rom, your call — document it); write `tracks`/`track_points` through `db` helpers; +solid color for hue markers, palette color for code markers. `POST /api/tracks/solve` and the +CLI now run detect→solve end-to-end. +**Accept:** synthetic recovery — median 3D error of both tracks < 0.3 scene units against +`track_truth.json`, ≥80 % temporal coverage of each marker's visible span; single-view +fallback exercised by a test that hides all-but-one camera's detections; degradation (no +detections → zero tracks + note); tests in `test_tracker_solve.py`. + +### M21 — Ribbons, moving labels, friends panel (lane J, frontend) +Fill `friendTracks.js`. On init (when `manifest.has_tracks`): unhide 👣; fetch +`/api/tracks` into `state.tracks`. Toggle renders per-track: a **ribbon** polyline through the +points (track color) + a **moving marker** (sphere + label sprite, position interpolated at +the current `t_global`, hidden outside the track's time span) — objects via +`scene3d.addObject/removeObject` only, registered with `scene3d.registerHelper` so photo mode +hides them. Panel in `#friends-panel` (anchorPanel.js is the style/UX reference — read it, +don't edit it): list tracks (dot, label, time span), rename/recolor via +`PATCH /api/tracks/{id}` (controls carry `write-ui`), delete, and a **Follow** button per row. +**Accept:** live-browser evidence — ribbons + moving labels track the synthetic markers in +play and scrub; rename/recolor/delete round-trip; photo mode hides ribbons; capsule mode +hides write controls but keeps ribbons/follow. + +### M22 — Follow-a-friend camera (lane J, frontend) +Chase cam on any track: `scene3d.enterPathMode()` hands you the camera (the camPath.js +pattern — read it); every `update(tGlobal)` compute the track position, place the camera at a +smoothed offset (behind/above, ~4 units, low-pass the position so it glides), look at the +friend (`applyCameraPose`). Esc / Free-roam / snap-to-camera exits (pathMode already handles +contention — mirror camPath's "someone else took the camera" check). +**Accept:** live-browser evidence — camera glides after the moving marker in play and scrub; +clean handoff to/from free-roam, snap-to-cam, and camPath playback; no fighting. + +### M23 — LED badge firmware + hardware kit (lane K, hardware/docs) +`hardware/badge/badge.ino` — ESP32/ATtiny-compatible Arduino sketch blinking the frozen OOK +protocol (contract #4): configurable 6-bit ID, bright single LED or WS2812; plus +`hardware/beacon/beacon.ino` — the sync/calibration beacon (distinct reserved ID 63, +double-brightness). `docs/hardware.md`: parts list (boards, LEDs, battery, diffuser), build +photos placeholders, wearing guidance (chest/hat height, diffusion, spacing), and the +camera-side truth: what ranges/framerates the protocol survives (≥24 fps, exposure caveats). +**Accept:** sketches compile (`arduino-cli compile` for esp32 + attiny if available — else +document exact compile commands and mark unverified); protocol constants byte-identical to +contract #4 (import/duplicate them in ONE header with a test-facing copy check documented). + +### M24 — Bench self-test tool (lane K, backend script) +`scripts/badge_selftest.py`: given a short phone clip of a badge on a desk, run the SAME +decoder path as lane H (`tracker_detect` blink mode) and print the decoded ID + confidence — +John's hardware smoke test before a shoot. Works on a synthetic self-test too: +`--synthetic` renders a blinking clip via the fixture helpers and asserts round-trip. +**Accept:** `uv run python scripts/badge_selftest.py --synthetic` decodes the rendered ID; +clear failure text for: no blob, ID mismatch, too-short clip. Small test in +`test_badge_selftest.py` (synthetic path only). + +## Contracts (frozen at foundation3 merge) + +1. **Marker hues (color mode):** magenta `#ff00ff` and cyan `#00ffff` are the two supported + fixture hues; real shoots may configure others via `FESTIVAL4D_MARKER_HUES` (comma-sep hex). +2. **Schema:** + `tracks(id INTEGER PK, marker_key TEXT NOT NULL, label TEXT, color TEXT, created_at TEXT)` + `track_points(id INTEGER PK, track_id FK, t_global_s FLOAT, x FLOAT, y FLOAT, z FLOAT, + quality FLOAT, views INTEGER)` — points ordered by `t_global_s`; `views` = cameras used + (1 = ground-plane fallback); coordinates in **Three.js scene space**. + `GET /api/tracks` → `[{id, marker_key, label, color, points: [{t_global_s,x,y,z,quality,views}]}]`. +3. **detections.json:** `{"videos": {"": [{"t_video_s": float, "marker_key": str, + "cx": float, "cy": float, "area": float, "conf": float 0..1}]}, "generated_by": + "festival4d.tracker_detect"}` — `cx`/`cy` normalized 0..1 (bbox-annotation convention). +4. **Blink protocol (OOK, camera-decodable):** bit period **133 ms** (4 frames @ 30 fps); + word = preamble `11100` + 6-bit ID (MSB first) + even parity = 12 bits, repeated + continuously (~1.6 s/word). ID 63 reserved for the sync beacon. Decoder needs ≥24 fps + video and ≥1 full word of visibility. +5. **Ground plane:** the synthetic fixture's ground is `y = 0` (scene space). For real + projects the solver estimates it as the 5th-percentile-y horizontal plane of the point + cloud and records it in its result dict. Single-view fallback intersects the pixel ray + with this plane; such points get `views = 1` and `quality ≤ 0.5`. +6. **Frozen during 6b:** everything frozen in phases 1–5 **plus** all phase-5 lane modules + (`director.js`, `exportVideo.js`, `anchorPanel.js`, `photoMode.js`, `fx.js`, + `pathsStore.js`, `audio_features.py`, `director.py`, `capsule.py`) and foundation3's edits + (`synthetic.py`, `main.js`, `index.html`, api/cli/db). Lanes fill their own new files only. + Believe a frozen file must change → `plan/CHANGE_REQUESTS.md`, work around, integration3 + adjudicates. Stub-era test assertions superseded by your milestone landing follow the + CR-1/-4/-5 pattern: minimal edit + CR entry. + +## integration3 scope (Phase 6c) +Merge order irrelevant; resolve CRs; full synthetic pass (evidence per milestone); raise the +suite floor; **tracked field test** — real clips with at least one badge/marker in frame +(combine with the still-open phase-5 field test); failures → `plan/ISSUES.md`; propose +`v0.3.0` (with `v0.2.0` if the phase-5 field test passes in the same session). + +## File ownership (Phase 6b) + +| Owner | Files | +|---|---| +| Lane H | `backend/festival4d/tracker_detect.py`, `tracker_solve.py`, `backend/tests/test_tracker_detect.py`, `test_tracker_solve.py` | +| Lane J | `frontend/src/friendTracks.js` | +| Lane K | `hardware/**`, `docs/hardware.md`, `scripts/badge_selftest.py`, `backend/tests/test_badge_selftest.py` | +| Every agent | its own `plan/status/.md` | +| **Coordinator only** | `plan/DIRECTIVES.md` | +| **Frozen during 6b** | contract #6 above | + +## Phase-6 pitfalls + +1. **Time alignment is the whole game.** Detections happen at `t_video`; association happens + at `t_global`. One `t_video_from_global` inversion mistake silently shears every track. + Use the frozen helpers; test association against ground truth, not against itself. +2. **Stage lighting lies about color.** The color detector must gate on saturation+value, not + hue alone, and the acceptance is on the *synthetic* fixture — note in your status file that + real-world hue robustness is a known field-test risk (that's why blink mode exists). +3. **Blink decoding vs. video fps:** cameras at 24–60 fps sample the 133 ms bit period + differently. Decode by integrating brightness per bit window using the video's actual fps + (`meta.fps`), never by counting frames. +4. **Don't fight the camera owners.** Follow-mode (M22) shares `pathMode` with camPath — + mirror camPath's contention checks exactly, or two drivers will interleave writes. +5. **rAF throttle (standing):** verify with the `__f4d` pump in hidden panes + (`localStorage f4dDebug=1`); the WebAudio timer exemption gives you ~50 Hz `setTimeout`. +6. **Fixture compatibility:** foundation3's synthetic edit runs UNDER the existing 189 tests — + markers must not move offsets, events, poses, or break SfM-adjacent tests. + +## Kickoff prompts (copy-paste one per Opus 4.8 session, in run order) + +**foundation3 (first, alone):** +> Clone ssh://git@100.71.119.27:222/monster/festifun.git. Read OPUS_BUILD_INSTRUCTIONS.md, +> plan/20-phase5.md, plan/30-phase6.md (M18 is yours), and the latest round of +> plan/DIRECTIVES.md. Execute foundation3 on branch `foundation3`; maintain +> plan/status/foundation3.md per plan/status/TEMPLATE.md; suite ≥ 189 + your tests; push the +> branch and report — the coordinator reviews and merges. + +**lane H / lane J / lane K (parallel, after foundation3 merges):** +> Clone ssh://git@100.71.119.27:222/monster/festifun.git and create your own git worktree +> before any edit. Read OPUS_BUILD_INSTRUCTIONS.md, plan/30-phase6.md, plan/lane-.md, and +> the latest round of plan/DIRECTIVES.md. Execute your lane on branch `lane/-`; +> maintain plan/status/lane-.md; obey the ownership table and frozen-files contract; push +> your branch and report — the coordinator reviews and merges. Do not merge to main. diff --git a/plan/DIRECTIVES.md b/plan/DIRECTIVES.md index 13888a9..bbbf157 100644 --- a/plan/DIRECTIVES.md +++ b/plan/DIRECTIVES.md @@ -4,6 +4,39 @@ Written **only** by the coordinator (the human's planning session). Agents: read --- +## Round 6 — 2026-07-17 — PHASE 6 planned: Friend Tracks (M18–M24) + +**Spec:** `plan/30-phase6.md` (canonical). Lane briefs: `plan/lane-H-tracker.md`, +`plan/lane-J-ribbons.md`, `plan/lane-K-badges.md`. Kickoff prompts are at the bottom of the +spec — the human launches each Opus session with one. + +**Directives:** +1. **Now: `foundation3` only** (one agent, branch `foundation3`, M18). Contracts correctness + beats speed — three lanes hard-depend on your fixture ground truth, schema, detections + shape, and blink protocol. The synthetic-fixture edit must keep all 189 existing tests + green (spec pitfall #6). Maintain `plan/status/foundation3.md`. **Push your branch; do NOT + merge to main** — this phase the coordinator merges after review (see #4). +2. **After foundation3 merges — lanes in priority order** (all three concurrently if capacity + allows): **H** (tracker — longest, validates every contract), **J** (ribbons/follow — the + visible payoff), **K** (badges — independent, hardware-facing). Per-agent worktrees remain + MANDATORY. +3. **Merge policy (changed from Round 4):** agents do NOT merge to main and do NOT tag. + Push your lane branch + status file, report completion to the human; the human pings the + coordinator session, which reviews (independent suite re-run, ownership check, CR + adjudication) and merges in completion order. Everything else from Round 0 protocol + (status cadence, blocker rule, evidence discipline) unchanged. +4. **Review loop:** coordinator reads `plan/status/.md` + the branch diff on each ping; + findings come back as a new DIRECTIVES round or direct merge. Stub-era assertions + superseded by a landing milestone: CR entry + minimal edit (CR-1/-4/-5 house pattern). +5. **integration3** (after all three merge): full synthetic pass, floor raise, tracked field + test (fold in the still-open phase-5 field test), `v0.3.0` proposal. `v0.2.0` remains held + per Round 5. + +**For the human:** (a) launch foundation3 first, alone, with its kickoff prompt; tell the +coordinator when its status file says ready; (b) then launch H, J, K in parallel; (c) the +field-test gate is unchanged — real clips in `data/raw/`, now ideally with a marker or badge +in frame; (d) if you order badge parts, `docs/hardware.md` (lane K) will have the list. + ## Round 5 — 2026-07-17 — PHASE 5 COMPLETE on synthetic; floor 189; v0.2.0 waits on the field test **Execution report (all same-day):** foundation2 → lanes E/G/F in parallel worktrees (per diff --git a/plan/README.md b/plan/README.md index ae8ff98..0110c92 100644 --- a/plan/README.md +++ b/plan/README.md @@ -2,9 +2,12 @@ **Read [`../OPUS_BUILD_INSTRUCTIONS.md`](../OPUS_BUILD_INSTRUCTIONS.md) first — it is the canonical spec** (architecture, repo layout, data model, milestone details M0–M9, pitfalls). This directory only organizes *who builds what, in what order, without colliding*. Lane briefs reference spec milestones by number; they do not restate them. -> **Current phase: 5** (M10–M17) — spec and organization in [`20-phase5.md`](20-phase5.md), lane -> briefs `lane-E/F/G-*.md`. Phases 1–4 below are complete (v0.1.0 + polish); their protocol -> sections (git, status, directives loop, evidence discipline) remain the standing rules. +> **Current phase: 6** (M18–M24, Friend Tracks) — spec and organization in +> [`30-phase6.md`](30-phase6.md), lane briefs `lane-H/J/K-*.md`, run order in DIRECTIVES +> Round 6 (note: this phase the **coordinator merges**, agents only push branches). +> Phases 1–5 below are complete (v0.1.0 shipped; phase 5 M10–M17 accepted on synthetic, +> suite floor 189, v0.2.0 held for the field test); their protocol sections (git, status, +> directives loop, evidence discipline) remain the standing rules. ## Phase map diff --git a/plan/lane-H-tracker.md b/plan/lane-H-tracker.md new file mode 100644 index 0000000..f006881 --- /dev/null +++ b/plan/lane-H-tracker.md @@ -0,0 +1,37 @@ +# Lane H — Tracker (M19 detection, M20 solving) + +You turn wearable markers in the videos into 3D friend tracks. Backend only. +Spec: `30-phase6.md` M19/M20 + contracts #1–#5. Branch `lane/h-tracker`, own worktree. + +## What foundation3 gives you +- Stubs `tracker_detect.py` / `tracker_solve.py` with the contracts in their docstrings; + API (`POST /api/tracks/solve` etc.), CLI (`track`), DB helpers, and the synthetic fixture's + two moving markers + `data/work/track_truth.json` ground truth — all already wired. +- You fill the two stub bodies and write the two test files. Nothing else. + +## M19 — detection, the parts that bite +- Sample frames with the `frames.py` machinery (~8 fps is plenty; document your rate). +- Color mode: convert to HSV, gate on **saturation AND value** before hue (pitfall #2); + connected components; centroid → normalized cx/cy. Hue buckets from contract #1 + + `FESTIVAL4D_MARKER_HUES`. +- Blink mode: track candidate bright blobs across the sampled window, integrate brightness + per 133 ms bit window **using the video's real fps** (pitfall #3), correlate against the + preamble, check parity, emit `code:`. +- Empty/dark/markerless video → valid-empty detections. Never crash. + +## M20 — solving, the parts that bite +- `t_video → t_global` through the frozen helpers ONLY (pitfall #1). +- Triangulation through `geometry` primitives exactly as `resolve.py` does — read it first. +- Bucket ≈0.25 s; 2+ cameras → triangulate (record `views`, `quality` from residual); + 1 camera → ray ∩ ground plane per contract #5 (`views=1`, `quality ≤ 0.5`). +- Smooth per track; write through `db` helpers; re-solve replaces that marker's tracks + (idempotent — don't stack duplicates on every run). + +## Acceptance (evidence in plan/status/lane-H.md) +- [ ] Detection: ≥90 % recall on visible synthetic markers, ≤0.008 normalized center error; + blink ID 5 decoded (`test_tracker_detect.py`) +- [ ] Solve: median 3D error < 0.3 vs `track_truth.json`, ≥80 % coverage; single-view + fallback test; idempotent re-solve; degradations (`test_tracker_solve.py`) +- [ ] CLI + `POST /api/tracks/solve` run detect→solve end-to-end on the fixture +- [ ] Suite ≥ foundation3's count + yours, green; stub-era assertions superseded by your + landing → CR entry (CR-1/-4/-5 pattern) diff --git a/plan/lane-J-ribbons.md b/plan/lane-J-ribbons.md new file mode 100644 index 0000000..050852e --- /dev/null +++ b/plan/lane-J-ribbons.md @@ -0,0 +1,38 @@ +# Lane J — Ribbons & follow-a-friend (M21, M22) + +You make tracks visible and rideable. Pure frontend — ONE file: `frontend/src/friendTracks.js`. +Spec: `30-phase6.md` M21/M22 + contract #2. Branch `lane/j-ribbons`, own worktree. + +## What foundation3 gives you +- Your stub module imported from main.js with `init()` + per-frame `update(tGlobal)` already + called; hidden `#btn-friends` + `#friends-panel` in index.html; `state.tracks` loaded at + boot when `manifest.has_tracks`. +- Read-only building blocks you may CALL but never edit: `scene3d.addObject/removeObject`, + `registerHelper` (photo mode hides you for free), `enterPathMode`/`exitPathMode`/ + `applyCameraPose` (the camPath.js pattern — read that file first), `focusOn`; + `PATCH/DELETE /api/tracks/{id}`. anchorPanel.js is your UX/style reference (inject your own +