874aaaefa3
FastAPI + PyAV + transformers SAM2.1 on MPS. Relative-URL frontend (works bare or behind digalot.fyi/rotogod/), ROTOGOD_ROOTS clip-path allowlist, fwd+rev propagation, ProRes 4444 / WebM / matte / greenscreen exports. deploy/rotogod.launchd.plist = always-on tailnet-bound instance. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
69 lines
4.9 KiB
Markdown
69 lines
4.9 KiB
Markdown
# ROTOGOD — design
|
||
|
||
Local rotoscoping tool for m3ultra. Load a video → scrub → cut out a scene → click actors →
|
||
SAM 2 propagates masks across the clip → export alpha (ProRes 4444 / WebM / matte / greenscreen).
|
||
|
||
## Why these choices
|
||
|
||
| Decision | Choice | Why |
|
||
|---|---|---|
|
||
| UI | Local web app (FastAPI + vanilla JS canvas), port **8484** | Matches the fleet pattern (fluxgod :8890, paradramorama :8474, m3panel :8790). 8474–8479 are already taken on this box. Browser canvas is plenty fast for 720–1080p preview; no Qt dependency. |
|
||
| Segmentation | **SAM 2.1 via `transformers`** (`Sam2VideoModel` / `Sam2VideoProcessor`) | pip-installable (no facebookresearch git clone, no hydra configs), weights from HF hub, actively maintained, pure-torch so MPS works. API verified against installed transformers 5.14. |
|
||
| Model size | `facebook/sam2.1-hiera-small` default, `-large` for finals | Small = snappy interactive loop; large = better masks. Both cached in `~/.cache/huggingface`. Switch with `ROTOGOD_SAM2=facebook/sam2.1-hiera-large`. |
|
||
| Python | 3.12 venv via uv (`.venv/`) | Homebrew python3 is 3.14 — no torch wheels yet. |
|
||
| Decode | PyAV (bundled ffmpeg), seek-to-keyframe + roll forward, RAM frame cache | Arbitrary scrubbing; 256GB UMA means the cache can be generous (default 600 frames ≈ 1.6GB at 1080p). |
|
||
| Precision | float32 on MPS | bf16 on MPS still has op gaps; fp32 is safe and fast enough on M3 Ultra. |
|
||
|
||
## Architecture
|
||
|
||
```
|
||
web/ (canvas UI: scrub, in/out, click prompts, overlays, export buttons)
|
||
│ JSON/JPEG/PNG over localhost:8484
|
||
rotogod/server.py FastAPI — endpoints below, serves web/
|
||
rotogod/video.py Clip: PyAV decode, frame(i) → RGB ndarray, JPEG previews, thumbstrip
|
||
rotogod/segment.py Sam2Engine (lazy model load) + RotoSession (frames, clicks, masks)
|
||
rotogod/jobs.py background threads with progress (propagate, exports)
|
||
rotogod/export.py ffmpeg: scene trim, ProRes 4444 alpha, VP9 alpha, matte, greenscreen
|
||
```
|
||
|
||
State is deliberately single-user/single-clip (module-level), same as fluxgod. Binds 127.0.0.1.
|
||
|
||
## The loop
|
||
|
||
1. **Open** a clip by path (`POST /api/open`). Meta + thumbstrip render; scrub via `GET /api/frame/{i}` (JPEG, cached decode).
|
||
2. **Cut the scene**: set in/out points (i/o keys), optional lossless keyframe-snapped trim or frame-exact re-encode (`POST /api/trim`).
|
||
3. **Roto session** (`POST /api/roto/start`): frames [in..out] are decoded once into RAM and handed to `processor.init_video_session(video=frames, inference_device="mps", video_storage_device="cpu")`.
|
||
4. **Click prompts**: left-click = positive, alt/right-click = negative, per object. The UI keeps the full point list per (frame, object) and sends it whole each time (SAM 2 replaces rather than accumulates per-frame inputs). Backend: `add_inputs_to_inference_session(...)` then `model(inference_session, frame_idx)` → instant mask overlay on that frame.
|
||
5. **Track** (`POST /api/roto/propagate`): background job runs `propagate_in_video_iterator` twice — forward from the prompted frame, then reverse to cover frames before it (SAM 2 only propagates one direction per pass); masks land per frame (packbits-compressed bools keyed by object) and the UI progress bar + overlay refresh as they stream in.
|
||
6. **Correct drift**: scrub to the bad frame, add +/- clicks there, re-track from that frame. SAM 2's memory bank handles re-propagation.
|
||
7. **Export** (`POST /api/export`): union of object masks → RGBA PNG sequence (optional edge feather) → ffmpeg:
|
||
- `prores`: ProRes 4444 `.mov`, `-pix_fmt yuva444p10le` (Resolve/FCP/Premiere-ready)
|
||
- `webm`: VP9 `yuva420p` (browser alpha)
|
||
- `matte`: grayscale h264 mp4
|
||
- `green`: comp over green, h264 mp4 (quick eyeball check)
|
||
|
||
## API surface
|
||
|
||
```
|
||
POST /api/open {path} GET /api/meta GET /api/frame/{i}?w=
|
||
GET /api/thumbs?n= POST /api/trim {in_f,out_f,exact}
|
||
POST /api/roto/start {in_f,out_f} POST /api/roto/click {frame,obj,points,labels}
|
||
POST /api/roto/propagate {start?} GET /api/roto/overlay/{frame}.png
|
||
GET /api/roto/status POST /api/roto/reset
|
||
POST /api/export {kind,feather} GET /api/job/{id} GET /api/status
|
||
```
|
||
|
||
Frame indices in the roto API are absolute clip frames; the session maps to local range internally.
|
||
Outputs land in `output/` (gitignored).
|
||
|
||
## Known limits / phase 2+
|
||
|
||
- **Hair & motion blur**: SAM 2 masks are binary. The feather param is a stopgap; real fix is a
|
||
matting pass — **MatAnyone** (preferred) or RVM taking SAM 2's mask as guidance → soft alpha.
|
||
Slots in as a second engine in `segment.py` + an export-time toggle.
|
||
- **Long clips**: propagate memory grows with range; UI warns > 2000 frames. Chunk-and-stitch later.
|
||
- **Audio** is dropped in roto exports (kept in scene trims).
|
||
- **Multi-clip / project files**: out of scope for v0; one clip + one session at a time.
|
||
- Possible future: farm propagation to MODELBEAST queue (:8777) as a job type; not needed while
|
||
this box is the only one running it.
|