rotogod/DESIGN.md

ROTOGOD — design

Local rotoscoping tool for m3ultra. Load a video → scrub → cut out a scene → click actors → SAM 2 propagates masks across the clip → export alpha (ProRes 4444 / WebM / matte / greenscreen).

Why these choices

Decision	Choice	Why
UI	Local web app (FastAPI + vanilla JS canvas), port 8484	Matches the fleet pattern (fluxgod :8890, paradramorama :8474, m3panel :8790). 8474–8479 are already taken on this box. Browser canvas is plenty fast for 720–1080p preview; no Qt dependency.
Segmentation	SAM 2.1 via transformers (Sam2VideoModel / Sam2VideoProcessor)	pip-installable (no facebookresearch git clone, no hydra configs), weights from HF hub, actively maintained, pure-torch so MPS works. API verified against installed transformers 5.14.
Model size	facebook/sam2.1-hiera-small default, -large for finals	Small = snappy interactive loop; large = better masks. Both cached in ~/.cache/huggingface. Switch with ROTOGOD_SAM2=facebook/sam2.1-hiera-large.
Python	3.12 venv via uv (.venv/)	Homebrew python3 is 3.14 — no torch wheels yet.
Decode	PyAV (bundled ffmpeg), seek-to-keyframe + roll forward, RAM frame cache	Arbitrary scrubbing; 256GB UMA means the cache can be generous (default 600 frames ≈ 1.6GB at 1080p).
Precision	float32 on MPS	bf16 on MPS still has op gaps; fp32 is safe and fast enough on M3 Ultra.

Architecture

web/  (canvas UI: scrub, in/out, click prompts, overlays, export buttons)
   │ JSON/JPEG/PNG over localhost:8484
rotogod/server.py   FastAPI — endpoints below, serves web/
rotogod/video.py    Clip: PyAV decode, frame(i) → RGB ndarray, JPEG previews, thumbstrip
rotogod/segment.py  Sam2Engine (lazy model load) + RotoSession (frames, clicks, masks)
rotogod/jobs.py     background threads with progress (propagate, exports)
rotogod/export.py   ffmpeg: scene trim, ProRes 4444 alpha, VP9 alpha, matte, greenscreen

State is deliberately single-user/single-clip (module-level), same as fluxgod. Binds 127.0.0.1.

The loop

1. Open a clip by path (POST /api/open). Meta + thumbstrip render; scrub via GET /api/frame/{i} (JPEG, cached decode).
2. Cut the scene: set in/out points (i/o keys), optional lossless keyframe-snapped trim or frame-exact re-encode (POST /api/trim).
3. Roto session (POST /api/roto/start): frames [in..out] are decoded once into RAM and handed to processor.init_video_session(video=frames, inference_device="mps", video_storage_device="cpu").
4. Click prompts: left-click = positive, alt/right-click = negative, per object. The UI keeps the full point list per (frame, object) and sends it whole each time (SAM 2 replaces rather than accumulates per-frame inputs). Backend: add_inputs_to_inference_session(...) then model(inference_session, frame_idx) → instant mask overlay on that frame.
5. Track (POST /api/roto/propagate): background job runs propagate_in_video_iterator twice — forward from the prompted frame, then reverse to cover frames before it (SAM 2 only propagates one direction per pass); masks land per frame (packbits-compressed bools keyed by object) and the UI progress bar + overlay refresh as they stream in.
6. Correct drift: scrub to the bad frame, add +/- clicks there, re-track from that frame. SAM 2's memory bank handles re-propagation.
7. Export (POST /api/export): union of object masks → RGBA PNG sequence (optional edge feather) → ffmpeg:
   • prores: ProRes 4444 .mov, -pix_fmt yuva444p10le (Resolve/FCP/Premiere-ready)
   • webm: VP9 yuva420p (browser alpha)
   • matte: grayscale h264 mp4
   • green: comp over green, h264 mp4 (quick eyeball check)

API surface

POST /api/open {path}                 GET /api/meta          GET /api/frame/{i}?w=
GET  /api/thumbs?n=                   POST /api/trim {in_f,out_f,exact}
POST /api/roto/start {in_f,out_f}     POST /api/roto/click {frame,obj,points,labels}
POST /api/roto/propagate {start?}     GET /api/roto/overlay/{frame}.png
GET  /api/roto/status                 POST /api/roto/reset
POST /api/export {kind,feather}       GET /api/job/{id}      GET /api/status

Frame indices in the roto API are absolute clip frames; the session maps to local range internally. Outputs land in output/ (gitignored).

Known limits / phase 2+

• Hair & motion blur: SAM 2 masks are binary. The feather param is a stopgap; real fix is a matting pass — MatAnyone (preferred) or RVM taking SAM 2's mask as guidance → soft alpha. Slots in as a second engine in segment.py + an export-time toggle.
• Long clips: propagate memory grows with range; UI warns > 2000 frames. Chunk-and-stitch later.
• Audio is dropped in roto exports (kept in scene trims).
• Multi-clip / project files: out of scope for v0; one clip + one session at a time.
• Possible future: farm propagation to MODELBEAST queue (:8777) as a job type; not needed while this box is the only one running it.