# VIDGOD

Open-source AI video production toolkit for music-video editing around DaVinci Resolve.
All inference is local on Apple Silicon (PyTorch MPS + MLX) — no cloud, no subscriptions.

Built for the workflow: hoard old footage → normalize it → index every shot and every
spoken word → search it → arrange to a beat grid in Resolve → cut actors out with clicks
instead of rotoscoping splines.

## The tools

| Command | What it does |
|---|---|
| `bin/vg-roto CLIP --point x,y` | Click an actor → tracked cutout as **ProRes 4444 with alpha** (SAM 2.1 + MatAnyone). `--grab-frame` saves the prompt frame so you can find coords. `--mode mask` for hard binary masks via SAM2 video propagation. |
| `bin/vg-transcode DIR` | Batch-convert old AVI/WMV/MPEG/etc → ProRes LT, mirrored dir tree, auto-deinterlace, skips already-done files. `--h264` for small proxies. |
| `bin/vg-index PATHS` | Shot-detect (PySceneDetect) + transcribe (mlx-whisper large-v3-turbo) + thumbnail everything into `library/clips.sqlite` (FTS5). Resumable. |
| `bin/vg-find WORDS` | Full-text search the dialogue; shows the enclosing shot. `--cut DIR` exports each matching shot as a ProRes clip. `--shots VIDEO` lists a video's shots. |
| `bin/vg-beats SONG` | Beat-track a music file (librosa) → `.beats.csv` + a **Resolve marker EDL** (import: right-click timeline in media pool → Timelines → Import → Timeline Markers from EDL). |

## Install

Needs: Apple Silicon Mac, `brew install ffmpeg uv`, ~4GB disk for venvs + models.

```bash
git clone ssh://git@100.71.119.27:222/monster/vidgod.git && cd vidgod
./setup/setup_venvs.sh        # venvs, SAM2 + MatAnyone clones, checkpoints, patches (~10 min)
./setup/smoke_test.sh         # fast lanes; add --roto for the full cutout pipeline
```

Everything heavy (venvs/, tools/, models/, library/, work/, tests/) lives untracked inside
the working copy; the repo is just the scripts. Long jobs heartbeat to `~/.jobs/vidgod-*.status`.

## Typical session

```bash
bin/vg-transcode ~/old-tv-rips                      # → ~/old-tv-rips-prores
bin/vg-index ~/old-tv-rips-prores                   # overnight for a big archive
bin/vg-find "radical" --cut ~/mv/pulls              # grab every shot where someone says it
bin/vg-beats ~/mv/song.mp3 --fps 25                 # markers for the Resolve timeline
bin/vg-roto ~/mv/pulls/003_*.mov --grab-frame       # find your click coords
bin/vg-roto ~/mv/pulls/003_*.mov --point 970,220    # → ProRes 4444 alpha cutout
```

## Notes & gotchas

- **MatAnyone patch**: torchvision ≥ 0.23 removed `torchvision.io.read_video`; setup applies
  `patches/matanyone-cv2-reader.patch` to its clone automatically. If you re-clone
  `tools/MatAnyone` by hand, re-run `setup/setup_venvs.sh`.
- MatAnyone's checkpoint comes from its GitHub release (the HF repo doesn't carry the
  file at the root path); its inference script insists on `pretrained_models/` inside its
  repo dir, so setup symlinks it there.
- MatAnyone propagates **forward from a first-frame mask** — `vg-roto --frame N` trims the
  clip at N first. Prompt on a frame where the target is clearly visible.
- Scripts set `/opt/homebrew/bin` in PATH themselves (non-interactive SSH doesn't) and
  `PYTORCH_ENABLE_MPS_FALLBACK=1` for stray unsupported MPS kernels.
- Rough speed on an M1 Ultra: MatAnyone ≈ 3.4 fr/s at 1920×804; whisper large-v3-turbo
  transcribes far faster than realtime; ProRes transcodes are ffmpeg-bound.

## Phase 2 (not built yet)

ProPainter (object/logo removal), RIFE interpolation, Cutie interactive segmentation GUI,
MODELBEAST farm operators for roto/index jobs, model mirror to NAS modelzoo. Upscaling
old footage already exists fleet-side (`seedvr2_upscale` on the farm); Resolve's Optical
Flow retime covers slow-mo.