VIDGOD: local AI video production toolkit
Five CLIs around DaVinci Resolve music-video editing, all inference local on Apple Silicon (MPS/MLX): - vg-roto: SAM 2.1 + MatAnyone click-to-cutout -> ProRes 4444 alpha - vg-index / vg-find: PySceneDetect + mlx-whisper searchable clip library - vg-beats: librosa beat grid -> Resolve marker EDL - vg-transcode: legacy codecs -> ProRes LT, deinterlaced, resumable setup/setup_venvs.sh rebuilds venvs, tool clones, checkpoints and applies patches/matanyone-cv2-reader.patch (torchvision >= 0.23 removed read_video). Verified end-to-end on ultra 2026-08-24; setup/smoke_test.sh covers the lanes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
type-two
commit
7911a59186
11
.gitignore
vendored
Normal file
11
.gitignore
vendored
Normal file
@ -0,0 +1,11 @@
|
||||
# rebuilt by setup/setup_venvs.sh — never committed
|
||||
venvs/
|
||||
tools/
|
||||
models/
|
||||
library/
|
||||
work/
|
||||
tests/
|
||||
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.DS_Store
|
||||
25
CLAUDE.md
Normal file
25
CLAUDE.md
Normal file
@ -0,0 +1,25 @@
|
||||
# VIDGOD — agent notes
|
||||
|
||||
Read README.md first for what the tools do. This file is the stuff an agent needs beyond it.
|
||||
|
||||
- **Canonical deploy: `~/Documents/VIDGOD` on ultra** (M1 Ultra 128GB, `johnking@100.91.239.7`).
|
||||
Remote: `ssh://git@100.71.119.27:222/monster/vidgod.git` (Gitea, tailnet-only URL — never a
|
||||
public one). The repo is scripts + patches only; venvs/tools/models/library are untracked
|
||||
and rebuilt by `setup/setup_venvs.sh` (idempotent, resumable, heartbeats to
|
||||
`~/.jobs/vidgod-setup.status`).
|
||||
- Paths are **repo-relative everywhere**: bin scripts locate the root via their own location
|
||||
(`#!/bin/sh` + exec-venv-python polyglot header; don't "fix" line 2 of the python CLIs —
|
||||
it's what makes them run inside the right venv from any checkout location).
|
||||
- Verified 2026-08-24 on ultra: all five lanes, including a real SAM2+MatAnyone cutout
|
||||
(1917 trench demo clip, 2 clicks, matte held over 6s; ~3.4 fr/s at 1920×804 on MPS).
|
||||
- The MatAnyone torchvision fix lives in `patches/matanyone-cv2-reader.patch`, applied by
|
||||
setup step 6 (guard: grep CAP_PROP_FPS). Upstream may eventually fix this — if the patch
|
||||
stops applying, check whether their `read_frame_from_videos` already avoids
|
||||
`torchvision.io.read_video`, and drop the patch step if so.
|
||||
- Test with `setup/smoke_test.sh` (fast) / `--roto` (full pipeline). Test media is synthesized
|
||||
(ffmpeg lavfi + macOS `say`) — nothing copyrighted is committed.
|
||||
- The clip library sqlite lives at `library/clips.sqlite` **per checkout** — the real library
|
||||
on ultra is the one that matters; don't index into a scratch clone and expect vg-find on
|
||||
ultra to see it.
|
||||
- Fleet context (SSH map, farm, heartbeat convention): `~/.claude/skills/fleet` +
|
||||
`~/.claude/skills/jobs` on JING5, GODVERSE_GUIDE.md §11 on ultra/JING5.
|
||||
64
README.md
Normal file
64
README.md
Normal file
@ -0,0 +1,64 @@
|
||||
# VIDGOD
|
||||
|
||||
Open-source AI video production toolkit for music-video editing around DaVinci Resolve.
|
||||
All inference is local on Apple Silicon (PyTorch MPS + MLX) — no cloud, no subscriptions.
|
||||
|
||||
Built for the workflow: hoard old footage → normalize it → index every shot and every
|
||||
spoken word → search it → arrange to a beat grid in Resolve → cut actors out with clicks
|
||||
instead of rotoscoping splines.
|
||||
|
||||
## The tools
|
||||
|
||||
| Command | What it does |
|
||||
|---|---|
|
||||
| `bin/vg-roto CLIP --point x,y` | Click an actor → tracked cutout as **ProRes 4444 with alpha** (SAM 2.1 + MatAnyone). `--grab-frame` saves the prompt frame so you can find coords. `--mode mask` for hard binary masks via SAM2 video propagation. |
|
||||
| `bin/vg-transcode DIR` | Batch-convert old AVI/WMV/MPEG/etc → ProRes LT, mirrored dir tree, auto-deinterlace, skips already-done files. `--h264` for small proxies. |
|
||||
| `bin/vg-index PATHS` | Shot-detect (PySceneDetect) + transcribe (mlx-whisper large-v3-turbo) + thumbnail everything into `library/clips.sqlite` (FTS5). Resumable. |
|
||||
| `bin/vg-find WORDS` | Full-text search the dialogue; shows the enclosing shot. `--cut DIR` exports each matching shot as a ProRes clip. `--shots VIDEO` lists a video's shots. |
|
||||
| `bin/vg-beats SONG` | Beat-track a music file (librosa) → `.beats.csv` + a **Resolve marker EDL** (import: right-click timeline in media pool → Timelines → Import → Timeline Markers from EDL). |
|
||||
|
||||
## Install
|
||||
|
||||
Needs: Apple Silicon Mac, `brew install ffmpeg uv`, ~4GB disk for venvs + models.
|
||||
|
||||
```bash
|
||||
git clone ssh://git@100.71.119.27:222/monster/vidgod.git && cd vidgod
|
||||
./setup/setup_venvs.sh # venvs, SAM2 + MatAnyone clones, checkpoints, patches (~10 min)
|
||||
./setup/smoke_test.sh # fast lanes; add --roto for the full cutout pipeline
|
||||
```
|
||||
|
||||
Everything heavy (venvs/, tools/, models/, library/, work/, tests/) lives untracked inside
|
||||
the working copy; the repo is just the scripts. Long jobs heartbeat to `~/.jobs/vidgod-*.status`.
|
||||
|
||||
## Typical session
|
||||
|
||||
```bash
|
||||
bin/vg-transcode ~/old-tv-rips # → ~/old-tv-rips-prores
|
||||
bin/vg-index ~/old-tv-rips-prores # overnight for a big archive
|
||||
bin/vg-find "radical" --cut ~/mv/pulls # grab every shot where someone says it
|
||||
bin/vg-beats ~/mv/song.mp3 --fps 25 # markers for the Resolve timeline
|
||||
bin/vg-roto ~/mv/pulls/003_*.mov --grab-frame # find your click coords
|
||||
bin/vg-roto ~/mv/pulls/003_*.mov --point 970,220 # → ProRes 4444 alpha cutout
|
||||
```
|
||||
|
||||
## Notes & gotchas
|
||||
|
||||
- **MatAnyone patch**: torchvision ≥ 0.23 removed `torchvision.io.read_video`; setup applies
|
||||
`patches/matanyone-cv2-reader.patch` to its clone automatically. If you re-clone
|
||||
`tools/MatAnyone` by hand, re-run `setup/setup_venvs.sh`.
|
||||
- MatAnyone's checkpoint comes from its GitHub release (the HF repo doesn't carry the
|
||||
file at the root path); its inference script insists on `pretrained_models/` inside its
|
||||
repo dir, so setup symlinks it there.
|
||||
- MatAnyone propagates **forward from a first-frame mask** — `vg-roto --frame N` trims the
|
||||
clip at N first. Prompt on a frame where the target is clearly visible.
|
||||
- Scripts set `/opt/homebrew/bin` in PATH themselves (non-interactive SSH doesn't) and
|
||||
`PYTORCH_ENABLE_MPS_FALLBACK=1` for stray unsupported MPS kernels.
|
||||
- Rough speed on an M1 Ultra: MatAnyone ≈ 3.4 fr/s at 1920×804; whisper large-v3-turbo
|
||||
transcribes far faster than realtime; ProRes transcodes are ffmpeg-bound.
|
||||
|
||||
## Phase 2 (not built yet)
|
||||
|
||||
ProPainter (object/logo removal), RIFE interpolation, Cutie interactive segmentation GUI,
|
||||
MODELBEAST farm operators for roto/index jobs, model mirror to NAS modelzoo. Upscaling
|
||||
old footage already exists fleet-side (`seedvr2_upscale` on the farm); Resolve's Optical
|
||||
Flow retime covers slow-mo.
|
||||
75
bin/vg-beats
Executable file
75
bin/vg-beats
Executable file
@ -0,0 +1,75 @@
|
||||
#!/bin/sh
|
||||
"exec" "`dirname $0`/../venvs/index/bin/python" "$0" "$@"
|
||||
"""vg-beats — extract a beat grid from a music track for cutting in Resolve.
|
||||
|
||||
Usage: vg-beats SONG.mp3 [--fps 25] [--tc-start 01:00:00:00] [--bpm HINT] [--out PREFIX]
|
||||
|
||||
Writes:
|
||||
PREFIX.beats.csv one beat time (seconds) per line, with beat number
|
||||
PREFIX.beats.edl Resolve timeline markers — import via
|
||||
Timeline > (right-click) > Timelines > Import > Timeline Markers from EDL
|
||||
Every 4th beat is a red marker (bar guess), others blue.
|
||||
"""
|
||||
import argparse, os, sys
|
||||
|
||||
os.environ["PATH"] = "/opt/homebrew/bin:" + os.environ.get("PATH", "")
|
||||
|
||||
|
||||
def tc(seconds, fps, start_frames):
|
||||
total = int(round(seconds * fps)) + start_frames
|
||||
f = total % int(fps)
|
||||
s = (total // int(fps)) % 60
|
||||
m = (total // (int(fps) * 60)) % 60
|
||||
h = total // (int(fps) * 3600)
|
||||
return f"{h:02d}:{m:02d}:{s:02d}:{f:02d}"
|
||||
|
||||
|
||||
def parse_tc(t, fps):
|
||||
h, m, s, f = (int(x) for x in t.split(":"))
|
||||
return ((h * 3600 + m * 60 + s) * int(fps)) + f
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("audio")
|
||||
ap.add_argument("--fps", type=float, default=25.0, help="timeline fps for the EDL (default 25)")
|
||||
ap.add_argument("--tc-start", default="01:00:00:00", help="timeline start timecode (Resolve default 01:00:00:00)")
|
||||
ap.add_argument("--bpm", type=float, default=None, help="starting tempo hint")
|
||||
ap.add_argument("--out", default=None, help="output prefix (default: alongside the audio file)")
|
||||
args = ap.parse_args()
|
||||
|
||||
import librosa
|
||||
import numpy as np
|
||||
|
||||
print(f"loading {args.audio} ...")
|
||||
y, sr = librosa.load(args.audio, sr=22050, mono=True)
|
||||
kw = {"start_bpm": args.bpm} if args.bpm else {}
|
||||
tempo, beats = librosa.beat.beat_track(y=y, sr=sr, units="time", **kw)
|
||||
tempo = float(np.atleast_1d(tempo)[0])
|
||||
if len(beats) == 0:
|
||||
sys.exit("no beats detected — is this a music track?")
|
||||
print(f"tempo ~{tempo:.1f} BPM, {len(beats)} beats over {len(y)/sr:.1f}s")
|
||||
|
||||
prefix = args.out or os.path.splitext(args.audio)[0]
|
||||
start_frames = parse_tc(args.tc_start, args.fps)
|
||||
|
||||
with open(prefix + ".beats.csv", "w") as f:
|
||||
f.write("beat,seconds\n")
|
||||
for i, t in enumerate(beats, 1):
|
||||
f.write(f"{i},{t:.4f}\n")
|
||||
|
||||
with open(prefix + ".beats.edl", "w") as f:
|
||||
f.write(f"TITLE: {os.path.basename(prefix)} beats ({tempo:.1f}bpm)\nFCM: NON-DROP FRAME\n\n")
|
||||
for i, t in enumerate(beats, 1):
|
||||
a = tc(t, args.fps, start_frames)
|
||||
b = tc(t + 1.0 / args.fps, args.fps, start_frames)
|
||||
color = "ResolveColorRed" if (i - 1) % 4 == 0 else "ResolveColorBlue"
|
||||
f.write(f"{i:03d} 001 V C {a} {b} {a} {b} \n")
|
||||
f.write(f" |C:{color} |M:Beat {i} |D:1\n\n")
|
||||
|
||||
print(f"wrote {prefix}.beats.csv and {prefix}.beats.edl")
|
||||
print("Resolve: right-click the timeline in the media pool > Timelines > Import > Timeline Markers from EDL")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
95
bin/vg-find
Executable file
95
bin/vg-find
Executable file
@ -0,0 +1,95 @@
|
||||
#!/bin/sh
|
||||
"exec" "`dirname $0`/../venvs/index/bin/python" "$0" "$@"
|
||||
"""vg-find — search the clip library built by vg-index.
|
||||
|
||||
Usage:
|
||||
vg-find radical search dialogue for a word/phrase (FTS5 syntax ok)
|
||||
vg-find "totally awesome" phrase search
|
||||
vg-find radical --cut DIR also export each matching shot as a ProRes clip into DIR
|
||||
vg-find --shots VIDEO list the detected shots of one video
|
||||
"""
|
||||
import argparse, os, sqlite3, subprocess, sys
|
||||
from pathlib import Path
|
||||
|
||||
os.environ["PATH"] = "/opt/homebrew/bin:" + os.environ.get("PATH", "")
|
||||
DB = Path(__file__).resolve().parent.parent / "library/clips.sqlite"
|
||||
|
||||
|
||||
def hms(t):
|
||||
t = max(0, t)
|
||||
return f"{int(t//3600):02d}:{int(t%3600//60):02d}:{t%60:06.3f}"
|
||||
|
||||
|
||||
def cut(src, a, b, dest):
|
||||
subprocess.run(["ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
|
||||
"-ss", f"{a:.3f}", "-to", f"{b:.3f}", "-i", src,
|
||||
"-c:v", "prores_ks", "-profile:v", "1", "-c:a", "pcm_s16le", str(dest)],
|
||||
check=False)
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("query", nargs="*")
|
||||
ap.add_argument("--cut", metavar="DIR", help="export matching shots as ProRes clips")
|
||||
ap.add_argument("--shots", metavar="VIDEO", help="list shots of one video (path substring ok)")
|
||||
ap.add_argument("--limit", type=int, default=40)
|
||||
args = ap.parse_args()
|
||||
|
||||
if not DB.exists():
|
||||
sys.exit("no library yet — run vg-index first")
|
||||
db = sqlite3.connect(DB)
|
||||
|
||||
if args.shots:
|
||||
rows = db.execute("""SELECT v.path, s.idx, s.start_s, s.end_s, s.thumb FROM shots s
|
||||
JOIN videos v ON v.id=s.video_id WHERE v.path LIKE ?
|
||||
ORDER BY v.path, s.idx""", (f"%{args.shots}%",)).fetchall()
|
||||
if not rows:
|
||||
sys.exit("no matching video in library")
|
||||
last = None
|
||||
for path, idx, a, b, thumb in rows:
|
||||
if path != last:
|
||||
print(f"\n{path}")
|
||||
last = path
|
||||
print(f" shot {idx:3d} {hms(a)} - {hms(b)} ({b-a:5.1f}s)")
|
||||
return
|
||||
|
||||
if not args.query:
|
||||
v, s, g = db.execute("""SELECT (SELECT COUNT(*) FROM videos),
|
||||
(SELECT COUNT(*) FROM shots),
|
||||
(SELECT COUNT(*) FROM segs)""").fetchone()
|
||||
print(f"library: {v} videos, {s} shots, {g} dialogue segments")
|
||||
return
|
||||
|
||||
q = " ".join(args.query)
|
||||
fts = f'"{q}"' if " " in q and not any(c in q for c in '"*') else q
|
||||
rows = db.execute("""
|
||||
SELECT v.path, g.start_s, g.end_s, g.text,
|
||||
(SELECT s.start_s FROM shots s WHERE s.video_id=g.video_id
|
||||
AND s.start_s <= (g.start_s+g.end_s)/2 ORDER BY s.start_s DESC LIMIT 1),
|
||||
(SELECT s.end_s FROM shots s WHERE s.video_id=g.video_id
|
||||
AND s.start_s <= (g.start_s+g.end_s)/2 ORDER BY s.start_s DESC LIMIT 1)
|
||||
FROM segs_fts f JOIN segs g ON g.id=f.rowid JOIN videos v ON v.id=g.video_id
|
||||
WHERE segs_fts MATCH ? ORDER BY v.path, g.start_s LIMIT ?""", (fts, args.limit)).fetchall()
|
||||
if not rows:
|
||||
print("no matches")
|
||||
return
|
||||
|
||||
outdir = None
|
||||
if args.cut:
|
||||
outdir = Path(args.cut).expanduser()
|
||||
outdir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
for i, (path, a, b, text, sa, sb) in enumerate(rows, 1):
|
||||
sa = a if sa is None else sa
|
||||
sb = b if sb is None else sb
|
||||
print(f"{hms(a)} {Path(path).name}")
|
||||
print(f" \"{text}\" (shot {hms(sa)} - {hms(sb)})")
|
||||
if outdir:
|
||||
name = f"{i:03d}_{Path(path).stem}_{int(sa)}s.mov"
|
||||
cut(path, sa, sb, outdir / name)
|
||||
print(f" -> {outdir/name}")
|
||||
print(f"\n{len(rows)} matches" + (f", clips in {outdir}" if outdir else ""))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
181
bin/vg-index
Executable file
181
bin/vg-index
Executable file
@ -0,0 +1,181 @@
|
||||
#!/bin/sh
|
||||
"exec" "`dirname $0`/../venvs/index/bin/python" "$0" "$@"
|
||||
"""vg-index — build a searchable clip library: shot detection + transcript + thumbnails.
|
||||
|
||||
Usage: vg-index PATH [PATH...] [--no-whisper] [--force] [--min-shot 0.5]
|
||||
|
||||
For every video found under the given paths:
|
||||
1. PySceneDetect splits it into shots
|
||||
2. a thumbnail is saved for each shot -> library/thumbs/
|
||||
3. Whisper (mlx) transcribes the audio -> timestamped segments
|
||||
4. everything lands in library/clips.sqlite (FTS5 full-text over dialogue)
|
||||
|
||||
Search it with vg-find. Re-running skips unchanged files. Heartbeat: ~/.jobs/vidgod-index.status
|
||||
"""
|
||||
import argparse, hashlib, os, sqlite3, subprocess, sys, time
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
os.environ["PATH"] = "/opt/homebrew/bin:" + os.environ.get("PATH", "")
|
||||
|
||||
VG = Path(__file__).resolve().parent.parent
|
||||
LIB = VG / "library"
|
||||
DB = LIB / "clips.sqlite"
|
||||
THUMBS = LIB / "thumbs"
|
||||
WHISPER_MODEL = "mlx-community/whisper-large-v3-turbo"
|
||||
VIDEO_EXTS = {".mov", ".mp4", ".mkv", ".avi", ".mpg", ".mpeg", ".wmv", ".webm", ".m4v", ".flv", ".mts", ".m2ts", ".vob"}
|
||||
|
||||
|
||||
def hb(msg, note=""):
|
||||
j = Path.home() / ".jobs"
|
||||
j.mkdir(exist_ok=True)
|
||||
(j / "vidgod-index.status").write_text(f"{datetime.now():%Y-%m-%d %H:%M:%S} | {msg} | {note}\n")
|
||||
|
||||
|
||||
def open_db():
|
||||
LIB.mkdir(parents=True, exist_ok=True)
|
||||
THUMBS.mkdir(parents=True, exist_ok=True)
|
||||
db = sqlite3.connect(DB)
|
||||
db.executescript("""
|
||||
CREATE TABLE IF NOT EXISTS videos(
|
||||
id INTEGER PRIMARY KEY, path TEXT UNIQUE, mtime REAL, duration REAL,
|
||||
fps REAL, width INT, height INT, has_transcript INT DEFAULT 0, indexed_at TEXT);
|
||||
CREATE TABLE IF NOT EXISTS shots(
|
||||
id INTEGER PRIMARY KEY, video_id INT, idx INT, start_s REAL, end_s REAL, thumb TEXT);
|
||||
CREATE TABLE IF NOT EXISTS segs(
|
||||
id INTEGER PRIMARY KEY, video_id INT, start_s REAL, end_s REAL, text TEXT);
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS segs_fts USING fts5(text, content='segs', content_rowid='id');
|
||||
CREATE INDEX IF NOT EXISTS idx_shots_video ON shots(video_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_segs_video ON segs(video_id);
|
||||
""")
|
||||
return db
|
||||
|
||||
|
||||
def probe(path):
|
||||
out = subprocess.run(
|
||||
["ffprobe", "-v", "error", "-select_streams", "v:0",
|
||||
"-show_entries", "stream=width,height,avg_frame_rate:format=duration",
|
||||
"-of", "csv=p=0", str(path)],
|
||||
capture_output=True, text=True).stdout.strip().splitlines()
|
||||
w = h = 0; fps = 25.0; dur = 0.0
|
||||
for line in out:
|
||||
parts = line.split(",")
|
||||
if len(parts) >= 3:
|
||||
w, h = int(parts[0]), int(parts[1])
|
||||
try:
|
||||
num, den = parts[2].split("/")
|
||||
fps = float(num) / float(den) if float(den) else 25.0
|
||||
except ValueError:
|
||||
pass
|
||||
elif len(parts) == 1 and parts[0]:
|
||||
try: dur = float(parts[0])
|
||||
except ValueError: pass
|
||||
return dur, fps, w, h
|
||||
|
||||
|
||||
def detect_shots(path, min_shot):
|
||||
from scenedetect import detect, ContentDetector
|
||||
scenes = detect(str(path), ContentDetector(min_scene_len=int(min_shot * 25)))
|
||||
if not scenes:
|
||||
return None # single-shot video; caller uses full duration
|
||||
return [(s.seconds, e.seconds) for s, e in scenes]
|
||||
|
||||
|
||||
def save_thumb(path, t, dest):
|
||||
subprocess.run(["ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
|
||||
"-ss", f"{t:.3f}", "-i", str(path),
|
||||
"-frames:v", "1", "-vf", "scale=480:-2", str(dest)],
|
||||
capture_output=True)
|
||||
return dest.exists()
|
||||
|
||||
|
||||
def transcribe(path):
|
||||
import mlx_whisper
|
||||
res = mlx_whisper.transcribe(str(path), path_or_hf_repo=WHISPER_MODEL)
|
||||
return [(s["start"], s["end"], s["text"].strip()) for s in res.get("segments", []) if s["text"].strip()]
|
||||
|
||||
|
||||
def index_file(db, path, args, pos, total):
|
||||
st = path.stat()
|
||||
row = db.execute("SELECT id, mtime FROM videos WHERE path=?", (str(path),)).fetchone()
|
||||
if row and abs(row[1] - st.st_mtime) < 1 and not args.force:
|
||||
return "skip"
|
||||
if row: # stale entry — wipe and redo
|
||||
vid = row[0]
|
||||
db.execute("DELETE FROM segs_fts WHERE rowid IN (SELECT id FROM segs WHERE video_id=?)", (vid,))
|
||||
db.execute("DELETE FROM segs WHERE video_id=?", (vid,))
|
||||
db.execute("DELETE FROM shots WHERE video_id=?", (vid,))
|
||||
db.execute("DELETE FROM videos WHERE id=?", (vid,))
|
||||
|
||||
hb(f"{pos}/{total}", f"probing {path.name}")
|
||||
dur, fps, w, h = probe(path)
|
||||
if dur <= 0:
|
||||
return "unreadable"
|
||||
cur = db.execute(
|
||||
"INSERT INTO videos(path, mtime, duration, fps, width, height, indexed_at) VALUES(?,?,?,?,?,?,?)",
|
||||
(str(path), st.st_mtime, dur, fps, w, h, datetime.now().isoformat(timespec="seconds")))
|
||||
vid = cur.lastrowid
|
||||
|
||||
hb(f"{pos}/{total}", f"shots: {path.name}")
|
||||
shots = detect_shots(path, args.min_shot) or [(0.0, dur)]
|
||||
tag = hashlib.md5(str(path).encode()).hexdigest()[:10]
|
||||
for i, (a, b) in enumerate(shots):
|
||||
thumb = THUMBS / f"{tag}_{i:04d}.jpg"
|
||||
save_thumb(path, a + (b - a) / 2, thumb)
|
||||
db.execute("INSERT INTO shots(video_id, idx, start_s, end_s, thumb) VALUES(?,?,?,?,?)",
|
||||
(vid, i, a, b, str(thumb)))
|
||||
|
||||
if not args.no_whisper:
|
||||
hb(f"{pos}/{total}", f"whisper: {path.name}")
|
||||
try:
|
||||
for a, b, text in transcribe(path):
|
||||
cur = db.execute("INSERT INTO segs(video_id, start_s, end_s, text) VALUES(?,?,?,?)",
|
||||
(vid, a, b, text))
|
||||
db.execute("INSERT INTO segs_fts(rowid, text) VALUES(?,?)", (cur.lastrowid, text))
|
||||
db.execute("UPDATE videos SET has_transcript=1 WHERE id=?", (vid,))
|
||||
except Exception as e: # no audio track etc — keep the shots
|
||||
print(f" whisper failed on {path.name}: {e}", file=sys.stderr)
|
||||
|
||||
db.commit()
|
||||
return f"{len(shots)} shots"
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("paths", nargs="+")
|
||||
ap.add_argument("--no-whisper", action="store_true")
|
||||
ap.add_argument("--force", action="store_true", help="re-index even if unchanged")
|
||||
ap.add_argument("--min-shot", type=float, default=0.5, help="minimum shot length, seconds")
|
||||
args = ap.parse_args()
|
||||
|
||||
files = []
|
||||
for p in args.paths:
|
||||
p = Path(p).expanduser().resolve()
|
||||
if p.is_dir():
|
||||
files += sorted(x for x in p.rglob("*") if x.suffix.lower() in VIDEO_EXTS and x.is_file())
|
||||
elif p.is_file():
|
||||
files.append(p)
|
||||
if not files:
|
||||
sys.exit("no video files found")
|
||||
|
||||
db = open_db()
|
||||
t0 = time.time()
|
||||
counts = {}
|
||||
for i, f in enumerate(files, 1):
|
||||
try:
|
||||
r = index_file(db, f, args, i, len(files))
|
||||
except KeyboardInterrupt:
|
||||
raise
|
||||
except Exception as e:
|
||||
r = "error"
|
||||
print(f"ERROR indexing {f}: {e}", file=sys.stderr)
|
||||
counts[r] = counts.get(r, 0) + 1
|
||||
print(f"[{i}/{len(files)}] {f.name}: {r}")
|
||||
hb(f"DONE {len(files)}/{len(files)}", str(counts))
|
||||
n = db.execute("SELECT COUNT(*) FROM shots").fetchone()[0]
|
||||
s = db.execute("SELECT COUNT(*) FROM segs").fetchone()[0]
|
||||
print(f"\nlibrary now: {n} shots, {s} dialogue segments ({time.time()-t0:.0f}s). Search with vg-find.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
222
bin/vg-roto
Executable file
222
bin/vg-roto
Executable file
@ -0,0 +1,222 @@
|
||||
#!/bin/sh
|
||||
"exec" "`dirname $0`/../venvs/roto/bin/python" "$0" "$@"
|
||||
"""vg-roto — one-click actor cutout: SAM 2.1 + MatAnyone on Apple Silicon.
|
||||
|
||||
Usage:
|
||||
vg-roto CLIP.mp4 --point 640,360 click on the actor (frame 0 coords)
|
||||
vg-roto CLIP.mp4 --point 640,360 --point 700,200 multiple clicks refine the pick
|
||||
vg-roto CLIP.mp4 --box 400,100,900,700 or a rough box around them
|
||||
vg-roto CLIP.mp4 --point ... --neg 100,100 negative click excludes a region
|
||||
vg-roto CLIP.mp4 --grab-frame just save frame 0 as a PNG so you
|
||||
can find click coords, then exit
|
||||
|
||||
Options: --out DIR (default CLIP_roto/), --frame N (prompt+start on frame N),
|
||||
--mode matte|mask (matte = soft alpha via MatAnyone, default;
|
||||
mask = hard binary via SAM2 video propagation),
|
||||
--max-size N (downscale min side for inference, e.g. 720 for speed)
|
||||
|
||||
Output in DIR:
|
||||
CLIP_alpha.mov ProRes 4444 with real alpha -> drop straight into Resolve
|
||||
CLIP_matte.mov the matte alone (grayscale)
|
||||
CLIP_green.mp4 preview comp over green
|
||||
"""
|
||||
import argparse, os, shutil, subprocess, sys
|
||||
from pathlib import Path
|
||||
|
||||
os.environ["PATH"] = "/opt/homebrew/bin:" + os.environ.get("PATH", "")
|
||||
os.environ.setdefault("PYTORCH_ENABLE_MPS_FALLBACK", "1")
|
||||
|
||||
VG = Path(__file__).resolve().parent.parent
|
||||
SAM2_CKPT = VG / "models/sam2.1_hiera_large.pt"
|
||||
SAM2_CFG = "configs/sam2.1/sam2.1_hiera_l.yaml"
|
||||
MATANYONE = VG / "tools/MatAnyone"
|
||||
RPY = VG / "venvs/roto/bin/python"
|
||||
|
||||
|
||||
def run(cmd, **kw):
|
||||
r = subprocess.run([str(c) for c in cmd], **kw)
|
||||
if r.returncode != 0:
|
||||
sys.exit(f"command failed: {' '.join(str(c) for c in cmd)}")
|
||||
return r
|
||||
|
||||
|
||||
def probe_video(path):
|
||||
out = subprocess.run(
|
||||
["ffprobe", "-v", "error", "-select_streams", "v:0",
|
||||
"-show_entries", "stream=width,height,avg_frame_rate,nb_frames",
|
||||
"-of", "csv=p=0", str(path)], capture_output=True, text=True).stdout.strip()
|
||||
w, h, fr = out.split(",")[:3]
|
||||
num, den = fr.split("/")
|
||||
return int(w), int(h), (float(num) / float(den) if float(den) else 25.0)
|
||||
|
||||
|
||||
def grab_frame(video, n, fps, dest):
|
||||
run(["ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
|
||||
"-ss", f"{n / fps:.6f}", "-i", video, "-frames:v", "1", dest])
|
||||
|
||||
|
||||
def sam2_first_frame_mask(frame_png, points, neg_points, box, mask_png):
|
||||
import numpy as np, torch
|
||||
from PIL import Image
|
||||
from sam2.build_sam import build_sam2
|
||||
from sam2.sam2_image_predictor import SAM2ImagePredictor
|
||||
|
||||
device = "mps" if torch.backends.mps.is_available() else "cpu"
|
||||
print(f"SAM2: loading {SAM2_CKPT.name} on {device} ...")
|
||||
model = build_sam2(SAM2_CFG, str(SAM2_CKPT), device=device)
|
||||
pred = SAM2ImagePredictor(model)
|
||||
img = np.array(Image.open(frame_png).convert("RGB"))
|
||||
pred.set_image(img)
|
||||
|
||||
pc = pl = bx = None
|
||||
if points or neg_points:
|
||||
pc = np.array(points + neg_points, dtype=np.float32)
|
||||
pl = np.array([1] * len(points) + [0] * len(neg_points), dtype=np.int32)
|
||||
if box:
|
||||
bx = np.array(box, dtype=np.float32)
|
||||
masks, scores, _ = pred.predict(point_coords=pc, point_labels=pl, box=bx, multimask_output=True)
|
||||
best = int(np.argmax(scores))
|
||||
m = (masks[best] > 0).astype(np.uint8) * 255
|
||||
Image.fromarray(m).save(mask_png)
|
||||
cov = 100.0 * (m > 0).mean()
|
||||
print(f"SAM2: mask score {scores[best]:.3f}, covers {cov:.1f}% of frame")
|
||||
if cov < 0.05 or cov > 95:
|
||||
print("WARNING: mask looks degenerate — check your click point (--grab-frame to inspect)")
|
||||
return mask_png
|
||||
|
||||
|
||||
def matanyone_matte(video, mask_png, outdir, max_size):
|
||||
env = dict(os.environ)
|
||||
cmd = [RPY, "inference_matanyone.py", "-i", str(video), "-m", str(mask_png),
|
||||
"-o", str(outdir), "--max_size", str(max_size)]
|
||||
print("MatAnyone: propagating matte (this is the slow part) ...")
|
||||
r = subprocess.run([str(c) for c in cmd], cwd=MATANYONE, env=env)
|
||||
if r.returncode != 0:
|
||||
sys.exit("MatAnyone failed")
|
||||
stem = Path(video).stem
|
||||
pha = outdir / f"{stem}_pha.mp4"
|
||||
if not pha.exists():
|
||||
cands = sorted(outdir.glob("*_pha.mp4"))
|
||||
if not cands:
|
||||
sys.exit(f"MatAnyone produced no *_pha.mp4 in {outdir}")
|
||||
pha = cands[-1]
|
||||
return pha
|
||||
|
||||
|
||||
def sam2_video_masks(video, points, neg_points, box, frame_n, workdir, fps):
|
||||
import numpy as np, torch
|
||||
from PIL import Image
|
||||
from sam2.build_sam import build_sam2_video_predictor
|
||||
|
||||
frames = workdir / "frames"
|
||||
frames.mkdir(parents=True, exist_ok=True)
|
||||
if not list(frames.glob("*.jpg")):
|
||||
run(["ffmpeg", "-hide_banner", "-loglevel", "error", "-y", "-i", video,
|
||||
"-q:v", "2", "-start_number", "0", frames / "%05d.jpg"])
|
||||
device = "mps" if torch.backends.mps.is_available() else "cpu"
|
||||
print(f"SAM2 video: loading on {device} ...")
|
||||
pred = build_sam2_video_predictor(SAM2_CFG, str(SAM2_CKPT), device=device)
|
||||
state = pred.init_state(video_path=str(frames), offload_video_to_cpu=True,
|
||||
offload_state_to_cpu=True)
|
||||
pc = pl = bx = None
|
||||
if points or neg_points:
|
||||
pc = np.array(points + neg_points, dtype=np.float32)
|
||||
pl = np.array([1] * len(points) + [0] * len(neg_points), dtype=np.int32)
|
||||
if box:
|
||||
bx = np.array(box, dtype=np.float32)
|
||||
pred.add_new_points_or_box(state, frame_idx=frame_n, obj_id=1,
|
||||
points=pc, labels=pl, box=bx)
|
||||
masks_dir = workdir / "masks"
|
||||
masks_dir.mkdir(exist_ok=True)
|
||||
n = 0
|
||||
for fidx, _, logits in pred.propagate_in_video(state):
|
||||
m = (logits[0] > 0).cpu().numpy().squeeze().astype(np.uint8) * 255
|
||||
Image.fromarray(m).save(masks_dir / f"{fidx:05d}.png")
|
||||
n += 1
|
||||
if n % 50 == 0:
|
||||
print(f" propagated {n} frames")
|
||||
print(f"SAM2 video: {n} frames masked")
|
||||
matte = workdir / "matte_raw.mp4"
|
||||
run(["ffmpeg", "-hide_banner", "-loglevel", "error", "-y", "-framerate", f"{fps}",
|
||||
"-start_number", str(frame_n), "-i", masks_dir / "%05d.png",
|
||||
"-c:v", "libx264", "-crf", "12", "-pix_fmt", "yuv420p", matte])
|
||||
return matte
|
||||
|
||||
|
||||
def compose_outputs(video, pha, outdir, stem, fps, w, h):
|
||||
alpha_mov = outdir / f"{stem}_alpha.mov"
|
||||
matte_mov = outdir / f"{stem}_matte.mov"
|
||||
green_mp4 = outdir / f"{stem}_green.mp4"
|
||||
scale = f"scale={w}:{h}:flags=bicubic,format=gray"
|
||||
print("composing ProRes 4444 + previews ...")
|
||||
run(["ffmpeg", "-hide_banner", "-loglevel", "error", "-y", "-i", video, "-i", pha,
|
||||
"-filter_complex", f"[1:v]{scale}[a];[0:v][a]alphamerge,format=yuva444p10le[out]",
|
||||
"-map", "[out]", "-map", "0:a:0?", "-c:v", "prores_ks", "-profile:v", "4444",
|
||||
"-c:a", "pcm_s16le", "-shortest", alpha_mov])
|
||||
run(["ffmpeg", "-hide_banner", "-loglevel", "error", "-y", "-i", pha,
|
||||
"-vf", scale, "-c:v", "prores_ks", "-profile:v", "1", matte_mov])
|
||||
run(["ffmpeg", "-hide_banner", "-loglevel", "error", "-y", "-i", video, "-i", pha,
|
||||
"-filter_complex",
|
||||
f"color=0x00b140:size={w}x{h}:rate={fps}[bg];"
|
||||
f"[1:v]{scale}[a];[0:v][a]alphamerge[fg];[bg][fg]overlay=shortest=1,format=yuv420p[out]",
|
||||
"-map", "[out]", "-c:v", "libx264", "-crf", "18", green_mp4])
|
||||
return alpha_mov, matte_mov, green_mp4
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("video")
|
||||
ap.add_argument("--point", action="append", default=[], help="x,y positive click (repeatable)")
|
||||
ap.add_argument("--neg", action="append", default=[], help="x,y negative click (repeatable)")
|
||||
ap.add_argument("--box", help="x1,y1,x2,y2")
|
||||
ap.add_argument("--frame", type=int, default=0, help="frame to prompt on / start from")
|
||||
ap.add_argument("--mode", choices=["matte", "mask"], default="matte")
|
||||
ap.add_argument("--max-size", type=int, default=-1, help="downscale min side for inference")
|
||||
ap.add_argument("--out", default=None)
|
||||
ap.add_argument("--grab-frame", action="store_true", help="save the prompt frame as PNG and exit")
|
||||
args = ap.parse_args()
|
||||
|
||||
video = Path(args.video).expanduser().resolve()
|
||||
if not video.exists():
|
||||
sys.exit(f"no such file: {video}")
|
||||
stem = video.stem
|
||||
outdir = Path(args.out).expanduser() if args.out else video.parent / f"{stem}_roto"
|
||||
outdir.mkdir(parents=True, exist_ok=True)
|
||||
work = outdir / "work"
|
||||
work.mkdir(exist_ok=True)
|
||||
|
||||
w, h, fps = probe_video(video)
|
||||
frame_png = work / f"frame{args.frame:05d}.png"
|
||||
grab_frame(video, args.frame, fps, frame_png)
|
||||
if args.grab_frame:
|
||||
print(f"prompt frame saved: {frame_png} ({w}x{h}) — open it, note x,y of your click")
|
||||
return
|
||||
|
||||
points = [tuple(float(v) for v in p.split(",")) for p in args.point]
|
||||
negs = [tuple(float(v) for v in p.split(",")) for p in args.neg]
|
||||
box = [float(v) for v in args.box.split(",")] if args.box else None
|
||||
if not points and not box:
|
||||
sys.exit("give me --point x,y or --box x1,y1,x2,y2 (use --grab-frame to find coords)")
|
||||
|
||||
src = video
|
||||
if args.frame > 0 and args.mode == "matte":
|
||||
src = work / f"{stem}_from{args.frame}.mp4"
|
||||
print(f"trimming from frame {args.frame} (MatAnyone propagates forward from its mask)")
|
||||
run(["ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
|
||||
"-ss", f"{args.frame / fps:.6f}", "-i", video,
|
||||
"-c:v", "libx264", "-crf", "12", "-c:a", "aac", src])
|
||||
|
||||
mask_png = work / "first_mask.png"
|
||||
sam2_first_frame_mask(frame_png, points, negs, box, mask_png)
|
||||
|
||||
if args.mode == "matte":
|
||||
pha = matanyone_matte(src, mask_png, work, args.max_size)
|
||||
else:
|
||||
pha = sam2_video_masks(src, points, negs, box, args.frame, work, fps)
|
||||
|
||||
alpha, matte, green = compose_outputs(src, pha, outdir, stem, fps, w, h)
|
||||
print(f"\ndone:\n {alpha} <- ProRes 4444 with alpha, drop into Resolve\n {matte}\n {green} <- preview")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
67
bin/vg-transcode
Executable file
67
bin/vg-transcode
Executable file
@ -0,0 +1,67 @@
|
||||
#!/bin/zsh
|
||||
# vg-transcode — batch-normalize old footage to ProRes LT for editing.
|
||||
# Usage: vg-transcode SRC_DIR [DEST_DIR] [--h264]
|
||||
# Recurses SRC_DIR, converts every video to ProRes LT .mov (or H.264 with --h264),
|
||||
# mirrors the directory structure, skips files already done, deinterlaces
|
||||
# interlaced sources (bwdif to progressive at double rate).
|
||||
# Heartbeat: ~/.jobs/vidgod-transcode.status
|
||||
export PATH=/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin
|
||||
set -u
|
||||
|
||||
SRC="${1:?usage: vg-transcode SRC_DIR [DEST_DIR] [--h264]}"
|
||||
DEST="${2:-}"
|
||||
CODEC=prores
|
||||
[[ "${2:-}" == "--h264" ]] && { DEST=""; CODEC=h264; }
|
||||
[[ "${3:-}" == "--h264" ]] && CODEC=h264
|
||||
SRC="${SRC:A}"
|
||||
[[ -d "$SRC" ]] || { echo "not a dir: $SRC" >&2; exit 1; }
|
||||
[[ -z "$DEST" ]] && DEST="${SRC}-prores"
|
||||
mkdir -p "$DEST"
|
||||
|
||||
hb(){ mkdir -p ~/.jobs; echo "$(date '+%F %T') | $1 | $2" > ~/.jobs/vidgod-transcode.status; }
|
||||
|
||||
typeset -a files
|
||||
files=()
|
||||
while IFS= read -r -d '' f; do files+=("$f"); done < <(
|
||||
find "$SRC" \( -iname '*.avi' -o -iname '*.wmv' -o -iname '*.mpg' -o -iname '*.mpeg' \
|
||||
-o -iname '*.asf' -o -iname '*.flv' -o -iname '*.rm' -o -iname '*.rmvb' -o -iname '*.vob' \
|
||||
-o -iname '*.m2v' -o -iname '*.divx' -o -iname '*.mov' -o -iname '*.mp4' -o -iname '*.mkv' \
|
||||
-o -iname '*.webm' -o -iname '*.m4v' -o -iname '*.3gp' -o -iname '*.mts' -o -iname '*.m2ts' \) \
|
||||
-not -path "$DEST/*" -print0 | sort -z)
|
||||
|
||||
total=${#files[@]}
|
||||
[[ $total -eq 0 ]] && { echo "no video files under $SRC"; exit 0; }
|
||||
echo "transcoding $total files: $SRC -> $DEST ($CODEC)"
|
||||
|
||||
n=0; ok=0; skip=0; fail=0
|
||||
for f in "${files[@]}"; do
|
||||
n=$((n+1))
|
||||
rel="${f#$SRC/}"
|
||||
out="$DEST/${rel%.*}.mov"
|
||||
[[ "$CODEC" == "h264" ]] && out="$DEST/${rel%.*}.mp4"
|
||||
mkdir -p "${out:h}"
|
||||
hb "$n/$total" "${rel:t}"
|
||||
if [[ -s "$out" && "$out" -nt "$f" ]]; then skip=$((skip+1)); continue; fi
|
||||
|
||||
# detect interlacing
|
||||
fo=$(ffprobe -v error -select_streams v:0 -show_entries stream=field_order -of csv=p=0 "$f" 2>/dev/null)
|
||||
vf="format=yuv422p10le"
|
||||
case "$fo" in
|
||||
tt|bb|tb|bt) vf="bwdif=mode=1:parity=auto,format=yuv422p10le" ;;
|
||||
esac
|
||||
|
||||
if [[ "$CODEC" == "prores" ]]; then
|
||||
ffmpeg -hide_banner -loglevel error -y -i "$f" -vf "$vf" \
|
||||
-c:v prores_ks -profile:v 1 -vendor apl0 \
|
||||
-c:a pcm_s16le -map 0:v:0 -map '0:a:0?' "$out" </dev/null
|
||||
else
|
||||
ffmpeg -hide_banner -loglevel error -y -i "$f" -vf "${vf%,format=yuv422p10le},format=yuv420p" \
|
||||
-c:v libx264 -preset fast -crf 18 -c:a aac -b:a 192k \
|
||||
-map 0:v:0 -map '0:a:0?' "$out" </dev/null
|
||||
fi
|
||||
if [[ $? -eq 0 && -s "$out" ]]; then ok=$((ok+1)); else fail=$((fail+1)); rm -f "$out"; echo "FAILED: $rel" >&2; fi
|
||||
done
|
||||
|
||||
hb "DONE $n/$total" "ok=$ok skip=$skip fail=$fail"
|
||||
echo "done: $ok converted, $skip skipped (already done), $fail failed"
|
||||
[[ $fail -eq 0 ]]
|
||||
24
patches/matanyone-cv2-reader.patch
Normal file
24
patches/matanyone-cv2-reader.patch
Normal file
@ -0,0 +1,24 @@
|
||||
diff --git a/matanyone/utils/inference_utils.py b/matanyone/utils/inference_utils.py
|
||||
index c5bb6a0..41c4c40 100644
|
||||
--- a/matanyone/utils/inference_utils.py
|
||||
+++ b/matanyone/utils/inference_utils.py
|
||||
@@ -12,8 +12,17 @@ VIDEO_EXTENSIONS = ('.mp4', '.mov', '.avi', '.MP4', '.MOV', '.AVI')
|
||||
def read_frame_from_videos(frame_root):
|
||||
if frame_root.endswith(VIDEO_EXTENSIONS): # Video file path
|
||||
video_name = os.path.basename(frame_root)[:-4]
|
||||
- frames, _, info = torchvision.io.read_video(filename=frame_root, pts_unit='sec', output_format='TCHW') # RGB
|
||||
- fps = info['video_fps']
|
||||
+ # torchvision.io.read_video was removed in torchvision >= 0.23 — read via cv2
|
||||
+ cap = cv2.VideoCapture(frame_root)
|
||||
+ fps = cap.get(cv2.CAP_PROP_FPS) or 24
|
||||
+ frames = []
|
||||
+ while True:
|
||||
+ ok, frame = cap.read()
|
||||
+ if not ok:
|
||||
+ break
|
||||
+ frames.append(frame[..., [2, 1, 0]]) # BGR -> RGB
|
||||
+ cap.release()
|
||||
+ frames = torch.from_numpy(np.array(frames)).permute(0, 3, 1, 2).contiguous() # TCHW
|
||||
else:
|
||||
video_name = os.path.basename(frame_root)
|
||||
frames = []
|
||||
86
setup/setup_venvs.sh
Executable file
86
setup/setup_venvs.sh
Executable file
@ -0,0 +1,86 @@
|
||||
#!/bin/zsh
|
||||
# VIDGOD bootstrap — venvs, repos, checkpoints, patches. Idempotent: re-run to resume.
|
||||
# Run from anywhere: paths derive from this script's location.
|
||||
# Heartbeat: ~/.jobs/vidgod-setup.status
|
||||
export PATH=/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin
|
||||
VG="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
JOB=vidgod-setup
|
||||
|
||||
hb(){ mkdir -p ~/.jobs; echo "$(date '+%F %T') | $1 | $2" > ~/.jobs/$JOB.status; }
|
||||
die(){ hb "FAILED" "$1"; echo "FATAL: $1" >&2; exit 1; }
|
||||
|
||||
command -v uv >/dev/null || die "uv not installed (brew install uv)"
|
||||
command -v ffmpeg >/dev/null || die "ffmpeg not installed (brew install ffmpeg)"
|
||||
|
||||
hb "1/10" "creating dirs"
|
||||
mkdir -p "$VG"/{venvs,tools,models,library/thumbs,work,tests} || die "mkdir"
|
||||
|
||||
# ---- roto venv (torch/MPS + SAM2 + MatAnyone) ----
|
||||
hb "2/10" "roto venv (py3.12)"
|
||||
[ -x "$VG/venvs/roto/bin/python" ] || uv venv "$VG/venvs/roto" --python 3.12 || die "uv venv roto"
|
||||
RPY="$VG/venvs/roto/bin/python"
|
||||
|
||||
hb "3/10" "torch+torchvision into roto (big download)"
|
||||
uv pip install --python "$RPY" torch torchvision numpy opencv-python pillow tqdm huggingface_hub imageio imageio-ffmpeg || die "torch install"
|
||||
|
||||
hb "4/10" "clone + install SAM2"
|
||||
if [ ! -d "$VG/tools/sam2" ]; then
|
||||
git clone --depth 1 https://github.com/facebookresearch/sam2 "$VG/tools/sam2" || die "sam2 clone"
|
||||
fi
|
||||
SAM2_BUILD_CUDA=0 uv pip install --python "$RPY" --no-build-isolation -e "$VG/tools/sam2" || die "sam2 install"
|
||||
|
||||
hb "5/10" "clone + install MatAnyone"
|
||||
if [ ! -d "$VG/tools/MatAnyone" ]; then
|
||||
git clone --depth 1 https://github.com/pq-yang/MatAnyone "$VG/tools/MatAnyone" || die "matanyone clone"
|
||||
fi
|
||||
if [ -f "$VG/tools/MatAnyone/requirements.txt" ]; then
|
||||
uv pip install --python "$RPY" -r "$VG/tools/MatAnyone/requirements.txt" || echo "WARN: some MatAnyone reqs failed, continuing"
|
||||
fi
|
||||
if [ -f "$VG/tools/MatAnyone/pyproject.toml" ] || [ -f "$VG/tools/MatAnyone/setup.py" ]; then
|
||||
uv pip install --python "$RPY" --no-build-isolation -e "$VG/tools/MatAnyone" || echo "WARN: matanyone -e install failed (will use sys.path)"
|
||||
fi
|
||||
|
||||
hb "6/10" "patch MatAnyone video reader (torchvision >= 0.23 removed read_video)"
|
||||
if ! grep -q "CAP_PROP_FPS" "$VG/tools/MatAnyone/matanyone/utils/inference_utils.py"; then
|
||||
git -C "$VG/tools/MatAnyone" apply "$VG/patches/matanyone-cv2-reader.patch" || die "matanyone patch failed to apply"
|
||||
echo "patch applied"
|
||||
else
|
||||
echo "patch already applied"
|
||||
fi
|
||||
|
||||
# ---- index venv (scene detect + whisper + beats) ----
|
||||
hb "7/10" "index venv + packages"
|
||||
[ -x "$VG/venvs/index/bin/python" ] || uv venv "$VG/venvs/index" --python 3.12 || die "uv venv index"
|
||||
IPY="$VG/venvs/index/bin/python"
|
||||
uv pip install --python "$IPY" mlx-whisper "scenedetect[opencv]" librosa soundfile numpy pillow tqdm || die "index pkgs"
|
||||
|
||||
# ---- checkpoints ----
|
||||
hb "8/10" "SAM2.1 large checkpoint (~856MB)"
|
||||
CKPT="$VG/models/sam2.1_hiera_large.pt"
|
||||
if [ ! -s "$CKPT" ]; then
|
||||
curl -fL --retry 3 -o "$CKPT.part" https://dl.fbaipublicfiles.com/segment_anything_2/092824/sam2.1_hiera_large.pt || die "sam2 ckpt download"
|
||||
mv "$CKPT.part" "$CKPT"
|
||||
fi
|
||||
|
||||
hb "9/10" "MatAnyone checkpoint (~135MB, github release)"
|
||||
MA="$VG/models/matanyone.pth"
|
||||
if [ ! -s "$MA" ]; then
|
||||
curl -fL --retry 3 -o "$MA.part" https://github.com/pq-yang/MatAnyone/releases/download/v1.0.0/matanyone.pth || die "matanyone ckpt download"
|
||||
mv "$MA.part" "$MA"
|
||||
fi
|
||||
# its inference script hard-codes pretrained_models/ relative to the repo dir
|
||||
mkdir -p "$VG/tools/MatAnyone/pretrained_models"
|
||||
ln -sf "$MA" "$VG/tools/MatAnyone/pretrained_models/matanyone.pth"
|
||||
|
||||
hb "10/10" "smoke: torch/MPS + imports"
|
||||
"$RPY" - <<'PYEOF' || die "roto smoke"
|
||||
import torch, sam2
|
||||
print("torch", torch.__version__, "mps:", torch.backends.mps.is_available())
|
||||
PYEOF
|
||||
"$IPY" - <<'PYEOF' || die "index smoke"
|
||||
import mlx_whisper, scenedetect, librosa
|
||||
print("index venv ok")
|
||||
PYEOF
|
||||
|
||||
hb "DONE" "venvs+repos+ckpts+patches ready"
|
||||
echo "SETUP DONE — try: $VG/setup/smoke_test.sh"
|
||||
53
setup/smoke_test.sh
Executable file
53
setup/smoke_test.sh
Executable file
@ -0,0 +1,53 @@
|
||||
#!/bin/zsh
|
||||
# VIDGOD smoke test — fast lanes only (beats, transcode, index, find). ~1-2 min.
|
||||
# Add --roto to also run the full SAM2+MatAnyone pipeline on a bundled demo clip (minutes).
|
||||
export PATH=/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin
|
||||
VG="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
T="$VG/tests/smoke"
|
||||
mkdir -p "$T/oldstyle"
|
||||
fails=0
|
||||
check(){ if [ $1 -eq 0 ]; then echo "PASS: $2"; else echo "FAIL: $2"; fails=$((fails+1)); fi }
|
||||
|
||||
echo "== 1. vg-beats (synthetic 120bpm click) =="
|
||||
"$VG/venvs/index/bin/python" - <<PYEOF
|
||||
import numpy as np, soundfile as sf
|
||||
sr=22050; dur=20; y=np.zeros(sr*dur)
|
||||
for b in np.arange(0,dur,0.5):
|
||||
i=int(b*sr); n=min(3000,len(y)-i)
|
||||
e=np.exp(-np.arange(n)/500); y[i:i+n]+=0.9*np.sin(2*np.pi*60*np.arange(n)/sr)*e
|
||||
sf.write("$T/click120.wav", y, sr)
|
||||
PYEOF
|
||||
"$VG/bin/vg-beats" "$T/click120.wav" --fps 25 >/dev/null 2>&1
|
||||
[ -s "$T/click120.beats.edl" ] && grep -q "ResolveColor" "$T/click120.beats.edl"
|
||||
check $? "vg-beats produced marker EDL"
|
||||
|
||||
echo "== 2. vg-transcode (fake xvid avi + wmv) =="
|
||||
ffmpeg -hide_banner -loglevel error -y -f lavfi -i "testsrc=duration=4:size=640x480:rate=25" -c:v mpeg4 -vtag xvid -qscale:v 5 "$T/oldstyle/fake_xvid.avi"
|
||||
ffmpeg -hide_banner -loglevel error -y -f lavfi -i "smptebars=duration=4:size=640x480:rate=25" -f lavfi -i "sine=frequency=440:duration=4" -c:v wmv2 -c:a wmav2 "$T/oldstyle/fake_wmv.wmv"
|
||||
"$VG/bin/vg-transcode" "$T/oldstyle" >/dev/null 2>&1
|
||||
[ -s "$T/oldstyle-prores/fake_xvid.mov" ] && [ -s "$T/oldstyle-prores/fake_wmv.mov" ]
|
||||
check $? "vg-transcode converted both to ProRes"
|
||||
|
||||
echo "== 3. vg-index + vg-find (speech clip with a hard cut) =="
|
||||
if command -v say >/dev/null; then
|
||||
say -o "$T/speech.aiff" "The radical robot destroyed the shopping mall. Totally awesome dude."
|
||||
ffmpeg -hide_banner -loglevel error -y -f lavfi -i "testsrc2=duration=2.5:size=640x360:rate=25" -f lavfi -i "smptebars=duration=2.5:size=640x360:rate=25" -i "$T/speech.aiff" -filter_complex "[0:v][1:v]concat=n=2:v=1[v]" -map "[v]" -map 2:a -c:v libx264 -crf 20 -c:a aac -shortest "$T/dialogue_test.mp4"
|
||||
"$VG/bin/vg-index" "$T/dialogue_test.mp4" --force >/dev/null 2>&1
|
||||
"$VG/bin/vg-find" radical | grep -qi radical
|
||||
check $? "vg-index + vg-find round-trip (whisper heard 'radical')"
|
||||
else
|
||||
echo "SKIP: no 'say' binary for speech synthesis"
|
||||
fi
|
||||
|
||||
if [ "${1:-}" = "--roto" ]; then
|
||||
echo "== 4. vg-roto (bundled demo clip, full SAM2+MatAnyone) =="
|
||||
DEMO="$VG/tools/MatAnyone/inputs/video/test-sample3.mp4"
|
||||
ffmpeg -hide_banner -loglevel error -y -t 4 -i "$DEMO" -c:v libx264 -crf 12 -an "$T/roto_in.mp4"
|
||||
"$VG/bin/vg-roto" "$T/roto_in.mp4" --point 970,220 --point 1150,500 --out "$T/roto_out"
|
||||
[ -s "$T/roto_out/roto_in_alpha.mov" ]
|
||||
check $? "vg-roto produced ProRes 4444 alpha"
|
||||
fi
|
||||
|
||||
echo
|
||||
if [ $fails -eq 0 ]; then echo "SMOKE TEST: all passed"; else echo "SMOKE TEST: $fails FAILED"; fi
|
||||
exit $fails
|
||||
Loading…
Reference in New Issue
Block a user