diff --git a/CORRIDORKEY.md b/CORRIDORKEY.md new file mode 100644 index 0000000..7744216 --- /dev/null +++ b/CORRIDORKEY.md @@ -0,0 +1,50 @@ +# CorridorKey in MODELBEAST — status, tuning findings, and use-case map + +*Working notes, 2026-07-16. We're just getting started — expect this doc to grow as we try things.* + +## What it is +[CorridorKey](https://github.com/nikopueringer/CorridorKey) (Corridor Digital, 14.4k★): a neural green/blue-screen keyer that recovers **true color in semi-transparent pixels** (motion blur, hair, defocus) instead of a binary mask. Native MLX on Apple Silicon. Outputs pro compositing passes: linear EXR alpha (`/Matte`), straight FG color (`/FG`), premultiplied RGBA (`/Processed`). + +## Current status in our fleet +- Installed + benchmarked on **M3 Ultra and M1 Ultra** (`vendor/corridorkey`, venv via `uv sync --extra mlx`; 399MB checkpoint in `CorridorKeyModule/checkpoints/`). +- **Our engine patch** (honor `compile` in tiled mode): `vendor/corridorkey-mlx` branch `modelbeast`, editable-installed into both nodes' venvs. **1.47× faster on M3, bit-identical output.** +- **Self-owned fork:** `monster/corridorkey-mrp-mlx` on the Gitea (full upstream history + patch). +- **Best config (both Ultras + any ≥8GB node):** `tile_size=512, overlap=64` → top quality (IoU 0.932 vs ground truth) at 2.3GB peak. Throughput mode: full-frame 1024. Full numbers: [BENCHMARKS.md](BENCHMARKS.md). +- **Deferred (deliberately, while we learn more):** upstream PR of the patch (check overlap with their open PRs #8/#11 first — maintainer is active and receptive); `corridorkey` queue operator; GVM trial. +- **MLX gaps today:** blue-screen checkpoint, despill/despeckle → torch/MPS backend covers those if needed. License: **CC BY-NC-SA (non-commercial)**. + +## The core question: keyed video → 3D models? + +**Yes — this works today with existing operators.** The chain: + +``` +green-screen video ──ffmpeg_frames──▶ frames ──CorridorKey──▶ per-frame RGBA cutouts + (true edge color, motion-blur-aware) │ + ├─▶ best frame ──hunyuan3d_mlx / trellis_mac──▶ textured GLB + ├─▶ all frames ──colmap_poses──▶ brush_train──▶ clean object splat + └─▶ frame set ──▶ sprite sheets / datasets / comps +``` + +Our #1 quality rule has always been "bg-remove before 3D-gen." CorridorKey is that rule **upgraded for video**: RMBG guesses at edges of a single image; CorridorKey *recovers* them from chroma, per frame, with motion blur handled. + +**The honest caveat:** it needs an actual green/blue screen behind the subject. RMBG keys anything; CorridorKey keys *filmed-for-keying* footage — but does it at a fidelity RMBG can't touch. (The no-green-screen version of this is GVM — see use case 7.) + +## Use-case map (ranked by fit to our pipeline) + +1. **Turntable capture rig → hero 3D** ⭐ *the killer combo.* Object on a turntable in front of a green screen → every rotation frame becomes a perfect cutout → best/sharpest frame (ffmpeg blur-cull) → `hunyuan3d_mlx`/`trellis_mac`. Beats single-photo capture: you pick the ideal angle *after* shooting, from hundreds of candidates. +2. **Clean gaussian splats / photogrammetry.** Background features poison SfM and splat training. Pre-keyed frames → `colmap_poses` → `brush_train` = object-only splats with no background floaters. (Apple Object Capture also accepts masks.) +3. **Multi-view 3D.** A rotating subject gives calibrated-ish multi-view cutouts → fal Hunyuan multi-view today; local multi-view later. +4. **Person/character reference capture.** Film someone on green → pristine per-frame mattes → reference sets for character modeling, texture projection, or the video→rigged-character track in [PLAN.md](PLAN.md). +5. **Sprite/billboard factories.** Keyed animation frames → RGBA sprite sheets for games — pairs with the FLUX "volume/sprites" lane. +6. **The reverse pipeline: composite INTO our 3D.** CorridorKey's native purpose — key real footage, then place the person *inside* renders of MODELBEAST-generated scenes (Blender comps). Assets out, humans in. +7. **GVM: keying WITHOUT a green screen** 🏔️ *the 256GB flex.* The optional ~80GB Generative Video Matting module mattes arbitrary footage automatically — no chroma needed. Almost nobody can run it (Niko built it on a 96GB RTX Pro 6000); the M3 Ultra can. If it works on MPS, "any video → clean subject → 3D" with no capture rig. Untested; genuine expedition. +8. **Dataset/training-data factory.** Volume RGBA cutouts from video → LoRA/fine-tune datasets, asset libraries — one turntable session = hundreds of clean training images. +9. **Key-then-upscale chain.** Key first, `seedvr2_upscale` the FG only (no background pollution in the upscale), then 3D-gen — the existing "upscale if thin" practice, sharpened. + +## Next experiments (in rough order) +1. Shoot/simulate a **real turntable clip** → run chain end-to-end → compare resulting mesh vs single-photo baseline. (Can simulate with a Blender-rendered turntable over green if no footage handy.) +2. `corridorkey` **operator** (video/frames in → RGBA sequence out, tiled-512 default) so the queue can fan keying across all three nodes. +3. **GVM on the M3** — install the ~80GB module, see if MPS survives it. +4. **Upstream PR** once we've lived with the patch a bit. + +*Fleet routing: [CLUSTER.md](CLUSTER.md) · RAM guide: [HARDWARE.md](HARDWARE.md) · numbers: [BENCHMARKS.md](BENCHMARKS.md)* diff --git a/README.md b/README.md index f4412fe..85d0daa 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,7 @@ Local-first web app that turns videos, images, and 3D files into meshes, splats, > 🖥️ **Multi-Mac fleet? Read [CLUSTER.md](CLUSTER.md) first** — the local-first policy, the M3/M1/M4 node roles, and how the central queue routes jobs across machines. > 💾 **What runs on which Mac?** [HARDWARE.md](HARDWARE.md) — min/recommended RAM per operator across every Apple Silicon tier (8GB → 256GB). +> 🎬 **Green-screen keying → 3D:** [CORRIDORKEY.md](CORRIDORKEY.md) — CorridorKey status, our Ultra tuning, and the video→cutout→mesh/splat use-case map. ## Run