51 lines
5.4 KiB
Markdown
51 lines
5.4 KiB
Markdown
# CorridorKey in MODELBEAST — status, tuning findings, and use-case map
|
||
|
||
*Working notes, 2026-07-16. We're just getting started — expect this doc to grow as we try things.*
|
||
|
||
## What it is
|
||
[CorridorKey](https://github.com/nikopueringer/CorridorKey) (Corridor Digital, 14.4k★): a neural green/blue-screen keyer that recovers **true color in semi-transparent pixels** (motion blur, hair, defocus) instead of a binary mask. Native MLX on Apple Silicon. Outputs pro compositing passes: linear EXR alpha (`/Matte`), straight FG color (`/FG`), premultiplied RGBA (`/Processed`).
|
||
|
||
## Current status in our fleet
|
||
- Installed + benchmarked on **M3 Ultra and M1 Ultra** (`vendor/corridorkey`, venv via `uv sync --extra mlx`; 399MB checkpoint in `CorridorKeyModule/checkpoints/`).
|
||
- **Our engine patch** (honor `compile` in tiled mode): `vendor/corridorkey-mlx` branch `modelbeast`, editable-installed into both nodes' venvs. **1.47× faster on M3, bit-identical output.**
|
||
- **Self-owned fork:** `monster/corridorkey-mrp-mlx` on the Gitea (full upstream history + patch).
|
||
- **Best config (both Ultras + any ≥8GB node):** `tile_size=512, overlap=64` → top quality (IoU 0.932 vs ground truth) at 2.3GB peak. Throughput mode: full-frame 1024. Full numbers: [BENCHMARKS.md](BENCHMARKS.md).
|
||
- **Deferred (deliberately, while we learn more):** upstream PR of the patch (check overlap with their open PRs #8/#11 first — maintainer is active and receptive); `corridorkey` queue operator; GVM trial.
|
||
- **MLX gaps today:** blue-screen checkpoint, despill/despeckle → torch/MPS backend covers those if needed. License: **CC BY-NC-SA (non-commercial)**.
|
||
|
||
## The core question: keyed video → 3D models?
|
||
|
||
**Yes — this works today with existing operators.** The chain:
|
||
|
||
```
|
||
green-screen video ──ffmpeg_frames──▶ frames ──CorridorKey──▶ per-frame RGBA cutouts
|
||
(true edge color, motion-blur-aware) │
|
||
├─▶ best frame ──hunyuan3d_mlx / trellis_mac──▶ textured GLB
|
||
├─▶ all frames ──colmap_poses──▶ brush_train──▶ clean object splat
|
||
└─▶ frame set ──▶ sprite sheets / datasets / comps
|
||
```
|
||
|
||
Our #1 quality rule has always been "bg-remove before 3D-gen." CorridorKey is that rule **upgraded for video**: RMBG guesses at edges of a single image; CorridorKey *recovers* them from chroma, per frame, with motion blur handled.
|
||
|
||
**The honest caveat:** it needs an actual green/blue screen behind the subject. RMBG keys anything; CorridorKey keys *filmed-for-keying* footage — but does it at a fidelity RMBG can't touch. (The no-green-screen version of this is GVM — see use case 7.)
|
||
|
||
## Use-case map (ranked by fit to our pipeline)
|
||
|
||
1. **Turntable capture rig → hero 3D** ⭐ *the killer combo.* Object on a turntable in front of a green screen → every rotation frame becomes a perfect cutout → best/sharpest frame (ffmpeg blur-cull) → `hunyuan3d_mlx`/`trellis_mac`. Beats single-photo capture: you pick the ideal angle *after* shooting, from hundreds of candidates.
|
||
2. **Clean gaussian splats / photogrammetry.** Background features poison SfM and splat training. Pre-keyed frames → `colmap_poses` → `brush_train` = object-only splats with no background floaters. (Apple Object Capture also accepts masks.)
|
||
3. **Multi-view 3D.** A rotating subject gives calibrated-ish multi-view cutouts → fal Hunyuan multi-view today; local multi-view later.
|
||
4. **Person/character reference capture.** Film someone on green → pristine per-frame mattes → reference sets for character modeling, texture projection, or the video→rigged-character track in [PLAN.md](PLAN.md).
|
||
5. **Sprite/billboard factories.** Keyed animation frames → RGBA sprite sheets for games — pairs with the FLUX "volume/sprites" lane.
|
||
6. **The reverse pipeline: composite INTO our 3D.** CorridorKey's native purpose — key real footage, then place the person *inside* renders of MODELBEAST-generated scenes (Blender comps). Assets out, humans in.
|
||
7. **GVM: keying WITHOUT a green screen** 🏔️ *the 256GB flex.* The optional ~80GB Generative Video Matting module mattes arbitrary footage automatically — no chroma needed. Almost nobody can run it (Niko built it on a 96GB RTX Pro 6000); the M3 Ultra can. If it works on MPS, "any video → clean subject → 3D" with no capture rig. Untested; genuine expedition.
|
||
8. **Dataset/training-data factory.** Volume RGBA cutouts from video → LoRA/fine-tune datasets, asset libraries — one turntable session = hundreds of clean training images.
|
||
9. **Key-then-upscale chain.** Key first, `seedvr2_upscale` the FG only (no background pollution in the upscale), then 3D-gen — the existing "upscale if thin" practice, sharpened.
|
||
|
||
## Next experiments (in rough order)
|
||
1. Shoot/simulate a **real turntable clip** → run chain end-to-end → compare resulting mesh vs single-photo baseline. (Can simulate with a Blender-rendered turntable over green if no footage handy.)
|
||
2. `corridorkey` **operator** (video/frames in → RGBA sequence out, tiled-512 default) so the queue can fan keying across all three nodes.
|
||
3. **GVM on the M3** — install the ~80GB module, see if MPS survives it.
|
||
4. **Upstream PR** once we've lived with the patch a bit.
|
||
|
||
*Fleet routing: [CLUSTER.md](CLUSTER.md) · RAM guide: [HARDWARE.md](HARDWARE.md) · numbers: [BENCHMARKS.md](BENCHMARKS.md)*
|