CorridorKey in MODELBEAST — status, tuning findings, and use-case map

Working notes, 2026-07-16. We're just getting started — expect this doc to grow as we try things.

What it is

CorridorKey (Corridor Digital, 14.4k★): a neural green/blue-screen keyer that recovers true color in semi-transparent pixels (motion blur, hair, defocus) instead of a binary mask. Native MLX on Apple Silicon. Outputs pro compositing passes: linear EXR alpha (/Matte), straight FG color (/FG), premultiplied RGBA (/Processed).

Current status in our fleet

Installed + benchmarked on M3 Ultra and M1 Ultra (vendor/corridorkey, venv via uv sync --extra mlx; 399MB checkpoint in CorridorKeyModule/checkpoints/).
Our engine patch (honor compile in tiled mode): vendor/corridorkey-mlx branch modelbeast, editable-installed into both nodes' venvs. 1.47× faster on M3, bit-identical output.
Self-owned fork: monster/corridorkey-mrp-mlx on the Gitea (full upstream history + patch).
Best config (both Ultras + any ≥8GB node): tile_size=512, overlap=64 → top quality (IoU 0.932 vs ground truth) at 2.3GB peak. Throughput mode: full-frame 1024. Full numbers: BENCHMARKS.md.
Deferred (deliberately, while we learn more): upstream PR of the patch (check overlap with their open PRs #8/#11 first — maintainer is active and receptive); corridorkey queue operator; GVM trial.
MLX gaps today: blue-screen checkpoint, despill/despeckle → torch/MPS backend covers those if needed. License: CC BY-NC-SA (non-commercial).

The core question: keyed video → 3D models?

Yes — this works today with existing operators. The chain:

green-screen video ──ffmpeg_frames──▶ frames ──CorridorKey──▶ per-frame RGBA cutouts
       (true edge color, motion-blur-aware)          │
                                                     ├─▶ best frame ──hunyuan3d_mlx / trellis_mac──▶ textured GLB
                                                     ├─▶ all frames ──colmap_poses──▶ brush_train──▶ clean object splat
                                                     └─▶ frame set ──▶ sprite sheets / datasets / comps

Our #1 quality rule has always been "bg-remove before 3D-gen." CorridorKey is that rule upgraded for video: RMBG guesses at edges of a single image; CorridorKey recovers them from chroma, per frame, with motion blur handled.

The honest caveat: it needs an actual green/blue screen behind the subject. RMBG keys anything; CorridorKey keys filmed-for-keying footage — but does it at a fidelity RMBG can't touch. (The no-green-screen version of this is GVM — see use case 7.)

Use-case map (ranked by fit to our pipeline)

Turntable capture rig → hero 3D ⭐ the killer combo. Object on a turntable in front of a green screen → every rotation frame becomes a perfect cutout → best/sharpest frame (ffmpeg blur-cull) → hunyuan3d_mlx/trellis_mac. Beats single-photo capture: you pick the ideal angle after shooting, from hundreds of candidates.
Clean gaussian splats / photogrammetry. Background features poison SfM and splat training. Pre-keyed frames → colmap_poses → brush_train = object-only splats with no background floaters. (Apple Object Capture also accepts masks.)
Multi-view 3D. A rotating subject gives calibrated-ish multi-view cutouts → fal Hunyuan multi-view today; local multi-view later.
Person/character reference capture. Film someone on green → pristine per-frame mattes → reference sets for character modeling, texture projection, or the video→rigged-character track in PLAN.md.
Sprite/billboard factories. Keyed animation frames → RGBA sprite sheets for games — pairs with the FLUX "volume/sprites" lane.
The reverse pipeline: composite INTO our 3D. CorridorKey's native purpose — key real footage, then place the person inside renders of MODELBEAST-generated scenes (Blender comps). Assets out, humans in.
GVM: keying WITHOUT a green screen 🏔️ the 256GB flex. The optional ~80GB Generative Video Matting module mattes arbitrary footage automatically — no chroma needed. Almost nobody can run it (Niko built it on a 96GB RTX Pro 6000); the M3 Ultra can. If it works on MPS, "any video → clean subject → 3D" with no capture rig. Untested; genuine expedition.
Dataset/training-data factory. Volume RGBA cutouts from video → LoRA/fine-tune datasets, asset libraries — one turntable session = hundreds of clean training images.
Key-then-upscale chain. Key first, seedvr2_upscale the FG only (no background pollution in the upscale), then 3D-gen — the existing "upscale if thin" practice, sharpened.

Next experiments (in rough order)

Shoot/simulate a real turntable clip → run chain end-to-end → compare resulting mesh vs single-photo baseline. (Can simulate with a Blender-rendered turntable over green if no footage handy.)
corridorkey operator (video/frames in → RGBA sequence out, tiled-512 default) so the queue can fan keying across all three nodes.
GVM on the M3 — install the ~80GB module, see if MPS survives it.
Upstream PR once we've lived with the patch a bit.

Fleet routing: CLUSTER.md · RAM guide: HARDWARE.md · numbers: BENCHMARKS.md

5.4 KiB Raw Blame History Unescape Escape