benchmarks: NEGATIVE result — I2V-synthesized orbits don't reconstruct in 3D

Tested single image -> I2V orbit -> colmap -> brush. COLMAP registered 26/49
frames but triangulated ZERO 3D points (vs 1006 for real iPad footage);
gauge fix failed, 'reconstruction' in 2.4s. Brush still emitted a 190MB .ply
from geometry-free input = garbage.
Cause: the subject isn't rigid (tail dissolves, orb appears, figure morphs).
Video models optimise temporal plausibility, not multi-view consistency.
You cannot synthesize your way out of capture. Documented so it isn't retried.
Also: TI2V-5B 704x704/49f/40steps = 337s.
This commit is contained in:
m3ultra 2026-07-17 02:07:19 +10:00
parent acec2cc6c5
commit a6a6d110b2

View File

@ -61,7 +61,21 @@ Quality: genuinely good — photoreal fox walking through snow, coherent across
Other mlx-video packaging bugs found (all upstream-PR-worthy): (1) `torch` is required to read the `.pth` T5/VAE but is **undeclared** — conversion dies *halfway*, leaving a plausible-looking broken model dir; (2) `librosa` is a hard dep though only LTX-2's *audio* VAE uses it — drags in ancient numba and **breaks install** on modern Python (should be an optional `[audio]` extra); (3) README documents `python -m mlx_video.wan2.convert` — that module doesn't exist (it's `mlx_video.models.wan_2.convert`). Other mlx-video packaging bugs found (all upstream-PR-worthy): (1) `torch` is required to read the `.pth` T5/VAE but is **undeclared** — conversion dies *halfway*, leaving a plausible-looking broken model dir; (2) `librosa` is a hard dep though only LTX-2's *audio* VAE uses it — drags in ancient numba and **breaks install** on modern Python (should be an optional `[audio]` extra); (3) README documents `python -m mlx_video.wan2.convert` — that module doesn't exist (it's `mlx_video.models.wan_2.convert`).
Also converted and ready: **Wan2.2-TI2V-5B** (23GB MLX, text+image→video, z_dim=48 VAE) for the I2V-orbit→3D experiment. Note `Wan2.2-I2V-A14B` is **126GB** — needs ~250GB with conversion; not attempted. **Wan2.2-TI2V-5B** (23GB MLX, text+image→video): 704², 49 frames, 40 steps = **337s (5.6 min)***faster than the 1.3B* at 832×480 (fewer steps + better-conditioned I2V). Note `Wan2.2-I2V-A14B` is **126GB** (~250GB with conversion) — not attempted.
### ❌ NEGATIVE RESULT: I2V-synthesized orbits do NOT reconstruct in 3D (2026-07-17)
Tested the appealing shortcut — *single image → I2V "camera orbit" → frames → colmap_poses → brush_train* — i.e. synthesize the turntable instead of filming it. Prompted TI2V-5B explicitly for camera orbit around a static subject (parallax being what COLMAP needs), fed the RMBG mermaid cutout on white.
| source | registered images | **3D points triangulated** |
|---|---|---|
| real iPad room video | 34 | **1006** ✅ |
| **synthesized I2V orbit** | 26 | **0** ❌ |
**Zero triangulated points.** COLMAP matched features and guessed poses but could not place a single consistent point in space (hence `Failed to fix Gauge … insufficient number of fixed points: 0`, and a "reconstruction" in 2.4s vs ~60s for real footage). Brush still emitted a 190MB .ply — trained faithfully on geometry-free input, i.e. garbage. **Bigger splat ≠ better.**
**Why:** the video visibly rotates (back of the head by frame 47) but the subject is **not rigid** — the tail dissolves, an orb materialises, the figure morphs into a different object. Video models optimise **temporal plausibility**, not **multi-view consistency**; 3D needs the latter. Each frame is individually gorgeous and mutually incompatible.
**Conclusion: you cannot synthesize your way out of capture.** For video→3D, film the real turntable (→ `bg_remove_local`/CorridorKey → colmap → brush/hunyuan3d_mlx). Use I2V for motion/B-roll, not as a multi-view source. Don't re-run this experiment — it's a property of the objective, not a tuning problem. (A purpose-built multi-view/NVS model — e.g. TRELLIS-class or a camera-controlled LoRA — is the thing that could work; a general I2V model can't.)
## CorridorKey (neural green-screen keyer, MLX) — Ultra tuning (2026-07-16) ## CorridorKey (neural green-screen keyer, MLX) — Ultra tuning (2026-07-16)
Corridor Digital's keyer (`vendor/corridorkey`, 14.4k★) with the native MLX backend (`corridorkey-mlx`, resolved from git — not on PyPI). Benchmarked on a synthetic 12-frame 2048² green-screen set with **exact ground-truth alpha** (RMBG mermaid cutout + soft-alpha stripes/disk over chroma green); hint = 8× downscaled truth. Scores = alpha MAE / soft-IoU vs truth, steady-state after 2-frame warmup. Corridor Digital's keyer (`vendor/corridorkey`, 14.4k★) with the native MLX backend (`corridorkey-mlx`, resolved from git — not on PyPI). Benchmarked on a synthetic 12-frame 2048² green-screen set with **exact ground-truth alpha** (RMBG mermaid cutout + soft-alpha stripes/disk over chroma green); hint = 8× downscaled truth. Scores = alpha MAE / soft-IoU vs truth, steady-state after 2-frame warmup.