From a6a6d110b2c363131debc9ea4ac503aad4c3bfba Mon Sep 17 00:00:00 2001 From: m3ultra Date: Fri, 17 Jul 2026 02:07:19 +1000 Subject: [PATCH] =?UTF-8?q?benchmarks:=20NEGATIVE=20result=20=E2=80=94=20I?= =?UTF-8?q?2V-synthesized=20orbits=20don't=20reconstruct=20in=203D?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tested single image -> I2V orbit -> colmap -> brush. COLMAP registered 26/49 frames but triangulated ZERO 3D points (vs 1006 for real iPad footage); gauge fix failed, 'reconstruction' in 2.4s. Brush still emitted a 190MB .ply from geometry-free input = garbage. Cause: the subject isn't rigid (tail dissolves, orb appears, figure morphs). Video models optimise temporal plausibility, not multi-view consistency. You cannot synthesize your way out of capture. Documented so it isn't retried. Also: TI2V-5B 704x704/49f/40steps = 337s. --- BENCHMARKS.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/BENCHMARKS.md b/BENCHMARKS.md index d4d8d83..1fd7f21 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -61,7 +61,21 @@ Quality: genuinely good — photoreal fox walking through snow, coherent across Other mlx-video packaging bugs found (all upstream-PR-worthy): (1) `torch` is required to read the `.pth` T5/VAE but is **undeclared** — conversion dies *halfway*, leaving a plausible-looking broken model dir; (2) `librosa` is a hard dep though only LTX-2's *audio* VAE uses it — drags in ancient numba and **breaks install** on modern Python (should be an optional `[audio]` extra); (3) README documents `python -m mlx_video.wan2.convert` — that module doesn't exist (it's `mlx_video.models.wan_2.convert`). -Also converted and ready: **Wan2.2-TI2V-5B** (23GB MLX, text+image→video, z_dim=48 VAE) for the I2V-orbit→3D experiment. Note `Wan2.2-I2V-A14B` is **126GB** — needs ~250GB with conversion; not attempted. +**Wan2.2-TI2V-5B** (23GB MLX, text+image→video): 704², 49 frames, 40 steps = **337s (5.6 min)** — *faster than the 1.3B* at 832×480 (fewer steps + better-conditioned I2V). Note `Wan2.2-I2V-A14B` is **126GB** (~250GB with conversion) — not attempted. + +### ❌ NEGATIVE RESULT: I2V-synthesized orbits do NOT reconstruct in 3D (2026-07-17) +Tested the appealing shortcut — *single image → I2V "camera orbit" → frames → colmap_poses → brush_train* — i.e. synthesize the turntable instead of filming it. Prompted TI2V-5B explicitly for camera orbit around a static subject (parallax being what COLMAP needs), fed the RMBG mermaid cutout on white. + +| source | registered images | **3D points triangulated** | +|---|---|---| +| real iPad room video | 34 | **1006** ✅ | +| **synthesized I2V orbit** | 26 | **0** ❌ | + +**Zero triangulated points.** COLMAP matched features and guessed poses but could not place a single consistent point in space (hence `Failed to fix Gauge … insufficient number of fixed points: 0`, and a "reconstruction" in 2.4s vs ~60s for real footage). Brush still emitted a 190MB .ply — trained faithfully on geometry-free input, i.e. garbage. **Bigger splat ≠ better.** + +**Why:** the video visibly rotates (back of the head by frame 47) but the subject is **not rigid** — the tail dissolves, an orb materialises, the figure morphs into a different object. Video models optimise **temporal plausibility**, not **multi-view consistency**; 3D needs the latter. Each frame is individually gorgeous and mutually incompatible. + +**Conclusion: you cannot synthesize your way out of capture.** For video→3D, film the real turntable (→ `bg_remove_local`/CorridorKey → colmap → brush/hunyuan3d_mlx). Use I2V for motion/B-roll, not as a multi-view source. Don't re-run this experiment — it's a property of the objective, not a tuning problem. (A purpose-built multi-view/NVS model — e.g. TRELLIS-class or a camera-controlled LoRA — is the thing that could work; a general I2V model can't.) ## CorridorKey (neural green-screen keyer, MLX) — Ultra tuning (2026-07-16) Corridor Digital's keyer (`vendor/corridorkey`, 14.4k★) with the native MLX backend (`corridorkey-mlx`, resolved from git — not on PyPI). Benchmarked on a synthetic 12-frame 2048² green-screen set with **exact ground-truth alpha** (RMBG mermaid cutout + soft-alpha stripes/disk over chroma green); hint = 8× downscaled truth. Scores = alpha MAE / soft-IoU vs truth, steady-state after 2-frame warmup.