monster/modelbeast
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

benchmarks: first M3 Ultra mlx-video/Wan numbers + the auto-tiling SIGKILL gotcha

Browse Source 
Wan2.1-T2V-1.3B: 3s @ 832x480/50 steps = 809.6s (denoise 780.7s, 15.6s/step).
Quality genuinely good (coherent 49-frame photoreal fox).
Default --tiling auto SIGKILLs past ~25 frames (bisect: 25f OK, 33f killed) --
NOT memory (4GB RSS, 195GB free, 179GB max_buffer). auto never enables temporal
tiling; fails AFTER denoising. Fix: --tiling aggressive. Plus 3 packaging bugs.
This commit is contained in:
m3ultra 2026-07-17 01:47:48 +10:00
parent 9c7164f77a
commit acec2cc6c5
1 changed files with 16 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
BENCHMARKS.md
View File

@ -47,6 +47,22 @@ First-run downloads (one-time): schnell/dev ~31GB & ~16–19 min each, Klein 9B
**Verdict:** at defaults `trellis_mac` wins on quality (crisper face + geometry, richer color); `hunyuan3d_mlx` is faster, ~3× lighter, and the only local 3D op that runs on M1. `sf3d` stays the ~5s draft tier. All free/offline; fal cloud for on-demand SOTA without the local wait. **Verdict:** at defaults `trellis_mac` wins on quality (crisper face + geometry, richer color); `hunyuan3d_mlx` is faster, ~3× lighter, and the only local 3D op that runs on M1. `sf3d` stays the ~5s draft tier. All free/offline; fal cloud for on-demand SOTA without the local wait.
## Video gen — mlx-video / Wan 2.1 on M3 Ultra (2026-07-17, first numbers)
[Blaizzy/mlx-video](https://github.com/Blaizzy/mlx-video) (native MLX, MIT) at `vendor/mlx-video`. Wan2.1-T2V-1.3B, PyTorch weights → MLX via their converter (56s), raw `.pth` deleted after (MLX copy is what runs; 14GB).
| run | config | time | note |
|---|---|---|---|
| **T2V 3s** | 832×480, 49 frames, 50 steps, `--tiling aggressive` | **809.6s (13.5 min)** | T5 6.5s · **denoise 780.7s (15.6s/step)** · VAE decode 21.7s |
| smoke | 832×480, 5 frames, 2 steps | 11.8s | pipeline sanity |
Quality: genuinely good — photoreal fox walking through snow, coherent across all 49 frames, correct gait, no flicker/melting. Output registered in the library (`wan_first.mp4`).
**⚠️ THE GOTCHA — default `--tiling auto` gets Wan SIGKILLed past ~25 frames.** Verified bisect (2 steps, 832×480): 13f (seq 6240) OK · 25f (seq 10920) OK · **33f (seq 14040) SIGKILL**. It is **not** memory: it dies at **4GB RSS with 195GB free**, and MLX reports `max_buffer_length: 179GB` / working set 239GB on this machine. The `auto` heuristic never enables **temporal** tiling, so the VAE decode explodes. Worse, it dies *after* denoising completes — with defaults you wait 14 min for the expensive part, then lose it at the decode with no error. **Fix: `--tiling aggressive`** (→ spatial=256px, temporal=32f) — 33f then saves fine. `--tiling temporal` does *not* help.
Other mlx-video packaging bugs found (all upstream-PR-worthy): (1) `torch` is required to read the `.pth` T5/VAE but is **undeclared** — conversion dies *halfway*, leaving a plausible-looking broken model dir; (2) `librosa` is a hard dep though only LTX-2's *audio* VAE uses it — drags in ancient numba and **breaks install** on modern Python (should be an optional `[audio]` extra); (3) README documents `python -m mlx_video.wan2.convert` — that module doesn't exist (it's `mlx_video.models.wan_2.convert`).
Also converted and ready: **Wan2.2-TI2V-5B** (23GB MLX, text+image→video, z_dim=48 VAE) for the I2V-orbit→3D experiment. Note `Wan2.2-I2V-A14B` is **126GB** — needs ~250GB with conversion; not attempted.
## CorridorKey (neural green-screen keyer, MLX) — Ultra tuning (2026-07-16) ## CorridorKey (neural green-screen keyer, MLX) — Ultra tuning (2026-07-16)
Corridor Digital's keyer (`vendor/corridorkey`, 14.4k★) with the native MLX backend (`corridorkey-mlx`, resolved from git — not on PyPI). Benchmarked on a synthetic 12-frame 2048² green-screen set with **exact ground-truth alpha** (RMBG mermaid cutout + soft-alpha stripes/disk over chroma green); hint = 8× downscaled truth. Scores = alpha MAE / soft-IoU vs truth, steady-state after 2-frame warmup. Corridor Digital's keyer (`vendor/corridorkey`, 14.4k★) with the native MLX backend (`corridorkey-mlx`, resolved from git — not on PyPI). Benchmarked on a synthetic 12-frame 2048² green-screen set with **exact ground-truth alpha** (RMBG mermaid cutout + soft-alpha stripes/disk over chroma green); hint = 8× downscaled truth. Scores = alpha MAE / soft-IoU vs truth, steady-state after 2-frame warmup.