modelbeast/BENCHMARKS.md

# MODELBEAST Benchmarks (M3 Ultra, 256GB)

First measurements on this machine, recorded 2026-07-12. Fixtures are synthetic
(Blender-rendered Suzanne), so quality numbers are not representative of real
photography — these validate that the pipeline *runs* and how fast.

## Scan track (fully validated, no gated weights)
| Stage | Input | Settings | Result |
|---|---|---|---|
| `colmap_poses` | 48 frames @ 800×600 | sequential matcher, global (GLOMAP) mapper, OPENCV, CPU SIFT | **48/48 images registered, 1479 points, 0.60px mean reprojection error**; global mapper step ~1.0s; full job a few seconds |
| `brush_train` | above colmap_dataset | 1500 steps, max_res 800, sh 2 | ~30–60s wall, 465KB splat.ply, renders in the in-app SplatViewer |

A full-quality `brush_train` run is 30000 steps (the default) — expect minutes,
and a much crisper splat than the 1500-step preview above.

## Image generation — local FLUX lineup (M3 Ultra, mflux/MLX, 1024×1024, validated 2026-07-13)

All five installed and generating. **Warm** = weights cached (the real per-image cost):

| Model | Steps | Warm gen | Peak MLX mem | Gated? | Best for |
|---|---|---|---|---|---|
| **FLUX.2 Klein 4B** | 4 | **9.1s** | 18.0 GB | no (Apache) | volume / sprites — the default |
| FLUX.1 schnell-4bit | 4 | 18.5s | 19.1 GB | no (community quant) | fast draft, FLUX.1 look |
| **FLUX.2 Klein 9B** | 4 | **18.7s** | 28.4 GB | yes | **hero assets — best object accuracy of the whole lineup, cloud included** |
| FLUX.1 schnell | 4 | 20.4s | 25.0 GB | yes | fast draft (Klein 9B beats it at same speed) |
| FLUX.1 dev | 25 | 108.1s | 25.1 GB | yes | cinematic mood / DoF when you can wait ~2 min |

First-run downloads (one-time): schnell/dev ~31GB & ~16–19 min each, Klein 9B ~32GB, Klein 4B ~15GB. Believed among the first published M3 Ultra mflux FLUX.1/FLUX.2 numbers.

**Sweet spot: Klein 9B.** Same ~19s as schnell but far better object coherence → it dominates schnell. Klein 4B when speed matters (2× faster), dev only when you want the cinematic atmosphere. Note the `hf_xet` chunked downloader fails on these repos ("Unable to parse string as hex hash value") — the operator sets `HF_HUB_DISABLE_XET=1` to force the reliable HTTP path.

**First A/B (same prompt, 2026-07-12):** FLUX.2 Klein 4B local (8s, $0) vs nano-banana via OpenRouter (`google/gemini-2.5-flash-image`, 7.7s, $0.0387 exact-billed). Klein: cleaner product-photo subject. nano-banana: richer scene dressing (books/inkwell/quill, dust motes) + finer engraving detail. Verdict: Klein is the volume workhorse; nano-banana wins on scene storytelling per prompt-adherence expectations (Elo 1154 vs ~1083).

## Mesh-gen (local, validated 2026-07-13)
| Operator | Config | Result |
|---|---|---|
| `sf3d` | image → GLB, MPS, tex 1024 | **~5s**, ~9GB peak, 1.5MB GLB. Fast draft tier — good on solid objects, struggles on thin/open geometry. Needs `OMP_NUM_THREADS=1`+`KMP_DUPLICATE_LIB_OK` (segfaults otherwise). |
| `trellis_mac` | TRELLIS.2-4B, pipeline 1024, tex 2048, MPS | **289s (~4.8 min) generation** + 16s bake, 18.4MB GLB with PBR. SOTA-tier local quality — clean coherent geometry even on a thin-ringed astrolabe (dramatically better than SF3D). First run adds a one-time ~15GB download (~30 min); cached after. Needs `HF_HUB_DISABLE_XET=1` + the OMP guards. |
| `hunyuan3d_mlx` | Hunyuan3D 2.1, native MLX (fp16), shape+PBR, tex 2048, remesh 40k | **260s (~4.3 min) total** (shape 149s + texture 112s), peak 20.2GB, 7.6MB GLB (40k faces, 2048² baseColor+MR PBR). Weights **public — no HF login**. Needs `diffusers`+`fast_simplification` in the venv. **MLX-native → the one local 3D op that runs on M1 Ultra** (trellis_mac's torch-MPS bf16 is unverified there). |
| `bg_remove_local` | RMBG-2.0, MPS, 1024 | seconds; clean transparent cutout. Run before SF3D for a big geometry improvement. Note: keeps original RGB under alpha (upscale before cutout). |

### Head-to-head, same mermaid cutout (M3 Ultra, 2026-07-16)
| | speed | GLB | faces | face/detail quality |
|---|---|---|---|---|
| `trellis_mac` | 318s | 23MB | 175,842 | **sharper** — defined eyes/nose/mouth, individually raised tail scales, vivid colors |
| `hunyuan3d_mlx` | 260s | 7.6MB | 40,000 | softer — melted face, smoothed scales, muted texture |

**Verdict:** at defaults `trellis_mac` wins on quality (crisper face + geometry, richer color); `hunyuan3d_mlx` is faster, ~3× lighter, and the only local 3D op that runs on M1. `sf3d` stays the ~5s draft tier. All free/offline; fal cloud for on-demand SOTA without the local wait.

## CorridorKey (neural green-screen keyer, MLX) — Ultra tuning (2026-07-16)
Corridor Digital's keyer (`vendor/corridorkey`, 14.4k★) with the native MLX backend (`corridorkey-mlx`, resolved from git — not on PyPI). Benchmarked on a synthetic 12-frame 2048² green-screen set with **exact ground-truth alpha** (RMBG mermaid cutout + soft-alpha stripes/disk over chroma green); hint = 8× downscaled truth. Scores = alpha MAE / soft-IoU vs truth, steady-state after 2-frame warmup.

| config | M3 s/f | M1 s/f | MAE ↓ | IoU ↑ | peak GB |
|---|---|---|---|---|---|
| full 2048 (stock default) | 3.81 | 4.97 | 0.0091 | 0.925 | **28.2** |
| full 1024 | 0.61 | 0.75 | 0.0111 | 0.908 | 3.7 |
| full 512 | 0.25 | 0.31 | 0.0139 | 0.885 | 2.6 |
| tiled 512 (stock: compile forced off) | 3.64 | 4.65 | **0.0082** | **0.932** | 2.5 |
| **tiled 512 + our compile patch** | **2.48** | 4.19 | **0.0082** | **0.932** | **2.3** |

**Findings:** (1) `compile` gains nothing full-frame at 2048 on Ultras (within noise) — the documented "1.5–2×" is a small-res/laptop figure. (2) **Tiled-512 is the QUALITY winner**, not just the memory fallback — best alpha accuracy, model at native tile scale over full-res input. (3) Upstream hard-codes `compile=False` in tiled mode; tiles are fixed-shape so compilation applies — our 1-line patch (`vendor/corridorkey-mlx`, branch `modelbeast`, editable-installed into the app venv on M3+M1) makes tiled **1.47× faster on M3** (3.64→2.48 s/f), 1.11× on M1, output bit-identical. (4) tile 1024 tested worse (quality + speed) — 512 is the sweet spot. (5) full-frame 2048 peaks **28 GB** → 32GB Macs (M1 Max) should run **tiled** (2.3 GB) — which is also the best-quality config anyway.

**Fleet verdict:** best-quality config = `tile_size=512, overlap=64` + our patch: M3 ~0.40 fps, M1 ~0.24 fps at 2048², IoU 0.932, 2.3 GB — runs on every node including the M4 24GB. Throughput mode: full-1024 (1.65/1.34 fps, IoU 0.908). MLX gaps: blue-screen checkpoint + despill/despeckle not yet on MLX (torch backend covers those). License: CC BY-NC-SA (non-commercial).

**Fleet ablation follow-up (2026-07-16, full report in the fork: `monster/corridorkey-mrp-mlx` → `docs/2026-07-16-m-series-fleet-ablation-results.md`):** ran upstream's own 6-toggle benchmark matrix on M3+M1+M4. New best config on BOTH Ultras = **tiled 768/64 + compile** (M3 1949ms, M1 3275ms per 2048² raster — vs 2788/5373 full-frame) at 2.4GB. Big discovery: **`sdpa` is a 4.5× regression on M1-class GPUs at 2048 (18.6s vs 4.1s) and free on M3; bf16 is neutral on both** — gate sdpa by GPU generation, the "M1 bf16 danger" assumption is wrong for this workload. `stage_gc` harmful on Ultras (0.53–0.66×). M4 24GB swaps at full-frame 2048 (~20–24s all configs) → tiled mandatory there; healthy ≤1024.

### Phase D — hunyuan Studio tuning (2026-07-16, M3 Ultra) → new operator defaults
Raised the config from the laptop-tuned defaults to `octree_resolution 384` + `remesh_faces 120000` + `texture_size 4096`. **4096² bake works on the Studio GPU — no Metal command-buffer watchdog** (the existing `extract_textiles` tiling handles it; `uv_feature_map` never needed patching). Result: **380s** (shape 160 + tex 221), peak 20.2GB, 21.5MB GLB, 78k verts / **120k faces**, 4096² baseColor+MR. Quality jump is real — the melted face gains defined eyes + structure, tail geometry sharpens, textures crisper; closes most of the gap to trellis (trellis still edges the face). Cost: ~46% slower + ~3× file size vs the 40k/2048 default. **These are now the `hunyuan3d_mlx` operator defaults** (all still param-overridable; drop to `remesh_faces 40000`/`texture_size 2048` for fast drafts). **4096 confirmed watchdog-free on the M1 Ultra too** (751s total — M1 runs it at ~2× M3 time, so speed-critical jobs prefer M3, which is first in the pool). Defaults are fleet-safe on both Ultras.

## Mesh-gen — earlier install notes (superseded by the table above)
| Operator | Install | Runtime status |
|---|---|---|
| `sf3d` | venv + Metal texture_baker/uv_unwrapper kernels compiled OK; torch 2.13 MPS available | Runs end-to-end; **weights gated** — `stabilityai/stable-fast-3d` returns `GatedRepoError` until the owner accepts the license + sets an HF token. Then expect seconds-to-a-minute on MPS. |
| `trellis_mac` | setup.sh built .venv (py3.11) + mtl* Metal kernels; torch 2.13 MPS available | Runs end-to-end; **weights gated** — needs HF access to `facebook/dinov3-vitl16-pretrain-lvd1689m` + `briaai/RMBG-2.0`. Expect ~3–5 min/gen once authed (M4 Pro reference; M3 Ultra should match or beat). |
| `fal_*` (trellis / trellis2 / hunyuan3d / rodin) | none (API) | Gated on `FAL_KEY`. Verified param surfaces; ~1s–1min server-side per fal docs. |

## To unblock the gated local operators
1. Accept the model licenses on HuggingFace (one-time, usually instant):
   - https://huggingface.co/stabilityai/stable-fast-3d
   - https://huggingface.co/facebook/dinov3-vitl16-pretrain-lvd1689m
   - https://huggingface.co/briaai/RMBG-2.0
2. Either `huggingface-cli login` on the machine, or paste an HF token into
   Settings → "HuggingFace token" (injected as `HF_TOKEN` for the operators).

## Method
Timings are wall-clock from the job runner (`started_at`→`finished_at`), single
job at a time (gpu lane = 1). Re-run `tests/smoke.sh` for the framework
regression suite (12 checks, ~30s).