monster/modelbeast
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
175e1eeac4
modelbeast/BENCHMARKS.md

9.1 KiB
Raw Blame History

MODELBEAST Benchmarks (M3 Ultra, 256GB)

First measurements on this machine, recorded 2026-07-12. Fixtures are synthetic (Blender-rendered Suzanne), so quality numbers are not representative of real photography — these validate that the pipeline runs and how fast.

Scan track (fully validated, no gated weights)

Stage Input Settings Result
colmap_poses 48 frames @ 800×600 sequential matcher, global (GLOMAP) mapper, OPENCV, CPU SIFT 48/48 images registered, 1479 points, 0.60px mean reprojection error; global mapper step ~1.0s; full job a few seconds
brush_train above colmap_dataset 1500 steps, max_res 800, sh 2 ~30–60s wall, 465KB splat.ply, renders in the in-app SplatViewer

A full-quality brush_train run is 30000 steps (the default) — expect minutes, and a much crisper splat than the 1500-step preview above.

Image generation — local FLUX lineup (M3 Ultra, mflux/MLX, 1024×1024, validated 2026-07-13)

All five installed and generating. Warm = weights cached (the real per-image cost):

Model Steps Warm gen Peak MLX mem Gated? Best for
FLUX.2 Klein 4B 4 9.1s 18.0 GB no (Apache) volume / sprites — the default
FLUX.1 schnell-4bit 4 18.5s 19.1 GB no (community quant) fast draft, FLUX.1 look
FLUX.2 Klein 9B 4 18.7s 28.4 GB yes hero assets — best object accuracy of the whole lineup, cloud included
FLUX.1 schnell 4 20.4s 25.0 GB yes fast draft (Klein 9B beats it at same speed)
FLUX.1 dev 25 108.1s 25.1 GB yes cinematic mood / DoF when you can wait ~2 min

First-run downloads (one-time): schnell/dev ~31GB & ~16–19 min each, Klein 9B ~32GB, Klein 4B ~15GB. Believed among the first published M3 Ultra mflux FLUX.1/FLUX.2 numbers.

Sweet spot: Klein 9B. Same ~19s as schnell but far better object coherence → it dominates schnell. Klein 4B when speed matters (2× faster), dev only when you want the cinematic atmosphere. Note the hf_xet chunked downloader fails on these repos ("Unable to parse string as hex hash value") — the operator sets HF_HUB_DISABLE_XET=1 to force the reliable HTTP path.

First A/B (same prompt, 2026-07-12): FLUX.2 Klein 4B local (8s, $0) vs nano-banana via OpenRouter (google/gemini-2.5-flash-image, 7.7s, $0.0387 exact-billed). Klein: cleaner product-photo subject. nano-banana: richer scene dressing (books/inkwell/quill, dust motes) + finer engraving detail. Verdict: Klein is the volume workhorse; nano-banana wins on scene storytelling per prompt-adherence expectations (Elo 1154 vs ~1083).

Mesh-gen (local, validated 2026-07-13)

Operator Config Result
sf3d image → GLB, MPS, tex 1024 ~5s, ~9GB peak, 1.5MB GLB. Fast draft tier — good on solid objects, struggles on thin/open geometry. Needs OMP_NUM_THREADS=1+KMP_DUPLICATE_LIB_OK (segfaults otherwise).
trellis_mac TRELLIS.2-4B, pipeline 1024, tex 2048, MPS 289s (~4.8 min) generation + 16s bake, 18.4MB GLB with PBR. SOTA-tier local quality — clean coherent geometry even on a thin-ringed astrolabe (dramatically better than SF3D). First run adds a one-time ~15GB download (~30 min); cached after. Needs HF_HUB_DISABLE_XET=1 + the OMP guards.
hunyuan3d_mlx Hunyuan3D 2.1, native MLX (fp16), shape+PBR, tex 2048, remesh 40k 260s (~4.3 min) total (shape 149s + texture 112s), peak 20.2GB, 7.6MB GLB (40k faces, 2048² baseColor+MR PBR). Weights public — no HF login. Needs diffusers+fast_simplification in the venv. MLX-native → the one local 3D op that runs on M1 Ultra (trellis_mac's torch-MPS bf16 is unverified there).
bg_remove_local RMBG-2.0, MPS, 1024 seconds; clean transparent cutout. Run before SF3D for a big geometry improvement. Note: keeps original RGB under alpha (upscale before cutout).

Head-to-head, same mermaid cutout (M3 Ultra, 2026-07-16)

speed GLB faces face/detail quality
trellis_mac 318s 23MB 175,842 sharper — defined eyes/nose/mouth, individually raised tail scales, vivid colors
hunyuan3d_mlx 260s 7.6MB 40,000 softer — melted face, smoothed scales, muted texture

Verdict: at defaults trellis_mac wins on quality (crisper face + geometry, richer color); hunyuan3d_mlx is faster, ~3× lighter, and the only local 3D op that runs on M1. sf3d stays the ~5s draft tier. All free/offline; fal cloud for on-demand SOTA without the local wait.

CorridorKey (neural green-screen keyer, MLX) — Ultra tuning (2026-07-16)

Corridor Digital's keyer (vendor/corridorkey, 14.4k★) with the native MLX backend (corridorkey-mlx, resolved from git — not on PyPI). Benchmarked on a synthetic 12-frame 2048² green-screen set with exact ground-truth alpha (RMBG mermaid cutout + soft-alpha stripes/disk over chroma green); hint = 8× downscaled truth. Scores = alpha MAE / soft-IoU vs truth, steady-state after 2-frame warmup.

config M3 s/f M1 s/f MAE ↓ IoU ↑ peak GB
full 2048 (stock default) 3.81 4.97 0.0091 0.925 28.2
full 1024 0.61 0.75 0.0111 0.908 3.7
full 512 0.25 0.31 0.0139 0.885 2.6
tiled 512 (stock: compile forced off) 3.64 4.65 0.0082 0.932 2.5
tiled 512 + our compile patch 2.48 4.19 0.0082 0.932 2.3

Findings: (1) compile gains nothing full-frame at 2048 on Ultras (within noise) — the documented "1.5–2×" is a small-res/laptop figure. (2) Tiled-512 is the QUALITY winner, not just the memory fallback — best alpha accuracy, model at native tile scale over full-res input. (3) Upstream hard-codes compile=False in tiled mode; tiles are fixed-shape so compilation applies — our 1-line patch (vendor/corridorkey-mlx, branch modelbeast, editable-installed into the app venv on M3+M1) makes tiled 1.47× faster on M3 (3.64→2.48 s/f), 1.11× on M1, output bit-identical. (4) tile 1024 tested worse (quality + speed) — 512 is the sweet spot. (5) full-frame 2048 peaks 28 GB → 32GB Macs (M1 Max) should run tiled (2.3 GB) — which is also the best-quality config anyway.

Fleet verdict: best-quality config = tile_size=512, overlap=64 + our patch: M3 ~0.40 fps, M1 ~0.24 fps at 2048², IoU 0.932, 2.3 GB — runs on every node including the M4 24GB. Throughput mode: full-1024 (1.65/1.34 fps, IoU 0.908). MLX gaps: blue-screen checkpoint + despill/despeckle not yet on MLX (torch backend covers those). License: CC BY-NC-SA (non-commercial).

Phase D — hunyuan Studio tuning (2026-07-16, M3 Ultra) → new operator defaults

Raised the config from the laptop-tuned defaults to octree_resolution 384 + remesh_faces 120000 + texture_size 4096. 4096² bake works on the Studio GPU — no Metal command-buffer watchdog (the existing extract_textiles tiling handles it; uv_feature_map never needed patching). Result: 380s (shape 160 + tex 221), peak 20.2GB, 21.5MB GLB, 78k verts / 120k faces, 4096² baseColor+MR. Quality jump is real — the melted face gains defined eyes + structure, tail geometry sharpens, textures crisper; closes most of the gap to trellis (trellis still edges the face). Cost: ~46% slower + ~3× file size vs the 40k/2048 default. These are now the hunyuan3d_mlx operator defaults (all still param-overridable; drop to remesh_faces 40000/texture_size 2048 for fast drafts). 4096 confirmed watchdog-free on the M1 Ultra too (751s total — M1 runs it at ~2× M3 time, so speed-critical jobs prefer M3, which is first in the pool). Defaults are fleet-safe on both Ultras.

Mesh-gen — earlier install notes (superseded by the table above)

Operator Install Runtime status
sf3d venv + Metal texture_baker/uv_unwrapper kernels compiled OK; torch 2.13 MPS available Runs end-to-end; weights gated — stabilityai/stable-fast-3d returns GatedRepoError until the owner accepts the license + sets an HF token. Then expect seconds-to-a-minute on MPS.
trellis_mac setup.sh built .venv (py3.11) + mtl* Metal kernels; torch 2.13 MPS available Runs end-to-end; weights gated — needs HF access to facebook/dinov3-vitl16-pretrain-lvd1689m + briaai/RMBG-2.0. Expect ~3–5 min/gen once authed (M4 Pro reference; M3 Ultra should match or beat).
fal_* (trellis / trellis2 / hunyuan3d / rodin) none (API) Gated on FAL_KEY. Verified param surfaces; ~1s–1min server-side per fal docs.

To unblock the gated local operators

  1. Accept the model licenses on HuggingFace (one-time, usually instant):
  2. Either huggingface-cli login on the machine, or paste an HF token into Settings → "HuggingFace token" (injected as HF_TOKEN for the operators).

Method

Timings are wall-clock from the job runner (started_at→finished_at), single job at a time (gpu lane = 1). Re-run tests/smoke.sh for the framework regression suite (12 checks, ~30s).