modelbeast/BENCHMARKS.md

MODELBEAST Benchmarks (M3 Ultra, 256GB)

First measurements on this machine, recorded 2026-07-12. Fixtures are synthetic (Blender-rendered Suzanne), so quality numbers are not representative of real photography — these validate that the pipeline runs and how fast.

Scan track (fully validated, no gated weights)

Stage	Input	Settings	Result
colmap_poses	48 frames @ 800×600	sequential matcher, global (GLOMAP) mapper, OPENCV, CPU SIFT	48/48 images registered, 1479 points, 0.60px mean reprojection error; global mapper step ~1.0s; full job a few seconds
brush_train	above colmap_dataset	1500 steps, max_res 800, sh 2	~30–60s wall, 465KB splat.ply, renders in the in-app SplatViewer

A full-quality brush_train run is 30000 steps (the default) — expect minutes, and a much crisper splat than the 1500-step preview above.

Image generation (local, validated)

Operator	Config	Result
flux_local	FLUX.2 Klein 4B, 4 steps, 1024×1024, seed 42, MLX	~8s generation (2.0 s/step), 17.95GB peak MLX memory, quality clearly production-viable for concept/product shots. One-time weights download ~15GB took ~4m50s (unauthenticated HF; an HF token in Settings speeds this up). Job adef87d4a5dd, 2026-07-12 — believed to be among the first published M3 Ultra mflux FLUX.2 numbers

Warm-model runs are the ~8s figure; each new model variant pays its download once.

First A/B (same prompt, 2026-07-12): FLUX.2 Klein 4B local (8s, $0) vs nano-banana via OpenRouter (google/gemini-2.5-flash-image, 7.7s, $0.0387 exact-billed). Klein: cleaner product-photo subject. nano-banana: richer scene dressing (books/inkwell/quill, dust motes) + finer engraving detail. Verdict: Klein is the volume workhorse; nano-banana wins on scene storytelling per prompt-adherence expectations (Elo 1154 vs ~1083).

Mesh-gen (installed; first real run blocked on owner HuggingFace auth)

Operator	Install	Runtime status
sf3d	venv + Metal texture_baker/uv_unwrapper kernels compiled OK; torch 2.13 MPS available	Runs end-to-end; weights gated — stabilityai/stable-fast-3d returns GatedRepoError until the owner accepts the license + sets an HF token. Then expect seconds-to-a-minute on MPS.
trellis_mac	setup.sh built .venv (py3.11) + mtl* Metal kernels; torch 2.13 MPS available	Runs end-to-end; weights gated — needs HF access to facebook/dinov3-vitl16-pretrain-lvd1689m + briaai/RMBG-2.0. Expect ~3–5 min/gen once authed (M4 Pro reference; M3 Ultra should match or beat).
fal_* (trellis / trellis2 / hunyuan3d / rodin)	none (API)	Gated on FAL_KEY. Verified param surfaces; ~1s–1min server-side per fal docs.

To unblock the gated local operators

1. Accept the model licenses on HuggingFace (one-time, usually instant):
   • https://huggingface.co/stabilityai/stable-fast-3d
   • https://huggingface.co/facebook/dinov3-vitl16-pretrain-lvd1689m
   • https://huggingface.co/briaai/RMBG-2.0
2. Either huggingface-cli login on the machine, or paste an HF token into Settings → "HuggingFace token" (injected as HF_TOKEN for the operators).

Method

Timings are wall-clock from the job runner (started_at→finished_at), single job at a time (gpu lane = 1). Re-run tests/smoke.sh for the framework regression suite (12 checks, ~30s).