44 lines
3.3 KiB
Markdown
44 lines
3.3 KiB
Markdown
# MODELBEAST Benchmarks (M3 Ultra, 256GB)
|
||
|
||
First measurements on this machine, recorded 2026-07-12. Fixtures are synthetic
|
||
(Blender-rendered Suzanne), so quality numbers are not representative of real
|
||
photography — these validate that the pipeline *runs* and how fast.
|
||
|
||
## Scan track (fully validated, no gated weights)
|
||
| Stage | Input | Settings | Result |
|
||
|---|---|---|---|
|
||
| `colmap_poses` | 48 frames @ 800×600 | sequential matcher, global (GLOMAP) mapper, OPENCV, CPU SIFT | **48/48 images registered, 1479 points, 0.60px mean reprojection error**; global mapper step ~1.0s; full job a few seconds |
|
||
| `brush_train` | above colmap_dataset | 1500 steps, max_res 800, sh 2 | ~30–60s wall, 465KB splat.ply, renders in the in-app SplatViewer |
|
||
|
||
A full-quality `brush_train` run is 30000 steps (the default) — expect minutes,
|
||
and a much crisper splat than the 1500-step preview above.
|
||
|
||
## Image generation (local, validated)
|
||
| Operator | Config | Result |
|
||
|---|---|---|
|
||
| `flux_local` | **FLUX.2 Klein 4B**, 4 steps, 1024×1024, seed 42, MLX | **~8s generation** (2.0 s/step), 17.95GB peak MLX memory, quality clearly production-viable for concept/product shots. One-time weights download ~15GB took ~4m50s (unauthenticated HF; an HF token in Settings speeds this up). Job `adef87d4a5dd`, 2026-07-12 — believed to be among the first published M3 Ultra mflux FLUX.2 numbers |
|
||
|
||
Warm-model runs are the ~8s figure; each new model variant pays its download once.
|
||
|
||
**First A/B (same prompt, 2026-07-12):** FLUX.2 Klein 4B local (8s, $0) vs nano-banana via OpenRouter (`google/gemini-2.5-flash-image`, 7.7s, $0.0387 exact-billed). Klein: cleaner product-photo subject. nano-banana: richer scene dressing (books/inkwell/quill, dust motes) + finer engraving detail. Verdict: Klein is the volume workhorse; nano-banana wins on scene storytelling per prompt-adherence expectations (Elo 1154 vs ~1083).
|
||
|
||
## Mesh-gen (installed; first real run blocked on owner HuggingFace auth)
|
||
| Operator | Install | Runtime status |
|
||
|---|---|---|
|
||
| `sf3d` | venv + Metal texture_baker/uv_unwrapper kernels compiled OK; torch 2.13 MPS available | Runs end-to-end; **weights gated** — `stabilityai/stable-fast-3d` returns `GatedRepoError` until the owner accepts the license + sets an HF token. Then expect seconds-to-a-minute on MPS. |
|
||
| `trellis_mac` | setup.sh built .venv (py3.11) + mtl* Metal kernels; torch 2.13 MPS available | Runs end-to-end; **weights gated** — needs HF access to `facebook/dinov3-vitl16-pretrain-lvd1689m` + `briaai/RMBG-2.0`. Expect ~3–5 min/gen once authed (M4 Pro reference; M3 Ultra should match or beat). |
|
||
| `fal_*` (trellis / trellis2 / hunyuan3d / rodin) | none (API) | Gated on `FAL_KEY`. Verified param surfaces; ~1s–1min server-side per fal docs. |
|
||
|
||
## To unblock the gated local operators
|
||
1. Accept the model licenses on HuggingFace (one-time, usually instant):
|
||
- https://huggingface.co/stabilityai/stable-fast-3d
|
||
- https://huggingface.co/facebook/dinov3-vitl16-pretrain-lvd1689m
|
||
- https://huggingface.co/briaai/RMBG-2.0
|
||
2. Either `huggingface-cli login` on the machine, or paste an HF token into
|
||
Settings → "HuggingFace token" (injected as `HF_TOKEN` for the operators).
|
||
|
||
## Method
|
||
Timings are wall-clock from the job runner (`started_at`→`finished_at`), single
|
||
job at a time (gpu lane = 1). Re-run `tests/smoke.sh` for the framework
|
||
regression suite (12 checks, ~30s).
|