diff --git a/BENCHMARKS.md b/BENCHMARKS.md index b83bc41..4f08cd6 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -13,12 +13,21 @@ photography — these validate that the pipeline *runs* and how fast. A full-quality `brush_train` run is 30000 steps (the default) — expect minutes, and a much crisper splat than the 1500-step preview above. -## Image generation (local, validated) -| Operator | Config | Result | -|---|---|---| -| `flux_local` | **FLUX.2 Klein 4B**, 4 steps, 1024×1024, seed 42, MLX | **~8s generation** (2.0 s/step), 17.95GB peak MLX memory, quality clearly production-viable for concept/product shots. One-time weights download ~15GB took ~4m50s (unauthenticated HF; an HF token in Settings speeds this up). Job `adef87d4a5dd`, 2026-07-12 — believed to be among the first published M3 Ultra mflux FLUX.2 numbers | +## Image generation — local FLUX lineup (M3 Ultra, mflux/MLX, 1024×1024, validated 2026-07-13) -Warm-model runs are the ~8s figure; each new model variant pays its download once. +All five installed and generating. **Warm** = weights cached (the real per-image cost): + +| Model | Steps | Warm gen | Peak MLX mem | Gated? | Best for | +|---|---|---|---|---|---| +| **FLUX.2 Klein 4B** | 4 | **9.1s** | 18.0 GB | no (Apache) | volume / sprites — the default | +| FLUX.1 schnell-4bit | 4 | 18.5s | 19.1 GB | no (community quant) | fast draft, FLUX.1 look | +| **FLUX.2 Klein 9B** | 4 | **18.7s** | 28.4 GB | yes | **hero assets — best object accuracy of the whole lineup, cloud included** | +| FLUX.1 schnell | 4 | 20.4s | 25.0 GB | yes | fast draft (Klein 9B beats it at same speed) | +| FLUX.1 dev | 25 | 108.1s | 25.1 GB | yes | cinematic mood / DoF when you can wait ~2 min | + +First-run downloads (one-time): schnell/dev ~31GB & ~16–19 min each, Klein 9B ~32GB, Klein 4B ~15GB. Believed among the first published M3 Ultra mflux FLUX.1/FLUX.2 numbers. + +**Sweet spot: Klein 9B.** Same ~19s as schnell but far better object coherence → it dominates schnell. Klein 4B when speed matters (2× faster), dev only when you want the cinematic atmosphere. Note the `hf_xet` chunked downloader fails on these repos ("Unable to parse string as hex hash value") — the operator sets `HF_HUB_DISABLE_XET=1` to force the reliable HTTP path. **First A/B (same prompt, 2026-07-12):** FLUX.2 Klein 4B local (8s, $0) vs nano-banana via OpenRouter (`google/gemini-2.5-flash-image`, 7.7s, $0.0387 exact-billed). Klein: cleaner product-photo subject. nano-banana: richer scene dressing (books/inkwell/quill, dust motes) + finer engraving detail. Verdict: Klein is the volume workhorse; nano-banana wins on scene storytelling per prompt-adherence expectations (Elo 1154 vs ~1083). diff --git a/server/operators/flux_local/run.py b/server/operators/flux_local/run.py index ced9ebd..b5337ef 100644 --- a/server/operators/flux_local/run.py +++ b/server/operators/flux_local/run.py @@ -1,5 +1,6 @@ import argparse import json +import os import subprocess import sys import time @@ -57,8 +58,12 @@ cmd = [str(cli), *model_args, print("+", " ".join(cmd), flush=True) print("(first run per model downloads weights from HuggingFace)", flush=True) +env = os.environ.copy() +# hf_xet's chunked downloader intermittently fails ("Unable to parse string as +# hex hash value"); the plain HTTP path is reliable. +env["HF_HUB_DISABLE_XET"] = "1" t0 = time.time() -res = subprocess.run(cmd, stdout=sys.stdout, stderr=subprocess.STDOUT) +res = subprocess.run(cmd, stdout=sys.stdout, stderr=subprocess.STDOUT, env=env) if res.returncode != 0: print("HINT: gated-repo/auth errors mean this model needs an HF license accept " "+ token (Settings → HuggingFace token). flux2-klein-4b and schnell-4bit "