flux_local: HF_HUB_DISABLE_XET fix + full local FLUX lineup benchmarks

- run.py: force HF_HUB_DISABLE_XET=1 (xet chunked downloader fails on BFL repos
  with 'Unable to parse string as hex hash value'; HTTP path is reliable)
- BENCHMARKS.md: warm generation times for all 5 local FLUX models on M3 Ultra
  (Klein 4B 9.1s, schnell-4bit 18.5s, Klein 9B 18.7s, schnell 20.4s, dev 108s).
  Verdict: Klein 9B = best hero-asset quality (beats schnell at same speed),
  Klein 4B = volume workhorse.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
MODELBEAST 2026-07-13 00:37:24 +10:00
parent 6c54091130
commit c42f066723
2 changed files with 20 additions and 6 deletions

View File

@ -13,12 +13,21 @@ photography — these validate that the pipeline *runs* and how fast.
A full-quality `brush_train` run is 30000 steps (the default) — expect minutes,
and a much crisper splat than the 1500-step preview above.
## Image generation (local, validated)
| Operator | Config | Result |
|---|---|---|
| `flux_local` | **FLUX.2 Klein 4B**, 4 steps, 1024×1024, seed 42, MLX | **~8s generation** (2.0 s/step), 17.95GB peak MLX memory, quality clearly production-viable for concept/product shots. One-time weights download ~15GB took ~4m50s (unauthenticated HF; an HF token in Settings speeds this up). Job `adef87d4a5dd`, 2026-07-12 — believed to be among the first published M3 Ultra mflux FLUX.2 numbers |
## Image generation — local FLUX lineup (M3 Ultra, mflux/MLX, 1024×1024, validated 2026-07-13)
Warm-model runs are the ~8s figure; each new model variant pays its download once.
All five installed and generating. **Warm** = weights cached (the real per-image cost):
| Model | Steps | Warm gen | Peak MLX mem | Gated? | Best for |
|---|---|---|---|---|---|
| **FLUX.2 Klein 4B** | 4 | **9.1s** | 18.0 GB | no (Apache) | volume / sprites — the default |
| FLUX.1 schnell-4bit | 4 | 18.5s | 19.1 GB | no (community quant) | fast draft, FLUX.1 look |
| **FLUX.2 Klein 9B** | 4 | **18.7s** | 28.4 GB | yes | **hero assets — best object accuracy of the whole lineup, cloud included** |
| FLUX.1 schnell | 4 | 20.4s | 25.0 GB | yes | fast draft (Klein 9B beats it at same speed) |
| FLUX.1 dev | 25 | 108.1s | 25.1 GB | yes | cinematic mood / DoF when you can wait ~2 min |
First-run downloads (one-time): schnell/dev ~31GB & ~1619 min each, Klein 9B ~32GB, Klein 4B ~15GB. Believed among the first published M3 Ultra mflux FLUX.1/FLUX.2 numbers.
**Sweet spot: Klein 9B.** Same ~19s as schnell but far better object coherence → it dominates schnell. Klein 4B when speed matters (2× faster), dev only when you want the cinematic atmosphere. Note the `hf_xet` chunked downloader fails on these repos ("Unable to parse string as hex hash value") — the operator sets `HF_HUB_DISABLE_XET=1` to force the reliable HTTP path.
**First A/B (same prompt, 2026-07-12):** FLUX.2 Klein 4B local (8s, $0) vs nano-banana via OpenRouter (`google/gemini-2.5-flash-image`, 7.7s, $0.0387 exact-billed). Klein: cleaner product-photo subject. nano-banana: richer scene dressing (books/inkwell/quill, dust motes) + finer engraving detail. Verdict: Klein is the volume workhorse; nano-banana wins on scene storytelling per prompt-adherence expectations (Elo 1154 vs ~1083).

View File

@ -1,5 +1,6 @@
import argparse
import json
import os
import subprocess
import sys
import time
@ -57,8 +58,12 @@ cmd = [str(cli), *model_args,
print("+", " ".join(cmd), flush=True)
print("(first run per model downloads weights from HuggingFace)", flush=True)
env = os.environ.copy()
# hf_xet's chunked downloader intermittently fails ("Unable to parse string as
# hex hash value"); the plain HTTP path is reliable.
env["HF_HUB_DISABLE_XET"] = "1"
t0 = time.time()
res = subprocess.run(cmd, stdout=sys.stdout, stderr=subprocess.STDOUT)
res = subprocess.run(cmd, stdout=sys.stdout, stderr=subprocess.STDOUT, env=env)
if res.returncode != 0:
print("HINT: gated-repo/auth errors mean this model needs an HF license accept "
"+ token (Settings → HuggingFace token). flux2-klein-4b and schnell-4bit "