modelbeast/docs/MRP_MLX.md

# MRP-MLX — the MonsterRobotParty MLX stack

**The short version:** every serious generative model we run — image, layered
image, image→3D, video keying — runs *locally, on Apple Silicon, in MLX or
Metal, from forks we control, at $0 per generation.* This document is the
canonical reference: what each piece is, where it lives, how fast it goes,
how to run it, and the hard-won rules that keep it fast. Written 2026-07-20
after the rung-3 kernel push; every number here was measured, not estimated.

---

## 1 · Philosophy (why this stack wins)

1. **Local-first.** Hosted GPU (fal) is for interactive one-offs and A/B
   references only. Volume work runs on the fleet. (July fal bill was
   ~$93/mo before this stack; the same workload is now ≈ $0.)
2. **Adopt-then-patch, never from-scratch.** Every port here started as the
   best community MLX effort, vendored to our Gitea, then patched where
   measurement said to. Proven 5×: hunyuan3d (dgrauet), corridorkey
   (cmoyates/Niko), trellis-2 (Jourloy PR#175), qwen-layered (mflux PR#302),
   flux (mflux upstream).
3. **Forks live on our Gitea** (`ssh://git@100.71.119.27:222/monster/…`)
   with upstream kept as `origin` for rebases and a pin tag on the adopted
   commit. Our infra, our history, our rollback.
4. **Parity-first.** No optimization ships without a numeric gate against
   the reference implementation (tolerances recorded), and e2e output
   stats compared at fixed seed. Fixture sets are kept for regression.
5. **Bench-everything, keep the negatives.** BENCHMARKS.md records failures
   (q4 = noise, mx.compile ≈ 0 on big GEMMs) with the same care as wins, so
   no future session re-chases a ghost.
6. **Env-gated changes.** Every behavioral change has an env knob and a
   default that preserves upstream behavior (or our validated best). Prod
   rollback is an environment variable, not a revert.

## 2 · The fleet it runs on

| box | chip | RAM | fp16 TFLOPS | bandwidth | role |
|---|---|---|---|---|---|
| m3ultra | M3 Ultra | 256GB | 23.5 | 400 GB/s | primary: TRELLIS.2, queue, heavy gen |
| m1ultra | M1 Ultra | 128GB | 16.3 | 297 GB/s | second lane: everything (post rung-3) |
| m1max | M1 Max | 32GB | 7.1 | 336 GB/s | bandwidth-friendly light ops |
| m4pro | M4 Pro | 24GB | 5.2 | 225 GB/s | MB light ops + Ollama (hands off) |

Full 7-machine MLX matrix + probe script: `BENCHMARKS.md` (2026-07-19) +
`scripts/mlxprobe.py`. Rules from it: **fp16 on M1-class** (bf16 −20%),
bf16 free on M3/M4/M5; pad attention head_dim→64 on M4/M5-class.

## 3 · The pillars

### 3.1 TRELLIS.2 — image → full-density 3D  ·  `monster/trellis-2-mrp-mlx`

- **Base:** Jourloy/TRELLIS.2 (upstream PR#175 head, pin `pr175-head-754d403`)
  — hybrid MLX (dense stages) + pedronaugusto Metal kernels (sparse ops).
- **Our commits:** `fa972aa` pure-MLX sampler loop (batched dense CFG,
  once-per-stage conversions, one eval/step; parity Δ0.03% e2e) ·
  `7860148` opt-in Metal spconv kernel + vertex-baker scheduler fix ·
  `0db816b` vertex-normal export + COLOR_0 asset class in the validator.
- **Weights:** original `microsoft/TRELLIS.2-4B` safetensors, loaded
  directly (`mx.load` + key renames; **no conversion step exists or should**
  — a converter would double-transpose ss_dec into silent garbage).
- **Performance (anatomy bench, seed 42, 1024-cascade, vertex baker):**

| box | e2e | peak mem |
|---|---|---|
| m3ultra | **72.0 s** | 26.5 GB |
| m1ultra | **181.5 s** | 17.2 GB |
| (old torch-MPS `trellis_mac`) | 124.8 s gen (248.8 wall) | 20.8 GB |
| (fal, same image) | 298.6 s wall | $0.30 |

- **Run:** `cd <fork> && .venv/bin/python scripts/generate_asset.py IMG
  --output-dir OUT --backend mlx-experimental --baker vertex
  --pipeline-type 1024_cascade --seed N`  ·  Farm: operator `trellis2_mlx`.
- **Knobs:** `TRELLIS2_MLX_SAMPLER=0` (revert to torch sampler) ·
  `TRELLIS2_MLX_COMPILE=1` (≈0 on M3U; try on small boxes) ·
  `TRELLIS2_METAL_SPCONV=1` (opt-in kernel; slower-but-leaner, reference).
- **Output class:** vertex-colored full-density GLB (albedo only). MR maps
  need the UV path (`--baker metal` — has the dark-patch sampling bug) or
  the future fast-UV bake. Game budgets: MESHGOD finish farm downstream.
- **Checkouts:** m3 `~/Documents/trellis2-mlx-staging/trellis2-jourloy` ·
  m1 `~/trellis2-mlx` · fresh boxes: `scripts/install_trellis2_mlx.sh`.

### 3.2 Hunyuan3D 2.1 — image → textured 3D  ·  `monster/Hunyuan3D-2.2-mrp-MLX`

- **Base:** dgrauet/Hunyuan3D-2.1-mlx (pure MLX, both stages; 13GB
  pre-converted MLX weights on HF). *Terminology note: this IS the
  v2-generation Hunyuan; fal's `/v2`,`/v21`,`/turbo` are hosted variants.*
- **Our branch `mlx-tune` @ `5a32240`** (deployed both Ultras): T1 UNet
  mx.compile + T2 whole-DiT compile (env `HY3D_MLX_COMPILE=1`; parity
  3e-5/7e-6) · T3 StaticMoELayer (bit-identical, kills ~4800 host syncs,
  makes the DiT compile-legal) · T4 fused-SDPA DINO attention (1.41× its
  stage, parity 1e-7).
- **Performance:** 260 s baseline → **243 s** tuned (m3ultra, defaults).
  Runs on every box M1-and-up (fp16). ~20GB peak.
- **Run:** `generate_e2e.py IMG --output OUT` in `vendor/hunyuan3d-mlx`
  (venv `.venv`) · Farm: operator `hunyuan3d_mlx`.

### 3.3 FLUX / image generation — `mflux` (upstream; not forked)

- mflux 0.18.0 (latest) in `venvs/mflux`, both Ultras + m4pro. Written
  from scratch in MLX by Filip Strand — no port debt to carry.
- **Audit (2026-07-19):** our default path (`flux2-klein-4b`, 4 steps,
  ~9 s/image) has mx.compile on the denoise step + a shapeless-compiled
  scheduler — healthy, keep. **Known gap:** the FLUX.1 family
  (dev/schnell/krea) has zero mx.compile — a backport is the same 10-line
  pattern as Hunyuan T1, expected win 1–4% on Ultras (more on small
  boxes), and belongs UPSTREAM as an mflux PR rather than a site-packages
  patch. Queued, not urgent.
- Operator `flux_local` (klein default) + `mflux_image_edit`.

### 3.4 CorridorKey — neural green-screen  ·  `monster/corridorkey-mrp-mlx`

- **Base:** nikopueringer/corridorkey-mlx (cmoyates' phased parity-first
  port — its `prompts/` folder is our template for any future from-scratch
  port). License CC BY-NC-SA — non-commercial only.
- **Our patch (`4c660df`, branch `modelbeast`):** honor the compile flag in
  tiled mode — 8 lines, **1.47×** on m3ultra (3.64→2.48 s/frame), output
  bit-identical. Plus the 8-machine ablation that found the head_dim→64
  fast-path rule and the "tiled beats full-frame on quality AND memory"
  result (2.3GB vs 28GB). Nightly perfcheck watches the fast path.

### 3.5 Qwen-Image-Layered — image → editable RGBA layers  ·  `monster/Qwen-Image-Layered-MRP-MLX`

- **Base:** mflux PR#302 (ZimengXiong; unmerged upstream), pin
  `pr302-head-a255e4f`. 20B, Apache 2.0 (ship-commercial OK).
- **Validated ladder:** **q8 is the quality FLOOR** — q6 breaks the layer
  semantics (silhouettes), q4 is noise; speed is FLAT across quants
  (compute-bound), so never quantize below q8. 20 steps ≈ 50 steps
  visually (2.9× faster). 1024-res = hero tier.
- **Performance (4 layers, 640, baked q8, 20 steps):** m3ultra **243 s** ·
  m1ultra 366 s · 1024-res hero: 858 s (m3). Peak ~37GB.
- **Baked models:** m3 `~/qwen-layered/qwen-layered-q8` · m1
  `~/qwen-layered-staging/qwen-layered-q8` (34GB; rebake via `mflux-save`).
- **Run:** `mflux-generate-qwen-layered --image IMG --layers 4
  --resolution 640 --steps 20 --model-path <baked>` · Farm: operator
  `qwen_layered_local`. Bonus: the subject layer is matting-grade — the
  premium tier of image preprocessing for hero assets.

## 4 · MODELBEAST integration

| operator | backs | speed (m3 / m1) | notes |
|---|---|---|---|
| `trellis2_mlx` | 3.1 | 72 s / 182 s | vertex baker default; **preferred 3D lane** |
| `trellis_mac` | torch-MPS port | 125 s gen / n/a | legacy; UV-textured (dark-patch bug) |
| `hunyuan3d_mlx` | 3.2 | 243 s / ~5-6 min | textured GLB; runs everywhere |
| `flux_local` | 3.3 | ~9 s/img | klein-4b 4-step default |
| `qwen_layered_local` | 3.5 | 243 s / 366 s | RGBA layer decomposition |

Routing lives in `nodes.json` (machine-local, gitignored — edit on the
m3 primary). MESHGOD's `local/` lane calls these via the MB REST API.

## 5 · The lessons ledger (do not re-learn these)

1. **Profile before optimizing — every time.** The rung-3 3× came from a
   scheduler bug fix + baker swap the profiler exposed; the glamorous
   kernel work measured ≈ 0. The 75GB "sparse conv" memory monster was
   actually the Metal texture bake.
2. **mx.compile ≈ 0 on GEMM-bound 2B+ models on M3 Ultra** (measured 4×:
   Hunyuan T1, T2, TRELLIS step, sampler). It's free insurance, not a win.
   Compile wins live on small models (corridorkey 1.47×) and maybe small
   chips.
3. **Quantization floors are model-specific and cliff-shaped.** Qwen-
   Layered: q8 fine, q6 semantically broken, q4 noise — and quant does
   NOT buy speed on compute-bound models, only footprint.
4. **glTF axis + gamma:** pipeline space is Z-up in [-0.5,0.5]³; glTF is
   Y-up (`(x,y,z)→(x,z,−y)`); COLOR_0 is LINEAR (never pre-gamma).
   A "mangled blob" render is usually just the figure viewed down its own
   axis after a convention miss.
5. **xatlas hangs (hours) on écorché-class meshes** (thousands of
   disconnected shells). Vertex bake sidesteps UV unwrap entirely.
6. **macOS TCC blocks launchd from ~/Documents** — services live in `~/`.
7. **Gated HF weights don't need logins on workers** — rsync the model
   dirs from a cache that has them.
8. **The fal-outage class of bug:** any `while True` polling a paid API
   needs a deadline + a cancel; an in-memory queue wedges silently.
9. **Adapters that convert per-call hide O(steps) waste** — convert at
   stage boundaries, hold state in the fast runtime.
10. **Sentinel fallthroughs bite:** a baker/scheduler that silently falls
    back can burn 5 hours before anyone notices. Fail loudly instead.

## 6 · Runbooks

**New box bring-up (any Apple Silicon, macOS 26+):**
1. Keys: box's SSH key added to Gitea (`monster` account).
2. `git clone` the MODELBEAST repo; run the relevant
   `scripts/install_*.sh` (trellis2_mlx, qwen_layered, hunyuan…) — each is
   idempotent, uses uv, and states its weight needs.
3. Weights: rsync the HF cache model dirs from m3ultra (see §5.7).
4. Bench: `uv run --with mlx python scripts/mlxprobe.py` → add the row to
   BENCHMARKS fleet matrix; set fp16/bf16 per §2 rules.
5. Register the box + operator list in `nodes.json` on the m3 primary.

**Rollback:** every MRP patch is env-gated (§3 knobs) or a pinned-tag fork
— `git checkout <pin-tag>` restores the adopted upstream exactly.

**Where results go:** every experiment appends to `BENCHMARKS.md` (wall
time, peak mem via `mx.get_peak_memory()`, parity numbers, and a verdict
line). Session-scale context lives in `docs/FABLE_STARTUP_PACKET.md`.

## 7 · Roadmap (scoped, in value order)

1. **DiT-internals deep-dive** — TRELLIS.2 sampling runs ≈2.3× above ideal
   FLOPs (54 s of the 72). Attention/MLP kernel work inside the forward;
   the corridorkey `prompts/` phased method is the template.
2. **Fast UV bake** — bucket rasterizer (written, in fast_bake_test.py) +
   an écorché-safe unwrap → texture+MR parity with fal on the vertex
   baker's quality. Unlocks `--baker vertex`-quality WITH UV maps.
3. **simdgroup spconv kernel** — upgrade the opt-in scalar kernel if the
   decoders ever matter for time (they're 10 s today).
4. **Upstreaming** — our sampler + baker + validator to Jourloy PR#175;
   FLUX.1 compile backport as an mflux PR; spconv kernel as reference.
5. **MESHGOD default flip** — route its `local/` 3D lane to `trellis2_mlx`
   once finish-farm verts→texture baking lands for game budgets.