Compare commits

...

39 Commits

Author SHA1 Message Date
m3ultra
03deb589c1 docs(BENCHMARKS): 1M-cap probe — 833k tris, hero tier; 500k stays default
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 16:29:49 +10:00
m3ultra
9004ce64bc fix(trellis ops): drop manifest python pins — remote dispatch to m1 broke on missing venv path
Remote runs exec the repo-relative manifest python via bash with no
existence check ('vendor/trellis2-mlx/.venv/bin/python: No such file'),
killing trellis2_mlx on the validated m1 lane (its fork lives at
~/trellis2-mlx). Both run.py launchers are stdlib-only and already
resolve the per-box fork checkout + venv themselves — run them on the
server/system python instead. trellis_mac/run.py now resolves its vendor
venv explicitly (sys.executable was only correct under the old pin).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 16:27:46 +10:00
m3ultra
529592e059 docs(BENCHMARKS): trellis2_mlx metal-PBR lane verified — 156s, 392k tris, fal parity
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 19:47:25 +10:00
m3ultra
ac817a9a05 feat(trellis2_mlx): metal PBR baker by default — the fast lane now ships meshgod-grade textures
Fork lane (72s vertex) gains the trellis_mac 2026-07-22 quality fixes:
metal UV-PBR bake with explicit RAM-sized face cap (the fork bakes FULL
density when no target passed — 75GB peak; always pass one), forced-
opaque alpha, texture_size/max_bake_faces/alpha_mode params. Verified
m3ultra: 156s e2e, 26.2GB peak, 391,782-tri candidate + 2.19M-tri raw
master. Vendor checkout synced to fork ba4e2ad.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 19:36:38 +10:00
m3ultra
a095f1a140 fix(runner): remap remote-worker output paths after collect
Pre-existing working-tree fix committed for the record: result.json from
a remote worker carries paths on ITS disk; remap by the out/-relative
tail (basename fallback) and log skipped outputs instead of silently
dropping them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 19:04:23 +10:00
m3ultra
b1d37b1629 feat(trellis_mac): close the fal quality gap — RAM-sized bake cap + forced-opaque alpha
Local GLBs were crushed to exactly 94,535 faces (fal ships ~495k) by the
200k laptop mtlbvh guard + o_voxel cleanup, and noisy baked alpha flipped
materials to BLEND (speckle-veil hair, 'broken face'). Neither was the
diffusion — local decode was already 2.2M tris.

- run.py: default TRELLIS2_MAX_BAKE_FACES by node RAM (>=96GB -> 500k,
  else 200k laptop-safe); pass max_bake_faces/alpha_mode/steps/baker
  params through (server forwards only client-sent params — manifest
  defaults are UI-only)
- manifest: new params, quality defaults
- BENCHMARKS.md: 2026-07-22 entry — 500k cap verified crash-free on M3
  Ultra, 351,898-face GLB, bake 7s->14s, farm-path e2e verified

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 19:04:23 +10:00
type-two
61c63e513e feat(mflux_image_edit): pass ALL inputs to --image-paths — multi-image compose
Qwen-Image-Edit takes up to ~3 images ('put the shirt from image 2 on the
person in image 1'); the operator only ever forwarded input[0]. Single-image
jobs unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 18:33:19 +10:00
type-two
3b8c88ce48 fix(sd_local): multi-dir LoRA search + UNet-only load
LORA_DIRS searches ~/Documents/localmodels/Lora (John's master) then the repose
copy (SD_LORA_DIRS to override). Load via filtered state_dict keeping only
lora_unet_* keys: this diffusers build IndexErrors on these kohya files'
text-encoder half (empty rank_dict in load_lora_into_text_encoder); the UNet
half carries the look. Verified: all 3 anatomy LoRAs load + generate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 18:20:12 +10:00
m3ultra
f374465216 docs: MRP_MLX.md — canonical reference for the MLX stack (pillars, perf, knobs, lessons ledger, runbooks, roadmap)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 17:07:56 +10:00
m3ultra
de0b509dfa operator: trellis2_mlx — the 72s/26GB MLX fork as a farm operator (vertex baker default; m1 routed via nodes.json)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 17:04:20 +10:00
m3ultra
eb3862bb8a rung 3 final: M1 Ultra validated — 181.5s / 17.2GB, both Ultras now run optimized MLX TRELLIS.2
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 04:16:29 +10:00
m3ultra
95aa214b1e rung 3 results: 3.0x e2e (216.6->72.0s), 2.8x memory, parity held — sampler+baker+validator on the fork
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 03:55:54 +10:00
m3ultra
d0b02003b8 benchmarks: Qwen-Layered complete ladder — q8 floor (q6/q4 quality cliff), 1024 hero tier, operator shipped; flux_local audit (FLUX.1 compile gap)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 22:20:49 +10:00
m3ultra
58a2a8177d operator: qwen_layered_local — image -> N editable RGBA layers (Qwen-Image-Layered 20B, MLX)
Vendored mflux PR#302 fork (monster/Qwen-Image-Layered-MRP-MLX), tuned
defaults 20 steps/640/baked-q8 (243s m3ultra, 366s m1ultra). Baked-model
resolution via QWEN_LAYERED_MODEL env or per-box paths, HF+q8 fallback.
nodes.json (gitignored, machine-local) updated on m3 to route to m1 too.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 22:06:05 +10:00
m3ultra
8849438bce benchmarks: Qwen-Layered cross-bench — m3ultra 243.0s (1.50x vs m1), both Ultras serving from baked q8
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 21:41:28 +10:00
m3ultra
0db2c90481 benchmarks: Qwen-Layered ladder — 20 steps holds quality, 3.2x to 365.6s on m1ultra
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 21:09:37 +10:00
m3ultra
64afe54474 benchmarks: Qwen-Image-Layered working on MLX (m1ultra) — 4-layer RGBA decomposition verified, PR302 fork vendored
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 20:40:02 +10:00
m3ultra
79bfd713c1 docs+benchmarks: vertex baker productionized in both pipelines — 130s e2e, 1s bake, zero dark patches
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 19:17:01 +10:00
m3ultra
232cc84420 bake fix PROVEN: KDTree-IDW vertex sampling kills the dark patches in 0.3s; axis + gamma gotchas documented; production path defined
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 18:07:21 +10:00
m3ultra
5d2a3f57e4 benchmarks: KDTree control post-mortem — 5.85h bake + broken export; fast-rasterizer rewrite required
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 15:52:26 +10:00
m3ultra
142a0ac5da docs: packet — trellis-2-mrp-mlx fork pushed to Gitea (pin tag pr175-head-754d403)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 15:30:39 +10:00
m3ultra
603efcf7e2 docs+benchmarks: dark-patch verdict — decoder attrs near-fal clean, Metal texel-sampling convicted; regular-grid gather fix designed; parity fixtures complete
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 14:03:53 +10:00
m3ultra
a0e1ef1f18 benchmarks: Hunyuan mlx-tune final — 243s vs 260s (1.07x), M1 rebench suggested
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 13:54:46 +10:00
m3ultra
d488d87490 benchmarks: Hunyuan mlx-tune e2e 246s vs 260s baseline (~1.06x, texture-led)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 12:51:24 +10:00
m3ultra
0476d1bc30 docs: packet §9 — Fable session 1 execution log (oracle pass, A0 ADOPT, mlx-tune branch, dark-patch narrowed)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 12:47:11 +10:00
m3ultra
0ac173d4fb benchmarks: A0 gate — MLX TRELLIS.2 (Jourloy hybrid) BEATS MPS 216.6s vs 248.8s at 14x bake density; dark patches reproduce -> Metal texel sampling prime suspect
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 12:46:16 +10:00
m3ultra
ace4be37ba benchmarks: Hunyuan T1 mx.compile micro gate — parity PASS, 1.04x (GEMM-bound; T3/T4 are the real levers)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 12:40:06 +10:00
m3ultra
eed732446c docs: FABLE_STARTUP_PACKET §8 — adopt path needs NO weight conversion (Jourloy loads originals directly; convert_weights.py retired as double-transpose trap)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 11:55:38 +10:00
m3ultra
3b6b9124f9 docs: FABLE_STARTUP_PACKET §7 — round-2 verification (ADOPT confirmed HIGH, transpose policy verified, parity oracle CPU-runnable, BiRefNet fetched; flags ss_dec double-transpose + bf16-crash + downcast in convert_weights)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 11:51:47 +10:00
m3ultra
abfa344565 docs: FABLE_STARTUP_PACKET — prep recon consolidated (adopt PR#175 MLX base, Hunyuan mx.compile targets, prep_local draft, weight-convert+parity plan)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 11:11:23 +10:00
m3ultra
ca4c802e40 docs: master build plan — TRELLIS.2+Hunyuan full MLX + kernel rewrites + preprocessing frontend (multi-day roadmap for Fable)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 10:46:38 +10:00
type-two
cf402e322c fix(stereo_depth): negative disparity window + percentile normalize — movie stereo works
Movie SBS converges on the screen plane, so real content sits at NEGATIVE
disparity; minDisparity=0 clipped everything behind convergence to black.
Window is now ±num_disp/2 with 2-98 percentile normalization and a validity
mask. Verified on Big Buck Bunny (half-OU) + One Night In Hell (half-SBS):
structured depth + colored clouds that render in Blender.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 01:51:55 +10:00
type-two
95708ad220 feat: sd_local (SD1.5 + anatomy LoRAs + pose/IP-Adapter) + stereo_depth (SBS → SGBM depth/cloud)
sd_local: the repose stack as an operator — Hyper_Realism + repose/models/lora
+ optional OpenPose T-pose template + IP-Adapter ref. Absolute venv path =
primary-only by design. Verified: 512x640 portrait in one shot.

stereo_depth: side-by-side stereo video/frame → per-frame SGBM disparity maps
+ colored midframe .ply for Blender displace/point-cloud work. venvs/stereo
(opencv-headless). Verified on a synthetic 24px-disparity plate: subject
bright, zero-disparity bg black, 1.4MB cloud.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 01:22:12 +10:00
m3ultra
fcb5e80702 benchmarks: trellis_mac sparse-conv ablation - Metal flex_gemm = 1.72x end-to-end vs pure-torch fallback
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 00:50:56 +10:00
m3ultra
793f8a88bb benchmarks: m1max studio added to fleet MLX matrix - 7.1 TF, 336 GB/s second-best bandwidth
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 00:35:15 +10:00
m3ultra
5bd7191b32 scripts: mlxprobe.py — fleet MLX capability probe
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 00:31:59 +10:00
m3ultra
ed2ee8dc16 benchmarks: 6-machine fleet MLX matrix — bf16-on-M1 answered, head_dim-64 padding critical on M4/M5
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 00:31:55 +10:00
m3ultra
daa6124dcf benchmarks: first trellis_mac vs fal trellis-2 head-to-head — local faster, quality gap = hole-fill + pre-simplify
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 00:22:41 +10:00
m3ultra
2734ef9107 docs: TRELLIS.2 Apple Silicon recon — trellis-mac is already TRELLIS.2, MLX ladder, PR #175 watch
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 23:44:30 +10:00
21 changed files with 2113 additions and 17 deletions

View File

@ -147,3 +147,347 @@ regression suite (12 checks, ~30s).
can make a node look 511x slower and it is invisible to load average (the M3 Ultra reads load
2.45 with its GPU pinned). Anything measured here should use min-of-runs and check
`ioreg … IOAccelerator` "Device Utilization %" *before* touching the GPU. See PERFCHECK.md.
## 2026-07-19 — fleet MLX capability matrix (mlx 0.32.0 everywhere)
Probe: 4096² matmul TFLOPS (fp16/bf16), 256MB elementwise GB/s, sdpa head_dim 56 vs 64 (B1·H8·L2048).
| host | chip | RAM | fp16 TF | bf16 TF | ~GB/s | sdpa d56/d64 |
|---|---|---|---|---|---|---|
| m3ultra | M3 Ultra | 256 | 23.48 | 23.54 | 400 | 1.51× |
| ultra | M1 Ultra | 128 | 16.31 | 13.16 | 297 | 0.94× |
| JING5 | M5 | 16 | 8.61 | 8.99 | 43* | **4.72×** |
| m4pro | M4 Pro | 24 | 5.24 | 3.86 | 225 | 1.29× |
| m4mini | M4 | 16 | 3.65 | 3.65 | 98 | 1.55× |
| AIRPRO | M3 | 16 | 1.55 | 1.64 | 87 | 0.95× |
*JING5 bandwidth measured under interactive load — retest idle before trusting.
Verdicts:
- **bf16-on-M1 question answered**: works, ~20% slower than fp16 (16.3→13.2). Ship fp16 on M1-class, bf16 free on M3 Ultra/M4/M5.
- **head_dim pad-to-64 matters MOST on newest chips** (M5 4.7×!, M4-class ~1.3-1.6×, M3 Ultra 1.5×) and ~zero on M1-class (no fused fast path there anyway — matches the corridorkey ablation).
- Fleet total ≈ 59 TFLOPS fp16; the two Ultras are 2/3 of it. TRELLIS.2-MLX lane assignment: m3ultra primary, ultra second (fp16), M4 Pro light ops.
- Probe script: reusable one-liner via `uv run --with mlx` — mlxprobe.py (this bench).
Addendum (same day) — m1max studio joins the matrix (login `studio@100.92.78.24`;
the fleet doc's `m1max@` user and "disk-critical" note are both stale — 170GB free):
| M1MAX | M1 Max | 32 | 7.09 | 6.04 | 336 | 1.22× |
Notable: 2nd-best bandwidth in the fleet (336 GB/s > M4 Pro's 225) despite mid-pack
compute — good fit for memory-bound MLX work (VAE decode, big-tensor elementwise).
Fleet total ≈ 66 TFLOPS fp16 across 7 machines.
## 2026-07-19 — trellis_mac rung 1: sparse-conv backend verified (m3ultra, anatomy image, seed 42, 1024-cascade)
| SPARSE_CONV_BACKEND | total gen+bake | wall | peak RSS |
|---|---|---|---|
| flex_gemm (Metal) | **124.8 s** | 248.8 s | 20.8 GB |
| none (pure-torch) | 214.4 s | 337.6 s | 20.8 GB |
Verdicts:
- The 124.8s baseline WAS the Metal fast path (log line "[SPARSE] Conv backend: flex_gemm") — no hidden 10x jackpot; metallib loads fine on macOS 26.5.
- Metal sparse-conv is worth **1.72x end-to-end** (saves 89.6s/gen). Keep flex_gemm the default; the conv_none fallback is a real 90s regression if the metallib ever breaks silently — worth a GODBOARD/log assertion.
- Remaining fat in the fast path: ~46s diffusion sampling (padded SDPA attention — next kernel frontier), ~50s decode/extract (incl. aten::segment_reduce falling back to CPU — patchable), 19s Metal bake.
- Decode actually produces 2.79M faces before the forced ~200k simplify — the quality-recovery headroom (rung 2) is bigger than the recon estimated.
## 2026-07-19 — Hunyuan paint-UNet mx.compile (Target 1, branch mlx-tune)
Micro gate (real arch, random fp16 weights, 6-view 64² latents, m3ultra):
eager 270.5 ms/fwd → compiled 260.5 ms/fwd = **1.04×**; parity max|diff|
3.0e-05 (PASS); first-trace 1.2 s; peak 4.9 GB. Env-gated
`HY3D_MLX_COMPILE=1`, default off (numerics untouched).
Verdict: keep (free, safe) but the paint UNet is GEMM-bound — fusion alone
is small. The bigger Hunyuan levers are Target 3 (MoE `.item()` sync-tax,
4800 syncs/gen, unlocks whole-DiT compile) and Target 4 (manual→fused
SDPA in paint DINO). E2E flag-on-vs-off measurement pending after those.
## 2026-07-19 — A0 GATE: first end-to-end MLX TRELLIS.2 (Jourloy PR#175 fork, m3ultra)
Same anatomy image, seed 42, 1024-cascade, tex 2048:
| path | wall (incl. load) | bake | peak mem | output |
|---|---|---|---|---|
| torch-MPS (trellis-mac) | 248.8 s | 19 s @191K faces | 20.8 GB | 21 MB GLB |
| **mlx-experimental (Jourloy)** | **216.6 s** | 83 s @**2.74M faces** | 75.4 GB | 131 MB PBR GLB |
**VERDICT: ADOPT — the MLX path BEAT the MPS baseline on wall-clock while
baking 14× the faces.** Key discovery: `mlx-experimental` is a HYBRID —
resolver log shows `Conv backend: flex_gemm; Attention: flex_gemm_sparse_attn`,
i.e. sparse ops still run pedronaugusto's Metal kernels; MLX runs the dense
stages. The feared naive-MLX-sparse-conv path never executes. Model load is
lazy/instant on the MLX side (vs 103 s torch load).
Quality: geometry excellent at full density; the DARK-PATCH texture defect
REPRODUCES on this second, independent pipeline — both paths share the
pedronaugusto mtldiffrast/BVH texel sampling. Culprit narrowed to those
shared Metal kernels OR the decoded tex attrs; the pure-Python KDTree bake
control (running) decides. fal renders clean with the same weights, so the
attrs are likely fine → Metal sampling is the prime suspect.
Caveat: 75 GB peak fits both Ultras but check m1 headroom under load.
## 2026-07-19 — Hunyuan mlx-tune e2e (T1+T3+T4, HY3D_MLX_COMPILE=1, m3ultra)
anatomy bench image, defaults (50 steps, octree 256): shape 146s + tex 101s
= **246s total, 17.5GB peak** vs recorded defaults baseline 260s (149+112).
**1.06× e2e** — texture stage carries the win (112→101s = T4 fused DINO
+ T1 UNet compile); shape delta noise-level. Caveat: baseline row used a
different input image; same settings though. Next lever: Target 2
whole-DiT compile (unblocked by T3).
Addendum — with T2 (whole-DiT compile) included: shape 144s + tex 99s =
**243s total**. Full mlx-tune vs 260s baseline ≈ **1.07× e2e**. Verdict:
on M3 Ultra these 2B GEMM-bound models get ~1% from mx.compile fusion;
the tune's real wins were T4 fused attention (texture stage) + T3
compile-safety. Branch mlx-tune @ 5a32240; flag HY3D_MLX_COMPILE=1.
Worth re-benching on M1 Ultra where launch overhead is proportionally larger.
## 10 · DARK-PATCH VERDICT: Metal texel-sampling convicted (2026-07-19)
Decisive experiment: dumped the decoded tex voxels (E2 fixture,
1,398,769 voxels × 6 PBR attrs) and rendered base_color as a direct
front-projection — `trellis2-bench/attrs_front.png`. **The attrs are
near-fal quality** (clean anatomy palette; dark voxels only 5.6%, all
interior). Both local pipelines (trellis-mac MPS and Jourloy MLX) produce
the same mottled dark bake from these clean attrs, and both share the
pedronaugusto mtldiffrast/mtlbvh texel→voxel sampling. ⇒ **the bake's
attr-sampling is the sole quality gap.** The 3h+ KDTree run is now merely
confirmatory (left running).
### Designed fix (next session's opening move)
The attrs live on a REGULAR voxel grid — no BVH/KDTree needed at all:
texel → (xatlas UV → triangle barycentric → world pos) →
`grid[(pos-origin)/voxel_size]` direct index (+8-neighbor occupancy
search). One vectorized gather over all texels; replaces the suspect
ray-cast sampling entirely; implementable in torch-MPS or MLX in ~a page.
Validate against `attrs_front.png` colors + the fal render. If it works,
BOTH local paths get fal-class texture quality at zero bake-speed risk.
### Parity fixture set: COMPLETE
All 8 fixtures captured at `trellis2-bench/parity_fixtures/` (cond_512/
1024, ss_coords, shape_slat, shape_dec, tex_slat, tex_voxels, mesh_final;
seed 42, 1024-cascade). Hook env needed: PYTORCH_ENABLE_MPS_FALLBACK=1
ATTN_BACKEND=sdpa SPARSE_ATTN_BACKEND=sdpa SPARSE_CONV_BACKEND=flex_gemm.
## 2026-07-19 — KDTree bake control: post-mortem (confirmatory run, superseded)
The TRELLIS2_FORCE_KDTREE control finally finished: **bake 21,063s (5.85h)**
and the exported GLB is GARBAGE geometry (mangled 395K-tri blob) — the
fork's KDTree export path has its own vertex-mapping bug on top of being
unusably slow (per-triangle Python rasterizer loop). Verdict unchanged —
the attrs-projection had already convicted the Metal texel-sampling — but
this rules out "just use the Python baker" as shipped: it needs the
fast-rasterizer rewrite + export fix (in progress as fast_bake_test.py).
## 11 · BAKE FIX PROVEN (2026-07-19 evening) — dark patches eliminated
`trellis2-bench/render_vbake5_front.png`: full anatomy figure, correct
colors, ZERO dark patches — from the same decoder attrs the Metal baker
mangles. Method: cKDTree IDW sampling of the voxel grid at mesh vertices
(k=4, distance² weights, 1.5-voxel radius, single-nearest fallback) →
vertex-colored GLB. **Sampling cost: 0.3s** for 270K verts / 1.4M voxels.
Two integration gotchas (cost hours; do not rediscover):
1. **Axis spaces.** Voxel grid + raw pipeline mesh = Z-up-long-axis in
[-0.5,0.5]³; GLB-loaded meshes = glTF Y-up. Transform before sampling:
`(x,y,z)_gltf → (x,z,y)_voxel`. The "mangled blob" renders were ONLY
this (figure viewed down its own axis) — no data corruption anywhere.
2. **glTF vertex colors are LINEAR** — do not pre-gamma (double-gamma =
washed out). Store linear, let the renderer do the transfer.
Also learned: raw `MeshWithVoxel.vertices/faces` need to_glb's weld/init
before use as a plain triangle soup; `output_3d.obj` inherits the raw
convention. And xatlas hangs (2h+) on écorché-topology meshes — the UV
path needs chart budgeting or per-island parallel unwrap.
### Production path (next session)
- Option A (fast to ship): full-density vertex-colored GLB as the raw
output; MESHGOD's Blender finish farm already decimates + can bake
verts→texture (standard Blender bake) — sidesteps xatlas entirely.
- Option B (fal-parity): fast bucket rasterizer (written, in
fast_bake_test.py) + xatlas-alternative UV → 2048² texture bake with
this sampler. Needs the écorché-safe unwrap.
- Wire as `--baker python-fast` in the Jourloy fork + trellis-mac, make it
the default over the convicted Metal texel path; port sampler to MLX
(trivial — gather + weighted sum) for the pure-MLX lane.
### §11 addendum — vertex baker PRODUCTIONIZED (2026-07-19 night)
Wired and validated in BOTH pipelines:
- `vendor/trellis-mac` @ 55fdd7b: `TRELLIS2_BAKER=vertex` env mode in
generate.py (Metal + kdtree paths untouched; sentinel-gated).
- Jourloy fork @ 746e727 (pushed to monster/trellis-2-mrp-mlx): CLI
`--baker vertex`.
E2E validation (anatomy, seed 42, 1024-cascade, uncapped density):
**130.0s total, bake 1s**, 232s wall, 20.8GB peak — full 2.79M-face
vertex-colored GLB, ZERO dark patches, best local quality yet
(`render_vprod_front.png`). vs Metal-bake baseline: same speed envelope,
quality gap closed for albedo. Not yet carried: metallic/roughness maps
(vertex colors are albedo-only) — MR needs the UV path (Option B) or a
COLOR_1 convention. MESHGOD finish-farm verts→texture bake = the
remaining Option-A step for game-budget assets.
## 2026-07-19 — Qwen-Image-Layered on MLX (m1ultra) — WORKING
New capability: image → N editable RGBA layers, fully local. Model:
Qwen/Qwen-Image-Layered (20B, BF16, Apache 2.0, 54GB in m1 HF cache).
Runtime: mflux PR#302 fork (unmerged upstream; vendored at
monster/Qwen-Image-Layered-MRP-MLX, pin tag pr302-head-a255e4f, isolated
venv ~/qwen-layered-staging/mflux-layered/.venv — production mflux venv
untouched). CLI: `mflux-generate-qwen-layered`.
Smoke (anatomy test image, 4 layers, 640, q8-on-the-fly): **1180.8s wall
first-run** (dominated by 20B load + live quantization), 27.1GB peak.
Output verified: 4 true RGBA PNGs — bg layer, atmosphere layer, clean
figure cutout (74.5% transparent, crisp to the fingers), detail layer.
The figure-cutout layer doubles as a high-end matting result — relevant
to the MESHGOD prep frontline for hero assets.
Optimization in progress: `mflux-save --quantize 8` pre-baked model →
skips load+quant cost; timed gen-only rerun to follow. Next: 1024-res,
q4/q6 quality ladder, m3ultra cross-bench.
### Qwen-Image-Layered optimization ladder (m1ultra, 640, q8, 4 layers)
| config | wall | note |
|---|---|---|
| cold: HF bf16 + on-the-fly q8, 50 steps | 1180.8 s | first smoke |
| baked q8 model, 50 steps | 1075.9 s | load cost was NOT dominant |
| **baked q8, 20 steps** | **365.6 s** | **quality held — figure cutout indistinguishable from 50-step. RECOMMENDED DEFAULT.** |
3.2× end-to-end. Peak 36.6GB. Baked model: ~/qwen-layered-staging/qwen-layered-q8 (34GB).
Remaining ladder (untested): q6/q4 quality/speed, 1024-res, fewer-steps floor (10?),
m3ultra cross-bench (~1.4× compute but needs 34GB disk — m3 disk currently tight),
MODELBEAST operator wrapper (qwen_layered_local) once defaults settle.
### Cross-bench addendum: m3ultra joins (same baked q8, 20 steps, 640, 4 layers)
| box | wall | note |
|---|---|---|
| m3ultra | **243.0 s** | 1.50× vs m1 — matches fleet-matrix TFLOPS ratio (1.44×) |
| m1ultra | 365.6 s | |
Both Ultras now serve Qwen-Image-Layered from the same 34GB baked-q8
artifact (m3: ~/qwen-layered/, m1: ~/qwen-layered-staging/) + the same
Gitea fork clone. Peak 36.7GB either box. m3 disk after install: ~82GB free.
### Qwen-Image-Layered — COMPLETE ladder + verdicts (2026-07-19/20)
| config | box | wall | verdict |
|---|---|---|---|
| q8 · 20 steps · 640 | m3ultra | **243.0 s** | **volume default** |
| q8 · 20 steps · 640 | m1ultra | 365.6 s | second lane |
| q8 · 20 steps · **1024** | m3ultra | 857.6 s | **hero tier** — superb detail, true RGBA (68% transp) |
| q6 · 20 steps · 640 | m1ultra | 366.4 s | ❌ REJECTED — decomposition semantics break (silhouette, no cutout) |
| q4 · 20 steps · 640 | m1ultra | 350.1 s | ❌ REJECTED — pure noise |
Quant findings: speed is FLAT across q4/q6/q8 (compute-bound at these
settings) — quantizing below q8 buys only disk/RAM and DESTROYS quality.
**q8 is the floor for this model.** q6/q4 bakes deleted (m1 back to 3.4T).
Operator `qwen_layered_local` shipped: vendored fork at
vendor/mflux-qwen-layered (install_qwen_layered.sh), tuned defaults
(20/640/baked-q8, resolution 1024 opt-in), m1 registered in nodes.json
(machine-local), e2e run.py contract smoke passed on m3.
Also audited flux_local (mflux 0.18.0): DEFAULT PATH HEALTHY — flux2-klein
family has mx.compile on denoise + compiled scheduler (the 9s/image path).
GAP: FLUX.1 family (dev/schnell/krea) has ZERO mx.compile — backport =
same env-gated pattern as Hunyuan T1/T2, modest expected win (1-4% M3U,
more on small boxes), upstreamable to mflux. Queued, not urgent.
## 12 · RUNG-3 KERNEL DAY — RESULTS (2026-07-20)
**Headline: 216.6s → 71.96s e2e (3.0×), peak 75.4GB → 26.5GB (2.8×),
parity held (raw mesh Δ0.03%), full gate EXIT=0.** All on the fork
(monster/trellis-2-mrp-mlx): fa972aa + 7860148 + 0db816b.
What actually moved the needle (in order of impact):
1. **Vertex baker properly wired into generate_asset** (scheduler
fallthrough fixed — `--baker vertex` was silently running the 5h
pure-python kdtree path): "other" bucket 69.4s → 8.3s, and the
75GB peak turned out to be the METAL BAKE's, not the conv's.
2. **Pure-MLX sampler loop** (mlx_samplers.py): torch CPU loop + per-
forward host bounces → one conversion per stage, batched dense CFG,
once-per-stage concat_cond. Sampling 56.8→54.4s (~4% — forwards are
compute-bound) but it is the correctness-proven, compile-ready
architecture, and removes ~72 host round-trips/gen.
3. **Validator taught the vertex-colored asset class** (COLOR_0 without
UVs = legit; normals materialized at export).
Honest negatives (documented so nobody re-chases them):
- mx.compile on the step forward: ~0 on M3 Ultra (GEMM-bound; matches
Hunyuan T1/T2 findings).
- Fused Metal sparse-conv kernel: built, parity 8e-4, but stock chunking
already bounds memory at decoder scale and beats the scalar kernel on
speed. Kept as TRELLIS2_METAL_SPCONV=1 opt-in reference (simdgroup
tiling = the future version if decoders ever matter for time).
- Remaining gap to H100 (~17s at 1024): inside the DiT forwards
(54s sampling ≈ 2.3× ideal FLOPs) — attention/MLP internals, a
future deep-dive, not orchestration.
Env knobs (all default-sane): TRELLIS2_MLX_SAMPLER=0 (revert sampler),
TRELLIS2_MLX_COMPILE=1, TRELLIS2_METAL_SPCONV=1.
M1 Ultra validation: in flight (26.5GB peak fits with 100GB headroom).
### Rung-3 addendum — M1 Ultra validated (2026-07-20)
| box | e2e wall | peak mem |
|---|---|---|
| m3ultra | 71.96 s | 26.5 GB |
| **m1ultra** | **181.5 s** | **17.2 GB** |
The M1 Ultra — which could not run TRELLIS.2 at all before this fork
(torch-MPS path was m3-only in practice) — now generates full-density
vertex-baked assets in ~3 min at 17GB. Install: clone
monster/trellis-2-mrp-mlx to ~/trellis2-mlx + scripts/setup_macos.sh
(PYTHON_BIN=uv 3.11) + gated weights rsynced from m3's HF cache
(no HF login needed on workers). Both Ultras now serve the fastest
local TRELLIS.2 in existence. TRELLIS.2 fleet: m3 primary (72s),
m1 second lane (182s), zero dollars per asset.
## 2026-07-22 — trellis_mac closes the fal quality gap (m3ultra, black-model image, seed 0, pipeline 1024)
Root-caused why local GLBs looked far worse than fal's TRELLIS.2 despite the
same 4B weights. Two independent bugs, neither in the diffusion (local decode
was already 2.23M tris / 101s):
1. **Decimation ceiling.** generate.py's `TRELLIS2_MAX_BAKE_FACES=200000`
laptop mtlbvh guard + o_voxel's cleanup pass (dup/degenerate removal +
small-component pruning) crushed every model to exactly **94,535 faces**
(fal ships ~495k). The 23× pre-bake simplify also shredded thin shells —
hair fragments then got deleted by `remove_small_connected_components`.
Fix: cap raised to **500k → 351,898-face GLB**, bake 7s→14s, no Metal
watchdog crash on the M3 Ultra Studio. Now the operator default
(`max_bake_faces` param; drop to 200k on laptops).
2. **Speckle-veil alpha.** o_voxel's Metal baker samples the decoder's alpha
attr noisily (same family as the 07-19 dark-patch bug); any texel <250
flips the material to BLEND → solid hair renders as a transparent speckle
veil, "broken face". Fix: generate.py now rewrites the exported GLB to
`alphaMode=OPAQUE` by default (`TRELLIS2_ALPHA_MODE=auto` keeps baked
alpha for glassy subjects; `alpha_mode` operator param).
Verified in Blender side-by-side (old 95k / new 352k opaque / fal 495k):
face + hair now read at fal level. Remaining deltas vs fal: slightly fuller
fal hair (try `max_bake_faces=1000000`), and the known Metal dark-patch
texels (workaround: `baker=vertex`). Also wired `steps` param passthrough.
### Addendum — trellis2_mlx gains the metal PBR lane (2026-07-22, m3ultra)
Item 1 of the post-quality-fix roadmap: the fast MLX fork lane now ships
meshgod-grade UV-PBR by default (baker=metal, RAM-sized cap, forced-opaque
alpha — fork commit ba4e2ad, operator ac817a9). Same image/seed as the
trellis_mac runs: **156s e2e, 26.2GB peak, candidate_pbr 391,782 tris +
2048² basecolor+MR, raw master 2.19M tris kept** (vs trellis_mac ~250s
wall for 351,898 tris; fal 495,176 @ $0.30). Farm-path e2e verified with
zero client params (defaults gap closed in run.py — server forwards only
client-sent params). Blender side-by-side vs fal: parity; local keeps
more hair curl detail. baker=vertex remains the 1s clean-albedo option.
1M-cap probe (same image/seed, farm path): candidate 832,883 tris, 43MB
GLB, OPAQUE+PBR — hair marginally fuller than 500k, subtle at viewing
distance. Verdict: 500k stays the default; 1M = hero-asset tier.

View File

@ -0,0 +1,518 @@
# FABLE STARTUP PACKET — TRELLIS.2 + Hunyuan MLX build
> Read this once, top to bottom, before touching anything. It consolidates five prep-only recon
> passes (all inspection, **no GPU job launched, no installs, no production code touched**). Every
> path here is on **m3ultra** (`m3ultra@100.89.131.57`) unless noted. Where a recon was uncertain,
> it says **(UNVERIFIED)** — do not treat those as facts.
>
> Two baselines you will see quoted, do **not** conflate them:
> - **124.8 s** = our current torch-MPS *production* path on the anatomy bench (the number to beat).
> - **1004.6 s / 3.09M tris / 17.82 GB** = PR#175's reported **MPS** E2E on an M4 Max 36 GB
> (different hardware, different mesh size, and NOT the MLX path). Not comparable to 124.8 s.
---
## 1 · STATE ON DISK — what prep already put on the box
Everything below already exists. **Do not redo it.**
### Staging dir: `~/Documents/trellis2-mlx-staging/`
| Path | What it is | Status |
|---|---|---|
| `trellis2-apple/` | Full clone of `pedronaugusto/trellis2-apple`, HEAD `6055b86` (2026-04-22), full 17-commit history deepened, MIT. 39 MB. **Nothing installed.** | read-only clone |
| `prep_local_draft.py` | 9638 B preprocessing draft (§4). `py_compile` OK under rmbg venv. Not wired in. | draft, do-not-run-yet-in-prod |
| `convert_weights.py` | Draft torch→MLX weight converter (§5). Loads ~14 GB — **do not run now.** | draft |
| `parity_dump_hook.py` | Draft monkeypatch fixture dumper (§5). Loads models+GPU — **do not run now.** | draft |
| `inspect_safetensors.py` | Stdlib safetensors header parser (used to produce the §5 layout table). | utility, safe |
| `__pycache__/` | Compile artifacts from the py_compile checks. | ignore |
### Venvs & operators already on the box (do not reinstall)
- **rmbg venv** `~/Documents/MODELBEAST/venvs/rmbg/` (uv, CPython 3.11). Has torch 2.13.0, torchvision 0.28.0, transformers 5.13.1, timm, kornia, einops, safetensors, hf_hub, Pillow, numpy, scipy, scikit-image. **No** cv2 / rembg / birefnet-pkg / upscaler. Installed by `~/Documents/MODELBEAST/scripts/install_rmbg.sh`.
- **`bg_remove_local` operator** `~/Documents/MODELBEAST/server/operators/bg_remove_local/run.py` — runs RMBG-2.0, pins `venvs/rmbg/bin/python`.
### Weights already in HF cache
- `microsoft/TRELLIS.2-4B``~/.cache/huggingface/hub/models--microsoft--TRELLIS.2-4B/snapshots/af44b45.../ckpts/` (7 of 8 pipeline modules).
- `sparse_structure_decoder` (8th module) comes from `microsoft/TRELLIS-image-large` snapshot `25e0d31.../ckpts/ss_dec_conv3d_16l8_fp16.safetensors`.
- `briaai/RMBG-2.0` — 844 MB, present.
- Hunyuan MLX converted weights — `~/.cache/huggingface/hub/models--dgrauet--hunyuan3d-2.1-mlx/`.
- **NOT present:** `ZhengPeng7/BiRefNet` (Apache/MIT, ~900 MB) — must be fetched once for the commercial preprocessing path.
---
## 2 · TRELLIS.2 MLX — ADOPT-vs-BUILD DECISION
### VERDICT: **ADOPT** the Jourloy PR#175 `mlx_backend/` tree as the MLX vendor base. Do **not** build from scratch, and do **not** wholesale-swap the 124.8 s production path — gate any promotion on a measured A0 run.
**Why PR#175 and not the already-cloned `trellis2-apple`:** the two share the *identical* 16-file / ~3,114-LOC `mlx_backend/` (same author — `pedronaugusto` co-authors PR#175). PR#175 (`Jourloy/TRELLIS.2`, opened 2026-07-17, +7,419/258 across 62 files) is a **strict superset**: same MLX modules **plus** the correctness oracle we lack (`tests/test_mlx_parity.py`, 4 unit parities at rtol 1e-63e-5), a backend probe/fallback resolver (`trellis2/backends.py`), a full CLI (`scripts/generate_asset.py`), macOS setup/probe/weight tooling, and the `--backend mlx-experimental` flag so optimized modules drop in behind a flag while MPS keeps E2E green. So PR#175 dominates `trellis2-apple` on every axis. Keep `vendor/trellis-mac` (shivampkumar, no MLX) only as the torch-MPS E2E oracle.
**Why NOT a blind swap (both recons agree):** the stock MLX path will very likely **regress** as-is, for two structural reasons:
- **Un-fused sparse conv.** `mlx_backend/sparse_conv.py` is the naive gather→matmul→sum, 512 MB-chunked — it does **not** use the `flex_gemm` Metal fast path, which our recon measured at **1.72× faster** on this shape. This is the ~10× crux, reimplemented slow in MLX.
- **CPU host bounce.** `mlx_backend/adapters.py` does `torch→numpy→mx→numpy→torch` on **every** model call — per sampler step (default ~12 steps × 3 stages × 2 for CFG) and at every sparse-conv boundary. The upstream torch pipeline still owns sampling orchestration + mesh extraction; MLX models are injected as adapters.
- **Zero benchmarks in either repo.** No evidence the MLX path beats — or matches — 124.8 s. Do **not** benchmark the stock `--backend mlx-experimental` path as representative; it's a correct scaffold, not a fast path.
- **Parity gap in `trellis2-apple`, closed by PR#175.** `trellis2-apple` has NO MLX parity test; PR#175 adds `test_mlx_parity.py`. Adopt PR#175 specifically to get that oracle.
**What is safe to reuse directly (low risk, no sparse-conv tax):** `dinov3.py`, `norm.py` (two-pass LayerNorm32 PT-parity), `rope.py` (interleaved complex-mul parity), `transformer_block.py`, `structure_decoder.py` (dense `mx.conv_general`). **Rewrite/replace first:** `sparse_conv.py` (fused Metal gather-GEMM-scatter), the `adapters.py` boundary (move the Euler/CFG loop *inside* MLX to kill the bounce), `sparse_ops.py` (numpy `np.where` host sync in upsample).
**Known smells to verify before trusting a module (UNVERIFIED in recon):** (1) `structure_decoder.py` `_remap_structure_decoder_weights` is a **no-op** feeding dict-stored conv weights — untested whether `nn.Module.load_weights` populates plain-dict children; (2) hand-rolled softplus/quad_lerp in `vae_decoders.py:1204` FlexiDualGrid decode; (3) full **non-windowed** attention (`attention.py`) may diverge from upstream windowed sparse attn; (4) docstrings claim "custom Metal kernel" but code is plain MLX built-in SDPA — **misleading comments, no hand-written kernel** anywhere in `mlx_backend/`. `mx.compile` *is* already applied to the DiT block loop and structure-decoder graph (with try/except fallback).
**Upstreamability: LOW — plan to fork, not to depend on merge.** PR#175 is unmerged, CLA-gated, 0 reviews, only the CLA bot has commented. Issue #74 (non-CUDA backends) has no maintainer response. Vendor it and **pin the commit**; basing our kernels on #175's layout keeps our work a clean, eventually-upstreamable diff.
### First 3 concrete commands
```bash
# 1. Get the superset fork (has test_mlx_parity.py; the cloned trellis2-apple does not)
cd ~/Documents/trellis2-mlx-staging && \
git clone https://github.com/Jourloy/TRELLIS.2 trellis2-jourloy && cd trellis2-jourloy
# 2. Run the correctness ORACLE first — cheap, no heavy GPU E2E, validates
# LayerNorm32 / apply_rope / SDPA / MlxSparseConv3d vs torch before any gen.
# (Needs an isolated venv with mlx + the deps; test no-ops off-Mac via importorskip.)
pytest tests/test_mlx_parity.py -v
# 3. The single GPU job — the measured A0 gate. Isolated venv, the 3 pedronaugusto
# Metal pkgs --no-build-isolation, reuse the existing TRELLIS.2-4B weights.
# ONE gen, anatomy bench, seed 42, 1024-cascade; compare wall-clock to 124.8 s
# and quality to the fal renders in ~/Documents/trellis2-bench/.
pip install -r requirements_macos.txt --no-build-isolation # torch>=2.11, mlx>=0.31, mtlgemm/mtldiffrast/cumesh
# then: scripts/generate_asset.py --backend mlx-experimental --seed 42 --pipeline-type 1024_cascade <anatomy_img>
```
**Decision rule after A0:** within ~1.5× of 124.8 s AND parity holds → patch-and-tune, adopt as production path behind the flag. If 35× slower (the likely outcome given un-fused conv + host bounce) → keep torch-MPS in `nodes.json` for production, lift `mlx_backend/` as the module-by-module scaffold, and land the kernel rewrites (fused Metal sparse conv → in-MLX sampler loop → parity fixtures) in that order. Also: fork to Gitea `monster/trellis2-mrp-mlx`, keep `origin` upstream for rebases (house pattern).
---
## 3 · HUNYUAN TUNE — edit-target list (ready to apply)
All paths under `~/Documents/MODELBEAST/vendor/hunyuan3d-mlx/`. Static inspection only (GPU was busy). Ship each edit with a PT-parity test (≤1e-4 rel) + a BENCHMARKS.md row.
**Headline: `head_dim` pad-to-64 does NOT apply** — every attention site is already a 64-multiple (64 or 128). The corridorkey padding win is **out of scope**; do not build pad wrappers. `mx.compile` count in the whole MLX port = **0** — that is the entire opportunity surface (corridorkey precedent: fixed-shape compile = 1.47× M3 / 1.11× M1, bit-identical, an 8-line change `vendor/corridorkey-mlx@4c660df`).
| # | Target | file:line | Change | Expected | Risk |
|---|---|---|---|---|---|
| 1 | **Paint UNet forward** (called 3×/step, ~15 steps) | `hy3dpaint/hunyuanpaintpbr_mlx/unet/unet_mlx.py:158` (`__call__`); loop `inference.py:254`, calls `:314,328,339`; steps default `inference.py:72` | Wrap forward in `mx.compile` (add `compile=True` ctor flag, `self._fwd = mx.compile(self._forward_impl)`). No MoE / no `.item()` → compile-safe. | ~1.11.3× on ~112 s paint | low; last chunk may have fewer views → 1 retrace (pad final chunk to fixed `n_chunk`) |
| 2 | **DiT dense blocks 014** (100 forwards = 50 steps × CFG 2) | `hy3dshape/.../denoisers/hunyuandit_mlx.py:379` (`HunYuanDiTBlock.__call__`); driver `pipeline_mlx.py:160`, loop `:143`, steps `:95` | `mx.compile` the dense-block forward (fixed shape `(B∈{1,2},4097,2048)`). Leave 6 MoE blocks (1520) eager until #3. | ~1.11.3× on ~149 s shape (dense = 15/21 depth) | low |
| 3 | **MoE `.item()` sync tax** (PREREQUISITE for full-DiT compile) | `mlx_arsenal/moe/moe.py:110``if not mx.any(w>0).item(): continue` | Remove the `.item()` early-skip; always run all 8 experts, let `w*out` (w=0) zero them. Deletes ~4800 syncs/gen AND makes the block static → unlocks compiling whole `HunYuanDiTPlain.__call__` (`hunyuandit_mlx.py:514`). | 4800 syncs/gen + unblocks #2 to 21/21 blocks | **`mlx_arsenal` is a SHARED dep** — do NOT edit in place. Vendor a local `MoELayer` subclass override, or upstream to arsenal. `MoEGate` routing is already static/compile-clean. |
| 4 | **Paint DINOv2-giant attn** (manual→fused, once/gen ×40 layers) | `hy3dpaint/hunyuanpaintpbr_mlx/dino_mlx.py:5860` (manual `q@kᵀ` softmax, fp32) | Replace with `mx.fast.scaled_dot_product_attention(q,k,v,scale=self.scale)` + existing transpose/reshape. `head_dim=64` hits fast path; fp32 supported. | one-shot; kills N² materialization (fused d64 = 1.51× on M3) | low |
| 5 | **Paint VAE decode** (once/gen) | `hy3dpaint/hunyuanpaintpbr_mlx/vae_mlx.py` decode; manual softmax `:8184` (`head_dim=512`) | `mx.compile` the decode (fuses the softmax). Prefer compile over `mx.fast` swap — `head_dim=512` likely **misses** MLX fused sdpa (kernel ≤256), would fall back anyway. | one-shot, minor | low |
| 6 | **(CONDITIONAL) Shape-VAE geo cross-attn** | `hy3dshape/.../autoencoders/model_mlx.py:206` (`CrossAttentionBlock`); driver `_query_sdf_volume` `:506`, chunk `:510` | `mx.compile` per-chunk cross-attn (fixed 10000-query × 4096-latent). Only final ragged chunk retraces. | smallmoderate | **Verify** decode isn't dominated by the Metal SDF/marching-cubes step before investing |
**Rasterizer: NO opportunity.** `hy3dpaint/DifferentiableRenderer/mesh_render_mlx.py` is a thin adapter over the `mlx_arsenal.rasterize` Metal kernel (already fused); Python side is glue + numpy round-trips. Skip for both levers.
**DTYPE: keep fp16 as the global default.** Port is fp16 everywhere (one deliberate exception: paint DINOv2-giant upcast to **fp32** in `load_model.py` — accumulation-depth fix, not a format preference). bf16 buys **zero** speed on M3/M4/M5 (equal TFLOPS) and is **~20% slower on M1** — and M1 Ultra (`johnking@100.91.239.7`) is the second production lane and hunyuan's headline "runs on M1" selling point. Do NOT globally switch to bf16. (This is the opposite of the TRELLIS.2 port, where bf16 range matters.)
**Execution order:** 1 → 3 → 2(dense, then whole-forward once 3 lands) → 4/5/6 (one-shot cleanups last).
---
## 4 · PREPROCESSING — draft, plug-in point, license, TODOs
**North star** (`meshgod/config.py:65-70`, `STYLE_3D`): a single isolated object, centered, fully visible (no crop); plain flat neutral light-gray bg; even soft shadowless studio light; neutral symmetrical A-pose.
**Draft: `~/Documents/trellis2-mlx-staging/prep_local_draft.py`** (9638 B, `py_compile` OK under rmbg venv, uses only PIL/numpy/torch/torchvision/transformers, not wired in). Signature `prep_local(in_path, out_path, commercial=False)`:
1. `_maybe_upscale` — if `min(w,h) < 1024`, Lanczos up to short-edge 1024, on RGB, **BEFORE** matte.
2. `_matte` — BiRefNet if `commercial` else RMBG-2.0; real inference mirroring `bg_remove_local/run.py` exactly (`Resize((1024,1024))→ToTensor→Normalize(imagenet)`; `model(inp)[-1].sigmoid()`; → `L` mask); model cached per-process.
37. `_compose_center_square` — composite onto `NEUTRAL_GRAY=(200,200,200)` RGB (not alpha) via mask, tight bbox crop from thresholded mask (`MASK_THRESH=8`), center, square-pad to `SUBJECT_FRAC=0.88`, resize 1024 Lanczos.
Run (GPU-free-time session): `venvs/rmbg/bin/python prep_local_draft.py --in X --out Y [--commercial]`.
**Order lesson (verified live 2026-07-13):** upscale FIRST, matte LAST — SeedVR strips alpha (RGBA→RGB), so matting first gets flattened. Draft honors this.
**Plug-in point:** `~/Documents/meshgod/meshgod/server.py:350` (local lane, `server.py:346-350`):
```python
paths[i] = stage3_reconstruct.mb_bg_remove(p, pp) # <- replace with prep_local(p, pp, commercial=...)
```
`prep_local` supersedes `mb_bg_remove` (does the full north-star transform in-process on rmbg venv, not just a farm-round-trip cutout to RMBG-2.0). Secondary optional precede-point: fal lane `server.py:359` (run `prep_local` locally, then skip fal `clean_bg`). Companion anchors to update: `config.py:143` (`BG_REMOVE_MODEL`), `web/index.html:229` label.
**LICENSE RULE (hard):**
| Model | HF id | License | Commercial ship? | Local? |
|---|---|---|---|---|
| RMBG-2.0 | `briaai/RMBG-2.0` | CC-BY-NC 4.0 | **NO** — non-commercial / personal one-offs only | Yes (844 MB) |
| BiRefNet | `ZhengPeng7/BiRefNet` | MIT | **YES** — safe for game/product assets | **No — fetch once** |
Default for anything game-bound = **BiRefNet**. RMBG-2.0 stays default only for local/personal one-offs. (Note: RMBG-2.0 *is itself* a BiRefNet-architecture checkpoint — same code, differs only in weights + license.)
**Open TODOs (in draft):**
1. **BiRefNet weights fetch** — first `--commercial` run must `huggingface-cli download ZhengPeng7/BiRefNet` (~900 MB, MIT, ungated). Until then only RMBG-2.0 works.
2. **Better upscaler** — PIL Lanczos is the zero-dep default (never invents detail). Swap for local faithful SR: MODELBEAST `seedvr2_upscale` operator (weights present: `models--numz--SeedVR2_comfyUI`) or Real-ESRGAN. Must stay BEFORE `_matte`.
3. **Lighting/exposure normalization (D2)** — optional auto-levels + gray-world WB on RGB before matte (fights TRELLIS dark-patch mode). Stubbed so v0 silhouette stays identical to the fal path for A/B.
4. **Tunables to A/B once GPU free:** `NEUTRAL_GRAY=(200,200,200)`, `SUBJECT_FRAC=0.88`, `MASK_THRESH=8` (all chosen from north-star text).
5. **Wire-in / flag plumbing** — add `req.commercial`, replace `server.py:350`, expose in `web/index.html`, decide local-lane upscale policy (`server.py:511` currently forces `upscale=False`).
---
## 5 · WEIGHT CONVERSION + PARITY
Drafts in staging: `convert_weights.py`, `parity_dump_hook.py`, `inspect_safetensors.py`. Both loaders touch ~14 GB / GPU — **run in a later session, not now.**
### Conversion pattern (from Hunyuan's `convert_realesrgan.py:34-58`)
`torch.load(weights_only=True)`**keys pass through unchanged** (module-tree names already match MLX list indexing, no remap dict) → **the only tensor mutation is conv transpose** (PyTorch `(O,I,H,W)`→MLX channels-last `(O,H,W,I)`, detected by `ndim==4`; generalizes to 5D Conv3d as `(0,2,3,4,1)`) → `mx.save_safetensors`.
### TRELLIS.2-4B layout — the transpose decision (resolved from source, not guessed)
There are **two Conv3d conventions on disk; the correct action is opposite for each:**
| pipeline key | file (ckpts/) | dtype | conv action |
|---|---|---|---|
| sparse_structure_flow_model | ss_flow_img_dit_1_3B_64_bf16 (640 t, 2.58 GB) | BF16 | none (pure Linear DiT) |
| **sparse_structure_decoder** (from TRELLIS-image-large) | ss_dec_conv3d_16l8_fp16 (74 t) | F32/F16 | **20 dense Conv3d → TRANSPOSE `(0,2,3,4,1)`** |
| shape_slat_flow_model_512 / _1024 | slat_flow_img2shape_dit_1_3B_{512,1024}_bf16 | BF16 | none |
| shape_slat_decoder | shape_dec_next_dc_f16c32_fp16 (292 t, 948 MB) | F16 | **40 sparse Conv3d → NO transpose (already channels-last)** |
| tex_slat_flow_model_512 / _1024 | slat_flow_imgshape2tex_dit_1_3B_{512,1024}_bf16 | BF16 | none |
| tex_slat_decoder | tex_dec_next_dc_f16c32_fp16 (284 t, 948 MB) | F16 | **40 sparse Conv3d → NO transpose** |
- **`ss_dec_conv3d` = dense `nn.Conv3d`**, on-disk `(Co,Ci,Kd,Kh,Kw)` → needs the 5D transpose. All 20 5-D tensors.
- **`*_next_dc` = o-voxel `SparseConv3d`**, on-disk **already** `(Co,Kd,Kh,Kw,Ci)`**load verbatim, NO transpose.** Proven by source: `conv_flex_gemm.py:33-34` and `conv_none.py:40-41` both `permute(0,2,3,4,1)` at construction, and the checkpoint was saved from that path. MLX forward must reshape `(Co,Kd,Kh,Kw,Ci)→(Kvol,Ci,Co)` for the per-offset gather-GEMM (contract at `conv_none.py:106`).
- **All 5 DiTs: pure `nn.Linear` (ndim 2) + attention + RMSNorm** — zero conv, zero transpose; only a BF16→target-dtype cast (**fp16 on M1** per fleet matrix). Block arch (30 blocks, identical across all 5): hidden 1536, 12 heads × **head_dim 128**, MLP 8192, cross-ctx 1024, adaLN-zero `modulation [9216]`. **head_dim 128 ≥ 64 → fused-SDPA fast path applies, NO padding.** Keep fused `to_qkv [4608,1536]` / `to_kv [3072,1024]` packed and slice in-kernel.
`convert_weights.py` encodes this per-file policy (dense-transpose for `ss_dec_conv3d`, keep for `*_next_dc`, passthrough for DiTs) with a kernel-dims sanity-assert. Uses `safetensors.torch.load_file` + `mx.save_safetensors`. Publish converted safetensors to Gitea/HF so re-runs skip conversion.
### Parity fixture plan
Stage boundaries all live in `run()` of `trellis2/pipelines/trellis2_image_to_3d.py:488-596`; each fixture = the **return value of a sub-method** → clean non-invasive monkeypatch, zero vendor edits. `parity_dump_hook.py` monkeypatches the 7 sub-methods, `torch.save`s each return to `$TRELLIS2_DUMP_DIR/<name>.pt`, then `runpy`-delegates to vendor `generate.py`. Unset `TRELLIS2_DUMP_DIR` → patches are no-ops.
| # | Fixture | Return / hook (file:line in trellis2_image_to_3d.py) | Shape/notes |
|---|---|---|---|
| A/A | Image-cond embed 512 / 1024 | `get_cond` return `:177` (called `:539`/`:540`; disambiguate by `resolution` arg → `cond_512.pt`/`cond_1024.pt`) | `[1,N,1024]` DINOv3 ViT-L/16 tokens; neg = zeros_like |
| B | Sparse-structure coords | `sample_sparse_structure` return `:235` (call `:542`) | int `[M,4]` (b,x,y,z); subs: `z_s [1,C,64³]` `:219`, occupancy `:227` |
| C | Shape-SLat latent | `sample_shape_slat_cascade` `:364` (`:567`) / non-cascade `:275` | SparseTensor `feats [K,32]`+`coords [K,4]`, **un-normalized** |
| D | Tex-SLat latent | `sample_tex_slat` `:432` (`:574`) | SparseTensor `feats [K,32]`, un-normalized, sampled w/ `concat_cond=shape_slat` |
| E1 | Decoded shape mesh + subs | `decode_shape_slat` `:389` (`:470`) | `List[Mesh]` + `List[SparseTensor]` |
| E2 | Decoded tex voxels | `decode_tex_slat` `:453` (`:471`) | SparseTensor `feats [K,6]` PBR (base_color 0:3, metallic 3:4, roughness 4:5, alpha 5:6) |
| E3 | Final mesh | `decode_latent` `:486` (`:592`) | `MeshWithVoxel`: `.vertices`,`.faces`,`.coords`,`.attrs`,`.voxel_size`,`.origin` |
Invocation (later, GPU): `TRELLIS2_DUMP_DIR=~/Documents/trellis2-bench/parity_fixtures python parity_dump_hook.py <anatomy_img> --pipeline-type 1024_cascade --seed 42 --texture-size 2048`. Fix `seed=42` (`torch.manual_seed` `:538`). Parity target **≤1e-4 rel**. Order cheap→expensive: A/A → B → C (the FlexGEMM sparse-conv crux) → D → E.
**GOTCHA:** C/D fixtures are captured **after** the std/mean de-normalization inside each sampler — the MLX port must apply the same `shape_slat_normalization`/`tex_slat_normalization` (32-dim mean/std, verbatim in `pipeline.json`) at the identical point or they diverge by a constant affine. `sample_tex_slat` also **re-normalizes** `shape_slat` back down (`:407-409`) before concatenating as `concat_cond` — replicate that round-trip.
---
## 6 · RECOMMENDED FIRST HOUR
1. **Read this packet + skim the 3 staging drafts** (`prep_local_draft.py`, `convert_weights.py`, `parity_dump_hook.py`) so you know what already exists. Do NOT re-clone `trellis2-apple` or re-inventory the rmbg venv.
2. **Clone the Jourloy superset** (`git clone https://github.com/Jourloy/TRELLIS.2 ~/Documents/trellis2-mlx-staging/trellis2-jourloy`) — §2 command 1. This is the vendor base; `trellis2-apple` lacks the parity oracle.
3. **Run the parity oracle** `pytest tests/test_mlx_parity.py -v` in an isolated venv (no heavy GPU E2E). Confirms LayerNorm32/RoPE/SDPA/SparseConv parity before trusting any module. This is the cheapest signal available and needs no A0.
4. **Fork to Gitea** `monster/trellis2-mrp-mlx`, keep `origin` upstream, **pin the PR#175 head commit** (it's unmerged/CLA-gated — do not depend on merge).
5. **Queue the single A0 GPU gate** for when the box is free — the ONE gen (anatomy bench, seed 42, 1024-cascade), compare to 124.8 s and the `~/Documents/trellis2-bench/` fal renders. Do not benchmark the stock MLX path as representative before the sparse-conv + adapter rewrites; expect a regression. Apply the §2 decision rule to the result.
6. **In parallel while GPU is busy** (all CPU/inspection, no contention): start Hunyuan **Target 1** (paint UNet `mx.compile`, §3) — cleanest, no blockers — with its PT-parity test; and/or dry-run `prep_local_draft.py` on a sample image under the rmbg venv (§4). Both are safe without the A0 result.
7. **Before any commercial preprocessing run:** `huggingface-cli download ZhengPeng7/BiRefNet` (§4 TODO 1) — otherwise only the non-commercial RMBG-2.0 path works.
**Prime directives:** parity-first (every kernel/edit ships a ≤1e-4-rel PT-parity test + a BENCHMARKS.md row); do NOT edit shared `mlx_arsenal` in place (§3 Target 3); default BiRefNet for game-bound assets (§4). The GPU A0 gen is the ONLY job that must wait for a free box.
---
## 7 · PREP ROUND 2 — VERIFICATION RESULTS (2026-07-19)
> Round-2 deep-verify of §1§6 against **live source + on-disk safetensors headers**. Still
> inspection-only: **no GPU gen, no weights loaded, no prod code touched** (`py_compile` /
> header-parse / `pytest --collect-only`). Where §7 says CONFIRMED / CORRECTION / CONTRADICTION,
> **trust §7 over the original section.**
### 7.1 · ADOPT confidence — §2 stance → **CONFIRMED (HIGH)**
- **Clone DONE & pinnable.** `~/Documents/trellis2-mlx-staging/trellis2-jourloy/`, HEAD `754d403`
(2026-07-17, Jourloy — matches "PR#175"), branch `main` **is** the PR head, tree clean.
**Pin `754d403`** when forking.
- **Superset CONFIRMED — stronger than §2 claimed.** All 4 named artifacts present
(`tests/test_mlx_parity.py`, `trellis2/backends.py`, `scripts/generate_asset.py`,
`requirements_macos.txt`) **plus 4 extra test files** (`test_backend_fallbacks`,
`test_background_preprocess`, `test_generate_asset`, `test_model_revisions`) **and** a new
`trellis2/model_revisions.py` (HF revision pins). `mlx_backend` = 3172 LOC / 16 files, all
`py_compile` clean.
- **`mlx_backend` numerics = `trellis2-apple` byte-for-byte.** `diff -rq`: 14/16 files identical;
the only 2 that differ (`dinov3.py`, `pipeline.py`) add **HF revision-pinning / offline kwargs
only — zero algorithm change.** §2's "identical mlx_backend" holds for numerics; jourloy is a
strict superset (adds reproducibility plumbing on top of the same math).
- **`--backend mlx-experimental` cleanly wired** (`generate_asset.py:333`→`create_mlx_pipeline`,
`:343`) behind a real **Metal-completeness probe gate** (`:170-187` → `backends.probe_metal_backends()`)
that refuses a half-built stack instead of silently degrading. MPS stays the E2E path.
- **§2's 4 UNVERIFIED smells — now resolved:**
| # | smell | round-2 disposition |
|---|---|---|
| 1 | structure_decoder no-op remap + dict-child conv load | **BENIGN by construction** (leaves pre-alloc'd `mx.zeros`; MLX param tree recurses dict/list; `load_weights(strict=True)` replaces leaves). Residual = exact key-name match only, **not** exercised by parity test → run a 2-sec non-GPU load smoke on ss_dec. |
| 2 | vae_decoders softplus/quad_lerp | **BENIGN** — it's the numerically-stable softplus identity `max(x,0)+log1p(exp(-|x|))` = torch softplus. Real line `vae_decoders.py:196` (§2's `:1204` misattributed). Needs one decode-boundary fixture (E1/E2). |
| 3 | non-windowed global attention | **NEEDS-RUNTIME-CHECK / POTENTIALLY REAL — highest-value item.** `_sdpa` does full global SDPA, **no window mask**. If upstream sparse flow is windowed → numeric divergence **and** O(N²) blowup at 1024-cascade. Tiny SDPA parity test validates the primitive, not the windowing semantics. **Verify vs upstream sparse-attn config before trusting shape/tex output.** |
| 4 | "custom Metal kernel" docstrings | **CONFIRMED BENIGN (cosmetic)** — grep `metal_kernel|@mx.custom|.metal|kernel_source` over `mlx_backend/` = zero matches; code calls MLX built-in `mx.fast.scaled_dot_product_attention`. No hand-written kernel anywhere. |
- **Bonus (confirms §2):** sparse-conv perf crux is **REAL**`sparse_conv.py` is naive
gather→matmul→sum, 512 MB-chunked with a per-chunk `mx.eval` host sync, O(batch·D³) dense LUT,
**no `flex_gemm`.** Matches the ~10× claim.
- **Confidence HIGH, not lowered:** nothing contradicted the ADOPT case; several findings
strengthened it. Two items still need a *runtime* signal (smell 3 windowed-attn, smell 1
dict-child load) — **both cheap, non-GPU; run before the A0 gate** (see 7.6).
### 7.2 · WEIGHT-CONVERT — transpose policy **VERIFIED vs disk**; converter has 2 bugs + 1 double-transpose contradiction
All 8 modules parsed **header-only** (no tensor bytes). Tensor counts match §5 **verbatim**
(640/640/640/640/640/292/284/74). Classifier found **ZERO ambiguous 5D tensors** — conventions
separate cleanly by shape.
| module(s) | dtype on disk | 5D convs | §5 policy | matches disk |
|---|---|---|---|---|
| 5 DiTs | **BF16** ×640 ea | 0 (215 Linear + 120 RMSNorm γ + 305 1D) | passthrough + dtype cast | ✅ |
| shape_dec / tex_dec | **F16** | 40 sparse ea, `(Co,K,K,K,Ci)` | **NO transpose** | ✅ |
| ss_dec | **F32×38 + F16×36** (2 F32 convs = `input_layer` + `out_layer.2`) | 20 dense, `(Co,Ci,K,K,K)` | **transpose `(0,2,3,4,1)`** | ✅ |
Transpose/convention classification in §5 holds on disk with **zero contradictions**. The
converter's `else: raise RuntimeError` on any unclassified 5D fails **loud**, so no silent
transpose-convention corruption is possible. **But three items must be fixed before any convert run:**
- 🔴 **BUG 1 (BLOCKING)** `convert_weights.py:59` `v.numpy()` on `torch.bfloat16`
`TypeError: unsupported ScalarType BFloat16`. **Crashes on all 5 BF16 DiTs** (the bulk of the
pipeline); only survives on F16/F32 decoders. Loud, not silent. Fix: `v.float().numpy()`, or
bit-reinterpret via `uint16`, or drop torch and use `mx.load`/`mx.save_safetensors`.
- 🟠 **BUG 2 (SILENT precision loss)** uniform `target_dtype` default `"bf16"` (`:46,81-83,94`)
downcasts the decoders under default args: shape_dec/tex_dec **F16→bf16** (lose 3 mantissa bits,
strictly worse + ~20% slower on M1); ss_dec's **2 F32 convs → bf16** (lose 16 bits on the most
sensitive first/last SDF layers). **Contradicts §5's own dtype column (decoders = F16).** Fix:
per-file dtype policy — DiTs→bf16 (fp16 on M1), decoders→preserve on-disk dtype.
- ⚠️ **CONTRADICTION — ss_dec DOUBLE-TRANSPOSE (silent-corruption risk).** §5/converter transpose
ss_dec `(0,2,3,4,1)` **at load**. But the live MLX `structure_decoder.py` stores PT-format
`(Co,Ci,kD,kH,kW)` **verbatim** and transposes **at forward-call time** (`conv3d`,
`structure_decoder.py:14`). If the converter *also* transposes ss_dec, the decoder
**double-transposes → garbage, silently.** **Resolve before converting:** convert ss_dec to
PT-format on disk (NO load transpose) to match this file, OR move the transpose — pick exactly
one, not both.
- **Port dependency (not the converter):** the MLX sparse-conv module MUST do the
`(Co,K,K,K,Ci)→(Kvol,Ci,Co)` reshape at load (`conv_none.py:106`) or the `*_next_dc` decoders
emit garbage. Converter correctly leaves this to the module.
### 7.3 · PARITY HOOKS — all 8 targets match live source, **ZERO corrections to the hook**
`parity_dump_hook.py` needs no edits. Source `trellis2_image_to_3d.py` is frozen at upstream
`75fbf01` (2026-06-05) + 1 benign local Mac cuda-guard (`:590-591`, downstream of every hook →
shifts no fixture line). **No post-packet drift.** Verified def/return/call (corrections to §5's
fixture table where it drifted):
| fixture | def | return | call-site(s) |
|---|---|---|---|
| get_cond (A/A) | 164 | **181** (no-neg) / **183-186** (dict) | 539, 540 |
| sample_sparse_structure (B) | 188 | 235 | 542 |
| sample_shape_slat (C, non-cascade) | 237 | 275 | 547, 557 |
| sample_shape_slat_cascade (C) | 277 | 364 (tuple `slat,hr_res`; `transform` takes `r[0]`) | 567, 579 |
| sample_tex_slat (D) | 391 | 432 | 551, 561, 574, 586 |
| decode_shape_slat (E1) | 366 | 389 | 470 |
| decode_tex_slat (E2) | 434 | 453 | 471 |
| decode_latent (E3) | 456 (`@torch.no_grad` :455) | 486 | 592 |
- **§5 CORRECTION:** the fixture table cites get_cond "return **:177**" — that line is the model
call `cond = self.image_cond_model(image)`, **not** the return. Real returns are **:181 / :183-186**.
Cosmetic (hook wraps by method name), but corrected.
- **Normalization gotcha CONFIRMED:** `shape_slat.pt` / `tex_slat.pt` captured **POST-denorm**
(`:271-273`, `:360-362`, `:428-430`); tex sampler **re-normalizes** shape_slat back down at
`:407-409` before `concat_cond` (:419). Port must replicate **both** the on-return de-norm and the
denorm→renorm round-trip or fixtures diverge by a constant affine (silent ≤1e-4 failure).
- **UNCERTAIN — flag for exec session:** the hook's `__main__` `runpy`-delegates to vendor
`generate.py`; that file's existence + CLI contract (`--pipeline-type` / `--seed` /
`--texture-size`) was **not** checked. Verify before the dump run — a missing/renamed delegate
means the hook installs but never fires.
### 7.4 · PREPROCESSING — BiRefNet **PRESENT**; D2 **IMPLEMENTED**
- **BiRefNet fetched (§1 "NOT present" TODO → DONE).** Cache:
`/Users/m3ultra/.cache/huggingface/hub/models--ZhengPeng7--BiRefNet/snapshots/e2bf8e4460fc8fa32bba5ea4d94b3233d367b0e4/`.
Size **424 MB** (`model.safetensors` = 444,473,596 B) — **§1/§4's "~900 MB" was high.**
License **MIT** (README `license: mit`) — **§1/§4's "Apache-2.0" label was wrong.** Ungated, no login.
- **BiRefNet branch VERIFIED CORRECT (not stubbed):** RMBG-2.0 *is* a BiRefNet checkpoint (identical
`auto_map`/`architectures`); forward `model(inp)[-1].sigmoid()` holds (`birefnet.py:2087/2090`);
deps satisfied by the rmbg venv — **no cv2 needed** (`birefnet.py` has zero opencv refs). Only
metadata was hardened; no code change.
- **D2 (light normalization) IMPLEMENTED** behind `normalize_light=False` (default **OFF** → v0
silhouette stays A/B-identical to fal). New `_normalize_light` (`prep_local_draft.py:84`, PIL/numpy
only): 1/99-pct auto-levels (flat-channel guarded) + gray-world WB; wired at step **1b** (after
upscale, before matte — honors "matte LAST"). CLI `--normalize-light`. Functional smoke passed on a
synthetic cast image.
- **Draft status:** `~/Documents/trellis2-mlx-staging/prep_local_draft.py` now 236 lines,
`py_compile` OK + imports clean under rmbg venv. Signature
`prep_local(in_path, out_path, commercial=False, normalize_light=False)`. **Staging only — still
NOT wired into prod** (§4.5 wire-in + tunable A/B still open; need a GPU-free box for real A/B).
### 7.5 · MLX TEST VENV — oracle is **one command from running**
- **Venv READY:** `/Users/m3ultra/Documents/trellis2-mlx-staging/mlx-test-venv` (CPython **3.12.13**,
uv). `mlx==0.32.0` (+`mlx-metal` 0.32.0), `torch==2.13.0` (prebuilt wheel, no compile),
`numpy` 2.5.1, `pytest` 9.1.1. (safetensors/transformers/einops also installed but the oracle uses
**none** of them — droppable for a leaner venv.)
- **Collects clean:** `--collect-only`**4 tests collected, 0 import errors, 0 skips** (0 skips
proves both `importorskip("mlx.core")` and `importorskip("torch")` succeeded on the box). **Not yet
executed** — a real pass is still pending (low-risk caveat: `np.asarray(mlx_array)` under numpy
2.5.1 surfaces only on the real run, not collection).
- **GPU/Metal?** 4 tests run microsecond ops on tiny tensors. MLX's default device is Metal (GPU), so
they *touch* the GPU — but this is trivial library compute, **NOT a generation/inference job**, uses
**no Metal build packages** (mtlgemm/mtldiffrast/cumesh) and no compile. Won't meaningfully contend
for the box; force CPU with `mx.set_default_device(mx.cpu)` if ever desired.
- **EXACT run command** (use the venv python — a bare `pytest` uses the wrong interpreter, correcting
§6/§2):
```
ssh -n m3ultra@100.89.131.57 'cd ~/Documents/trellis2-mlx-staging/trellis2-jourloy && \
../mlx-test-venv/bin/python -m pytest tests/test_mlx_parity.py -v'
```
Expected: **4 passed.** Baked tolerances: LayerNorm `2e-5`, RoPE `1e-6`, SDPA `3e-5`,
SparseConv `2e-5` (all tighter than the ≤1e-4 target).
### 7.6 · UPDATED FIRST MOVES — deltas to §6
- **§6 step 2 (clone Jourloy): DONE** — `trellis2-jourloy/` @ `754d403`, tree clean. Skip the clone;
just `git checkout 754d403` / pin it when forking.
- **§6 step 3 (run parity oracle): venv READY, oracle collects clean.** Now a one-command move — run
the exact command in **7.5**. **Do NOT use the bare `pytest tests/...` from §6/§2** (wrong
interpreter).
- **§6 step 7 / §4 TODO 1 (BiRefNet fetch): DONE** — skip. Present at the 7.4 path (424 MB, MIT).
- **NEW — before any weight-convert run (§5):** fix converter **Bug 1** (blocking bf16 crash) +
**Bug 2** (silent decoder downcast), AND resolve the **ss_dec double-transpose** contradiction (7.2).
Else DiT convert crashes, decoders lose precision, or ss_dec silently corrupts.
- **NEW — before the A0 GPU gate:** run the two cheap **non-GPU** runtime checks — (a) ss_dec
dict-child weight-load smoke, and especially **(c) global-vs-windowed attention vs the upstream
sparse-attn config** (the one smell that can be a true correctness bug, not just slow). Both fit
CPU/inspection while the box is busy.
- **Unchanged:** §6 step 5 (single A0 GPU gate — seed 42, 1024-cascade, compare 124.8 s) is still the
**only** job that must wait for a free box; §6 step 6 (Hunyuan Target 1) still safe in parallel.
## 8 · CORRECTION — the adopt path needs NO weight conversion (2026-07-19, verified in source)
Round-2 flagged three bugs in `convert_weights.py` (bf16 crash, decoder downcast, ss_dec double-transpose). **All three are moot: the converter is not needed for the adopted Jourloy base, and using it would corrupt.** Verified by reading Jourloy `mlx_backend/`:
- **Weights load directly from the original TRELLIS.2-4B safetensors.** `pipeline.py:116/141/162``load_safetensors("<path>.safetensors")``model.load_weights(...)`. `load_safetensors` (`__init__.py:26`) is just `mx.load(path)`, keeping native bf16/fp16; an optional `dtype=` casts float weights (this is the M1 fp16 path — **not** a separate converter).
- **Remap = key-rename ONLY, zero tensor transform.** `remap_flow_model_weights` / `remap_vae_decoder_weights` only rewrite key strings (nn.Sequential `.N.`→`.layers.N.`; MlxSparseLinear `.linear.` wrap). No transpose, no dtype change to any tensor.
- **All conv transposing is IN-MODULE at forward.** `structure_decoder.py:14` transposes the dense ss_dec (Co,Ci,K,K,K)→(Co,K,K,K,Ci) every forward; `sparse_conv.py:130` `w.transpose(1,2,0)` for the sparse GEMM. So the modules EXPECT PyTorch-format weights.
- **Why the converter is a trap:** it transposes ss_dec at load; the module transposes again at forward → double-transpose. For conv1/conv2 (Ci==Co==channels) the SHAPE still matches, so `load_weights` succeeds silently with wrong values; for input_layer (Ci=latent≠Co) it would shape-mismatch and error. Mixed silent/loud corruption.
**ACTION for Fable:** do **not** run `convert_weights.py` for the Jourloy path (it's now banner-warned in staging). For M1 fp16, pass `dtype=mx.float16` to `load_safetensors`. §5's transpose table remains correct as a *description of the on-disk layout*, but the CONVERSION step it implied is unnecessary — Jourloy consumes originals directly. A converter is only relevant if you ever build a from-scratch decoder that does not transpose at forward.
## 9 · EXECUTION LOG — Fable session 1 (2026-07-19)
Done this session (all committed; BENCHMARKS.md has the rows):
1. **Parity oracle: 4/4 PASS** (2.55s, CPU-only, mlx-test-venv).
2. **Jourloy venv built** via scripts/setup_macos.sh (PYTHON_BIN=uv 3.11; Metal pkgs compiled).
3. **A0 GATE PASSED — ADOPT CONFIRMED**: mlx-experimental beat MPS (216.6s vs 248.8s wall) at 14× bake density. It's a HYBRID (Metal flex_gemm sparse ops + MLX dense); naive MLX sparse-conv never runs. 75.4GB peak.
4. **Hunyuan mlx-tune branch @ 4b368ee** (pushed to partly): T1 UNet mx.compile (env-gated HY3D_MLX_COMPILE, parity 3e-5, 1.04× micro), T3 StaticMoELayer (bit-identical, kills ~4800 syncs, compile-unlocked), T4 fused DINO SDPA (1.41× micro, parity 1e-7). E2E flag-on run in flight vs the 260s defaults baseline.
5. **Dark-patch culprit narrowed**: reproduces on BOTH independent pipelines (trellis-mac MPS + Jourloy MLX) → shared pedronaugusto Metal texel-sampling (mtldiffrast/BVH) or decoded attrs. fal is clean on the same weights → attrs likely fine → **Metal sampling = prime suspect**. KDTree pure-Python bake control still grinding (2h+, alive, 300% CPU) — its render decides.
NEXT (in order): read KDTree verdict when it lands → Hunyuan e2e A/B number → wire prep_local into MESHGOD behind a flag → Target 2 (whole-DiT compile now that T3 unblocked it) → promote mlx-experimental into MODELBEAST as a `trellis2_mlx` operator behind a flag once the bake-quality question is settled.
## 10 · DARK-PATCH VERDICT: Metal texel-sampling convicted (2026-07-19)
Decisive experiment: dumped the decoded tex voxels (E2 fixture,
1,398,769 voxels × 6 PBR attrs) and rendered base_color as a direct
front-projection — `trellis2-bench/attrs_front.png`. **The attrs are
near-fal quality** (clean anatomy palette; dark voxels only 5.6%, all
interior). Both local pipelines (trellis-mac MPS and Jourloy MLX) produce
the same mottled dark bake from these clean attrs, and both share the
pedronaugusto mtldiffrast/mtlbvh texel→voxel sampling. ⇒ **the bake's
attr-sampling is the sole quality gap.** The 3h+ KDTree run is now merely
confirmatory (left running).
### Designed fix (next session's opening move)
The attrs live on a REGULAR voxel grid — no BVH/KDTree needed at all:
texel → (xatlas UV → triangle barycentric → world pos) →
`grid[(pos-origin)/voxel_size]` direct index (+8-neighbor occupancy
search). One vectorized gather over all texels; replaces the suspect
ray-cast sampling entirely; implementable in torch-MPS or MLX in ~a page.
Validate against `attrs_front.png` colors + the fal render. If it works,
BOTH local paths get fal-class texture quality at zero bake-speed risk.
### Parity fixture set: COMPLETE
All 8 fixtures captured at `trellis2-bench/parity_fixtures/` (cond_512/
1024, ss_coords, shape_slat, shape_dec, tex_slat, tex_voxels, mesh_final;
seed 42, 1024-cascade). Hook env needed: PYTORCH_ENABLE_MPS_FALLBACK=1
ATTN_BACKEND=sdpa SPARSE_ATTN_BACKEND=sdpa SPARSE_CONV_BACKEND=flex_gemm.
### Fork home (added 2026-07-19, post-§10)
The vendored Jourloy/PR#175 tree now lives on our Gitea:
`ssh://git@100.71.119.27:222/monster/trellis-2-mrp-mlx.git` (note the
hyphens), branch `main`, pin tag `pr175-head-754d403`. m3 staging clone has
it as remote `partly`; `origin` stays github/Jourloy for rebases. All our
kernel/bake work lands as commits on this fork.
## 11 · BAKE FIX PROVEN (2026-07-19 evening) — dark patches eliminated
`trellis2-bench/render_vbake5_front.png`: full anatomy figure, correct
colors, ZERO dark patches — from the same decoder attrs the Metal baker
mangles. Method: cKDTree IDW sampling of the voxel grid at mesh vertices
(k=4, distance² weights, 1.5-voxel radius, single-nearest fallback) →
vertex-colored GLB. **Sampling cost: 0.3s** for 270K verts / 1.4M voxels.
Two integration gotchas (cost hours; do not rediscover):
1. **Axis spaces.** Voxel grid + raw pipeline mesh = Z-up-long-axis in
[-0.5,0.5]³; GLB-loaded meshes = glTF Y-up. Transform before sampling:
`(x,y,z)_gltf → (x,z,y)_voxel`. The "mangled blob" renders were ONLY
this (figure viewed down its own axis) — no data corruption anywhere.
2. **glTF vertex colors are LINEAR** — do not pre-gamma (double-gamma =
washed out). Store linear, let the renderer do the transfer.
Also learned: raw `MeshWithVoxel.vertices/faces` need to_glb's weld/init
before use as a plain triangle soup; `output_3d.obj` inherits the raw
convention. And xatlas hangs (2h+) on écorché-topology meshes — the UV
path needs chart budgeting or per-island parallel unwrap.
### Production path (next session)
- Option A (fast to ship): full-density vertex-colored GLB as the raw
output; MESHGOD's Blender finish farm already decimates + can bake
verts→texture (standard Blender bake) — sidesteps xatlas entirely.
- Option B (fal-parity): fast bucket rasterizer (written, in
fast_bake_test.py) + xatlas-alternative UV → 2048² texture bake with
this sampler. Needs the écorché-safe unwrap.
- Wire as `--baker python-fast` in the Jourloy fork + trellis-mac, make it
the default over the convicted Metal texel path; port sampler to MLX
(trivial — gather + weighted sum) for the pure-MLX lane.
### §11 addendum — vertex baker PRODUCTIONIZED (2026-07-19 night)
Wired and validated in BOTH pipelines:
- `vendor/trellis-mac` @ 55fdd7b: `TRELLIS2_BAKER=vertex` env mode in
generate.py (Metal + kdtree paths untouched; sentinel-gated).
- Jourloy fork @ 746e727 (pushed to monster/trellis-2-mrp-mlx): CLI
`--baker vertex`.
E2E validation (anatomy, seed 42, 1024-cascade, uncapped density):
**130.0s total, bake 1s**, 232s wall, 20.8GB peak — full 2.79M-face
vertex-colored GLB, ZERO dark patches, best local quality yet
(`render_vprod_front.png`). vs Metal-bake baseline: same speed envelope,
quality gap closed for albedo. Not yet carried: metallic/roughness maps
(vertex colors are albedo-only) — MR needs the UV path (Option B) or a
COLOR_1 convention. MESHGOD finish-farm verts→texture bake = the
remaining Option-A step for game-budget assets.
## 12 · RUNG-3 KERNEL DAY — RESULTS (2026-07-20)
**Headline: 216.6s → 71.96s e2e (3.0×), peak 75.4GB → 26.5GB (2.8×),
parity held (raw mesh Δ0.03%), full gate EXIT=0.** All on the fork
(monster/trellis-2-mrp-mlx): fa972aa + 7860148 + 0db816b.
What actually moved the needle (in order of impact):
1. **Vertex baker properly wired into generate_asset** (scheduler
fallthrough fixed — `--baker vertex` was silently running the 5h
pure-python kdtree path): "other" bucket 69.4s → 8.3s, and the
75GB peak turned out to be the METAL BAKE's, not the conv's.
2. **Pure-MLX sampler loop** (mlx_samplers.py): torch CPU loop + per-
forward host bounces → one conversion per stage, batched dense CFG,
once-per-stage concat_cond. Sampling 56.8→54.4s (~4% — forwards are
compute-bound) but it is the correctness-proven, compile-ready
architecture, and removes ~72 host round-trips/gen.
3. **Validator taught the vertex-colored asset class** (COLOR_0 without
UVs = legit; normals materialized at export).
Honest negatives (documented so nobody re-chases them):
- mx.compile on the step forward: ~0 on M3 Ultra (GEMM-bound; matches
Hunyuan T1/T2 findings).
- Fused Metal sparse-conv kernel: built, parity 8e-4, but stock chunking
already bounds memory at decoder scale and beats the scalar kernel on
speed. Kept as TRELLIS2_METAL_SPCONV=1 opt-in reference (simdgroup
tiling = the future version if decoders ever matter for time).
- Remaining gap to H100 (~17s at 1024): inside the DiT forwards
(54s sampling ≈ 2.3× ideal FLOPs) — attention/MLP internals, a
future deep-dive, not orchestration.
Env knobs (all default-sane): TRELLIS2_MLX_SAMPLER=0 (revert sampler),
TRELLIS2_MLX_COMPILE=1, TRELLIS2_METAL_SPCONV=1.
M1 Ultra validation: in flight (26.5GB peak fits with 100GB headroom).
### Rung-3 addendum — M1 Ultra validated (2026-07-20)
| box | e2e wall | peak mem |
|---|---|---|
| m3ultra | 71.96 s | 26.5 GB |
| **m1ultra** | **181.5 s** | **17.2 GB** |
The M1 Ultra — which could not run TRELLIS.2 at all before this fork
(torch-MPS path was m3-only in practice) — now generates full-density
vertex-baked assets in ~3 min at 17GB. Install: clone
monster/trellis-2-mrp-mlx to ~/trellis2-mlx + scripts/setup_macos.sh
(PYTHON_BIN=uv 3.11) + gated weights rsynced from m3's HF cache
(no HF login needed on workers). Both Ultras now serve the fastest
local TRELLIS.2 in existence. TRELLIS.2 fleet: m3 primary (72s),
m1 second lane (182s), zero dollars per asset.

212
docs/MRP_MLX.md Normal file
View File

@ -0,0 +1,212 @@
# MRP-MLX — the MonsterRobotParty MLX stack
**The short version:** every serious generative model we run — image, layered
image, image→3D, video keying — runs *locally, on Apple Silicon, in MLX or
Metal, from forks we control, at $0 per generation.* This document is the
canonical reference: what each piece is, where it lives, how fast it goes,
how to run it, and the hard-won rules that keep it fast. Written 2026-07-20
after the rung-3 kernel push; every number here was measured, not estimated.
---
## 1 · Philosophy (why this stack wins)
1. **Local-first.** Hosted GPU (fal) is for interactive one-offs and A/B
references only. Volume work runs on the fleet. (July fal bill was
~$93/mo before this stack; the same workload is now ≈ $0.)
2. **Adopt-then-patch, never from-scratch.** Every port here started as the
best community MLX effort, vendored to our Gitea, then patched where
measurement said to. Proven 5×: hunyuan3d (dgrauet), corridorkey
(cmoyates/Niko), trellis-2 (Jourloy PR#175), qwen-layered (mflux PR#302),
flux (mflux upstream).
3. **Forks live on our Gitea** (`ssh://git@100.71.119.27:222/monster/…`)
with upstream kept as `origin` for rebases and a pin tag on the adopted
commit. Our infra, our history, our rollback.
4. **Parity-first.** No optimization ships without a numeric gate against
the reference implementation (tolerances recorded), and e2e output
stats compared at fixed seed. Fixture sets are kept for regression.
5. **Bench-everything, keep the negatives.** BENCHMARKS.md records failures
(q4 = noise, mx.compile ≈ 0 on big GEMMs) with the same care as wins, so
no future session re-chases a ghost.
6. **Env-gated changes.** Every behavioral change has an env knob and a
default that preserves upstream behavior (or our validated best). Prod
rollback is an environment variable, not a revert.
## 2 · The fleet it runs on
| box | chip | RAM | fp16 TFLOPS | bandwidth | role |
|---|---|---|---|---|---|
| m3ultra | M3 Ultra | 256GB | 23.5 | 400 GB/s | primary: TRELLIS.2, queue, heavy gen |
| m1ultra | M1 Ultra | 128GB | 16.3 | 297 GB/s | second lane: everything (post rung-3) |
| m1max | M1 Max | 32GB | 7.1 | 336 GB/s | bandwidth-friendly light ops |
| m4pro | M4 Pro | 24GB | 5.2 | 225 GB/s | MB light ops + Ollama (hands off) |
Full 7-machine MLX matrix + probe script: `BENCHMARKS.md` (2026-07-19) +
`scripts/mlxprobe.py`. Rules from it: **fp16 on M1-class** (bf16 20%),
bf16 free on M3/M4/M5; pad attention head_dim→64 on M4/M5-class.
## 3 · The pillars
### 3.1 TRELLIS.2 — image → full-density 3D · `monster/trellis-2-mrp-mlx`
- **Base:** Jourloy/TRELLIS.2 (upstream PR#175 head, pin `pr175-head-754d403`)
— hybrid MLX (dense stages) + pedronaugusto Metal kernels (sparse ops).
- **Our commits:** `fa972aa` pure-MLX sampler loop (batched dense CFG,
once-per-stage conversions, one eval/step; parity Δ0.03% e2e) ·
`7860148` opt-in Metal spconv kernel + vertex-baker scheduler fix ·
`0db816b` vertex-normal export + COLOR_0 asset class in the validator.
- **Weights:** original `microsoft/TRELLIS.2-4B` safetensors, loaded
directly (`mx.load` + key renames; **no conversion step exists or should**
— a converter would double-transpose ss_dec into silent garbage).
- **Performance (anatomy bench, seed 42, 1024-cascade, vertex baker):**
| box | e2e | peak mem |
|---|---|---|
| m3ultra | **72.0 s** | 26.5 GB |
| m1ultra | **181.5 s** | 17.2 GB |
| (old torch-MPS `trellis_mac`) | 124.8 s gen (248.8 wall) | 20.8 GB |
| (fal, same image) | 298.6 s wall | $0.30 |
- **Run:** `cd <fork> && .venv/bin/python scripts/generate_asset.py IMG
--output-dir OUT --backend mlx-experimental --baker vertex
--pipeline-type 1024_cascade --seed N` · Farm: operator `trellis2_mlx`.
- **Knobs:** `TRELLIS2_MLX_SAMPLER=0` (revert to torch sampler) ·
`TRELLIS2_MLX_COMPILE=1` (≈0 on M3U; try on small boxes) ·
`TRELLIS2_METAL_SPCONV=1` (opt-in kernel; slower-but-leaner, reference).
- **Output class:** vertex-colored full-density GLB (albedo only). MR maps
need the UV path (`--baker metal` — has the dark-patch sampling bug) or
the future fast-UV bake. Game budgets: MESHGOD finish farm downstream.
- **Checkouts:** m3 `~/Documents/trellis2-mlx-staging/trellis2-jourloy` ·
m1 `~/trellis2-mlx` · fresh boxes: `scripts/install_trellis2_mlx.sh`.
### 3.2 Hunyuan3D 2.1 — image → textured 3D · `monster/Hunyuan3D-2.2-mrp-MLX`
- **Base:** dgrauet/Hunyuan3D-2.1-mlx (pure MLX, both stages; 13GB
pre-converted MLX weights on HF). *Terminology note: this IS the
v2-generation Hunyuan; fal's `/v2`,`/v21`,`/turbo` are hosted variants.*
- **Our branch `mlx-tune` @ `5a32240`** (deployed both Ultras): T1 UNet
mx.compile + T2 whole-DiT compile (env `HY3D_MLX_COMPILE=1`; parity
3e-5/7e-6) · T3 StaticMoELayer (bit-identical, kills ~4800 host syncs,
makes the DiT compile-legal) · T4 fused-SDPA DINO attention (1.41× its
stage, parity 1e-7).
- **Performance:** 260 s baseline → **243 s** tuned (m3ultra, defaults).
Runs on every box M1-and-up (fp16). ~20GB peak.
- **Run:** `generate_e2e.py IMG --output OUT` in `vendor/hunyuan3d-mlx`
(venv `.venv`) · Farm: operator `hunyuan3d_mlx`.
### 3.3 FLUX / image generation — `mflux` (upstream; not forked)
- mflux 0.18.0 (latest) in `venvs/mflux`, both Ultras + m4pro. Written
from scratch in MLX by Filip Strand — no port debt to carry.
- **Audit (2026-07-19):** our default path (`flux2-klein-4b`, 4 steps,
~9 s/image) has mx.compile on the denoise step + a shapeless-compiled
scheduler — healthy, keep. **Known gap:** the FLUX.1 family
(dev/schnell/krea) has zero mx.compile — a backport is the same 10-line
pattern as Hunyuan T1, expected win 14% on Ultras (more on small
boxes), and belongs UPSTREAM as an mflux PR rather than a site-packages
patch. Queued, not urgent.
- Operator `flux_local` (klein default) + `mflux_image_edit`.
### 3.4 CorridorKey — neural green-screen · `monster/corridorkey-mrp-mlx`
- **Base:** nikopueringer/corridorkey-mlx (cmoyates' phased parity-first
port — its `prompts/` folder is our template for any future from-scratch
port). License CC BY-NC-SA — non-commercial only.
- **Our patch (`4c660df`, branch `modelbeast`):** honor the compile flag in
tiled mode — 8 lines, **1.47×** on m3ultra (3.64→2.48 s/frame), output
bit-identical. Plus the 8-machine ablation that found the head_dim→64
fast-path rule and the "tiled beats full-frame on quality AND memory"
result (2.3GB vs 28GB). Nightly perfcheck watches the fast path.
### 3.5 Qwen-Image-Layered — image → editable RGBA layers · `monster/Qwen-Image-Layered-MRP-MLX`
- **Base:** mflux PR#302 (ZimengXiong; unmerged upstream), pin
`pr302-head-a255e4f`. 20B, Apache 2.0 (ship-commercial OK).
- **Validated ladder:** **q8 is the quality FLOOR** — q6 breaks the layer
semantics (silhouettes), q4 is noise; speed is FLAT across quants
(compute-bound), so never quantize below q8. 20 steps ≈ 50 steps
visually (2.9× faster). 1024-res = hero tier.
- **Performance (4 layers, 640, baked q8, 20 steps):** m3ultra **243 s** ·
m1ultra 366 s · 1024-res hero: 858 s (m3). Peak ~37GB.
- **Baked models:** m3 `~/qwen-layered/qwen-layered-q8` · m1
`~/qwen-layered-staging/qwen-layered-q8` (34GB; rebake via `mflux-save`).
- **Run:** `mflux-generate-qwen-layered --image IMG --layers 4
--resolution 640 --steps 20 --model-path <baked>` · Farm: operator
`qwen_layered_local`. Bonus: the subject layer is matting-grade — the
premium tier of image preprocessing for hero assets.
## 4 · MODELBEAST integration
| operator | backs | speed (m3 / m1) | notes |
|---|---|---|---|
| `trellis2_mlx` | 3.1 | 72 s / 182 s | vertex baker default; **preferred 3D lane** |
| `trellis_mac` | torch-MPS port | 125 s gen / n/a | legacy; UV-textured (dark-patch bug) |
| `hunyuan3d_mlx` | 3.2 | 243 s / ~5-6 min | textured GLB; runs everywhere |
| `flux_local` | 3.3 | ~9 s/img | klein-4b 4-step default |
| `qwen_layered_local` | 3.5 | 243 s / 366 s | RGBA layer decomposition |
Routing lives in `nodes.json` (machine-local, gitignored — edit on the
m3 primary). MESHGOD's `local/` lane calls these via the MB REST API.
## 5 · The lessons ledger (do not re-learn these)
1. **Profile before optimizing — every time.** The rung-3 3× came from a
scheduler bug fix + baker swap the profiler exposed; the glamorous
kernel work measured ≈ 0. The 75GB "sparse conv" memory monster was
actually the Metal texture bake.
2. **mx.compile ≈ 0 on GEMM-bound 2B+ models on M3 Ultra** (measured 4×:
Hunyuan T1, T2, TRELLIS step, sampler). It's free insurance, not a win.
Compile wins live on small models (corridorkey 1.47×) and maybe small
chips.
3. **Quantization floors are model-specific and cliff-shaped.** Qwen-
Layered: q8 fine, q6 semantically broken, q4 noise — and quant does
NOT buy speed on compute-bound models, only footprint.
4. **glTF axis + gamma:** pipeline space is Z-up in [-0.5,0.5]³; glTF is
Y-up (`(x,y,z)→(x,z,y)`); COLOR_0 is LINEAR (never pre-gamma).
A "mangled blob" render is usually just the figure viewed down its own
axis after a convention miss.
5. **xatlas hangs (hours) on écorché-class meshes** (thousands of
disconnected shells). Vertex bake sidesteps UV unwrap entirely.
6. **macOS TCC blocks launchd from ~/Documents** — services live in `~/`.
7. **Gated HF weights don't need logins on workers** — rsync the model
dirs from a cache that has them.
8. **The fal-outage class of bug:** any `while True` polling a paid API
needs a deadline + a cancel; an in-memory queue wedges silently.
9. **Adapters that convert per-call hide O(steps) waste** — convert at
stage boundaries, hold state in the fast runtime.
10. **Sentinel fallthroughs bite:** a baker/scheduler that silently falls
back can burn 5 hours before anyone notices. Fail loudly instead.
## 6 · Runbooks
**New box bring-up (any Apple Silicon, macOS 26+):**
1. Keys: box's SSH key added to Gitea (`monster` account).
2. `git clone` the MODELBEAST repo; run the relevant
`scripts/install_*.sh` (trellis2_mlx, qwen_layered, hunyuan…) — each is
idempotent, uses uv, and states its weight needs.
3. Weights: rsync the HF cache model dirs from m3ultra (see §5.7).
4. Bench: `uv run --with mlx python scripts/mlxprobe.py` → add the row to
BENCHMARKS fleet matrix; set fp16/bf16 per §2 rules.
5. Register the box + operator list in `nodes.json` on the m3 primary.
**Rollback:** every MRP patch is env-gated (§3 knobs) or a pinned-tag fork
`git checkout <pin-tag>` restores the adopted upstream exactly.
**Where results go:** every experiment appends to `BENCHMARKS.md` (wall
time, peak mem via `mx.get_peak_memory()`, parity numbers, and a verdict
line). Session-scale context lives in `docs/FABLE_STARTUP_PACKET.md`.
## 7 · Roadmap (scoped, in value order)
1. **DiT-internals deep-dive** — TRELLIS.2 sampling runs ≈2.3× above ideal
FLOPs (54 s of the 72). Attention/MLP kernel work inside the forward;
the corridorkey `prompts/` phased method is the template.
2. **Fast UV bake** — bucket rasterizer (written, in fast_bake_test.py) +
an écorché-safe unwrap → texture+MR parity with fal on the vertex
baker's quality. Unlocks `--baker vertex`-quality WITH UV maps.
3. **simdgroup spconv kernel** — upgrade the opt-in scalar kernel if the
decoders ever matter for time (they're 10 s today).
4. **Upstreaming** — our sampler + baker + validator to Jourloy PR#175;
FLUX.1 compile backport as an mflux PR; spconv kernel as reference.
5. **MESHGOD default flip** — route its `local/` 3D lane to `trellis2_mlx`
once finish-farm verts→texture baking lands for game budgets.

View File

@ -0,0 +1,143 @@
# Local 3D → full MLX + preprocessing frontend — master build plan
**For:** a fresh long-running Fable session on m3ultra (no execution-time cap).
**Author:** Opus planning pass, 2026-07-19. **Read first:** `docs/TRELLIS2_MLX_RECON.md`,
`BENCHMARKS.md` (fleet MLX matrix + rung 1/2), `HANDOFF_HY3D_MLX.md`, `CORRIDORKEY.md`.
**Prime directive:** parity-first. Every kernel/port change ships with a PT-parity test
(≤1e-4 rel) + a BENCHMARKS.md row (per-stage time + `mx.get_peak_memory()`). No silent
regressions — that discipline is why hunyuan3d-mlx and corridorkey-mlx worked.
**Two goals, independent, run in parallel:**
- **G1 — the models go fully MLX + fast** (TRELLIS.2 port from scratch-ish; Hunyuan tune).
- **G2 — a local preprocessing frontend** that turns any messy photo into the input the
models want. Model-agnostic; helps every generation from here on.
---
## Ground truth (measured, don't re-derive)
- Local TRELLIS.2 = `vendor/trellis-mac` (shivampkumar fork), **torch-MPS, NOT MLX**.
124.8s/gen on m3ultra (1024-cascade, tex2048), 20.8GB peak. Weights `TRELLIS.2-4B` 14GB.
- Stage breakdown of the 124.8s: **~46s diffusion sampling** (padded SDPA attention),
**~50s decode + mesh extract** (incl. `aten::segment_reduce`**CPU fallback**),
**19s Metal PBR bake**. Decoder emits **2.79M faces** pre-simplify.
- Sparse-conv backend already on the Metal fast path (`flex_gemm`); worth **1.72×** vs the
pure-torch fallback (rung 1). Metallib loads on macOS 26.5.
- Hunyuan3D-2.1 = `vendor/hunyuan3d-mlx` (dgrauet), **already MLX**, ~260s m3ultra, runs on M1.
- Quality gap vs fal = dark shattered patches; density is NOT the cause (rung 2: 200K/800K/2.79M
all show it) → **texture bake or decoder attrs**. KDTree-baker A/B pending (`TRELLIS2_FORCE_KDTREE`).
- Fleet MLX matrix (BENCHMARKS.md): bf16 fine except M1 (~20% slower → ship fp16 on M1);
**pad attention head_dim→64** (up to 4.7× M5, 1.5× M3U, ~0 on M1); `mx.compile` on fixed shapes.
- Knobs already added to `vendor/trellis-mac/generate.py`: `TRELLIS2_MAX_BAKE_FACES` (0=uncapped),
`TRELLIS2_FORCE_KDTREE` (A/B the baker). Bench artifacts in `~/Documents/trellis2-bench/`.
---
## G1 · WORKSTREAM A — TRELLIS.2 → MLX
### A0. Scout the existing MLX efforts (do FIRST — decides adopt vs build)
- Vendor `pedronaugusto/trellis2-apple` (has an `mlx_backend/`) and upstream **PR #175**
(MPS+Metal + experimental `mlx` flag, 28 tests green on M4 Max). Fork both to Gitea as
`monster/trellis2-mrp-*` (house pattern; keep upstream remote for rebases).
- Install each, run ONE gen on the anatomy bench image, seed 42, 1024-cascade. Compare to the
124.8s MPS baseline **and** to fal quality (renders already in `trellis2-bench/`).
- **Decision gate:** if trellis2-apple's MLX path is correct + faster → we ADOPT and jump to A3
(patch, not rebuild). If it's partial/slower/buggy → we BUILD (A1→A2), using it + the
corridorkey `prompts/` 6-phase template as reference.
### A1. Scaffold the MLX port (if building)
- New repo `monster/trellis2-mlx`. Mirror the corridorkey-mlx phased layout: `prompts/` phase docs,
`src/`, `tests/test_parity.py`, `scripts/convert_weights.py`.
- Weight conversion: TRELLIS.2-4B safetensors → MLX. Reuse the hunyuan pattern
(`hy3dpaint/utils/convert_realesrgan.py`): torch load → key remap → **transpose Conv (O,I,H,W)→(O,H,W,I)** → save. fp16 (M1-safe); publish converted weights to HF/Gitea so re-runs skip conversion.
- Build a PT reference harness first (dump per-stage activations from the working MPS build as
fixtures) — every MLX module validates against these.
### A2. The kernel rewrites (the real work — this is where the days go)
Priority = by measured cost. Each = an `mx.fast.metal_kernel` + parity test + bench row.
1. **Sparse-conv (FlexGEMM → fused Metal).** THE crux and the ~10× lever. Triton has no Metal
target; implement gather→GEMM→scatter as a fused Metal kernel. Reference: the existing
`mtlgemm` Metal port in trellis-mac/deps and `backends/conv_none.py` (the slow pure-torch
version) for correct semantics. MLX has no sparse tensor type — hand-roll SparseTensor as
coord-list + hash (same as CUDA/MPS side).
2. **Varlen/sparse attention (padded SDPA → fused).** ~46s of the run. Kill the pad-to-maxlen
waste; pad head_dim→64 for the fused fast path (fleet matrix). This + #1 close most of the gap.
3. **`segment_reduce` MPS→CPU fallback → MLX-native.** Currently silently on CPU each step;
pure MLX scatter-add removes a real tax in the decode path.
4. **o-voxel hashmap + dual-grid mesh extract.** Keep the Metal/CPU fork if it's not hot; port to
MLX only if profiling says so.
Leave the PBR bake on the existing Metal path (`mtldiffrast`) unless G1-C convicts it.
### A3. Fleet-tune + the prize
- Apply corridorkey ablation moves across the port: head_dim→64, `mx.compile` fixed shapes,
sdpa gating by GPU generation, tiled-vs-full sweeps. Nightly perfcheck like corridorkey.
- **Prize: M1 Ultra compatibility.** MLX-native runs where torch-MPS/bf16 is shaky → a *second*
free TRELLIS box (297 GB/s, 128GB). Validate a full gen on `johnking@100.91.239.7`.
- Wire `trellis_mac`→MLX into MODELBEAST `nodes.json` (m3ultra primary, m1ultra fp16 second) and
add `trellis2` to MESHGOD's `local/` model list beside `hunyuan3d_mlx`.
## G1 · WORKSTREAM B — Hunyuan3D-2.1-MLX optimization (already MLX; just tune)
- It's a *tune*, not a port. Apply the same corridorkey wins to `vendor/hunyuan3d-mlx`:
pad attention head_dim→64, `mx.compile` on fixed-shape blocks (DiT denoiser, VAE), sdpa gating.
- Check `mlx_arsenal` rasterizer (`mesh_render_mlx.py`, 1137 LOC) for the same fixed-shape
compile opportunity. Bench each on the fleet matrix; expect corridorkey-scale wins (1.11.5×).
- Low risk, high certainty — good "warm-up" task while A0 scouting runs.
## G1 · WORKSTREAM C — quality (rung 2 close-out; feeds BOTH MPS + MLX)
- **KDTree-baker A/B** (`TRELLIS2_FORCE_KDTREE=1`, running now). If its surface is clean → the
**Metal baker (mtldiffrast/mtlbvh texel→voxel sampling) is the dark-patch culprit** → fix or
default-swap the baker. If still dirty → decoder voxel attrs are suspect → investigate upstream.
- Keep `TRELLIS2_MAX_BAKE_FACES` (geometry win, free).
- Add a **hole-fill pass** (CuMesh replacement) pre-bake — Blender fill or a voxel-remesh.
- Loud assertion if `flex_gemm` metallib ever fails to load (silent → +90s/gen forever).
---
## G2 · WORKSTREAM D — image preprocessing frontend ("best chance of success")
North star already written in `meshgod/config.py`: *centered · plain neutral light-gray bg ·
even shadowless studio light · neutral symmetric A-pose*. The job = transform any user photo → that.
Model-agnostic; helps TRELLIS.2 AND Hunyuan equally; no fal/network.
### D1. `prep_local()` — the front line (highest ROI, pure glue)
- RMBG-2.0 (local `rmbg` venv) or BiRefNet matte → **composite on neutral gray (NOT alpha**
config's own spec) → bbox crop → center → square-pad → upscale to ≥1024.
- Wire as a LOCAL replacement for MESHGOD's current `clean_bg` (today it's a fal call). Respect the
learned order: **upscale FIRST, matte LAST** (SeedVR strips alpha).
- ⚠️ **Licensing:** RMBG-2.0 is CC-BY-NC → local/personal only. Ship-commercial path = **BiRefNet
(Apache)**. Expose both; default BiRefNet for anything headed to a game.
### D2. Lighting/exposure normalization
- Auto-levels + gray-world WB before recon. TRELLIS bakes light into albedo → even input = cleaner
texture, and directly fights the dark-patch failure mode. Trivial PIL; measurable.
### D3. Multi-view synthesis — the RIGHT fix for touching limbs
- Single-view can't separate an arm fused to a torso in silhouette. Generate back + ¾ views (image
model) → feed **multi-view reconstruction** (both models support it; MESHGOD already wires the MV
path). The extra angle disambiguates. Prototype: photo → gen back view → MV recon → compare.
### D4. Input QA gate (save wasted gens)
- Pre-flight heuristic/VLM (local Ollama VLM or DINOv3-embedding rules) flags too-small, busy-bg,
cropped-limb, multi-subject BEFORE spending a generation. Cheap insurance.
### D5. T-pose normalization = POST-process via MIRPAMO (NOT a preprocess)
- Reposing a 2D human hallucinates. Correct pipeline: reconstruct in-pose → **MIRPAMO auto-rig +
retarget to T/A-pose**. Design-doc it; tie in MOCAPGOD/HSMR (SMPL fit, on m1) as the body prior.
---
## Suggested execution order for the long session
1. **G1-C** close-out (read KDTree result, act) — unblocks the quality question. *(minutes)*
2. **G1-B** Hunyuan tune — certain win, warms up the parity/bench rhythm. *(hours)*
3. **G2-D1+D2** preprocessing front line — compounding ROI on every future gen. *(hours)*
4. **G1-A0** scout MLX ports — the adopt-vs-build decision gate. *(hour)*
5. **G1-A1→A3** the TRELLIS.2 MLX port + kernels — the multi-day main event. *(days)*
6. **G2-D3/D4/D5** as reach goals.
## Definition of done
- TRELLIS.2 runs MLX on BOTH Ultras, faster than today's 124.8s MPS, parity ≤1e-4, in `nodes.json`.
- Hunyuan tuned with a measured fleet-matrix win.
- Dark-patch quality gap root-caused and closed (or a clear upstream bug filed).
- `prep_local()` live as MESHGOD's default local pre-pass; BiRefNet for commercial, multi-view
path proven on a touching-limbs case.
- Every step has a BENCHMARKS.md row. Upstreamable kernel work offered to PR #175.

106
docs/TRELLIS2_MLX_RECON.md Normal file
View File

@ -0,0 +1,106 @@
# TRELLIS.2 on Apple Silicon — recon (2026-07-18)
**Verdict up front:** we already run TRELLIS.2 locally — `vendor/trellis-mac`
(torch-MPS + Metal kernels) IS TRELLIS.2-4B and is wired as the
`trellis_mac` operator. There is no "MLX interface" shortcut for CUDA code
(CUDA kernels must be *rewritten*, not wrapped), but ~90% of TRELLIS.2 is
standard tensor math that MPS/MLX already runs; only four custom CUDA
pieces matter, and the community has replaced all four. The open work is
**speed** (unfused sparse ops ≈ 10× slower than CUDA) and **quality gaps**
(hole-filling disabled, forced pre-simplification) — not feasibility.
## What we run today (`vendor/trellis-mac`, shivampkumar fork)
- torch-MPS + `PYTORCH_ENABLE_MPS_FALLBACK`, Metal kernels by @pedronaugusto
(mtlgemm / mtldiffrast / mtlbvh / mtlmesh), SDPA attention.
- Full inference: sparse structure → shape SLat → tex SLat → decode →
dual-grid mesh extract → simplify → Metal PBR bake → GLB.
- Our benchmark: **318 s/gen on m3ultra**, ~18 GB peak (M4 Pro 24GB: 5m13s
cold at 512). H100 does 317 s — the gap is almost entirely the unfused
sparse conv + padded attention.
- Known gaps vs fal's trellis-2: CuMesh skipped → **no hole filling**,
meshes pre-simplified ~858K→200K faces before baking, Metal BVH
instability, macOS GPU-watchdog can kill long kernels (detected +
workarounds printed by generate.py). macOS 26 needed for the metallib
(fleet is on 26.5 ✓).
- Licensing: DINOv3 is Meta-gated; **RMBG-2.0 preprocessing is CC BY-NC**.
## What TRELLIS.2 actually needs (upstream: Linux, CUDA 12.4, ≥24 GB)
| CUDA dep | Role | Mac status |
|---|---|---|
| o-voxel ext (hash/convert/rasterize) | the O-Voxel representation | pure-Python reimpl (trellis-mac `backends/`) + CPU fork |
| FlexGEMM (**Triton**) | all sparse conv | **the crux** — Triton ≠ Metal; Metal `mtlgemm` or slow pure-torch gather/scatter |
| flash-attn / xformers | 4B flow transformer attention | SDPA (padded → unfused, big cost) |
| CuMesh | decimate/remesh/**hole-fill**/UV | skipped → `fast_simplification`; hole-fill lost |
| nvdiffrast | texture bake | `mtldiffrast` (Metal) |
| nvdiffrec | preview renders only | not needed |
## The MLX question, answered
- **No shim exists or can exist**: CUDA kernels are NVIDIA-machine code;
"an MLX interface" means rewriting each custom op in MLX/Metal. MLX can
express them (`mx.fast.metal_kernel` JIT-compiles Metal from Python) —
but MLX has **no sparse-tensor type and no sparse-voxel precedent**;
SparseTensor/varlen semantics must be hand-rolled. A TRELLIS.2-MLX would
be a first.
- **Our own playbook (proven 2×) is adopt-then-patch, not from-scratch**:
hunyuan3d-mlx = dgrauet's 8.4k-LOC port + our thin packaging layer;
corridorkey-mlx = cmoyates/Niko's 5.3k-LOC port + our **8-line**
`mx.compile`-in-tiled-mode patch = the 1.47× win. From-scratch dense→MLX
ports of this scale are months of solo work (mflux, mlx-video authors).
- **Starting points already exist**: upstream **PR #175** (Jourloy,
2026-07-17 — MPS + Metal + an *experimental `mlx` backend flag*, 28
tests green on M4 Max) and **pedronaugusto/trellis2-apple** (an
`mlx_backend/` dir, no benchmarks yet).
## Recommended ladder (effort-ordered)
1. **Hours — tune what we have**: benchmark `trellis_mac` 1024_cascade vs
fal trellis-2 on identical inputs (BENCHMARKS.md format); route
MESHGOD's batch/overnight work to the local lane (m3ultra clears 18 GB
~14× over; fal stays for interactive one-offs). $93/mo → mostly $0.
2. **Days — adopt + fleet-patch**: vendor PR #175 / trellis2-apple as
`monster/trellis2-*` Gitea forks (house pattern); run the corridorkey
ablation moves on them: attention head_dim → pad to 64 fast-path,
`mx.compile` on fixed shapes, sdpa gating by GPU generation, tiled-vs-
full sweeps. Prize: **M1 Ultra compatibility** (MLX-native, like
hunyuan3d — today trellis is m3-only in practice) = 2nd free 3D box.
3. **Weeks — the real kernel work** (only if we want fal-class speed):
fused Metal sparse-conv (gather-GEMM-scatter) + varlen attention via
`mx.fast.metal_kernel`; port CuMesh hole-filling. Closes most of the
10× gap; genuinely novel, upstreamable to PR #175.
4. **Quality parity misc**: raise simplification budget on 256 GB boxes,
swap RMBG-2.0 → our licensed bg-remove lane, wire `trellis_mac` into
MESHGOD's `local/` model list next to hunyuan3d-mlx.
## Sources
- github.com/microsoft/TRELLIS.2 (setup.sh = dep manifest) · PR #175 ·
issue #74 · shivampkumar/trellis-mac · pedronaugusto/trellis2-apple
- ml-explore.github.io/mlx custom-metal-kernels docs
- Local: HANDOFF_HY3D_MLX.md, CORRIDORKEY.md, BENCHMARKS.md,
vendor/trellis-mac/README, vendor/corridorkey-mlx/prompts/ (the 6-phase
parity-first port template — the blueprint if we ever do ladder step 3)
## First benchmark (2026-07-19, m3ultra, anatomy écorché test image)
| run | tris | time | peak mem | cost |
|---|---|---|---|---|
| trellis_mac 1024-cascade, tex 2048 | 191,336 | **124.8 s** compute (248.8 s wall, 103 s load) | 20.8 GB | $0 |
| fal trellis-2, same image | 468,049 | 298.6 s wall (queue incl.) | — | $0.30 |
**Local already beats fal on wall-clock.** Quality is the gap, not speed:
the local master shows shattered dark patches — CuMesh hole-fill skipped +
forced 858k→191k pre-simplify before baking (the un-simplified master IS
saved as .obj alongside the GLB). Fix quality first, then fuse kernels.
Tri-budget ladder (Blender decimate on the fal master): 2k = confetti
(ratio floors at 0.01 → 4,674 min), 8k = torn, 30k = good, 50k ≈ master.
**Écorché-class meshes (many disconnected thin shells) cannot pure-decimate
below ~30k** — game budgets need the MESHGOD solidify/remesh route.
Discovered in passing: MESHGOD finish_glb.py's tri budget is silently
ignored on Blender 5 (modifier_apply cancels — fix task spawned 2026-07-19).
Bench artifacts: m3ultra ~/Documents/trellis2-bench/ (masters, sweeps,
renders, logs, sweep2.csv / sweep_fal.csv).

25
scripts/install_qwen_layered.sh Executable file
View File

@ -0,0 +1,25 @@
#!/bin/bash
# Install the Qwen-Image-Layered MLX runtime (mflux PR#302 fork, vendored at
# our Gitea) into vendor/mflux-qwen-layered + a uv venv. Idempotent.
#
# Model weights are NOT fetched here. run.py resolves, in order:
# $QWEN_LAYERED_MODEL -> ~/qwen-layered/qwen-layered-q8 (m3)
# -> ~/qwen-layered-staging/qwen-layered-q8 (m1)
# -> HF Qwen/Qwen-Image-Layered with on-the-fly -q8 (54GB download).
# Bake a local q8 once per box: .venv/bin/mflux-save \
# --model Qwen/Qwen-Image-Layered --base-model qwen-image-layered \
# --quantize 8 --path ~/qwen-layered/qwen-layered-q8
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
DEST="$ROOT/vendor/mflux-qwen-layered"
REPO="ssh://git@100.71.119.27:222/monster/Qwen-Image-Layered-MRP-MLX.git"
if [ ! -d "$DEST/.git" ]; then
git clone "$REPO" "$DEST"
else
echo "already cloned: $DEST"
fi
cd "$DEST"
/opt/homebrew/bin/uv venv .venv --python 3.12
/opt/homebrew/bin/uv pip install -q --python .venv/bin/python -e .
.venv/bin/mflux-generate-qwen-layered --help >/dev/null && echo "qwen-layered OK: $DEST"

20
scripts/install_trellis2_mlx.sh Executable file
View File

@ -0,0 +1,20 @@
#!/bin/bash
# Vendor the MRP MLX fork of TRELLIS.2 (monster/trellis-2-mrp-mlx) for the
# trellis2_mlx operator. Idempotent. Existing per-box clones also work
# (run.py resolves m3 staging / m1 ~/trellis2-mlx automatically) — this
# script is for fresh boxes.
# Weights: TRELLIS.2-4B + gated facebook/dinov3 + briaai/RMBG-2.0 — rsync
# from m3ultra's HF cache (no HF login):
# for M in models--facebook--dinov3-vitl16-pretrain-lvd1689m \
# models--microsoft--TRELLIS.2-4B models--briaai--RMBG-2.0; do
# rsync -a m3ultra@100.89.131.57:.cache/huggingface/hub/$M ~/.cache/huggingface/hub/
# done
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
DEST="$ROOT/vendor/trellis2-mlx"
REPO="ssh://git@100.71.119.27:222/monster/trellis-2-mrp-mlx.git"
[ -d "$DEST/.git" ] && echo "already cloned: $DEST" || git clone "$REPO" "$DEST"
cd "$DEST"
PB="${PYTHON_BIN:-$(/opt/homebrew/bin/uv python find 3.11)}"
PYTHON_BIN="$PB" bash scripts/setup_macos.sh
.venv/bin/python -c "import mlx_backend" && echo "trellis2-mlx OK: $DEST"

90
scripts/mlxprobe.py Normal file
View File

@ -0,0 +1,90 @@
"""MLX fleet probe — one JSON line of capability + micro-bench per machine.
Benches (each ~seconds, sized for 16GB boxes):
matmul fp16/bf16 4096^2 -> TFLOPS (bf16-on-M1 question)
elementwise add 256MB -> GB/s (memory bandwidth proxy)
sdpa head_dim 56 vs 64 -> ratio (fused fast-path check, corridorkey lesson)
"""
import json
import platform
import subprocess
import time
R = {"host": platform.node().split(".")[0]}
try:
R["chip"] = subprocess.check_output(
["sysctl", "-n", "machdep.cpu.brand_string"], text=True).strip()
R["ram_gb"] = round(int(subprocess.check_output(
["sysctl", "-n", "hw.memsize"], text=True)) / 2**30)
R["macos"] = subprocess.check_output(
["sw_vers", "-productVersion"], text=True).strip()
except Exception:
pass
try:
import mlx.core as mx
R["mlx"] = getattr(mx, "__version__", "?")
except Exception as e:
R["mlx"] = None
R["error"] = f"mlx import failed: {e}"
print(json.dumps(R))
raise SystemExit(0)
try:
di = mx.metal.device_info() if hasattr(mx, "metal") else {}
R["gpu"] = {k: di[k] for k in ("architecture", "max_recommended_working_set_size",
"memory_size") if k in di}
except Exception:
pass
def timeit(fn, warmup=3, iters=10):
for _ in range(warmup):
mx.eval(fn())
t0 = time.perf_counter()
for _ in range(iters):
mx.eval(fn())
return (time.perf_counter() - t0) / iters
N = 4096
flops = 2 * N * N * N
for dt, name in ((mx.float16, "fp16"), (mx.bfloat16, "bf16")):
try:
a = mx.random.normal((N, N)).astype(dt)
b = mx.random.normal((N, N)).astype(dt)
mx.eval(a, b)
t = timeit(lambda: a @ b)
R[f"matmul_{name}_tflops"] = round(flops / t / 1e12, 2)
except Exception as e:
R[f"matmul_{name}_tflops"] = f"fail: {e}"
try:
M = 64 * 1024 * 1024 # 64M floats = 256MB per array
x = mx.random.normal((M,))
y = mx.random.normal((M,))
mx.eval(x, y)
t = timeit(lambda: x + y, warmup=2, iters=8)
R["bandwidth_gbs"] = round(3 * M * 4 / t / 1e9) # 2 reads + 1 write, fp32
except Exception as e:
R["bandwidth_gbs"] = f"fail: {e}"
try:
B, H, L = 1, 8, 2048
sdpa = {}
for D in (56, 64):
q = mx.random.normal((B, H, L, D)).astype(mx.float16)
k = mx.random.normal((B, H, L, D)).astype(mx.float16)
v = mx.random.normal((B, H, L, D)).astype(mx.float16)
mx.eval(q, k, v)
t = timeit(lambda: mx.fast.scaled_dot_product_attention(
q, k, v, scale=D ** -0.5), warmup=3, iters=20)
sdpa[f"d{D}_us"] = round(t * 1e6)
sdpa["d56_vs_d64"] = round(sdpa["d56_us"] / max(sdpa["d64_us"], 1), 2)
R["sdpa"] = sdpa
except Exception as e:
R["sdpa"] = f"fail: {e}"
R["peak_mem_gb"] = round(mx.get_peak_memory() / 2**30, 2) \
if hasattr(mx, "get_peak_memory") else None
print(json.dumps(R))

View File

@ -2,19 +2,43 @@
"id": "mflux_image_edit",
"name": "Edit Image (local, Qwen)",
"category": "image-prep",
"description": "Image + instruction → edited image entirely on this Mac via mflux Qwen-Image-Edit (ungated/Apache-2.0). Free local alternative to fal_image_edit / nano-banana edit — 'remove the sticker', 'make it studio-lit on white', material/lighting changes. First run downloads weights.",
"accepts": ["image"],
"produces": ["image"],
"description": "1-3 images + instruction \u2192 edited image (multi-image composes: 'put the shirt from image 2 on the person in image 1') entirely on this Mac via mflux Qwen-Image-Edit (ungated/Apache-2.0). Free local alternative to fal_image_edit / nano-banana edit \u2014 'remove the sticker', 'make it studio-lit on white', material/lighting changes. First run downloads weights.",
"accepts": [
"image"
],
"produces": [
"image"
],
"resources": "gpu",
"entry": "run.py",
"python": "venvs/mflux/bin/python",
"params_schema": {
"type": "object",
"properties": {
"prompt": {"type": "string", "default": "", "description": "Edit instruction (what to change)"},
"steps": {"type": "integer", "default": 25, "minimum": 1, "maximum": 50, "description": "Inference steps"},
"seed": {"type": "integer", "default": 42, "description": "Random seed"},
"guidance": {"type": "number", "default": 4.0, "minimum": 0, "maximum": 10, "description": "Guidance scale"}
"prompt": {
"type": "string",
"default": "",
"description": "Edit instruction (what to change)"
},
"steps": {
"type": "integer",
"default": 25,
"minimum": 1,
"maximum": 50,
"description": "Inference steps"
},
"seed": {
"type": "integer",
"default": 42,
"description": "Random seed"
},
"guidance": {
"type": "number",
"default": 4.0,
"minimum": 0,
"maximum": 10,
"description": "Guidance scale"
}
}
}
}
}

View File

@ -25,7 +25,9 @@ if not cli.exists():
src = Path(a.input[0])
outdir = Path(a.outdir)
out = outdir / f"{src.stem}_edited.png"
cmd = [str(cli), "--image-paths", str(src.resolve()),
# ALL inputs go to Qwen (it composes up to ~3 images: "put the shirt from image 2 on the person
# in image 1"). Single-image jobs behave exactly as before.
cmd = [str(cli), "--image-paths", *[str(Path(x).resolve()) for x in a.input],
"--prompt", p["prompt"],
"--steps", str(p.get("steps", 25)),
"--seed", str(p.get("seed", 42)),

View File

@ -0,0 +1,21 @@
{
"id": "qwen_layered_local",
"name": "Qwen-Image-Layered (local)",
"category": "image-edit",
"description": "Image → N editable RGBA layers (bg / subject / detail separation), fully local via the Qwen-Image-Layered 20B MLX port (mflux PR#302 fork). ~4 min on m3ultra, ~6 min on m1ultra at the tuned defaults (20 steps, 640, baked q8). The subject layer doubles as matting-grade background removal. Bake a local q8 model once per box (see scripts/install_qwen_layered.sh) or first run falls back to a 54GB HF download + on-the-fly quantization.",
"accepts": ["image"],
"produces": ["image"],
"resources": "gpu",
"entry": "run.py",
"python": "vendor/mflux-qwen-layered/.venv/bin/python",
"params_schema": {
"type": "object",
"properties": {
"layers": {"type": "integer", "default": 4, "description": "Number of RGBA layers to decompose into"},
"steps": {"type": "integer", "default": 20, "description": "Denoising steps (20 = tuned default; 50 = upstream default, ~2.9x slower, no visible gain on tested inputs)"},
"resolution": {"type": "integer", "enum": [640, 1024], "default": 640, "description": "Working resolution"},
"seed": {"type": "integer", "default": 0, "description": "Random seed"},
"prompt": {"type": "string", "default": "", "description": "Optional prompt to guide the decomposition"}
}
}
}

View File

@ -0,0 +1,71 @@
import argparse
import json
import os
import subprocess
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[3]
VENDOR = ROOT / "vendor" / "mflux-qwen-layered"
CLI = VENDOR / ".venv" / "bin" / "mflux-generate-qwen-layered"
ap = argparse.ArgumentParser()
ap.add_argument("--input", action="append", default=[])
ap.add_argument("--outdir", required=True)
ap.add_argument("--params", default="{}")
a = ap.parse_args()
p = json.loads(a.params)
if not a.input:
print("ERROR: no input image")
sys.exit(1)
if not CLI.exists():
print(f"ERROR: qwen-layered not installed at {VENDOR}. "
"Run scripts/install_qwen_layered.sh")
sys.exit(1)
# Baked-model resolution: env override, then per-box conventional paths,
# else fall back to the HF repo with on-the-fly q8 (slow first run: 54GB).
candidates = [os.environ.get("QWEN_LAYERED_MODEL", "")]
candidates += [str(Path.home() / "qwen-layered" / "qwen-layered-q8"),
str(Path.home() / "qwen-layered-staging" / "qwen-layered-q8")]
model_path = next((c for c in candidates if c and Path(c).is_dir()), None)
outdir = Path(a.outdir)
cmd = [
str(CLI), "--image", str(Path(a.input[0]).resolve()),
"--layers", str(p.get("layers", 4)),
"--steps", str(p.get("steps", 20)),
"--resolution", str(p.get("resolution", 640)),
"--seed", str(p.get("seed", 0)),
"--output-dir", str(outdir.resolve()),
]
if model_path:
cmd += ["--model-path", model_path]
print(f"using baked model: {model_path}", flush=True)
else:
cmd += ["-q", "8"]
print("no baked model found — HF fallback with on-the-fly q8 "
"(first run downloads 54GB)", flush=True)
if p.get("prompt"):
cmd += ["--prompt", str(p["prompt"])]
env = os.environ.copy()
env["HF_HUB_DISABLE_XET"] = "1"
print("+", " ".join(cmd), flush=True)
res = subprocess.run(cmd, cwd=str(VENDOR), stdout=sys.stdout,
stderr=subprocess.STDOUT, env=env)
if res.returncode != 0:
sys.exit(res.returncode)
layers = sorted(outdir.glob("layer_*.png"))
if not layers:
print("ERROR: qwen-layered produced no layer PNGs")
sys.exit(1)
stem = Path(a.input[0]).stem
outputs = [{"path": str(f), "name": f"{stem}_{f.stem}.png",
"meta": {"tool": "qwen_layered_local", "layer": i}}
for i, f in enumerate(layers)]
(outdir / "result.json").write_text(json.dumps({"outputs": outputs}))
print(f"done: {len(layers)} layers", flush=True)

View File

@ -0,0 +1,26 @@
{
"id": "sd_local",
"name": "SD1.5 + anatomy LoRAs (local, primary-only)",
"category": "generate",
"description": "Prompt → image via the repose stack on the primary: SD1.5 Hyper_Realism + John's anatomy LoRAs (~/Documents/repose/models/lora) + optional OpenPose T-pose template (front/back/side) + optional IP-Adapter reference image. The no-cloud, no-content-filter lane FLUX can't cover — LoRA-styled anatomy, exact pose control, reference-guided identity. Primary-only: needs ~/Documents/repose (absolute venv path keeps it off remote dispatch by design).",
"accepts": ["image"],
"produces": ["image"],
"resources": "gpu",
"entry": "run.py",
"python": "/Users/m3ultra/Documents/repose/venv/bin/python",
"params_schema": {
"type": "object",
"properties": {
"prompt": {"type": "string", "default": "", "description": "What to generate"},
"negative": {"type": "string", "default": "deformed, extra limbs, extra fingers, mutated hands, missing limbs, cropped, blurry, lowres, watermark, text, multiple people, child", "description": "Negative prompt"},
"loras": {"type": "array", "items": {"type": "string"}, "default": [], "description": "'stem=weight' from repose/models/lora, e.g. 'GodPussy1 v4=0.6'"},
"pose": {"type": "string", "enum": ["none", "front", "back", "side"], "default": "none", "description": "OpenPose T-pose template via ControlNet"},
"ip": {"type": "number", "default": 0.7, "description": "IP-Adapter strength when a reference image is attached"},
"steps": {"type": "integer", "default": 28, "minimum": 1, "maximum": 60},
"cfg": {"type": "number", "default": 7.0},
"seed": {"type": "integer", "default": 7},
"width": {"type": "integer", "default": 512, "description": "SD1.5 native ≈512-768; upscale after via seedvr2"},
"height": {"type": "integer", "default": 768}
}
}
}

View File

@ -0,0 +1,97 @@
"""sd_local — SD1.5 Hyper_Realism + anatomy LoRAs (+ optional pose ControlNet / IP-Adapter ref).
Runs UNDER the repose venv (manifest python is that venv, absolute primary-only).
Generalizes scripts/repose.py from MESHGOD: pose template optional, plain txt2img otherwise.
"""
import argparse
import json
import os
import sys
os.environ.setdefault("PYTORCH_ENABLE_MPS_FALLBACK", "1")
import torch # noqa: E402
from diffusers import DPMSolverMultistepScheduler # noqa: E402
from PIL import Image # noqa: E402
ap = argparse.ArgumentParser()
ap.add_argument("--input", action="append", default=[])
ap.add_argument("--outdir", required=True)
ap.add_argument("--params", default="{}")
a = ap.parse_args()
p = json.loads(a.params)
HERE = os.path.expanduser("~/Documents/repose")
BASE = os.path.join(HERE, "models", "Hyper_Realism_1.2_fp16.safetensors")
# LoRA search path: John's master dir first, the repose copy second (SD_LORA_DIRS to override)
LORA_DIRS = [os.path.expanduser(d) for d in os.environ.get(
"SD_LORA_DIRS", "~/Documents/localmodels/Lora:~/Documents/repose/models/lora").split(":")]
def find_lora(stem):
for d in LORA_DIRS:
cand = os.path.join(d, stem + ".safetensors")
if os.path.exists(cand):
return cand
return None
if not os.path.exists(BASE):
print(f"ERROR: {BASE} missing — sd_local runs on the primary only")
sys.exit(1)
dev = "mps" if torch.backends.mps.is_available() else "cpu"
dtype = torch.float16 if dev == "mps" else torch.float32
pose = p.get("pose", "none")
kw = {}
if pose != "none":
from diffusers import ControlNetModel, StableDiffusionControlNetPipeline
tpl = os.path.join(HERE, f"tpose_{pose}.png")
cn = ControlNetModel.from_pretrained("lllyasviel/control_v11p_sd15_openpose", torch_dtype=dtype)
pipe = StableDiffusionControlNetPipeline.from_single_file(
BASE, controlnet=cn, torch_dtype=dtype, safety_checker=None, requires_safety_checker=False)
kw["image"] = Image.open(tpl).convert("RGB")
else:
from diffusers import StableDiffusionPipeline
pipe = StableDiffusionPipeline.from_single_file(
BASE, torch_dtype=dtype, safety_checker=None, requires_safety_checker=False)
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config, use_karras_sigmas=True)
pipe = pipe.to(dev)
# no attention slicing — it crashes IP-Adapter attn-processor injection (see repose.py)
if p.get("loras"):
from safetensors.torch import load_file
names, weights = [], []
for spec in p["loras"]:
stem, _, w = str(spec).partition("=")
path = find_lora(stem.strip())
if not path:
print(f"[sd] LORA MISS '{stem}' — searched {LORA_DIRS}", flush=True)
continue
# UNet-only: this diffusers build trips an IndexError on these kohya files' text-encoder
# half (empty rank_dict in load_lora_into_text_encoder). The UNet half carries the look.
sd_l = load_file(path)
unet_only = {k: v for k, v in sd_l.items() if not k.startswith("lora_te")}
pipe.load_lora_weights(unet_only, adapter_name=stem.replace(" ", "_"))
names.append(stem.replace(" ", "_"))
weights.append(float(w or 0.6))
if names:
pipe.set_adapters(names, weights)
print(f"[sd] loras: {dict(zip(names, weights))}", flush=True)
if a.input: # reference image → IP-Adapter
pipe.load_ip_adapter("h94/IP-Adapter", subfolder="models", weight_name="ip-adapter_sd15.bin")
pipe.set_ip_adapter_scale(float(p.get("ip", 0.7)))
kw["ip_adapter_image"] = Image.open(a.input[0]).convert("RGB")
print(f"[sd] ip-adapter ref @ {p.get('ip', 0.7)}", flush=True)
gen = torch.Generator("cpu").manual_seed(int(p.get("seed", 7)))
img = pipe(prompt=p.get("prompt", ""), negative_prompt=p.get("negative", ""),
num_inference_steps=int(p.get("steps", 28)), guidance_scale=float(p.get("cfg", 7.0)),
generator=gen, width=int(p.get("width", 512)), height=int(p.get("height", 768)),
**kw).images[0]
out = os.path.join(a.outdir, "sd_local.png")
img.save(out)
json.dump({"outputs": [{"path": "sd_local.png", "meta": {"tool": "sd_local"}}]},
open(os.path.join(a.outdir, "result.json"), "w"))
print("done: sd_local.png", flush=True)

View File

@ -0,0 +1,20 @@
{
"id": "stereo_depth",
"name": "Stereo SBS → depth + point cloud (local)",
"category": "video-prep",
"description": "Side-by-side stereoscopic video (or one SBS frame) → per-frame disparity/depth maps via OpenCV StereoSGBM + a colored .ply point cloud of the middle frame for Blender. TRUE geometric depth from the L/R eye shift — no AI hallucination. Uncalibrated SBS rips aren't rectified, so expect relief-map depth (great for Displace-modifier planes), not survey-grade geometry. Output: depth preview mp4 + mid-frame .ply + zipped depth PNGs.",
"accepts": ["video", "image"],
"produces": ["archive"],
"resources": "cpu",
"entry": "run.py",
"python": "venvs/stereo/bin/python",
"params_schema": {
"type": "object",
"properties": {
"max_frames": {"type": "integer", "default": 120, "description": "Cap processed frames"},
"num_disp": {"type": "integer", "default": 128, "description": "Disparity search range (multiple of 16; raise for close subjects)"},
"block": {"type": "integer", "default": 5, "description": "SGBM block size (odd, 3-11)"},
"layout": {"type": "string", "enum": ["sbs", "tab"], "default": "sbs", "description": "side-by-side or top-and-bottom stereo"}
}
}
}

View File

@ -0,0 +1,112 @@
"""stereo_depth — SBS stereoscopic video/frame → SGBM disparity maps + mid-frame point cloud.
True geometric depth from the built-in L/R parallax (cv2.StereoSGBM). ffmpeg (system)
decodes/splits; depth PNGs + preview mp4 + a colored .ply land as the outputs.
ponytail: uncalibrated disparity (SBS rips have no camera intrinsics) z is relative,
perfect for Blender displace planes; not metric reconstruction.
"""
import argparse
import glob
import json
import os
import shutil
import subprocess
import sys
import tempfile
import zipfile
import cv2
import numpy as np
FFMPEG = shutil.which("ffmpeg") or "/opt/homebrew/bin/ffmpeg" # headless ssh has no brew PATH
ap = argparse.ArgumentParser()
ap.add_argument("--input", action="append", default=[])
ap.add_argument("--outdir", required=True)
ap.add_argument("--params", default="{}")
a = ap.parse_args()
p = json.loads(a.params)
if not a.input:
print("ERROR: no input")
sys.exit(1)
src = a.input[0]
outdir = a.outdir
max_frames = int(p.get("max_frames", 120))
num_disp = max(16, (int(p.get("num_disp", 128)) // 16) * 16)
block = int(p.get("block", 5)) | 1
tab = p.get("layout", "sbs") == "tab"
work = tempfile.mkdtemp(prefix="stereo_")
frames_dir = os.path.join(work, "frames")
depth_dir = os.path.join(work, "depth")
os.makedirs(frames_dir)
os.makedirs(depth_dir)
# decode → frames (a single image just becomes frame 1)
subprocess.run([FFMPEG, "-y", "-loglevel", "error", "-i", src,
"-frames:v", str(max_frames), os.path.join(frames_dir, "f%05d.png")], check=True)
frames = sorted(glob.glob(os.path.join(frames_dir, "*.png")))
print(f"[stereo] {len(frames)} frames, num_disp={num_disp} block={block}", flush=True)
min_disp = -(num_disp // 2) # movie stereo converges on the screen plane — content
sgbm = cv2.StereoSGBM_create( # lives BOTH sides of it, so the window must go negative
minDisparity=min_disp, numDisparities=num_disp, blockSize=block,
P1=8 * 3 * block * block, P2=32 * 3 * block * block,
disp12MaxDiff=1, uniquenessRatio=10, speckleWindowSize=100, speckleRange=2,
mode=cv2.STEREO_SGBM_MODE_SGBM_3WAY)
mid = frames[len(frames) // 2]
for i, f in enumerate(frames):
img = cv2.imread(f)
h, w = img.shape[:2]
if tab:
L, R = img[: h // 2], img[h // 2:]
else:
L, R = img[:, : w // 2], img[:, w // 2:]
gl, gr = cv2.cvtColor(L, cv2.COLOR_BGR2GRAY), cv2.cvtColor(R, cv2.COLOR_BGR2GRAY)
disp = sgbm.compute(gl, gr).astype(np.float32) / 16.0
valid = disp > (min_disp - 0.5) # cv2 marks unmatched pixels min_disp-1
if valid.sum() > 100:
lo, hi = np.percentile(disp[valid], [2, 98])
else:
lo, hi = 0.0, 1.0
norm = np.zeros(disp.shape, np.uint8)
norm[valid] = (np.clip((disp[valid] - lo) / max(1e-6, hi - lo), 0, 1) * 255).astype(np.uint8)
cv2.imwrite(os.path.join(depth_dir, os.path.basename(f)), norm)
if f == mid: # colored point cloud of the middle frame
ys, xs = np.where(valid & (norm > 2))
step = max(1, len(ys) // 400_000) # cap cloud size
ys, xs = ys[::step], xs[::step]
z = disp[ys, xs]
rgb = L[ys, xs][:, ::-1] # BGR→RGB
ply = os.path.join(outdir, "midframe_cloud.ply")
with open(ply, "w") as fh:
fh.write("ply\nformat ascii 1.0\n"
f"element vertex {len(ys)}\n"
"property float x\nproperty float y\nproperty float z\n"
"property uchar red\nproperty uchar green\nproperty uchar blue\nend_header\n")
for (yy, xx, zz, (r, g, b)) in zip(ys, xs, z, rgb):
fh.write(f"{xx} {-yy} {zz:.1f} {r} {g} {b}\n")
print(f"[stereo] wrote midframe_cloud.ply ({len(ys)} pts)", flush=True)
if i % 24 == 0:
print(f"[stereo] frame {i + 1}/{len(frames)}", flush=True)
outputs = [{"path": "midframe_cloud.ply", "meta": {"tool": "stereo_sgbm"}}]
if len(frames) > 1: # depth preview video
prev = os.path.join(outdir, "depth_preview.mp4")
subprocess.run([FFMPEG, "-y", "-loglevel", "error", "-framerate", "24",
"-i", os.path.join(depth_dir, "f%05d.png"),
"-pix_fmt", "yuv420p", prev], check=True)
outputs.append({"path": "depth_preview.mp4", "meta": {"tool": "stereo_sgbm"}})
zpath = os.path.join(outdir, "depth_frames.zip")
with zipfile.ZipFile(zpath, "w", zipfile.ZIP_DEFLATED) as zf:
for f in sorted(glob.glob(os.path.join(depth_dir, "*.png"))):
zf.write(f, os.path.basename(f))
outputs.append({"path": "depth_frames.zip", "meta": {"tool": "stereo_sgbm"}})
json.dump({"outputs": outputs}, open(os.path.join(outdir, "result.json"), "w"))
shutil.rmtree(work, ignore_errors=True)
print("done:", ", ".join(o["path"] for o in outputs), flush=True)

View File

@ -0,0 +1,79 @@
{
"id": "trellis2_mlx",
"name": "TRELLIS.2 MLX (local, fastest)",
"category": "mesh-gen",
"description": "Image → PBR GLB via the MRP MLX fork of TRELLIS.2 (monster/trellis-2-mrp-mlx): pure-MLX sampler, Metal PBR bake with RAM-sized face cap + forced-opaque alpha (2026-07-22 quality fixes — fal-parity detail), kdtree/vertex fallbacks. ~72s vertex / ~2min metal-PBR on m3ultra — still the fastest local TRELLIS.2. baker=vertex for the 1s clean-albedo bake (no MR maps). Weights: TRELLIS.2-4B + gated DINOv3/RMBG (rsync from m3 HF cache to workers; no HF login needed).",
"accepts": [
"image"
],
"produces": [
"model"
],
"resources": "gpu",
"entry": "run.py",
"params_schema": {
"type": "object",
"properties": {
"pipeline_type": {
"type": "string",
"enum": [
"512",
"1024",
"1024_cascade"
],
"default": "1024_cascade",
"description": "Resolution tier"
},
"seed": {
"type": "integer",
"default": 0,
"description": "Random seed"
},
"steps": {
"type": "integer",
"default": 12,
"description": "Sampler steps per stage (12 = upstream default)"
},
"baker": {
"type": "string",
"enum": [
"metal",
"vertex",
"kdtree"
],
"default": "metal",
"description": "metal = UV PBR textures (meshgod-grade; minor dark-patch texels remain); vertex = 1s bake, cleanest colors, no MR maps"
},
"texture_size": {
"type": "integer",
"enum": [
512,
1024,
2048,
4096
],
"default": 2048,
"description": "Baked texture resolution (metal/kdtree bakers)"
},
"max_bake_faces": {
"type": "integer",
"enum": [
200000,
500000,
1000000
],
"default": 500000,
"description": "Pre-bake face cap. 500k ≈ fal parity (M3 Ultra verified); node RAM sizes the default when unset — laptops get 200k"
},
"alpha_mode": {
"type": "string",
"enum": [
"opaque",
"auto"
],
"default": "opaque",
"description": "opaque = force alphaMode OPAQUE (fixes speckle-veil hair); auto = keep baked alpha for glassy subjects"
}
}
}
}

View File

@ -0,0 +1,86 @@
import argparse
import json
import os
import subprocess
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[3]
# fork checkout resolution: vendored path first, then the per-box conventional
# clones (m3 staging, m1 home) so existing installs work without re-vendoring.
CANDIDATES = [
ROOT / "vendor" / "trellis2-mlx",
Path.home() / "Documents" / "trellis2-mlx-staging" / "trellis2-jourloy",
Path.home() / "trellis2-mlx",
]
FORK = next((c for c in CANDIDATES if (c / "scripts" / "generate_asset.py").exists()), None)
ap = argparse.ArgumentParser()
ap.add_argument("--input", action="append", default=[])
ap.add_argument("--outdir", required=True)
ap.add_argument("--params", default="{}")
a = ap.parse_args()
p = json.loads(a.params)
if not a.input:
print("ERROR: no input image")
sys.exit(1)
if FORK is None:
print("ERROR: trellis-2-mrp-mlx not found. Run scripts/install_trellis2_mlx.sh")
sys.exit(1)
py = FORK / ".venv" / "bin" / "python"
if not py.exists():
print(f"ERROR: no venv at {FORK}/.venv — run its scripts/setup_macos.sh")
sys.exit(1)
outdir = Path(a.outdir)
# Meshgod-grade PBR by default (metal baker + capped bake), matching the
# trellis_mac lane's 2026-07-22 quality fixes. The fork's schedule bakes the
# FULL-density mesh when no target is passed (75GB peak, 131MB GLB) — always
# pass an explicit cap; RAM-sized default like trellis_mac (500k ≈ fal parity
# on ≥96GB Studios, 200k mtlbvh-safe elsewhere).
faces = p.get("max_bake_faces")
if faces is None:
mem_gb = os.sysconf("SC_PAGE_SIZE") * os.sysconf("SC_PHYS_PAGES") / 1e9
faces = 500000 if mem_gb >= 96 else 200000
cmd = [
str(py), str(FORK / "scripts" / "generate_asset.py"),
str(Path(a.input[0]).resolve()),
"--output-dir", str(outdir.resolve()),
"--backend", "mlx-experimental",
"--baker", str(p.get("baker", "metal")),
"--pipeline-type", str(p.get("pipeline_type", "1024_cascade")),
"--seed", str(p.get("seed", 0)),
"--texture-size", str(p.get("texture_size", 2048)),
"--pbr-decimation-target", str(faces),
"--force",
]
if p.get("steps"):
cmd += ["--steps", str(p["steps"])]
env = os.environ.copy()
env["HF_HUB_DISABLE_XET"] = "1"
# "auto" keeps o_voxel's baked-alpha BLEND detection (glass etc.); default
# opaque — noisy baked alpha renders solid hair as a speckle veil.
if p.get("alpha_mode"):
env["TRELLIS2_ALPHA_MODE"] = str(p["alpha_mode"])
print("+", " ".join(cmd), flush=True)
res = subprocess.run(cmd, cwd=str(FORK), stdout=sys.stdout,
stderr=subprocess.STDOUT, env=env)
if res.returncode != 0:
sys.exit(res.returncode)
glbs = sorted(outdir.rglob("candidate_pbr.glb")) or sorted(outdir.rglob("*.glb"))
if not glbs:
print("ERROR: no GLB produced")
sys.exit(1)
stem = Path(a.input[0]).stem
outputs = [{"path": str(glbs[0]), "name": f"trellis2mlx_{stem}.glb",
"meta": {"tool": "trellis2_mlx", "baker": p.get("baker", "vertex")}}]
(outdir / "result.json").write_text(json.dumps({"outputs": outputs}))
print("done:", glbs[0], flush=True)

View File

@ -3,18 +3,80 @@
"name": "TRELLIS.2 (local, SOTA)",
"category": "mesh-gen",
"description": "Image → GLB + baked PBR locally via the TRELLIS.2 MPS/Metal port (best open-weights quality on Apple Silicon, ~3-5 min/gen). Weights need a one-time HuggingFace login (facebook/dinov3-vitl16, briaai/RMBG-2.0). Note: 1536 is upstream-only; Mac port maxes at 1024_cascade.",
"accepts": ["image"],
"produces": ["model"],
"accepts": [
"image"
],
"produces": [
"model"
],
"resources": "gpu",
"entry": "run.py",
"python": "vendor/trellis-mac/.venv/bin/python",
"params_schema": {
"type": "object",
"properties": {
"pipeline_type": {"type": "string", "enum": ["512", "1024", "1024_cascade"], "default": "1024", "description": "Resolution tier (1024_cascade = highest quality on Mac)"},
"texture_size": {"type": "integer", "enum": [512, 1024, 2048], "default": 2048, "description": "Baked texture resolution"},
"seed": {"type": "integer", "default": 0, "description": "Random seed"},
"no_texture": {"type": "boolean", "default": false, "description": "Geometry only (skip texture stage, much faster)"}
"pipeline_type": {
"type": "string",
"enum": [
"512",
"1024",
"1024_cascade"
],
"default": "1024",
"description": "Resolution tier (1024_cascade = highest quality on Mac)"
},
"texture_size": {
"type": "integer",
"enum": [
512,
1024,
2048
],
"default": 2048,
"description": "Baked texture resolution"
},
"seed": {
"type": "integer",
"default": 0,
"description": "Random seed"
},
"no_texture": {
"type": "boolean",
"default": false,
"description": "Geometry only (skip texture stage, much faster)"
},
"max_bake_faces": {
"type": "integer",
"enum": [
200000,
500000,
1000000
],
"default": 500000,
"description": "Pre-bake simplify cap. 500k ≈ fal parity, verified crash-free on the M3 Ultra (2026-07-22); drop to 200k on laptops (mtlbvh guard)"
},
"steps": {
"type": "integer",
"default": 0,
"description": "Sampler steps override for all 3 flow phases (0 = pipeline default, 12)"
},
"baker": {
"type": "string",
"enum": [
"",
"vertex"
],
"default": "",
"description": "'vertex' = KDTree vertex-color bake (dark-patch workaround, no PBR maps)"
},
"alpha_mode": {
"type": "string",
"enum": [
"opaque",
"auto"
],
"default": "opaque",
"description": "opaque = force alphaMode OPAQUE in the GLB (fixes speckle-veil hair from noisy baked alpha); auto = keep o_voxel's BLEND detection for glassy subjects"
}
}
}
}

View File

@ -21,12 +21,18 @@ if not a.input:
if not (VENDOR / "generate.py").exists():
print(f"ERROR: trellis-mac not installed at {VENDOR}. Run scripts/install_trellis_mac.sh")
sys.exit(1)
# manifest no longer pins "python" (broke remote dispatch when the venv path
# didn't exist on the worker) — resolve the vendor venv here instead.
VENV_PY = VENDOR / ".venv" / "bin" / "python"
if not VENV_PY.exists():
print(f"ERROR: no venv at {VENV_PY} — run scripts/install_trellis_mac.sh")
sys.exit(1)
outdir = Path(a.outdir)
out_stem = outdir / f"trellis2_{Path(a.input[0]).stem}"
cmd = [
sys.executable, "generate.py", str(Path(a.input[0]).resolve()),
str(VENV_PY), "generate.py", str(Path(a.input[0]).resolve()),
"--pipeline-type", str(p.get("pipeline_type", "1024")),
"--texture-size", str(p.get("texture_size", 2048)),
"--seed", str(p.get("seed", 0)),
@ -34,6 +40,8 @@ cmd = [
]
if p.get("no_texture"):
cmd.append("--no-texture")
if p.get("steps"):
cmd += ["--steps", str(p["steps"])]
env = os.environ.copy()
# xet chunked downloader fails on these BFL/Meta repos ("Unable to parse string
@ -43,6 +51,22 @@ env["HF_HUB_DISABLE_XET"] = "1"
env["KMP_DUPLICATE_LIB_OK"] = "TRUE"
env.setdefault("OMP_NUM_THREADS", "1")
env.setdefault("MKL_NUM_THREADS", "1")
# Pre-bake simplify cap. The 200k generate.py default is a laptop-derived
# mtlbvh guard that costs most of the mesh detail (2.2M-tri decode → 95k GLB
# vs fal's 495k). 500k verified crash-free on the M3 Ultra Studio 2026-07-22.
# The server passes only client-sent params (manifest defaults are UI-only),
# so default here, sized by node RAM: big-memory Studios get the quality cap.
if p.get("max_bake_faces") is not None:
env["TRELLIS2_MAX_BAKE_FACES"] = str(p["max_bake_faces"])
else:
mem_gb = os.sysconf("SC_PAGE_SIZE") * os.sysconf("SC_PHYS_PAGES") / 1e9
env.setdefault("TRELLIS2_MAX_BAKE_FACES", "500000" if mem_gb >= 96 else "200000")
if p.get("baker"):
env["TRELLIS2_BAKER"] = str(p["baker"])
# "auto" keeps o_voxel's baked-alpha BLEND detection (glass etc.); default
# opaque — the Metal baker's noisy alpha turns solid hair into speckles.
if p.get("alpha_mode"):
env["TRELLIS2_ALPHA_MODE"] = str(p["alpha_mode"])
print("+", " ".join(cmd), flush=True)
print("(first run downloads TRELLIS.2-4B + dinov3 + RMBG-2.0 weights)", flush=True)

View File

@ -341,11 +341,25 @@ class Runner:
p = Path(out["path"])
if not p.is_absolute():
p = outdir / p
if not p.exists():
# a remote worker writes result.json with paths on ITS disk
# (<root>/data/remote/<job>/out/...); after collect() those
# files sit in the local outdir — remap by the out/-relative
# tail, falling back to the basename
marker = f"/data/remote/{job_id}/out/"
_, _, tail = str(out["path"]).partition(marker)
if tail and (outdir / tail).exists():
p = outdir / tail
else:
p = outdir / Path(out["path"]).name
if p.exists():
a = store.register_file(con, p, name=out.get("name"),
parent_job=job_id, move=True, meta=out.get("meta"),
user_id=owner_id)
registered.append(a["id"])
else:
print(f"[runner] {job_id}: result.json output not found after "
f"collect, skipped: {out.get('path')}", flush=True)
else:
for p in sorted(outdir.iterdir()):
if p.name == "result.json" or p.name.startswith("."):