Compare commits
1 Commits
master
...
corridorke
| Author | SHA1 | Date | |
|---|---|---|---|
|
d5217575eb |
344
BENCHMARKS.md
344
BENCHMARKS.md
@ -147,347 +147,3 @@ regression suite (12 checks, ~30s).
|
||||
can make a node look 5–11x slower and it is invisible to load average (the M3 Ultra reads load
|
||||
2.45 with its GPU pinned). Anything measured here should use min-of-runs and check
|
||||
`ioreg … IOAccelerator` "Device Utilization %" *before* touching the GPU. See PERFCHECK.md.
|
||||
|
||||
## 2026-07-19 — fleet MLX capability matrix (mlx 0.32.0 everywhere)
|
||||
|
||||
Probe: 4096² matmul TFLOPS (fp16/bf16), 256MB elementwise GB/s, sdpa head_dim 56 vs 64 (B1·H8·L2048).
|
||||
|
||||
| host | chip | RAM | fp16 TF | bf16 TF | ~GB/s | sdpa d56/d64 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| m3ultra | M3 Ultra | 256 | 23.48 | 23.54 | 400 | 1.51× |
|
||||
| ultra | M1 Ultra | 128 | 16.31 | 13.16 | 297 | 0.94× |
|
||||
| JING5 | M5 | 16 | 8.61 | 8.99 | 43* | **4.72×** |
|
||||
| m4pro | M4 Pro | 24 | 5.24 | 3.86 | 225 | 1.29× |
|
||||
| m4mini | M4 | 16 | 3.65 | 3.65 | 98 | 1.55× |
|
||||
| AIRPRO | M3 | 16 | 1.55 | 1.64 | 87 | 0.95× |
|
||||
|
||||
*JING5 bandwidth measured under interactive load — retest idle before trusting.
|
||||
|
||||
Verdicts:
|
||||
- **bf16-on-M1 question answered**: works, ~20% slower than fp16 (16.3→13.2). Ship fp16 on M1-class, bf16 free on M3 Ultra/M4/M5.
|
||||
- **head_dim pad-to-64 matters MOST on newest chips** (M5 4.7×!, M4-class ~1.3-1.6×, M3 Ultra 1.5×) and ~zero on M1-class (no fused fast path there anyway — matches the corridorkey ablation).
|
||||
- Fleet total ≈ 59 TFLOPS fp16; the two Ultras are 2/3 of it. TRELLIS.2-MLX lane assignment: m3ultra primary, ultra second (fp16), M4 Pro light ops.
|
||||
- Probe script: reusable one-liner via `uv run --with mlx` — mlxprobe.py (this bench).
|
||||
|
||||
Addendum (same day) — m1max studio joins the matrix (login `studio@100.92.78.24`;
|
||||
the fleet doc's `m1max@` user and "disk-critical" note are both stale — 170GB free):
|
||||
|
||||
| M1MAX | M1 Max | 32 | 7.09 | 6.04 | 336 | 1.22× |
|
||||
|
||||
Notable: 2nd-best bandwidth in the fleet (336 GB/s > M4 Pro's 225) despite mid-pack
|
||||
compute — good fit for memory-bound MLX work (VAE decode, big-tensor elementwise).
|
||||
Fleet total ≈ 66 TFLOPS fp16 across 7 machines.
|
||||
|
||||
## 2026-07-19 — trellis_mac rung 1: sparse-conv backend verified (m3ultra, anatomy image, seed 42, 1024-cascade)
|
||||
|
||||
| SPARSE_CONV_BACKEND | total gen+bake | wall | peak RSS |
|
||||
|---|---|---|---|
|
||||
| flex_gemm (Metal) | **124.8 s** | 248.8 s | 20.8 GB |
|
||||
| none (pure-torch) | 214.4 s | 337.6 s | 20.8 GB |
|
||||
|
||||
Verdicts:
|
||||
- The 124.8s baseline WAS the Metal fast path (log line "[SPARSE] Conv backend: flex_gemm") — no hidden 10x jackpot; metallib loads fine on macOS 26.5.
|
||||
- Metal sparse-conv is worth **1.72x end-to-end** (saves 89.6s/gen). Keep flex_gemm the default; the conv_none fallback is a real 90s regression if the metallib ever breaks silently — worth a GODBOARD/log assertion.
|
||||
- Remaining fat in the fast path: ~46s diffusion sampling (padded SDPA attention — next kernel frontier), ~50s decode/extract (incl. aten::segment_reduce falling back to CPU — patchable), 19s Metal bake.
|
||||
- Decode actually produces 2.79M faces before the forced ~200k simplify — the quality-recovery headroom (rung 2) is bigger than the recon estimated.
|
||||
|
||||
## 2026-07-19 — Hunyuan paint-UNet mx.compile (Target 1, branch mlx-tune)
|
||||
|
||||
Micro gate (real arch, random fp16 weights, 6-view 64² latents, m3ultra):
|
||||
eager 270.5 ms/fwd → compiled 260.5 ms/fwd = **1.04×**; parity max|diff|
|
||||
3.0e-05 (PASS); first-trace 1.2 s; peak 4.9 GB. Env-gated
|
||||
`HY3D_MLX_COMPILE=1`, default off (numerics untouched).
|
||||
|
||||
Verdict: keep (free, safe) but the paint UNet is GEMM-bound — fusion alone
|
||||
is small. The bigger Hunyuan levers are Target 3 (MoE `.item()` sync-tax,
|
||||
−4800 syncs/gen, unlocks whole-DiT compile) and Target 4 (manual→fused
|
||||
SDPA in paint DINO). E2E flag-on-vs-off measurement pending after those.
|
||||
|
||||
## 2026-07-19 — A0 GATE: first end-to-end MLX TRELLIS.2 (Jourloy PR#175 fork, m3ultra)
|
||||
|
||||
Same anatomy image, seed 42, 1024-cascade, tex 2048:
|
||||
|
||||
| path | wall (incl. load) | bake | peak mem | output |
|
||||
|---|---|---|---|---|
|
||||
| torch-MPS (trellis-mac) | 248.8 s | 19 s @191K faces | 20.8 GB | 21 MB GLB |
|
||||
| **mlx-experimental (Jourloy)** | **216.6 s** | 83 s @**2.74M faces** | 75.4 GB | 131 MB PBR GLB |
|
||||
|
||||
**VERDICT: ADOPT — the MLX path BEAT the MPS baseline on wall-clock while
|
||||
baking 14× the faces.** Key discovery: `mlx-experimental` is a HYBRID —
|
||||
resolver log shows `Conv backend: flex_gemm; Attention: flex_gemm_sparse_attn`,
|
||||
i.e. sparse ops still run pedronaugusto's Metal kernels; MLX runs the dense
|
||||
stages. The feared naive-MLX-sparse-conv path never executes. Model load is
|
||||
lazy/instant on the MLX side (vs 103 s torch load).
|
||||
|
||||
Quality: geometry excellent at full density; the DARK-PATCH texture defect
|
||||
REPRODUCES on this second, independent pipeline — both paths share the
|
||||
pedronaugusto mtldiffrast/BVH texel sampling. Culprit narrowed to those
|
||||
shared Metal kernels OR the decoded tex attrs; the pure-Python KDTree bake
|
||||
control (running) decides. fal renders clean with the same weights, so the
|
||||
attrs are likely fine → Metal sampling is the prime suspect.
|
||||
|
||||
Caveat: 75 GB peak fits both Ultras but check m1 headroom under load.
|
||||
|
||||
## 2026-07-19 — Hunyuan mlx-tune e2e (T1+T3+T4, HY3D_MLX_COMPILE=1, m3ultra)
|
||||
|
||||
anatomy bench image, defaults (50 steps, octree 256): shape 146s + tex 101s
|
||||
= **246s total, 17.5GB peak** vs recorded defaults baseline 260s (149+112).
|
||||
≈ **1.06× e2e** — texture stage carries the win (112→101s = T4 fused DINO
|
||||
+ T1 UNet compile); shape delta noise-level. Caveat: baseline row used a
|
||||
different input image; same settings though. Next lever: Target 2
|
||||
whole-DiT compile (unblocked by T3).
|
||||
|
||||
Addendum — with T2 (whole-DiT compile) included: shape 144s + tex 99s =
|
||||
**243s total**. Full mlx-tune vs 260s baseline ≈ **1.07× e2e**. Verdict:
|
||||
on M3 Ultra these 2B GEMM-bound models get ~1% from mx.compile fusion;
|
||||
the tune's real wins were T4 fused attention (texture stage) + T3
|
||||
compile-safety. Branch mlx-tune @ 5a32240; flag HY3D_MLX_COMPILE=1.
|
||||
Worth re-benching on M1 Ultra where launch overhead is proportionally larger.
|
||||
|
||||
## 10 · DARK-PATCH VERDICT: Metal texel-sampling convicted (2026-07-19)
|
||||
|
||||
Decisive experiment: dumped the decoded tex voxels (E2 fixture,
|
||||
1,398,769 voxels × 6 PBR attrs) and rendered base_color as a direct
|
||||
front-projection — `trellis2-bench/attrs_front.png`. **The attrs are
|
||||
near-fal quality** (clean anatomy palette; dark voxels only 5.6%, all
|
||||
interior). Both local pipelines (trellis-mac MPS and Jourloy MLX) produce
|
||||
the same mottled dark bake from these clean attrs, and both share the
|
||||
pedronaugusto mtldiffrast/mtlbvh texel→voxel sampling. ⇒ **the bake's
|
||||
attr-sampling is the sole quality gap.** The 3h+ KDTree run is now merely
|
||||
confirmatory (left running).
|
||||
|
||||
### Designed fix (next session's opening move)
|
||||
The attrs live on a REGULAR voxel grid — no BVH/KDTree needed at all:
|
||||
texel → (xatlas UV → triangle barycentric → world pos) →
|
||||
`grid[(pos-origin)/voxel_size]` direct index (+8-neighbor occupancy
|
||||
search). One vectorized gather over all texels; replaces the suspect
|
||||
ray-cast sampling entirely; implementable in torch-MPS or MLX in ~a page.
|
||||
Validate against `attrs_front.png` colors + the fal render. If it works,
|
||||
BOTH local paths get fal-class texture quality at zero bake-speed risk.
|
||||
|
||||
### Parity fixture set: COMPLETE
|
||||
All 8 fixtures captured at `trellis2-bench/parity_fixtures/` (cond_512/
|
||||
1024, ss_coords, shape_slat, shape_dec, tex_slat, tex_voxels, mesh_final;
|
||||
seed 42, 1024-cascade). Hook env needed: PYTORCH_ENABLE_MPS_FALLBACK=1
|
||||
ATTN_BACKEND=sdpa SPARSE_ATTN_BACKEND=sdpa SPARSE_CONV_BACKEND=flex_gemm.
|
||||
|
||||
## 2026-07-19 — KDTree bake control: post-mortem (confirmatory run, superseded)
|
||||
|
||||
The TRELLIS2_FORCE_KDTREE control finally finished: **bake 21,063s (5.85h)**
|
||||
and the exported GLB is GARBAGE geometry (mangled 395K-tri blob) — the
|
||||
fork's KDTree export path has its own vertex-mapping bug on top of being
|
||||
unusably slow (per-triangle Python rasterizer loop). Verdict unchanged —
|
||||
the attrs-projection had already convicted the Metal texel-sampling — but
|
||||
this rules out "just use the Python baker" as shipped: it needs the
|
||||
fast-rasterizer rewrite + export fix (in progress as fast_bake_test.py).
|
||||
|
||||
## 11 · BAKE FIX PROVEN (2026-07-19 evening) — dark patches eliminated
|
||||
|
||||
`trellis2-bench/render_vbake5_front.png`: full anatomy figure, correct
|
||||
colors, ZERO dark patches — from the same decoder attrs the Metal baker
|
||||
mangles. Method: cKDTree IDW sampling of the voxel grid at mesh vertices
|
||||
(k=4, distance² weights, 1.5-voxel radius, single-nearest fallback) →
|
||||
vertex-colored GLB. **Sampling cost: 0.3s** for 270K verts / 1.4M voxels.
|
||||
|
||||
Two integration gotchas (cost hours; do not rediscover):
|
||||
1. **Axis spaces.** Voxel grid + raw pipeline mesh = Z-up-long-axis in
|
||||
[-0.5,0.5]³; GLB-loaded meshes = glTF Y-up. Transform before sampling:
|
||||
`(x,y,z)_gltf → (x,−z,y)_voxel`. The "mangled blob" renders were ONLY
|
||||
this (figure viewed down its own axis) — no data corruption anywhere.
|
||||
2. **glTF vertex colors are LINEAR** — do not pre-gamma (double-gamma =
|
||||
washed out). Store linear, let the renderer do the transfer.
|
||||
|
||||
Also learned: raw `MeshWithVoxel.vertices/faces` need to_glb's weld/init
|
||||
before use as a plain triangle soup; `output_3d.obj` inherits the raw
|
||||
convention. And xatlas hangs (2h+) on écorché-topology meshes — the UV
|
||||
path needs chart budgeting or per-island parallel unwrap.
|
||||
|
||||
### Production path (next session)
|
||||
- Option A (fast to ship): full-density vertex-colored GLB as the raw
|
||||
output; MESHGOD's Blender finish farm already decimates + can bake
|
||||
verts→texture (standard Blender bake) — sidesteps xatlas entirely.
|
||||
- Option B (fal-parity): fast bucket rasterizer (written, in
|
||||
fast_bake_test.py) + xatlas-alternative UV → 2048² texture bake with
|
||||
this sampler. Needs the écorché-safe unwrap.
|
||||
- Wire as `--baker python-fast` in the Jourloy fork + trellis-mac, make it
|
||||
the default over the convicted Metal texel path; port sampler to MLX
|
||||
(trivial — gather + weighted sum) for the pure-MLX lane.
|
||||
|
||||
### §11 addendum — vertex baker PRODUCTIONIZED (2026-07-19 night)
|
||||
|
||||
Wired and validated in BOTH pipelines:
|
||||
- `vendor/trellis-mac` @ 55fdd7b: `TRELLIS2_BAKER=vertex` env mode in
|
||||
generate.py (Metal + kdtree paths untouched; sentinel-gated).
|
||||
- Jourloy fork @ 746e727 (pushed to monster/trellis-2-mrp-mlx): CLI
|
||||
`--baker vertex`.
|
||||
|
||||
E2E validation (anatomy, seed 42, 1024-cascade, uncapped density):
|
||||
**130.0s total, bake 1s**, 232s wall, 20.8GB peak — full 2.79M-face
|
||||
vertex-colored GLB, ZERO dark patches, best local quality yet
|
||||
(`render_vprod_front.png`). vs Metal-bake baseline: same speed envelope,
|
||||
quality gap closed for albedo. Not yet carried: metallic/roughness maps
|
||||
(vertex colors are albedo-only) — MR needs the UV path (Option B) or a
|
||||
COLOR_1 convention. MESHGOD finish-farm verts→texture bake = the
|
||||
remaining Option-A step for game-budget assets.
|
||||
|
||||
## 2026-07-19 — Qwen-Image-Layered on MLX (m1ultra) — WORKING
|
||||
|
||||
New capability: image → N editable RGBA layers, fully local. Model:
|
||||
Qwen/Qwen-Image-Layered (20B, BF16, Apache 2.0, 54GB in m1 HF cache).
|
||||
Runtime: mflux PR#302 fork (unmerged upstream; vendored at
|
||||
monster/Qwen-Image-Layered-MRP-MLX, pin tag pr302-head-a255e4f, isolated
|
||||
venv ~/qwen-layered-staging/mflux-layered/.venv — production mflux venv
|
||||
untouched). CLI: `mflux-generate-qwen-layered`.
|
||||
|
||||
Smoke (anatomy test image, 4 layers, 640, q8-on-the-fly): **1180.8s wall
|
||||
first-run** (dominated by 20B load + live quantization), 27.1GB peak.
|
||||
Output verified: 4 true RGBA PNGs — bg layer, atmosphere layer, clean
|
||||
figure cutout (74.5% transparent, crisp to the fingers), detail layer.
|
||||
The figure-cutout layer doubles as a high-end matting result — relevant
|
||||
to the MESHGOD prep frontline for hero assets.
|
||||
|
||||
Optimization in progress: `mflux-save --quantize 8` pre-baked model →
|
||||
skips load+quant cost; timed gen-only rerun to follow. Next: 1024-res,
|
||||
q4/q6 quality ladder, m3ultra cross-bench.
|
||||
|
||||
### Qwen-Image-Layered optimization ladder (m1ultra, 640, q8, 4 layers)
|
||||
|
||||
| config | wall | note |
|
||||
|---|---|---|
|
||||
| cold: HF bf16 + on-the-fly q8, 50 steps | 1180.8 s | first smoke |
|
||||
| baked q8 model, 50 steps | 1075.9 s | load cost was NOT dominant |
|
||||
| **baked q8, 20 steps** | **365.6 s** | **quality held — figure cutout indistinguishable from 50-step. RECOMMENDED DEFAULT.** |
|
||||
|
||||
3.2× end-to-end. Peak 36.6GB. Baked model: ~/qwen-layered-staging/qwen-layered-q8 (34GB).
|
||||
Remaining ladder (untested): q6/q4 quality/speed, 1024-res, fewer-steps floor (10?),
|
||||
m3ultra cross-bench (~1.4× compute but needs 34GB disk — m3 disk currently tight),
|
||||
MODELBEAST operator wrapper (qwen_layered_local) once defaults settle.
|
||||
|
||||
### Cross-bench addendum: m3ultra joins (same baked q8, 20 steps, 640, 4 layers)
|
||||
|
||||
| box | wall | note |
|
||||
|---|---|---|
|
||||
| m3ultra | **243.0 s** | 1.50× vs m1 — matches fleet-matrix TFLOPS ratio (1.44×) |
|
||||
| m1ultra | 365.6 s | |
|
||||
|
||||
Both Ultras now serve Qwen-Image-Layered from the same 34GB baked-q8
|
||||
artifact (m3: ~/qwen-layered/, m1: ~/qwen-layered-staging/) + the same
|
||||
Gitea fork clone. Peak 36.7GB either box. m3 disk after install: ~82GB free.
|
||||
|
||||
### Qwen-Image-Layered — COMPLETE ladder + verdicts (2026-07-19/20)
|
||||
|
||||
| config | box | wall | verdict |
|
||||
|---|---|---|---|
|
||||
| q8 · 20 steps · 640 | m3ultra | **243.0 s** | **volume default** |
|
||||
| q8 · 20 steps · 640 | m1ultra | 365.6 s | second lane |
|
||||
| q8 · 20 steps · **1024** | m3ultra | 857.6 s | **hero tier** — superb detail, true RGBA (68% transp) |
|
||||
| q6 · 20 steps · 640 | m1ultra | 366.4 s | ❌ REJECTED — decomposition semantics break (silhouette, no cutout) |
|
||||
| q4 · 20 steps · 640 | m1ultra | 350.1 s | ❌ REJECTED — pure noise |
|
||||
|
||||
Quant findings: speed is FLAT across q4/q6/q8 (compute-bound at these
|
||||
settings) — quantizing below q8 buys only disk/RAM and DESTROYS quality.
|
||||
**q8 is the floor for this model.** q6/q4 bakes deleted (m1 back to 3.4T).
|
||||
|
||||
Operator `qwen_layered_local` shipped: vendored fork at
|
||||
vendor/mflux-qwen-layered (install_qwen_layered.sh), tuned defaults
|
||||
(20/640/baked-q8, resolution 1024 opt-in), m1 registered in nodes.json
|
||||
(machine-local), e2e run.py contract smoke passed on m3.
|
||||
|
||||
Also audited flux_local (mflux 0.18.0): DEFAULT PATH HEALTHY — flux2-klein
|
||||
family has mx.compile on denoise + compiled scheduler (the 9s/image path).
|
||||
GAP: FLUX.1 family (dev/schnell/krea) has ZERO mx.compile — backport =
|
||||
same env-gated pattern as Hunyuan T1/T2, modest expected win (1-4% M3U,
|
||||
more on small boxes), upstreamable to mflux. Queued, not urgent.
|
||||
|
||||
## 12 · RUNG-3 KERNEL DAY — RESULTS (2026-07-20)
|
||||
|
||||
**Headline: 216.6s → 71.96s e2e (3.0×), peak 75.4GB → 26.5GB (2.8×),
|
||||
parity held (raw mesh Δ0.03%), full gate EXIT=0.** All on the fork
|
||||
(monster/trellis-2-mrp-mlx): fa972aa + 7860148 + 0db816b.
|
||||
|
||||
What actually moved the needle (in order of impact):
|
||||
1. **Vertex baker properly wired into generate_asset** (scheduler
|
||||
fallthrough fixed — `--baker vertex` was silently running the 5h
|
||||
pure-python kdtree path): "other" bucket 69.4s → 8.3s, and the
|
||||
75GB peak turned out to be the METAL BAKE's, not the conv's.
|
||||
2. **Pure-MLX sampler loop** (mlx_samplers.py): torch CPU loop + per-
|
||||
forward host bounces → one conversion per stage, batched dense CFG,
|
||||
once-per-stage concat_cond. Sampling 56.8→54.4s (~4% — forwards are
|
||||
compute-bound) but it is the correctness-proven, compile-ready
|
||||
architecture, and removes ~72 host round-trips/gen.
|
||||
3. **Validator taught the vertex-colored asset class** (COLOR_0 without
|
||||
UVs = legit; normals materialized at export).
|
||||
|
||||
Honest negatives (documented so nobody re-chases them):
|
||||
- mx.compile on the step forward: ~0 on M3 Ultra (GEMM-bound; matches
|
||||
Hunyuan T1/T2 findings).
|
||||
- Fused Metal sparse-conv kernel: built, parity 8e-4, but stock chunking
|
||||
already bounds memory at decoder scale and beats the scalar kernel on
|
||||
speed. Kept as TRELLIS2_METAL_SPCONV=1 opt-in reference (simdgroup
|
||||
tiling = the future version if decoders ever matter for time).
|
||||
- Remaining gap to H100 (~17s at 1024): inside the DiT forwards
|
||||
(54s sampling ≈ 2.3× ideal FLOPs) — attention/MLP internals, a
|
||||
future deep-dive, not orchestration.
|
||||
|
||||
Env knobs (all default-sane): TRELLIS2_MLX_SAMPLER=0 (revert sampler),
|
||||
TRELLIS2_MLX_COMPILE=1, TRELLIS2_METAL_SPCONV=1.
|
||||
M1 Ultra validation: in flight (26.5GB peak fits with 100GB headroom).
|
||||
|
||||
### Rung-3 addendum — M1 Ultra validated (2026-07-20)
|
||||
|
||||
| box | e2e wall | peak mem |
|
||||
|---|---|---|
|
||||
| m3ultra | 71.96 s | 26.5 GB |
|
||||
| **m1ultra** | **181.5 s** | **17.2 GB** |
|
||||
|
||||
The M1 Ultra — which could not run TRELLIS.2 at all before this fork
|
||||
(torch-MPS path was m3-only in practice) — now generates full-density
|
||||
vertex-baked assets in ~3 min at 17GB. Install: clone
|
||||
monster/trellis-2-mrp-mlx to ~/trellis2-mlx + scripts/setup_macos.sh
|
||||
(PYTHON_BIN=uv 3.11) + gated weights rsynced from m3's HF cache
|
||||
(no HF login needed on workers). Both Ultras now serve the fastest
|
||||
local TRELLIS.2 in existence. TRELLIS.2 fleet: m3 primary (72s),
|
||||
m1 second lane (182s), zero dollars per asset.
|
||||
|
||||
## 2026-07-22 — trellis_mac closes the fal quality gap (m3ultra, black-model image, seed 0, pipeline 1024)
|
||||
|
||||
Root-caused why local GLBs looked far worse than fal's TRELLIS.2 despite the
|
||||
same 4B weights. Two independent bugs, neither in the diffusion (local decode
|
||||
was already 2.23M tris / 101s):
|
||||
|
||||
1. **Decimation ceiling.** generate.py's `TRELLIS2_MAX_BAKE_FACES=200000`
|
||||
laptop mtlbvh guard + o_voxel's cleanup pass (dup/degenerate removal +
|
||||
small-component pruning) crushed every model to exactly **94,535 faces**
|
||||
(fal ships ~495k). The 23× pre-bake simplify also shredded thin shells —
|
||||
hair fragments then got deleted by `remove_small_connected_components`.
|
||||
Fix: cap raised to **500k → 351,898-face GLB**, bake 7s→14s, no Metal
|
||||
watchdog crash on the M3 Ultra Studio. Now the operator default
|
||||
(`max_bake_faces` param; drop to 200k on laptops).
|
||||
|
||||
2. **Speckle-veil alpha.** o_voxel's Metal baker samples the decoder's alpha
|
||||
attr noisily (same family as the 07-19 dark-patch bug); any texel <250
|
||||
flips the material to BLEND → solid hair renders as a transparent speckle
|
||||
veil, "broken face". Fix: generate.py now rewrites the exported GLB to
|
||||
`alphaMode=OPAQUE` by default (`TRELLIS2_ALPHA_MODE=auto` keeps baked
|
||||
alpha for glassy subjects; `alpha_mode` operator param).
|
||||
|
||||
Verified in Blender side-by-side (old 95k / new 352k opaque / fal 495k):
|
||||
face + hair now read at fal level. Remaining deltas vs fal: slightly fuller
|
||||
fal hair (try `max_bake_faces=1000000`), and the known Metal dark-patch
|
||||
texels (workaround: `baker=vertex`). Also wired `steps` param passthrough.
|
||||
|
||||
### Addendum — trellis2_mlx gains the metal PBR lane (2026-07-22, m3ultra)
|
||||
|
||||
Item 1 of the post-quality-fix roadmap: the fast MLX fork lane now ships
|
||||
meshgod-grade UV-PBR by default (baker=metal, RAM-sized cap, forced-opaque
|
||||
alpha — fork commit ba4e2ad, operator ac817a9). Same image/seed as the
|
||||
trellis_mac runs: **156s e2e, 26.2GB peak, candidate_pbr 391,782 tris +
|
||||
2048² basecolor+MR, raw master 2.19M tris kept** (vs trellis_mac ~250s
|
||||
wall for 351,898 tris; fal 495,176 @ $0.30). Farm-path e2e verified with
|
||||
zero client params (defaults gap closed in run.py — server forwards only
|
||||
client-sent params). Blender side-by-side vs fal: parity; local keeps
|
||||
more hair curl detail. baker=vertex remains the 1s clean-albedo option.
|
||||
|
||||
1M-cap probe (same image/seed, farm path): candidate 832,883 tris, 43MB
|
||||
GLB, OPAQUE+PBR — hair marginally fuller than 500k, subtle at viewing
|
||||
distance. Verdict: 500k stays the default; 1M = hero-asset tier.
|
||||
|
||||
@ -1,518 +0,0 @@
|
||||
# FABLE STARTUP PACKET — TRELLIS.2 + Hunyuan MLX build
|
||||
|
||||
> Read this once, top to bottom, before touching anything. It consolidates five prep-only recon
|
||||
> passes (all inspection, **no GPU job launched, no installs, no production code touched**). Every
|
||||
> path here is on **m3ultra** (`m3ultra@100.89.131.57`) unless noted. Where a recon was uncertain,
|
||||
> it says **(UNVERIFIED)** — do not treat those as facts.
|
||||
>
|
||||
> Two baselines you will see quoted, do **not** conflate them:
|
||||
> - **124.8 s** = our current torch-MPS *production* path on the anatomy bench (the number to beat).
|
||||
> - **1004.6 s / 3.09M tris / 17.82 GB** = PR#175's reported **MPS** E2E on an M4 Max 36 GB
|
||||
> (different hardware, different mesh size, and NOT the MLX path). Not comparable to 124.8 s.
|
||||
|
||||
---
|
||||
|
||||
## 1 · STATE ON DISK — what prep already put on the box
|
||||
|
||||
Everything below already exists. **Do not redo it.**
|
||||
|
||||
### Staging dir: `~/Documents/trellis2-mlx-staging/`
|
||||
| Path | What it is | Status |
|
||||
|---|---|---|
|
||||
| `trellis2-apple/` | Full clone of `pedronaugusto/trellis2-apple`, HEAD `6055b86` (2026-04-22), full 17-commit history deepened, MIT. 39 MB. **Nothing installed.** | read-only clone |
|
||||
| `prep_local_draft.py` | 9638 B preprocessing draft (§4). `py_compile` OK under rmbg venv. Not wired in. | draft, do-not-run-yet-in-prod |
|
||||
| `convert_weights.py` | Draft torch→MLX weight converter (§5). Loads ~14 GB — **do not run now.** | draft |
|
||||
| `parity_dump_hook.py` | Draft monkeypatch fixture dumper (§5). Loads models+GPU — **do not run now.** | draft |
|
||||
| `inspect_safetensors.py` | Stdlib safetensors header parser (used to produce the §5 layout table). | utility, safe |
|
||||
| `__pycache__/` | Compile artifacts from the py_compile checks. | ignore |
|
||||
|
||||
### Venvs & operators already on the box (do not reinstall)
|
||||
- **rmbg venv** `~/Documents/MODELBEAST/venvs/rmbg/` (uv, CPython 3.11). Has torch 2.13.0, torchvision 0.28.0, transformers 5.13.1, timm, kornia, einops, safetensors, hf_hub, Pillow, numpy, scipy, scikit-image. **No** cv2 / rembg / birefnet-pkg / upscaler. Installed by `~/Documents/MODELBEAST/scripts/install_rmbg.sh`.
|
||||
- **`bg_remove_local` operator** `~/Documents/MODELBEAST/server/operators/bg_remove_local/run.py` — runs RMBG-2.0, pins `venvs/rmbg/bin/python`.
|
||||
|
||||
### Weights already in HF cache
|
||||
- `microsoft/TRELLIS.2-4B` — `~/.cache/huggingface/hub/models--microsoft--TRELLIS.2-4B/snapshots/af44b45.../ckpts/` (7 of 8 pipeline modules).
|
||||
- `sparse_structure_decoder` (8th module) comes from `microsoft/TRELLIS-image-large` snapshot `25e0d31.../ckpts/ss_dec_conv3d_16l8_fp16.safetensors`.
|
||||
- `briaai/RMBG-2.0` — 844 MB, present.
|
||||
- Hunyuan MLX converted weights — `~/.cache/huggingface/hub/models--dgrauet--hunyuan3d-2.1-mlx/`.
|
||||
- **NOT present:** `ZhengPeng7/BiRefNet` (Apache/MIT, ~900 MB) — must be fetched once for the commercial preprocessing path.
|
||||
|
||||
---
|
||||
|
||||
## 2 · TRELLIS.2 MLX — ADOPT-vs-BUILD DECISION
|
||||
|
||||
### VERDICT: **ADOPT** the Jourloy PR#175 `mlx_backend/` tree as the MLX vendor base. Do **not** build from scratch, and do **not** wholesale-swap the 124.8 s production path — gate any promotion on a measured A0 run.
|
||||
|
||||
**Why PR#175 and not the already-cloned `trellis2-apple`:** the two share the *identical* 16-file / ~3,114-LOC `mlx_backend/` (same author — `pedronaugusto` co-authors PR#175). PR#175 (`Jourloy/TRELLIS.2`, opened 2026-07-17, +7,419/−258 across 62 files) is a **strict superset**: same MLX modules **plus** the correctness oracle we lack (`tests/test_mlx_parity.py`, 4 unit parities at rtol 1e-6–3e-5), a backend probe/fallback resolver (`trellis2/backends.py`), a full CLI (`scripts/generate_asset.py`), macOS setup/probe/weight tooling, and the `--backend mlx-experimental` flag so optimized modules drop in behind a flag while MPS keeps E2E green. So PR#175 dominates `trellis2-apple` on every axis. Keep `vendor/trellis-mac` (shivampkumar, no MLX) only as the torch-MPS E2E oracle.
|
||||
|
||||
**Why NOT a blind swap (both recons agree):** the stock MLX path will very likely **regress** as-is, for two structural reasons:
|
||||
- **Un-fused sparse conv.** `mlx_backend/sparse_conv.py` is the naive gather→matmul→sum, 512 MB-chunked — it does **not** use the `flex_gemm` Metal fast path, which our recon measured at **1.72× faster** on this shape. This is the ~10× crux, reimplemented slow in MLX.
|
||||
- **CPU host bounce.** `mlx_backend/adapters.py` does `torch→numpy→mx→numpy→torch` on **every** model call — per sampler step (default ~12 steps × 3 stages × 2 for CFG) and at every sparse-conv boundary. The upstream torch pipeline still owns sampling orchestration + mesh extraction; MLX models are injected as adapters.
|
||||
- **Zero benchmarks in either repo.** No evidence the MLX path beats — or matches — 124.8 s. Do **not** benchmark the stock `--backend mlx-experimental` path as representative; it's a correct scaffold, not a fast path.
|
||||
- **Parity gap in `trellis2-apple`, closed by PR#175.** `trellis2-apple` has NO MLX parity test; PR#175 adds `test_mlx_parity.py`. Adopt PR#175 specifically to get that oracle.
|
||||
|
||||
**What is safe to reuse directly (low risk, no sparse-conv tax):** `dinov3.py`, `norm.py` (two-pass LayerNorm32 PT-parity), `rope.py` (interleaved complex-mul parity), `transformer_block.py`, `structure_decoder.py` (dense `mx.conv_general`). **Rewrite/replace first:** `sparse_conv.py` (fused Metal gather-GEMM-scatter), the `adapters.py` boundary (move the Euler/CFG loop *inside* MLX to kill the bounce), `sparse_ops.py` (numpy `np.where` host sync in upsample).
|
||||
|
||||
**Known smells to verify before trusting a module (UNVERIFIED in recon):** (1) `structure_decoder.py` `_remap_structure_decoder_weights` is a **no-op** feeding dict-stored conv weights — untested whether `nn.Module.load_weights` populates plain-dict children; (2) hand-rolled softplus/quad_lerp in `vae_decoders.py:1204` FlexiDualGrid decode; (3) full **non-windowed** attention (`attention.py`) may diverge from upstream windowed sparse attn; (4) docstrings claim "custom Metal kernel" but code is plain MLX built-in SDPA — **misleading comments, no hand-written kernel** anywhere in `mlx_backend/`. `mx.compile` *is* already applied to the DiT block loop and structure-decoder graph (with try/except fallback).
|
||||
|
||||
**Upstreamability: LOW — plan to fork, not to depend on merge.** PR#175 is unmerged, CLA-gated, 0 reviews, only the CLA bot has commented. Issue #74 (non-CUDA backends) has no maintainer response. Vendor it and **pin the commit**; basing our kernels on #175's layout keeps our work a clean, eventually-upstreamable diff.
|
||||
|
||||
### First 3 concrete commands
|
||||
```bash
|
||||
# 1. Get the superset fork (has test_mlx_parity.py; the cloned trellis2-apple does not)
|
||||
cd ~/Documents/trellis2-mlx-staging && \
|
||||
git clone https://github.com/Jourloy/TRELLIS.2 trellis2-jourloy && cd trellis2-jourloy
|
||||
|
||||
# 2. Run the correctness ORACLE first — cheap, no heavy GPU E2E, validates
|
||||
# LayerNorm32 / apply_rope / SDPA / MlxSparseConv3d vs torch before any gen.
|
||||
# (Needs an isolated venv with mlx + the deps; test no-ops off-Mac via importorskip.)
|
||||
pytest tests/test_mlx_parity.py -v
|
||||
|
||||
# 3. The single GPU job — the measured A0 gate. Isolated venv, the 3 pedronaugusto
|
||||
# Metal pkgs --no-build-isolation, reuse the existing TRELLIS.2-4B weights.
|
||||
# ONE gen, anatomy bench, seed 42, 1024-cascade; compare wall-clock to 124.8 s
|
||||
# and quality to the fal renders in ~/Documents/trellis2-bench/.
|
||||
pip install -r requirements_macos.txt --no-build-isolation # torch>=2.11, mlx>=0.31, mtlgemm/mtldiffrast/cumesh
|
||||
# then: scripts/generate_asset.py --backend mlx-experimental --seed 42 --pipeline-type 1024_cascade <anatomy_img>
|
||||
```
|
||||
**Decision rule after A0:** within ~1.5× of 124.8 s AND parity holds → patch-and-tune, adopt as production path behind the flag. If 3–5× slower (the likely outcome given un-fused conv + host bounce) → keep torch-MPS in `nodes.json` for production, lift `mlx_backend/` as the module-by-module scaffold, and land the kernel rewrites (fused Metal sparse conv → in-MLX sampler loop → parity fixtures) in that order. Also: fork to Gitea `monster/trellis2-mrp-mlx`, keep `origin` upstream for rebases (house pattern).
|
||||
|
||||
---
|
||||
|
||||
## 3 · HUNYUAN TUNE — edit-target list (ready to apply)
|
||||
|
||||
All paths under `~/Documents/MODELBEAST/vendor/hunyuan3d-mlx/`. Static inspection only (GPU was busy). Ship each edit with a PT-parity test (≤1e-4 rel) + a BENCHMARKS.md row.
|
||||
|
||||
**Headline: `head_dim` pad-to-64 does NOT apply** — every attention site is already a 64-multiple (64 or 128). The corridorkey padding win is **out of scope**; do not build pad wrappers. `mx.compile` count in the whole MLX port = **0** — that is the entire opportunity surface (corridorkey precedent: fixed-shape compile = 1.47× M3 / 1.11× M1, bit-identical, an 8-line change `vendor/corridorkey-mlx@4c660df`).
|
||||
|
||||
| # | Target | file:line | Change | Expected | Risk |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | **Paint UNet forward** (called 3×/step, ~15 steps) | `hy3dpaint/hunyuanpaintpbr_mlx/unet/unet_mlx.py:158` (`__call__`); loop `inference.py:254`, calls `:314,328,339`; steps default `inference.py:72` | Wrap forward in `mx.compile` (add `compile=True` ctor flag, `self._fwd = mx.compile(self._forward_impl)`). No MoE / no `.item()` → compile-safe. | ~1.1–1.3× on ~112 s paint | low; last chunk may have fewer views → 1 retrace (pad final chunk to fixed `n_chunk`) |
|
||||
| 2 | **DiT dense blocks 0–14** (100 forwards = 50 steps × CFG 2) | `hy3dshape/.../denoisers/hunyuandit_mlx.py:379` (`HunYuanDiTBlock.__call__`); driver `pipeline_mlx.py:160`, loop `:143`, steps `:95` | `mx.compile` the dense-block forward (fixed shape `(B∈{1,2},4097,2048)`). Leave 6 MoE blocks (15–20) eager until #3. | ~1.1–1.3× on ~149 s shape (dense = 15/21 depth) | low |
|
||||
| 3 | **MoE `.item()` sync tax** (PREREQUISITE for full-DiT compile) | `mlx_arsenal/moe/moe.py:110` — `if not mx.any(w>0).item(): continue` | Remove the `.item()` early-skip; always run all 8 experts, let `w*out` (w=0) zero them. Deletes ~4800 syncs/gen AND makes the block static → unlocks compiling whole `HunYuanDiTPlain.__call__` (`hunyuandit_mlx.py:514`). | −4800 syncs/gen + unblocks #2 to 21/21 blocks | **`mlx_arsenal` is a SHARED dep** — do NOT edit in place. Vendor a local `MoELayer` subclass override, or upstream to arsenal. `MoEGate` routing is already static/compile-clean. |
|
||||
| 4 | **Paint DINOv2-giant attn** (manual→fused, once/gen ×40 layers) | `hy3dpaint/hunyuanpaintpbr_mlx/dino_mlx.py:58–60` (manual `q@kᵀ` softmax, fp32) | Replace with `mx.fast.scaled_dot_product_attention(q,k,v,scale=self.scale)` + existing transpose/reshape. `head_dim=64` hits fast path; fp32 supported. | one-shot; kills N² materialization (fused d64 = 1.51× on M3) | low |
|
||||
| 5 | **Paint VAE decode** (once/gen) | `hy3dpaint/hunyuanpaintpbr_mlx/vae_mlx.py` decode; manual softmax `:81–84` (`head_dim=512`) | `mx.compile` the decode (fuses the softmax). Prefer compile over `mx.fast` swap — `head_dim=512` likely **misses** MLX fused sdpa (kernel ≤256), would fall back anyway. | one-shot, minor | low |
|
||||
| 6 | **(CONDITIONAL) Shape-VAE geo cross-attn** | `hy3dshape/.../autoencoders/model_mlx.py:206` (`CrossAttentionBlock`); driver `_query_sdf_volume` `:506`, chunk `:510` | `mx.compile` per-chunk cross-attn (fixed 10000-query × 4096-latent). Only final ragged chunk retraces. | small–moderate | **Verify** decode isn't dominated by the Metal SDF/marching-cubes step before investing |
|
||||
|
||||
**Rasterizer: NO opportunity.** `hy3dpaint/DifferentiableRenderer/mesh_render_mlx.py` is a thin adapter over the `mlx_arsenal.rasterize` Metal kernel (already fused); Python side is glue + numpy round-trips. Skip for both levers.
|
||||
|
||||
**DTYPE: keep fp16 as the global default.** Port is fp16 everywhere (one deliberate exception: paint DINOv2-giant upcast to **fp32** in `load_model.py` — accumulation-depth fix, not a format preference). bf16 buys **zero** speed on M3/M4/M5 (equal TFLOPS) and is **~20% slower on M1** — and M1 Ultra (`johnking@100.91.239.7`) is the second production lane and hunyuan's headline "runs on M1" selling point. Do NOT globally switch to bf16. (This is the opposite of the TRELLIS.2 port, where bf16 range matters.)
|
||||
|
||||
**Execution order:** 1 → 3 → 2(dense, then whole-forward once 3 lands) → 4/5/6 (one-shot cleanups last).
|
||||
|
||||
---
|
||||
|
||||
## 4 · PREPROCESSING — draft, plug-in point, license, TODOs
|
||||
|
||||
**North star** (`meshgod/config.py:65-70`, `STYLE_3D`): a single isolated object, centered, fully visible (no crop); plain flat neutral light-gray bg; even soft shadowless studio light; neutral symmetrical A-pose.
|
||||
|
||||
**Draft: `~/Documents/trellis2-mlx-staging/prep_local_draft.py`** (9638 B, `py_compile` OK under rmbg venv, uses only PIL/numpy/torch/torchvision/transformers, not wired in). Signature `prep_local(in_path, out_path, commercial=False)`:
|
||||
1. `_maybe_upscale` — if `min(w,h) < 1024`, Lanczos up to short-edge 1024, on RGB, **BEFORE** matte.
|
||||
2. `_matte` — BiRefNet if `commercial` else RMBG-2.0; real inference mirroring `bg_remove_local/run.py` exactly (`Resize((1024,1024))→ToTensor→Normalize(imagenet)`; `model(inp)[-1].sigmoid()`; → `L` mask); model cached per-process.
|
||||
3–7. `_compose_center_square` — composite onto `NEUTRAL_GRAY=(200,200,200)` RGB (not alpha) via mask, tight bbox crop from thresholded mask (`MASK_THRESH=8`), center, square-pad to `SUBJECT_FRAC=0.88`, resize 1024 Lanczos.
|
||||
|
||||
Run (GPU-free-time session): `venvs/rmbg/bin/python prep_local_draft.py --in X --out Y [--commercial]`.
|
||||
|
||||
**Order lesson (verified live 2026-07-13):** upscale FIRST, matte LAST — SeedVR strips alpha (RGBA→RGB), so matting first gets flattened. Draft honors this.
|
||||
|
||||
**Plug-in point:** `~/Documents/meshgod/meshgod/server.py:350` (local lane, `server.py:346-350`):
|
||||
```python
|
||||
paths[i] = stage3_reconstruct.mb_bg_remove(p, pp) # <- replace with prep_local(p, pp, commercial=...)
|
||||
```
|
||||
`prep_local` supersedes `mb_bg_remove` (does the full north-star transform in-process on rmbg venv, not just a farm-round-trip cutout to RMBG-2.0). Secondary optional precede-point: fal lane `server.py:359` (run `prep_local` locally, then skip fal `clean_bg`). Companion anchors to update: `config.py:143` (`BG_REMOVE_MODEL`), `web/index.html:229` label.
|
||||
|
||||
**LICENSE RULE (hard):**
|
||||
| Model | HF id | License | Commercial ship? | Local? |
|
||||
|---|---|---|---|---|
|
||||
| RMBG-2.0 | `briaai/RMBG-2.0` | CC-BY-NC 4.0 | **NO** — non-commercial / personal one-offs only | Yes (844 MB) |
|
||||
| BiRefNet | `ZhengPeng7/BiRefNet` | MIT | **YES** — safe for game/product assets | **No — fetch once** |
|
||||
Default for anything game-bound = **BiRefNet**. RMBG-2.0 stays default only for local/personal one-offs. (Note: RMBG-2.0 *is itself* a BiRefNet-architecture checkpoint — same code, differs only in weights + license.)
|
||||
|
||||
**Open TODOs (in draft):**
|
||||
1. **BiRefNet weights fetch** — first `--commercial` run must `huggingface-cli download ZhengPeng7/BiRefNet` (~900 MB, MIT, ungated). Until then only RMBG-2.0 works.
|
||||
2. **Better upscaler** — PIL Lanczos is the zero-dep default (never invents detail). Swap for local faithful SR: MODELBEAST `seedvr2_upscale` operator (weights present: `models--numz--SeedVR2_comfyUI`) or Real-ESRGAN. Must stay BEFORE `_matte`.
|
||||
3. **Lighting/exposure normalization (D2)** — optional auto-levels + gray-world WB on RGB before matte (fights TRELLIS dark-patch mode). Stubbed so v0 silhouette stays identical to the fal path for A/B.
|
||||
4. **Tunables to A/B once GPU free:** `NEUTRAL_GRAY=(200,200,200)`, `SUBJECT_FRAC=0.88`, `MASK_THRESH=8` (all chosen from north-star text).
|
||||
5. **Wire-in / flag plumbing** — add `req.commercial`, replace `server.py:350`, expose in `web/index.html`, decide local-lane upscale policy (`server.py:511` currently forces `upscale=False`).
|
||||
|
||||
---
|
||||
|
||||
## 5 · WEIGHT CONVERSION + PARITY
|
||||
|
||||
Drafts in staging: `convert_weights.py`, `parity_dump_hook.py`, `inspect_safetensors.py`. Both loaders touch ~14 GB / GPU — **run in a later session, not now.**
|
||||
|
||||
### Conversion pattern (from Hunyuan's `convert_realesrgan.py:34-58`)
|
||||
`torch.load(weights_only=True)` → **keys pass through unchanged** (module-tree names already match MLX list indexing, no remap dict) → **the only tensor mutation is conv transpose** (PyTorch `(O,I,H,W)`→MLX channels-last `(O,H,W,I)`, detected by `ndim==4`; generalizes to 5D Conv3d as `(0,2,3,4,1)`) → `mx.save_safetensors`.
|
||||
|
||||
### TRELLIS.2-4B layout — the transpose decision (resolved from source, not guessed)
|
||||
There are **two Conv3d conventions on disk; the correct action is opposite for each:**
|
||||
| pipeline key | file (ckpts/) | dtype | conv action |
|
||||
|---|---|---|---|
|
||||
| sparse_structure_flow_model | ss_flow_img_dit_1_3B_64_bf16 (640 t, 2.58 GB) | BF16 | none (pure Linear DiT) |
|
||||
| **sparse_structure_decoder** (from TRELLIS-image-large) | ss_dec_conv3d_16l8_fp16 (74 t) | F32/F16 | **20 dense Conv3d → TRANSPOSE `(0,2,3,4,1)`** |
|
||||
| shape_slat_flow_model_512 / _1024 | slat_flow_img2shape_dit_1_3B_{512,1024}_bf16 | BF16 | none |
|
||||
| shape_slat_decoder | shape_dec_next_dc_f16c32_fp16 (292 t, 948 MB) | F16 | **40 sparse Conv3d → NO transpose (already channels-last)** |
|
||||
| tex_slat_flow_model_512 / _1024 | slat_flow_imgshape2tex_dit_1_3B_{512,1024}_bf16 | BF16 | none |
|
||||
| tex_slat_decoder | tex_dec_next_dc_f16c32_fp16 (284 t, 948 MB) | F16 | **40 sparse Conv3d → NO transpose** |
|
||||
|
||||
- **`ss_dec_conv3d` = dense `nn.Conv3d`**, on-disk `(Co,Ci,Kd,Kh,Kw)` → needs the 5D transpose. All 20 5-D tensors.
|
||||
- **`*_next_dc` = o-voxel `SparseConv3d`**, on-disk **already** `(Co,Kd,Kh,Kw,Ci)` → **load verbatim, NO transpose.** Proven by source: `conv_flex_gemm.py:33-34` and `conv_none.py:40-41` both `permute(0,2,3,4,1)` at construction, and the checkpoint was saved from that path. MLX forward must reshape `(Co,Kd,Kh,Kw,Ci)→(Kvol,Ci,Co)` for the per-offset gather-GEMM (contract at `conv_none.py:106`).
|
||||
- **All 5 DiTs: pure `nn.Linear` (ndim 2) + attention + RMSNorm** — zero conv, zero transpose; only a BF16→target-dtype cast (**fp16 on M1** per fleet matrix). Block arch (30 blocks, identical across all 5): hidden 1536, 12 heads × **head_dim 128**, MLP 8192, cross-ctx 1024, adaLN-zero `modulation [9216]`. **head_dim 128 ≥ 64 → fused-SDPA fast path applies, NO padding.** Keep fused `to_qkv [4608,1536]` / `to_kv [3072,1024]` packed and slice in-kernel.
|
||||
|
||||
`convert_weights.py` encodes this per-file policy (dense-transpose for `ss_dec_conv3d`, keep for `*_next_dc`, passthrough for DiTs) with a kernel-dims sanity-assert. Uses `safetensors.torch.load_file` + `mx.save_safetensors`. Publish converted safetensors to Gitea/HF so re-runs skip conversion.
|
||||
|
||||
### Parity fixture plan
|
||||
Stage boundaries all live in `run()` of `trellis2/pipelines/trellis2_image_to_3d.py:488-596`; each fixture = the **return value of a sub-method** → clean non-invasive monkeypatch, zero vendor edits. `parity_dump_hook.py` monkeypatches the 7 sub-methods, `torch.save`s each return to `$TRELLIS2_DUMP_DIR/<name>.pt`, then `runpy`-delegates to vendor `generate.py`. Unset `TRELLIS2_DUMP_DIR` → patches are no-ops.
|
||||
|
||||
| # | Fixture | Return / hook (file:line in trellis2_image_to_3d.py) | Shape/notes |
|
||||
|---|---|---|---|
|
||||
| A/A′ | Image-cond embed 512 / 1024 | `get_cond` return `:177` (called `:539`/`:540`; disambiguate by `resolution` arg → `cond_512.pt`/`cond_1024.pt`) | `[1,N,1024]` DINOv3 ViT-L/16 tokens; neg = zeros_like |
|
||||
| B | Sparse-structure coords | `sample_sparse_structure` return `:235` (call `:542`) | int `[M,4]` (b,x,y,z); subs: `z_s [1,C,64³]` `:219`, occupancy `:227` |
|
||||
| C | Shape-SLat latent | `sample_shape_slat_cascade` `:364` (`:567`) / non-cascade `:275` | SparseTensor `feats [K,32]`+`coords [K,4]`, **un-normalized** |
|
||||
| D | Tex-SLat latent | `sample_tex_slat` `:432` (`:574`) | SparseTensor `feats [K,32]`, un-normalized, sampled w/ `concat_cond=shape_slat` |
|
||||
| E1 | Decoded shape mesh + subs | `decode_shape_slat` `:389` (`:470`) | `List[Mesh]` + `List[SparseTensor]` |
|
||||
| E2 | Decoded tex voxels | `decode_tex_slat` `:453` (`:471`) | SparseTensor `feats [K,6]` PBR (base_color 0:3, metallic 3:4, roughness 4:5, alpha 5:6) |
|
||||
| E3 | Final mesh | `decode_latent` `:486` (`:592`) | `MeshWithVoxel`: `.vertices`,`.faces`,`.coords`,`.attrs`,`.voxel_size`,`.origin` |
|
||||
|
||||
Invocation (later, GPU): `TRELLIS2_DUMP_DIR=~/Documents/trellis2-bench/parity_fixtures python parity_dump_hook.py <anatomy_img> --pipeline-type 1024_cascade --seed 42 --texture-size 2048`. Fix `seed=42` (`torch.manual_seed` `:538`). Parity target **≤1e-4 rel**. Order cheap→expensive: A/A′ → B → C (the FlexGEMM sparse-conv crux) → D → E.
|
||||
|
||||
**GOTCHA:** C/D fixtures are captured **after** the std/mean de-normalization inside each sampler — the MLX port must apply the same `shape_slat_normalization`/`tex_slat_normalization` (32-dim mean/std, verbatim in `pipeline.json`) at the identical point or they diverge by a constant affine. `sample_tex_slat` also **re-normalizes** `shape_slat` back down (`:407-409`) before concatenating as `concat_cond` — replicate that round-trip.
|
||||
|
||||
---
|
||||
|
||||
## 6 · RECOMMENDED FIRST HOUR
|
||||
|
||||
1. **Read this packet + skim the 3 staging drafts** (`prep_local_draft.py`, `convert_weights.py`, `parity_dump_hook.py`) so you know what already exists. Do NOT re-clone `trellis2-apple` or re-inventory the rmbg venv.
|
||||
2. **Clone the Jourloy superset** (`git clone https://github.com/Jourloy/TRELLIS.2 ~/Documents/trellis2-mlx-staging/trellis2-jourloy`) — §2 command 1. This is the vendor base; `trellis2-apple` lacks the parity oracle.
|
||||
3. **Run the parity oracle** `pytest tests/test_mlx_parity.py -v` in an isolated venv (no heavy GPU E2E). Confirms LayerNorm32/RoPE/SDPA/SparseConv parity before trusting any module. This is the cheapest signal available and needs no A0.
|
||||
4. **Fork to Gitea** `monster/trellis2-mrp-mlx`, keep `origin` upstream, **pin the PR#175 head commit** (it's unmerged/CLA-gated — do not depend on merge).
|
||||
5. **Queue the single A0 GPU gate** for when the box is free — the ONE gen (anatomy bench, seed 42, 1024-cascade), compare to 124.8 s and the `~/Documents/trellis2-bench/` fal renders. Do not benchmark the stock MLX path as representative before the sparse-conv + adapter rewrites; expect a regression. Apply the §2 decision rule to the result.
|
||||
6. **In parallel while GPU is busy** (all CPU/inspection, no contention): start Hunyuan **Target 1** (paint UNet `mx.compile`, §3) — cleanest, no blockers — with its PT-parity test; and/or dry-run `prep_local_draft.py` on a sample image under the rmbg venv (§4). Both are safe without the A0 result.
|
||||
7. **Before any commercial preprocessing run:** `huggingface-cli download ZhengPeng7/BiRefNet` (§4 TODO 1) — otherwise only the non-commercial RMBG-2.0 path works.
|
||||
|
||||
**Prime directives:** parity-first (every kernel/edit ships a ≤1e-4-rel PT-parity test + a BENCHMARKS.md row); do NOT edit shared `mlx_arsenal` in place (§3 Target 3); default BiRefNet for game-bound assets (§4). The GPU A0 gen is the ONLY job that must wait for a free box.
|
||||
|
||||
---
|
||||
|
||||
## 7 · PREP ROUND 2 — VERIFICATION RESULTS (2026-07-19)
|
||||
|
||||
> Round-2 deep-verify of §1–§6 against **live source + on-disk safetensors headers**. Still
|
||||
> inspection-only: **no GPU gen, no weights loaded, no prod code touched** (`py_compile` /
|
||||
> header-parse / `pytest --collect-only`). Where §7 says CONFIRMED / CORRECTION / CONTRADICTION,
|
||||
> **trust §7 over the original section.**
|
||||
|
||||
### 7.1 · ADOPT confidence — §2 stance → **CONFIRMED (HIGH)**
|
||||
|
||||
- **Clone DONE & pinnable.** `~/Documents/trellis2-mlx-staging/trellis2-jourloy/`, HEAD `754d403`
|
||||
(2026-07-17, Jourloy — matches "PR#175"), branch `main` **is** the PR head, tree clean.
|
||||
**Pin `754d403`** when forking.
|
||||
- **Superset CONFIRMED — stronger than §2 claimed.** All 4 named artifacts present
|
||||
(`tests/test_mlx_parity.py`, `trellis2/backends.py`, `scripts/generate_asset.py`,
|
||||
`requirements_macos.txt`) **plus 4 extra test files** (`test_backend_fallbacks`,
|
||||
`test_background_preprocess`, `test_generate_asset`, `test_model_revisions`) **and** a new
|
||||
`trellis2/model_revisions.py` (HF revision pins). `mlx_backend` = 3172 LOC / 16 files, all
|
||||
`py_compile` clean.
|
||||
- **`mlx_backend` numerics = `trellis2-apple` byte-for-byte.** `diff -rq`: 14/16 files identical;
|
||||
the only 2 that differ (`dinov3.py`, `pipeline.py`) add **HF revision-pinning / offline kwargs
|
||||
only — zero algorithm change.** §2's "identical mlx_backend" holds for numerics; jourloy is a
|
||||
strict superset (adds reproducibility plumbing on top of the same math).
|
||||
- **`--backend mlx-experimental` cleanly wired** (`generate_asset.py:333`→`create_mlx_pipeline`,
|
||||
`:343`) behind a real **Metal-completeness probe gate** (`:170-187` → `backends.probe_metal_backends()`)
|
||||
that refuses a half-built stack instead of silently degrading. MPS stays the E2E path.
|
||||
- **§2's 4 UNVERIFIED smells — now resolved:**
|
||||
|
||||
| # | smell | round-2 disposition |
|
||||
|---|---|---|
|
||||
| 1 | structure_decoder no-op remap + dict-child conv load | **BENIGN by construction** (leaves pre-alloc'd `mx.zeros`; MLX param tree recurses dict/list; `load_weights(strict=True)` replaces leaves). Residual = exact key-name match only, **not** exercised by parity test → run a 2-sec non-GPU load smoke on ss_dec. |
|
||||
| 2 | vae_decoders softplus/quad_lerp | **BENIGN** — it's the numerically-stable softplus identity `max(x,0)+log1p(exp(-|x|))` = torch softplus. Real line `vae_decoders.py:196` (§2's `:1204` misattributed). Needs one decode-boundary fixture (E1/E2). |
|
||||
| 3 | non-windowed global attention | **NEEDS-RUNTIME-CHECK / POTENTIALLY REAL — highest-value item.** `_sdpa` does full global SDPA, **no window mask**. If upstream sparse flow is windowed → numeric divergence **and** O(N²) blowup at 1024-cascade. Tiny SDPA parity test validates the primitive, not the windowing semantics. **Verify vs upstream sparse-attn config before trusting shape/tex output.** |
|
||||
| 4 | "custom Metal kernel" docstrings | **CONFIRMED BENIGN (cosmetic)** — grep `metal_kernel|@mx.custom|.metal|kernel_source` over `mlx_backend/` = zero matches; code calls MLX built-in `mx.fast.scaled_dot_product_attention`. No hand-written kernel anywhere. |
|
||||
|
||||
- **Bonus (confirms §2):** sparse-conv perf crux is **REAL** — `sparse_conv.py` is naive
|
||||
gather→matmul→sum, 512 MB-chunked with a per-chunk `mx.eval` host sync, O(batch·D³) dense LUT,
|
||||
**no `flex_gemm`.** Matches the ~10× claim.
|
||||
- **Confidence HIGH, not lowered:** nothing contradicted the ADOPT case; several findings
|
||||
strengthened it. Two items still need a *runtime* signal (smell 3 windowed-attn, smell 1
|
||||
dict-child load) — **both cheap, non-GPU; run before the A0 gate** (see 7.6).
|
||||
|
||||
### 7.2 · WEIGHT-CONVERT — transpose policy **VERIFIED vs disk**; converter has 2 bugs + 1 double-transpose contradiction
|
||||
|
||||
All 8 modules parsed **header-only** (no tensor bytes). Tensor counts match §5 **verbatim**
|
||||
(640/640/640/640/640/292/284/74). Classifier found **ZERO ambiguous 5D tensors** — conventions
|
||||
separate cleanly by shape.
|
||||
|
||||
| module(s) | dtype on disk | 5D convs | §5 policy | matches disk |
|
||||
|---|---|---|---|---|
|
||||
| 5 DiTs | **BF16** ×640 ea | 0 (215 Linear + 120 RMSNorm γ + 305 1D) | passthrough + dtype cast | ✅ |
|
||||
| shape_dec / tex_dec | **F16** | 40 sparse ea, `(Co,K,K,K,Ci)` | **NO transpose** | ✅ |
|
||||
| ss_dec | **F32×38 + F16×36** (2 F32 convs = `input_layer` + `out_layer.2`) | 20 dense, `(Co,Ci,K,K,K)` | **transpose `(0,2,3,4,1)`** | ✅ |
|
||||
|
||||
Transpose/convention classification in §5 holds on disk with **zero contradictions**. The
|
||||
converter's `else: raise RuntimeError` on any unclassified 5D fails **loud**, so no silent
|
||||
transpose-convention corruption is possible. **But three items must be fixed before any convert run:**
|
||||
|
||||
- 🔴 **BUG 1 (BLOCKING)** `convert_weights.py:59` `v.numpy()` on `torch.bfloat16` →
|
||||
`TypeError: unsupported ScalarType BFloat16`. **Crashes on all 5 BF16 DiTs** (the bulk of the
|
||||
pipeline); only survives on F16/F32 decoders. Loud, not silent. Fix: `v.float().numpy()`, or
|
||||
bit-reinterpret via `uint16`, or drop torch and use `mx.load`/`mx.save_safetensors`.
|
||||
- 🟠 **BUG 2 (SILENT precision loss)** uniform `target_dtype` default `"bf16"` (`:46,81-83,94`)
|
||||
downcasts the decoders under default args: shape_dec/tex_dec **F16→bf16** (lose 3 mantissa bits,
|
||||
strictly worse + ~20% slower on M1); ss_dec's **2 F32 convs → bf16** (lose 16 bits on the most
|
||||
sensitive first/last SDF layers). **Contradicts §5's own dtype column (decoders = F16).** Fix:
|
||||
per-file dtype policy — DiTs→bf16 (fp16 on M1), decoders→preserve on-disk dtype.
|
||||
- ⚠️ **CONTRADICTION — ss_dec DOUBLE-TRANSPOSE (silent-corruption risk).** §5/converter transpose
|
||||
ss_dec `(0,2,3,4,1)` **at load**. But the live MLX `structure_decoder.py` stores PT-format
|
||||
`(Co,Ci,kD,kH,kW)` **verbatim** and transposes **at forward-call time** (`conv3d`,
|
||||
`structure_decoder.py:14`). If the converter *also* transposes ss_dec, the decoder
|
||||
**double-transposes → garbage, silently.** **Resolve before converting:** convert ss_dec to
|
||||
PT-format on disk (NO load transpose) to match this file, OR move the transpose — pick exactly
|
||||
one, not both.
|
||||
- **Port dependency (not the converter):** the MLX sparse-conv module MUST do the
|
||||
`(Co,K,K,K,Ci)→(Kvol,Ci,Co)` reshape at load (`conv_none.py:106`) or the `*_next_dc` decoders
|
||||
emit garbage. Converter correctly leaves this to the module.
|
||||
|
||||
### 7.3 · PARITY HOOKS — all 8 targets match live source, **ZERO corrections to the hook**
|
||||
|
||||
`parity_dump_hook.py` needs no edits. Source `trellis2_image_to_3d.py` is frozen at upstream
|
||||
`75fbf01` (2026-06-05) + 1 benign local Mac cuda-guard (`:590-591`, downstream of every hook →
|
||||
shifts no fixture line). **No post-packet drift.** Verified def/return/call (corrections to §5's
|
||||
fixture table where it drifted):
|
||||
|
||||
| fixture | def | return | call-site(s) |
|
||||
|---|---|---|---|
|
||||
| get_cond (A/A′) | 164 | **181** (no-neg) / **183-186** (dict) | 539, 540 |
|
||||
| sample_sparse_structure (B) | 188 | 235 | 542 |
|
||||
| sample_shape_slat (C, non-cascade) | 237 | 275 | 547, 557 |
|
||||
| sample_shape_slat_cascade (C′) | 277 | 364 (tuple `slat,hr_res`; `transform` takes `r[0]`) | 567, 579 |
|
||||
| sample_tex_slat (D) | 391 | 432 | 551, 561, 574, 586 |
|
||||
| decode_shape_slat (E1) | 366 | 389 | 470 |
|
||||
| decode_tex_slat (E2) | 434 | 453 | 471 |
|
||||
| decode_latent (E3) | 456 (`@torch.no_grad` :455) | 486 | 592 |
|
||||
|
||||
- **§5 CORRECTION:** the fixture table cites get_cond "return **:177**" — that line is the model
|
||||
call `cond = self.image_cond_model(image)`, **not** the return. Real returns are **:181 / :183-186**.
|
||||
Cosmetic (hook wraps by method name), but corrected.
|
||||
- **Normalization gotcha CONFIRMED:** `shape_slat.pt` / `tex_slat.pt` captured **POST-denorm**
|
||||
(`:271-273`, `:360-362`, `:428-430`); tex sampler **re-normalizes** shape_slat back down at
|
||||
`:407-409` before `concat_cond` (:419). Port must replicate **both** the on-return de-norm and the
|
||||
denorm→renorm round-trip or fixtures diverge by a constant affine (silent ≤1e-4 failure).
|
||||
- **UNCERTAIN — flag for exec session:** the hook's `__main__` `runpy`-delegates to vendor
|
||||
`generate.py`; that file's existence + CLI contract (`--pipeline-type` / `--seed` /
|
||||
`--texture-size`) was **not** checked. Verify before the dump run — a missing/renamed delegate
|
||||
means the hook installs but never fires.
|
||||
|
||||
### 7.4 · PREPROCESSING — BiRefNet **PRESENT**; D2 **IMPLEMENTED**
|
||||
|
||||
- **BiRefNet fetched (§1 "NOT present" TODO → DONE).** Cache:
|
||||
`/Users/m3ultra/.cache/huggingface/hub/models--ZhengPeng7--BiRefNet/snapshots/e2bf8e4460fc8fa32bba5ea4d94b3233d367b0e4/`.
|
||||
Size **424 MB** (`model.safetensors` = 444,473,596 B) — **§1/§4's "~900 MB" was high.**
|
||||
License **MIT** (README `license: mit`) — **§1/§4's "Apache-2.0" label was wrong.** Ungated, no login.
|
||||
- **BiRefNet branch VERIFIED CORRECT (not stubbed):** RMBG-2.0 *is* a BiRefNet checkpoint (identical
|
||||
`auto_map`/`architectures`); forward `model(inp)[-1].sigmoid()` holds (`birefnet.py:2087/2090`);
|
||||
deps satisfied by the rmbg venv — **no cv2 needed** (`birefnet.py` has zero opencv refs). Only
|
||||
metadata was hardened; no code change.
|
||||
- **D2 (light normalization) IMPLEMENTED** behind `normalize_light=False` (default **OFF** → v0
|
||||
silhouette stays A/B-identical to fal). New `_normalize_light` (`prep_local_draft.py:84`, PIL/numpy
|
||||
only): 1/99-pct auto-levels (flat-channel guarded) + gray-world WB; wired at step **1b** (after
|
||||
upscale, before matte — honors "matte LAST"). CLI `--normalize-light`. Functional smoke passed on a
|
||||
synthetic cast image.
|
||||
- **Draft status:** `~/Documents/trellis2-mlx-staging/prep_local_draft.py` now 236 lines,
|
||||
`py_compile` OK + imports clean under rmbg venv. Signature
|
||||
`prep_local(in_path, out_path, commercial=False, normalize_light=False)`. **Staging only — still
|
||||
NOT wired into prod** (§4.5 wire-in + tunable A/B still open; need a GPU-free box for real A/B).
|
||||
|
||||
### 7.5 · MLX TEST VENV — oracle is **one command from running**
|
||||
|
||||
- **Venv READY:** `/Users/m3ultra/Documents/trellis2-mlx-staging/mlx-test-venv` (CPython **3.12.13**,
|
||||
uv). `mlx==0.32.0` (+`mlx-metal` 0.32.0), `torch==2.13.0` (prebuilt wheel, no compile),
|
||||
`numpy` 2.5.1, `pytest` 9.1.1. (safetensors/transformers/einops also installed but the oracle uses
|
||||
**none** of them — droppable for a leaner venv.)
|
||||
- **Collects clean:** `--collect-only` → **4 tests collected, 0 import errors, 0 skips** (0 skips
|
||||
proves both `importorskip("mlx.core")` and `importorskip("torch")` succeeded on the box). **Not yet
|
||||
executed** — a real pass is still pending (low-risk caveat: `np.asarray(mlx_array)` under numpy
|
||||
2.5.1 surfaces only on the real run, not collection).
|
||||
- **GPU/Metal?** 4 tests run microsecond ops on tiny tensors. MLX's default device is Metal (GPU), so
|
||||
they *touch* the GPU — but this is trivial library compute, **NOT a generation/inference job**, uses
|
||||
**no Metal build packages** (mtlgemm/mtldiffrast/cumesh) and no compile. Won't meaningfully contend
|
||||
for the box; force CPU with `mx.set_default_device(mx.cpu)` if ever desired.
|
||||
- **EXACT run command** (use the venv python — a bare `pytest` uses the wrong interpreter, correcting
|
||||
§6/§2):
|
||||
```
|
||||
ssh -n m3ultra@100.89.131.57 'cd ~/Documents/trellis2-mlx-staging/trellis2-jourloy && \
|
||||
../mlx-test-venv/bin/python -m pytest tests/test_mlx_parity.py -v'
|
||||
```
|
||||
Expected: **4 passed.** Baked tolerances: LayerNorm `2e-5`, RoPE `1e-6`, SDPA `3e-5`,
|
||||
SparseConv `2e-5` (all tighter than the ≤1e-4 target).
|
||||
|
||||
### 7.6 · UPDATED FIRST MOVES — deltas to §6
|
||||
|
||||
- **§6 step 2 (clone Jourloy): DONE** — `trellis2-jourloy/` @ `754d403`, tree clean. Skip the clone;
|
||||
just `git checkout 754d403` / pin it when forking.
|
||||
- **§6 step 3 (run parity oracle): venv READY, oracle collects clean.** Now a one-command move — run
|
||||
the exact command in **7.5**. **Do NOT use the bare `pytest tests/...` from §6/§2** (wrong
|
||||
interpreter).
|
||||
- **§6 step 7 / §4 TODO 1 (BiRefNet fetch): DONE** — skip. Present at the 7.4 path (424 MB, MIT).
|
||||
- **NEW — before any weight-convert run (§5):** fix converter **Bug 1** (blocking bf16 crash) +
|
||||
**Bug 2** (silent decoder downcast), AND resolve the **ss_dec double-transpose** contradiction (7.2).
|
||||
Else DiT convert crashes, decoders lose precision, or ss_dec silently corrupts.
|
||||
- **NEW — before the A0 GPU gate:** run the two cheap **non-GPU** runtime checks — (a) ss_dec
|
||||
dict-child weight-load smoke, and especially **(c) global-vs-windowed attention vs the upstream
|
||||
sparse-attn config** (the one smell that can be a true correctness bug, not just slow). Both fit
|
||||
CPU/inspection while the box is busy.
|
||||
- **Unchanged:** §6 step 5 (single A0 GPU gate — seed 42, 1024-cascade, compare 124.8 s) is still the
|
||||
**only** job that must wait for a free box; §6 step 6 (Hunyuan Target 1) still safe in parallel.
|
||||
|
||||
## 8 · CORRECTION — the adopt path needs NO weight conversion (2026-07-19, verified in source)
|
||||
|
||||
Round-2 flagged three bugs in `convert_weights.py` (bf16 crash, decoder downcast, ss_dec double-transpose). **All three are moot: the converter is not needed for the adopted Jourloy base, and using it would corrupt.** Verified by reading Jourloy `mlx_backend/`:
|
||||
|
||||
- **Weights load directly from the original TRELLIS.2-4B safetensors.** `pipeline.py:116/141/162` → `load_safetensors("<path>.safetensors")` → `model.load_weights(...)`. `load_safetensors` (`__init__.py:26`) is just `mx.load(path)`, keeping native bf16/fp16; an optional `dtype=` casts float weights (this is the M1 fp16 path — **not** a separate converter).
|
||||
- **Remap = key-rename ONLY, zero tensor transform.** `remap_flow_model_weights` / `remap_vae_decoder_weights` only rewrite key strings (nn.Sequential `.N.`→`.layers.N.`; MlxSparseLinear `.linear.` wrap). No transpose, no dtype change to any tensor.
|
||||
- **All conv transposing is IN-MODULE at forward.** `structure_decoder.py:14` transposes the dense ss_dec (Co,Ci,K,K,K)→(Co,K,K,K,Ci) every forward; `sparse_conv.py:130` `w.transpose(1,2,0)` for the sparse GEMM. So the modules EXPECT PyTorch-format weights.
|
||||
- **Why the converter is a trap:** it transposes ss_dec at load; the module transposes again at forward → double-transpose. For conv1/conv2 (Ci==Co==channels) the SHAPE still matches, so `load_weights` succeeds silently with wrong values; for input_layer (Ci=latent≠Co) it would shape-mismatch and error. Mixed silent/loud corruption.
|
||||
|
||||
**ACTION for Fable:** do **not** run `convert_weights.py` for the Jourloy path (it's now banner-warned in staging). For M1 fp16, pass `dtype=mx.float16` to `load_safetensors`. §5's transpose table remains correct as a *description of the on-disk layout*, but the CONVERSION step it implied is unnecessary — Jourloy consumes originals directly. A converter is only relevant if you ever build a from-scratch decoder that does not transpose at forward.
|
||||
|
||||
## 9 · EXECUTION LOG — Fable session 1 (2026-07-19)
|
||||
|
||||
Done this session (all committed; BENCHMARKS.md has the rows):
|
||||
1. **Parity oracle: 4/4 PASS** (2.55s, CPU-only, mlx-test-venv).
|
||||
2. **Jourloy venv built** via scripts/setup_macos.sh (PYTHON_BIN=uv 3.11; Metal pkgs compiled).
|
||||
3. **A0 GATE PASSED — ADOPT CONFIRMED**: mlx-experimental beat MPS (216.6s vs 248.8s wall) at 14× bake density. It's a HYBRID (Metal flex_gemm sparse ops + MLX dense); naive MLX sparse-conv never runs. 75.4GB peak.
|
||||
4. **Hunyuan mlx-tune branch @ 4b368ee** (pushed to partly): T1 UNet mx.compile (env-gated HY3D_MLX_COMPILE, parity 3e-5, 1.04× micro), T3 StaticMoELayer (bit-identical, kills ~4800 syncs, compile-unlocked), T4 fused DINO SDPA (1.41× micro, parity 1e-7). E2E flag-on run in flight vs the 260s defaults baseline.
|
||||
5. **Dark-patch culprit narrowed**: reproduces on BOTH independent pipelines (trellis-mac MPS + Jourloy MLX) → shared pedronaugusto Metal texel-sampling (mtldiffrast/BVH) or decoded attrs. fal is clean on the same weights → attrs likely fine → **Metal sampling = prime suspect**. KDTree pure-Python bake control still grinding (2h+, alive, 300% CPU) — its render decides.
|
||||
|
||||
NEXT (in order): read KDTree verdict when it lands → Hunyuan e2e A/B number → wire prep_local into MESHGOD behind a flag → Target 2 (whole-DiT compile now that T3 unblocked it) → promote mlx-experimental into MODELBEAST as a `trellis2_mlx` operator behind a flag once the bake-quality question is settled.
|
||||
|
||||
## 10 · DARK-PATCH VERDICT: Metal texel-sampling convicted (2026-07-19)
|
||||
|
||||
Decisive experiment: dumped the decoded tex voxels (E2 fixture,
|
||||
1,398,769 voxels × 6 PBR attrs) and rendered base_color as a direct
|
||||
front-projection — `trellis2-bench/attrs_front.png`. **The attrs are
|
||||
near-fal quality** (clean anatomy palette; dark voxels only 5.6%, all
|
||||
interior). Both local pipelines (trellis-mac MPS and Jourloy MLX) produce
|
||||
the same mottled dark bake from these clean attrs, and both share the
|
||||
pedronaugusto mtldiffrast/mtlbvh texel→voxel sampling. ⇒ **the bake's
|
||||
attr-sampling is the sole quality gap.** The 3h+ KDTree run is now merely
|
||||
confirmatory (left running).
|
||||
|
||||
### Designed fix (next session's opening move)
|
||||
The attrs live on a REGULAR voxel grid — no BVH/KDTree needed at all:
|
||||
texel → (xatlas UV → triangle barycentric → world pos) →
|
||||
`grid[(pos-origin)/voxel_size]` direct index (+8-neighbor occupancy
|
||||
search). One vectorized gather over all texels; replaces the suspect
|
||||
ray-cast sampling entirely; implementable in torch-MPS or MLX in ~a page.
|
||||
Validate against `attrs_front.png` colors + the fal render. If it works,
|
||||
BOTH local paths get fal-class texture quality at zero bake-speed risk.
|
||||
|
||||
### Parity fixture set: COMPLETE
|
||||
All 8 fixtures captured at `trellis2-bench/parity_fixtures/` (cond_512/
|
||||
1024, ss_coords, shape_slat, shape_dec, tex_slat, tex_voxels, mesh_final;
|
||||
seed 42, 1024-cascade). Hook env needed: PYTORCH_ENABLE_MPS_FALLBACK=1
|
||||
ATTN_BACKEND=sdpa SPARSE_ATTN_BACKEND=sdpa SPARSE_CONV_BACKEND=flex_gemm.
|
||||
|
||||
### Fork home (added 2026-07-19, post-§10)
|
||||
The vendored Jourloy/PR#175 tree now lives on our Gitea:
|
||||
`ssh://git@100.71.119.27:222/monster/trellis-2-mrp-mlx.git` (note the
|
||||
hyphens), branch `main`, pin tag `pr175-head-754d403`. m3 staging clone has
|
||||
it as remote `partly`; `origin` stays github/Jourloy for rebases. All our
|
||||
kernel/bake work lands as commits on this fork.
|
||||
|
||||
## 11 · BAKE FIX PROVEN (2026-07-19 evening) — dark patches eliminated
|
||||
|
||||
`trellis2-bench/render_vbake5_front.png`: full anatomy figure, correct
|
||||
colors, ZERO dark patches — from the same decoder attrs the Metal baker
|
||||
mangles. Method: cKDTree IDW sampling of the voxel grid at mesh vertices
|
||||
(k=4, distance² weights, 1.5-voxel radius, single-nearest fallback) →
|
||||
vertex-colored GLB. **Sampling cost: 0.3s** for 270K verts / 1.4M voxels.
|
||||
|
||||
Two integration gotchas (cost hours; do not rediscover):
|
||||
1. **Axis spaces.** Voxel grid + raw pipeline mesh = Z-up-long-axis in
|
||||
[-0.5,0.5]³; GLB-loaded meshes = glTF Y-up. Transform before sampling:
|
||||
`(x,y,z)_gltf → (x,−z,y)_voxel`. The "mangled blob" renders were ONLY
|
||||
this (figure viewed down its own axis) — no data corruption anywhere.
|
||||
2. **glTF vertex colors are LINEAR** — do not pre-gamma (double-gamma =
|
||||
washed out). Store linear, let the renderer do the transfer.
|
||||
|
||||
Also learned: raw `MeshWithVoxel.vertices/faces` need to_glb's weld/init
|
||||
before use as a plain triangle soup; `output_3d.obj` inherits the raw
|
||||
convention. And xatlas hangs (2h+) on écorché-topology meshes — the UV
|
||||
path needs chart budgeting or per-island parallel unwrap.
|
||||
|
||||
### Production path (next session)
|
||||
- Option A (fast to ship): full-density vertex-colored GLB as the raw
|
||||
output; MESHGOD's Blender finish farm already decimates + can bake
|
||||
verts→texture (standard Blender bake) — sidesteps xatlas entirely.
|
||||
- Option B (fal-parity): fast bucket rasterizer (written, in
|
||||
fast_bake_test.py) + xatlas-alternative UV → 2048² texture bake with
|
||||
this sampler. Needs the écorché-safe unwrap.
|
||||
- Wire as `--baker python-fast` in the Jourloy fork + trellis-mac, make it
|
||||
the default over the convicted Metal texel path; port sampler to MLX
|
||||
(trivial — gather + weighted sum) for the pure-MLX lane.
|
||||
|
||||
### §11 addendum — vertex baker PRODUCTIONIZED (2026-07-19 night)
|
||||
|
||||
Wired and validated in BOTH pipelines:
|
||||
- `vendor/trellis-mac` @ 55fdd7b: `TRELLIS2_BAKER=vertex` env mode in
|
||||
generate.py (Metal + kdtree paths untouched; sentinel-gated).
|
||||
- Jourloy fork @ 746e727 (pushed to monster/trellis-2-mrp-mlx): CLI
|
||||
`--baker vertex`.
|
||||
|
||||
E2E validation (anatomy, seed 42, 1024-cascade, uncapped density):
|
||||
**130.0s total, bake 1s**, 232s wall, 20.8GB peak — full 2.79M-face
|
||||
vertex-colored GLB, ZERO dark patches, best local quality yet
|
||||
(`render_vprod_front.png`). vs Metal-bake baseline: same speed envelope,
|
||||
quality gap closed for albedo. Not yet carried: metallic/roughness maps
|
||||
(vertex colors are albedo-only) — MR needs the UV path (Option B) or a
|
||||
COLOR_1 convention. MESHGOD finish-farm verts→texture bake = the
|
||||
remaining Option-A step for game-budget assets.
|
||||
|
||||
## 12 · RUNG-3 KERNEL DAY — RESULTS (2026-07-20)
|
||||
|
||||
**Headline: 216.6s → 71.96s e2e (3.0×), peak 75.4GB → 26.5GB (2.8×),
|
||||
parity held (raw mesh Δ0.03%), full gate EXIT=0.** All on the fork
|
||||
(monster/trellis-2-mrp-mlx): fa972aa + 7860148 + 0db816b.
|
||||
|
||||
What actually moved the needle (in order of impact):
|
||||
1. **Vertex baker properly wired into generate_asset** (scheduler
|
||||
fallthrough fixed — `--baker vertex` was silently running the 5h
|
||||
pure-python kdtree path): "other" bucket 69.4s → 8.3s, and the
|
||||
75GB peak turned out to be the METAL BAKE's, not the conv's.
|
||||
2. **Pure-MLX sampler loop** (mlx_samplers.py): torch CPU loop + per-
|
||||
forward host bounces → one conversion per stage, batched dense CFG,
|
||||
once-per-stage concat_cond. Sampling 56.8→54.4s (~4% — forwards are
|
||||
compute-bound) but it is the correctness-proven, compile-ready
|
||||
architecture, and removes ~72 host round-trips/gen.
|
||||
3. **Validator taught the vertex-colored asset class** (COLOR_0 without
|
||||
UVs = legit; normals materialized at export).
|
||||
|
||||
Honest negatives (documented so nobody re-chases them):
|
||||
- mx.compile on the step forward: ~0 on M3 Ultra (GEMM-bound; matches
|
||||
Hunyuan T1/T2 findings).
|
||||
- Fused Metal sparse-conv kernel: built, parity 8e-4, but stock chunking
|
||||
already bounds memory at decoder scale and beats the scalar kernel on
|
||||
speed. Kept as TRELLIS2_METAL_SPCONV=1 opt-in reference (simdgroup
|
||||
tiling = the future version if decoders ever matter for time).
|
||||
- Remaining gap to H100 (~17s at 1024): inside the DiT forwards
|
||||
(54s sampling ≈ 2.3× ideal FLOPs) — attention/MLP internals, a
|
||||
future deep-dive, not orchestration.
|
||||
|
||||
Env knobs (all default-sane): TRELLIS2_MLX_SAMPLER=0 (revert sampler),
|
||||
TRELLIS2_MLX_COMPILE=1, TRELLIS2_METAL_SPCONV=1.
|
||||
M1 Ultra validation: in flight (26.5GB peak fits with 100GB headroom).
|
||||
|
||||
### Rung-3 addendum — M1 Ultra validated (2026-07-20)
|
||||
|
||||
| box | e2e wall | peak mem |
|
||||
|---|---|---|
|
||||
| m3ultra | 71.96 s | 26.5 GB |
|
||||
| **m1ultra** | **181.5 s** | **17.2 GB** |
|
||||
|
||||
The M1 Ultra — which could not run TRELLIS.2 at all before this fork
|
||||
(torch-MPS path was m3-only in practice) — now generates full-density
|
||||
vertex-baked assets in ~3 min at 17GB. Install: clone
|
||||
monster/trellis-2-mrp-mlx to ~/trellis2-mlx + scripts/setup_macos.sh
|
||||
(PYTHON_BIN=uv 3.11) + gated weights rsynced from m3's HF cache
|
||||
(no HF login needed on workers). Both Ultras now serve the fastest
|
||||
local TRELLIS.2 in existence. TRELLIS.2 fleet: m3 primary (72s),
|
||||
m1 second lane (182s), zero dollars per asset.
|
||||
212
docs/MRP_MLX.md
212
docs/MRP_MLX.md
@ -1,212 +0,0 @@
|
||||
# MRP-MLX — the MonsterRobotParty MLX stack
|
||||
|
||||
**The short version:** every serious generative model we run — image, layered
|
||||
image, image→3D, video keying — runs *locally, on Apple Silicon, in MLX or
|
||||
Metal, from forks we control, at $0 per generation.* This document is the
|
||||
canonical reference: what each piece is, where it lives, how fast it goes,
|
||||
how to run it, and the hard-won rules that keep it fast. Written 2026-07-20
|
||||
after the rung-3 kernel push; every number here was measured, not estimated.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Philosophy (why this stack wins)
|
||||
|
||||
1. **Local-first.** Hosted GPU (fal) is for interactive one-offs and A/B
|
||||
references only. Volume work runs on the fleet. (July fal bill was
|
||||
~$93/mo before this stack; the same workload is now ≈ $0.)
|
||||
2. **Adopt-then-patch, never from-scratch.** Every port here started as the
|
||||
best community MLX effort, vendored to our Gitea, then patched where
|
||||
measurement said to. Proven 5×: hunyuan3d (dgrauet), corridorkey
|
||||
(cmoyates/Niko), trellis-2 (Jourloy PR#175), qwen-layered (mflux PR#302),
|
||||
flux (mflux upstream).
|
||||
3. **Forks live on our Gitea** (`ssh://git@100.71.119.27:222/monster/…`)
|
||||
with upstream kept as `origin` for rebases and a pin tag on the adopted
|
||||
commit. Our infra, our history, our rollback.
|
||||
4. **Parity-first.** No optimization ships without a numeric gate against
|
||||
the reference implementation (tolerances recorded), and e2e output
|
||||
stats compared at fixed seed. Fixture sets are kept for regression.
|
||||
5. **Bench-everything, keep the negatives.** BENCHMARKS.md records failures
|
||||
(q4 = noise, mx.compile ≈ 0 on big GEMMs) with the same care as wins, so
|
||||
no future session re-chases a ghost.
|
||||
6. **Env-gated changes.** Every behavioral change has an env knob and a
|
||||
default that preserves upstream behavior (or our validated best). Prod
|
||||
rollback is an environment variable, not a revert.
|
||||
|
||||
## 2 · The fleet it runs on
|
||||
|
||||
| box | chip | RAM | fp16 TFLOPS | bandwidth | role |
|
||||
|---|---|---|---|---|---|
|
||||
| m3ultra | M3 Ultra | 256GB | 23.5 | 400 GB/s | primary: TRELLIS.2, queue, heavy gen |
|
||||
| m1ultra | M1 Ultra | 128GB | 16.3 | 297 GB/s | second lane: everything (post rung-3) |
|
||||
| m1max | M1 Max | 32GB | 7.1 | 336 GB/s | bandwidth-friendly light ops |
|
||||
| m4pro | M4 Pro | 24GB | 5.2 | 225 GB/s | MB light ops + Ollama (hands off) |
|
||||
|
||||
Full 7-machine MLX matrix + probe script: `BENCHMARKS.md` (2026-07-19) +
|
||||
`scripts/mlxprobe.py`. Rules from it: **fp16 on M1-class** (bf16 −20%),
|
||||
bf16 free on M3/M4/M5; pad attention head_dim→64 on M4/M5-class.
|
||||
|
||||
## 3 · The pillars
|
||||
|
||||
### 3.1 TRELLIS.2 — image → full-density 3D · `monster/trellis-2-mrp-mlx`
|
||||
|
||||
- **Base:** Jourloy/TRELLIS.2 (upstream PR#175 head, pin `pr175-head-754d403`)
|
||||
— hybrid MLX (dense stages) + pedronaugusto Metal kernels (sparse ops).
|
||||
- **Our commits:** `fa972aa` pure-MLX sampler loop (batched dense CFG,
|
||||
once-per-stage conversions, one eval/step; parity Δ0.03% e2e) ·
|
||||
`7860148` opt-in Metal spconv kernel + vertex-baker scheduler fix ·
|
||||
`0db816b` vertex-normal export + COLOR_0 asset class in the validator.
|
||||
- **Weights:** original `microsoft/TRELLIS.2-4B` safetensors, loaded
|
||||
directly (`mx.load` + key renames; **no conversion step exists or should**
|
||||
— a converter would double-transpose ss_dec into silent garbage).
|
||||
- **Performance (anatomy bench, seed 42, 1024-cascade, vertex baker):**
|
||||
|
||||
| box | e2e | peak mem |
|
||||
|---|---|---|
|
||||
| m3ultra | **72.0 s** | 26.5 GB |
|
||||
| m1ultra | **181.5 s** | 17.2 GB |
|
||||
| (old torch-MPS `trellis_mac`) | 124.8 s gen (248.8 wall) | 20.8 GB |
|
||||
| (fal, same image) | 298.6 s wall | $0.30 |
|
||||
|
||||
- **Run:** `cd <fork> && .venv/bin/python scripts/generate_asset.py IMG
|
||||
--output-dir OUT --backend mlx-experimental --baker vertex
|
||||
--pipeline-type 1024_cascade --seed N` · Farm: operator `trellis2_mlx`.
|
||||
- **Knobs:** `TRELLIS2_MLX_SAMPLER=0` (revert to torch sampler) ·
|
||||
`TRELLIS2_MLX_COMPILE=1` (≈0 on M3U; try on small boxes) ·
|
||||
`TRELLIS2_METAL_SPCONV=1` (opt-in kernel; slower-but-leaner, reference).
|
||||
- **Output class:** vertex-colored full-density GLB (albedo only). MR maps
|
||||
need the UV path (`--baker metal` — has the dark-patch sampling bug) or
|
||||
the future fast-UV bake. Game budgets: MESHGOD finish farm downstream.
|
||||
- **Checkouts:** m3 `~/Documents/trellis2-mlx-staging/trellis2-jourloy` ·
|
||||
m1 `~/trellis2-mlx` · fresh boxes: `scripts/install_trellis2_mlx.sh`.
|
||||
|
||||
### 3.2 Hunyuan3D 2.1 — image → textured 3D · `monster/Hunyuan3D-2.2-mrp-MLX`
|
||||
|
||||
- **Base:** dgrauet/Hunyuan3D-2.1-mlx (pure MLX, both stages; 13GB
|
||||
pre-converted MLX weights on HF). *Terminology note: this IS the
|
||||
v2-generation Hunyuan; fal's `/v2`,`/v21`,`/turbo` are hosted variants.*
|
||||
- **Our branch `mlx-tune` @ `5a32240`** (deployed both Ultras): T1 UNet
|
||||
mx.compile + T2 whole-DiT compile (env `HY3D_MLX_COMPILE=1`; parity
|
||||
3e-5/7e-6) · T3 StaticMoELayer (bit-identical, kills ~4800 host syncs,
|
||||
makes the DiT compile-legal) · T4 fused-SDPA DINO attention (1.41× its
|
||||
stage, parity 1e-7).
|
||||
- **Performance:** 260 s baseline → **243 s** tuned (m3ultra, defaults).
|
||||
Runs on every box M1-and-up (fp16). ~20GB peak.
|
||||
- **Run:** `generate_e2e.py IMG --output OUT` in `vendor/hunyuan3d-mlx`
|
||||
(venv `.venv`) · Farm: operator `hunyuan3d_mlx`.
|
||||
|
||||
### 3.3 FLUX / image generation — `mflux` (upstream; not forked)
|
||||
|
||||
- mflux 0.18.0 (latest) in `venvs/mflux`, both Ultras + m4pro. Written
|
||||
from scratch in MLX by Filip Strand — no port debt to carry.
|
||||
- **Audit (2026-07-19):** our default path (`flux2-klein-4b`, 4 steps,
|
||||
~9 s/image) has mx.compile on the denoise step + a shapeless-compiled
|
||||
scheduler — healthy, keep. **Known gap:** the FLUX.1 family
|
||||
(dev/schnell/krea) has zero mx.compile — a backport is the same 10-line
|
||||
pattern as Hunyuan T1, expected win 1–4% on Ultras (more on small
|
||||
boxes), and belongs UPSTREAM as an mflux PR rather than a site-packages
|
||||
patch. Queued, not urgent.
|
||||
- Operator `flux_local` (klein default) + `mflux_image_edit`.
|
||||
|
||||
### 3.4 CorridorKey — neural green-screen · `monster/corridorkey-mrp-mlx`
|
||||
|
||||
- **Base:** nikopueringer/corridorkey-mlx (cmoyates' phased parity-first
|
||||
port — its `prompts/` folder is our template for any future from-scratch
|
||||
port). License CC BY-NC-SA — non-commercial only.
|
||||
- **Our patch (`4c660df`, branch `modelbeast`):** honor the compile flag in
|
||||
tiled mode — 8 lines, **1.47×** on m3ultra (3.64→2.48 s/frame), output
|
||||
bit-identical. Plus the 8-machine ablation that found the head_dim→64
|
||||
fast-path rule and the "tiled beats full-frame on quality AND memory"
|
||||
result (2.3GB vs 28GB). Nightly perfcheck watches the fast path.
|
||||
|
||||
### 3.5 Qwen-Image-Layered — image → editable RGBA layers · `monster/Qwen-Image-Layered-MRP-MLX`
|
||||
|
||||
- **Base:** mflux PR#302 (ZimengXiong; unmerged upstream), pin
|
||||
`pr302-head-a255e4f`. 20B, Apache 2.0 (ship-commercial OK).
|
||||
- **Validated ladder:** **q8 is the quality FLOOR** — q6 breaks the layer
|
||||
semantics (silhouettes), q4 is noise; speed is FLAT across quants
|
||||
(compute-bound), so never quantize below q8. 20 steps ≈ 50 steps
|
||||
visually (2.9× faster). 1024-res = hero tier.
|
||||
- **Performance (4 layers, 640, baked q8, 20 steps):** m3ultra **243 s** ·
|
||||
m1ultra 366 s · 1024-res hero: 858 s (m3). Peak ~37GB.
|
||||
- **Baked models:** m3 `~/qwen-layered/qwen-layered-q8` · m1
|
||||
`~/qwen-layered-staging/qwen-layered-q8` (34GB; rebake via `mflux-save`).
|
||||
- **Run:** `mflux-generate-qwen-layered --image IMG --layers 4
|
||||
--resolution 640 --steps 20 --model-path <baked>` · Farm: operator
|
||||
`qwen_layered_local`. Bonus: the subject layer is matting-grade — the
|
||||
premium tier of image preprocessing for hero assets.
|
||||
|
||||
## 4 · MODELBEAST integration
|
||||
|
||||
| operator | backs | speed (m3 / m1) | notes |
|
||||
|---|---|---|---|
|
||||
| `trellis2_mlx` | 3.1 | 72 s / 182 s | vertex baker default; **preferred 3D lane** |
|
||||
| `trellis_mac` | torch-MPS port | 125 s gen / n/a | legacy; UV-textured (dark-patch bug) |
|
||||
| `hunyuan3d_mlx` | 3.2 | 243 s / ~5-6 min | textured GLB; runs everywhere |
|
||||
| `flux_local` | 3.3 | ~9 s/img | klein-4b 4-step default |
|
||||
| `qwen_layered_local` | 3.5 | 243 s / 366 s | RGBA layer decomposition |
|
||||
|
||||
Routing lives in `nodes.json` (machine-local, gitignored — edit on the
|
||||
m3 primary). MESHGOD's `local/` lane calls these via the MB REST API.
|
||||
|
||||
## 5 · The lessons ledger (do not re-learn these)
|
||||
|
||||
1. **Profile before optimizing — every time.** The rung-3 3× came from a
|
||||
scheduler bug fix + baker swap the profiler exposed; the glamorous
|
||||
kernel work measured ≈ 0. The 75GB "sparse conv" memory monster was
|
||||
actually the Metal texture bake.
|
||||
2. **mx.compile ≈ 0 on GEMM-bound 2B+ models on M3 Ultra** (measured 4×:
|
||||
Hunyuan T1, T2, TRELLIS step, sampler). It's free insurance, not a win.
|
||||
Compile wins live on small models (corridorkey 1.47×) and maybe small
|
||||
chips.
|
||||
3. **Quantization floors are model-specific and cliff-shaped.** Qwen-
|
||||
Layered: q8 fine, q6 semantically broken, q4 noise — and quant does
|
||||
NOT buy speed on compute-bound models, only footprint.
|
||||
4. **glTF axis + gamma:** pipeline space is Z-up in [-0.5,0.5]³; glTF is
|
||||
Y-up (`(x,y,z)→(x,z,−y)`); COLOR_0 is LINEAR (never pre-gamma).
|
||||
A "mangled blob" render is usually just the figure viewed down its own
|
||||
axis after a convention miss.
|
||||
5. **xatlas hangs (hours) on écorché-class meshes** (thousands of
|
||||
disconnected shells). Vertex bake sidesteps UV unwrap entirely.
|
||||
6. **macOS TCC blocks launchd from ~/Documents** — services live in `~/`.
|
||||
7. **Gated HF weights don't need logins on workers** — rsync the model
|
||||
dirs from a cache that has them.
|
||||
8. **The fal-outage class of bug:** any `while True` polling a paid API
|
||||
needs a deadline + a cancel; an in-memory queue wedges silently.
|
||||
9. **Adapters that convert per-call hide O(steps) waste** — convert at
|
||||
stage boundaries, hold state in the fast runtime.
|
||||
10. **Sentinel fallthroughs bite:** a baker/scheduler that silently falls
|
||||
back can burn 5 hours before anyone notices. Fail loudly instead.
|
||||
|
||||
## 6 · Runbooks
|
||||
|
||||
**New box bring-up (any Apple Silicon, macOS 26+):**
|
||||
1. Keys: box's SSH key added to Gitea (`monster` account).
|
||||
2. `git clone` the MODELBEAST repo; run the relevant
|
||||
`scripts/install_*.sh` (trellis2_mlx, qwen_layered, hunyuan…) — each is
|
||||
idempotent, uses uv, and states its weight needs.
|
||||
3. Weights: rsync the HF cache model dirs from m3ultra (see §5.7).
|
||||
4. Bench: `uv run --with mlx python scripts/mlxprobe.py` → add the row to
|
||||
BENCHMARKS fleet matrix; set fp16/bf16 per §2 rules.
|
||||
5. Register the box + operator list in `nodes.json` on the m3 primary.
|
||||
|
||||
**Rollback:** every MRP patch is env-gated (§3 knobs) or a pinned-tag fork
|
||||
— `git checkout <pin-tag>` restores the adopted upstream exactly.
|
||||
|
||||
**Where results go:** every experiment appends to `BENCHMARKS.md` (wall
|
||||
time, peak mem via `mx.get_peak_memory()`, parity numbers, and a verdict
|
||||
line). Session-scale context lives in `docs/FABLE_STARTUP_PACKET.md`.
|
||||
|
||||
## 7 · Roadmap (scoped, in value order)
|
||||
|
||||
1. **DiT-internals deep-dive** — TRELLIS.2 sampling runs ≈2.3× above ideal
|
||||
FLOPs (54 s of the 72). Attention/MLP kernel work inside the forward;
|
||||
the corridorkey `prompts/` phased method is the template.
|
||||
2. **Fast UV bake** — bucket rasterizer (written, in fast_bake_test.py) +
|
||||
an écorché-safe unwrap → texture+MR parity with fal on the vertex
|
||||
baker's quality. Unlocks `--baker vertex`-quality WITH UV maps.
|
||||
3. **simdgroup spconv kernel** — upgrade the opt-in scalar kernel if the
|
||||
decoders ever matter for time (they're 10 s today).
|
||||
4. **Upstreaming** — our sampler + baker + validator to Jourloy PR#175;
|
||||
FLUX.1 compile backport as an mflux PR; spconv kernel as reference.
|
||||
5. **MESHGOD default flip** — route its `local/` 3D lane to `trellis2_mlx`
|
||||
once finish-farm verts→texture baking lands for game budgets.
|
||||
@ -1,143 +0,0 @@
|
||||
# Local 3D → full MLX + preprocessing frontend — master build plan
|
||||
|
||||
**For:** a fresh long-running Fable session on m3ultra (no execution-time cap).
|
||||
**Author:** Opus planning pass, 2026-07-19. **Read first:** `docs/TRELLIS2_MLX_RECON.md`,
|
||||
`BENCHMARKS.md` (fleet MLX matrix + rung 1/2), `HANDOFF_HY3D_MLX.md`, `CORRIDORKEY.md`.
|
||||
**Prime directive:** parity-first. Every kernel/port change ships with a PT-parity test
|
||||
(≤1e-4 rel) + a BENCHMARKS.md row (per-stage time + `mx.get_peak_memory()`). No silent
|
||||
regressions — that discipline is why hunyuan3d-mlx and corridorkey-mlx worked.
|
||||
|
||||
**Two goals, independent, run in parallel:**
|
||||
- **G1 — the models go fully MLX + fast** (TRELLIS.2 port from scratch-ish; Hunyuan tune).
|
||||
- **G2 — a local preprocessing frontend** that turns any messy photo into the input the
|
||||
models want. Model-agnostic; helps every generation from here on.
|
||||
|
||||
---
|
||||
|
||||
## Ground truth (measured, don't re-derive)
|
||||
|
||||
- Local TRELLIS.2 = `vendor/trellis-mac` (shivampkumar fork), **torch-MPS, NOT MLX**.
|
||||
124.8s/gen on m3ultra (1024-cascade, tex2048), 20.8GB peak. Weights `TRELLIS.2-4B` 14GB.
|
||||
- Stage breakdown of the 124.8s: **~46s diffusion sampling** (padded SDPA attention),
|
||||
**~50s decode + mesh extract** (incl. `aten::segment_reduce` → **CPU fallback**),
|
||||
**19s Metal PBR bake**. Decoder emits **2.79M faces** pre-simplify.
|
||||
- Sparse-conv backend already on the Metal fast path (`flex_gemm`); worth **1.72×** vs the
|
||||
pure-torch fallback (rung 1). Metallib loads on macOS 26.5.
|
||||
- Hunyuan3D-2.1 = `vendor/hunyuan3d-mlx` (dgrauet), **already MLX**, ~260s m3ultra, runs on M1.
|
||||
- Quality gap vs fal = dark shattered patches; density is NOT the cause (rung 2: 200K/800K/2.79M
|
||||
all show it) → **texture bake or decoder attrs**. KDTree-baker A/B pending (`TRELLIS2_FORCE_KDTREE`).
|
||||
- Fleet MLX matrix (BENCHMARKS.md): bf16 fine except M1 (~20% slower → ship fp16 on M1);
|
||||
**pad attention head_dim→64** (up to 4.7× M5, 1.5× M3U, ~0 on M1); `mx.compile` on fixed shapes.
|
||||
- Knobs already added to `vendor/trellis-mac/generate.py`: `TRELLIS2_MAX_BAKE_FACES` (0=uncapped),
|
||||
`TRELLIS2_FORCE_KDTREE` (A/B the baker). Bench artifacts in `~/Documents/trellis2-bench/`.
|
||||
|
||||
---
|
||||
|
||||
## G1 · WORKSTREAM A — TRELLIS.2 → MLX
|
||||
|
||||
### A0. Scout the existing MLX efforts (do FIRST — decides adopt vs build)
|
||||
- Vendor `pedronaugusto/trellis2-apple` (has an `mlx_backend/`) and upstream **PR #175**
|
||||
(MPS+Metal + experimental `mlx` flag, 28 tests green on M4 Max). Fork both to Gitea as
|
||||
`monster/trellis2-mrp-*` (house pattern; keep upstream remote for rebases).
|
||||
- Install each, run ONE gen on the anatomy bench image, seed 42, 1024-cascade. Compare to the
|
||||
124.8s MPS baseline **and** to fal quality (renders already in `trellis2-bench/`).
|
||||
- **Decision gate:** if trellis2-apple's MLX path is correct + faster → we ADOPT and jump to A3
|
||||
(patch, not rebuild). If it's partial/slower/buggy → we BUILD (A1→A2), using it + the
|
||||
corridorkey `prompts/` 6-phase template as reference.
|
||||
|
||||
### A1. Scaffold the MLX port (if building)
|
||||
- New repo `monster/trellis2-mlx`. Mirror the corridorkey-mlx phased layout: `prompts/` phase docs,
|
||||
`src/`, `tests/test_parity.py`, `scripts/convert_weights.py`.
|
||||
- Weight conversion: TRELLIS.2-4B safetensors → MLX. Reuse the hunyuan pattern
|
||||
(`hy3dpaint/utils/convert_realesrgan.py`): torch load → key remap → **transpose Conv (O,I,H,W)→(O,H,W,I)** → save. fp16 (M1-safe); publish converted weights to HF/Gitea so re-runs skip conversion.
|
||||
- Build a PT reference harness first (dump per-stage activations from the working MPS build as
|
||||
fixtures) — every MLX module validates against these.
|
||||
|
||||
### A2. The kernel rewrites (the real work — this is where the days go)
|
||||
Priority = by measured cost. Each = an `mx.fast.metal_kernel` + parity test + bench row.
|
||||
1. **Sparse-conv (FlexGEMM → fused Metal).** THE crux and the ~10× lever. Triton has no Metal
|
||||
target; implement gather→GEMM→scatter as a fused Metal kernel. Reference: the existing
|
||||
`mtlgemm` Metal port in trellis-mac/deps and `backends/conv_none.py` (the slow pure-torch
|
||||
version) for correct semantics. MLX has no sparse tensor type — hand-roll SparseTensor as
|
||||
coord-list + hash (same as CUDA/MPS side).
|
||||
2. **Varlen/sparse attention (padded SDPA → fused).** ~46s of the run. Kill the pad-to-maxlen
|
||||
waste; pad head_dim→64 for the fused fast path (fleet matrix). This + #1 close most of the gap.
|
||||
3. **`segment_reduce` MPS→CPU fallback → MLX-native.** Currently silently on CPU each step;
|
||||
pure MLX scatter-add removes a real tax in the decode path.
|
||||
4. **o-voxel hashmap + dual-grid mesh extract.** Keep the Metal/CPU fork if it's not hot; port to
|
||||
MLX only if profiling says so.
|
||||
Leave the PBR bake on the existing Metal path (`mtldiffrast`) unless G1-C convicts it.
|
||||
|
||||
### A3. Fleet-tune + the prize
|
||||
- Apply corridorkey ablation moves across the port: head_dim→64, `mx.compile` fixed shapes,
|
||||
sdpa gating by GPU generation, tiled-vs-full sweeps. Nightly perfcheck like corridorkey.
|
||||
- **Prize: M1 Ultra compatibility.** MLX-native runs where torch-MPS/bf16 is shaky → a *second*
|
||||
free TRELLIS box (297 GB/s, 128GB). Validate a full gen on `johnking@100.91.239.7`.
|
||||
- Wire `trellis_mac`→MLX into MODELBEAST `nodes.json` (m3ultra primary, m1ultra fp16 second) and
|
||||
add `trellis2` to MESHGOD's `local/` model list beside `hunyuan3d_mlx`.
|
||||
|
||||
## G1 · WORKSTREAM B — Hunyuan3D-2.1-MLX optimization (already MLX; just tune)
|
||||
- It's a *tune*, not a port. Apply the same corridorkey wins to `vendor/hunyuan3d-mlx`:
|
||||
pad attention head_dim→64, `mx.compile` on fixed-shape blocks (DiT denoiser, VAE), sdpa gating.
|
||||
- Check `mlx_arsenal` rasterizer (`mesh_render_mlx.py`, 1137 LOC) for the same fixed-shape
|
||||
compile opportunity. Bench each on the fleet matrix; expect corridorkey-scale wins (1.1–1.5×).
|
||||
- Low risk, high certainty — good "warm-up" task while A0 scouting runs.
|
||||
|
||||
## G1 · WORKSTREAM C — quality (rung 2 close-out; feeds BOTH MPS + MLX)
|
||||
- **KDTree-baker A/B** (`TRELLIS2_FORCE_KDTREE=1`, running now). If its surface is clean → the
|
||||
**Metal baker (mtldiffrast/mtlbvh texel→voxel sampling) is the dark-patch culprit** → fix or
|
||||
default-swap the baker. If still dirty → decoder voxel attrs are suspect → investigate upstream.
|
||||
- Keep `TRELLIS2_MAX_BAKE_FACES` (geometry win, free).
|
||||
- Add a **hole-fill pass** (CuMesh replacement) pre-bake — Blender fill or a voxel-remesh.
|
||||
- Loud assertion if `flex_gemm` metallib ever fails to load (silent → +90s/gen forever).
|
||||
|
||||
---
|
||||
|
||||
## G2 · WORKSTREAM D — image preprocessing frontend ("best chance of success")
|
||||
|
||||
North star already written in `meshgod/config.py`: *centered · plain neutral light-gray bg ·
|
||||
even shadowless studio light · neutral symmetric A-pose*. The job = transform any user photo → that.
|
||||
Model-agnostic; helps TRELLIS.2 AND Hunyuan equally; no fal/network.
|
||||
|
||||
### D1. `prep_local()` — the front line (highest ROI, pure glue)
|
||||
- RMBG-2.0 (local `rmbg` venv) or BiRefNet matte → **composite on neutral gray (NOT alpha** —
|
||||
config's own spec) → bbox crop → center → square-pad → upscale to ≥1024.
|
||||
- Wire as a LOCAL replacement for MESHGOD's current `clean_bg` (today it's a fal call). Respect the
|
||||
learned order: **upscale FIRST, matte LAST** (SeedVR strips alpha).
|
||||
- ⚠️ **Licensing:** RMBG-2.0 is CC-BY-NC → local/personal only. Ship-commercial path = **BiRefNet
|
||||
(Apache)**. Expose both; default BiRefNet for anything headed to a game.
|
||||
|
||||
### D2. Lighting/exposure normalization
|
||||
- Auto-levels + gray-world WB before recon. TRELLIS bakes light into albedo → even input = cleaner
|
||||
texture, and directly fights the dark-patch failure mode. Trivial PIL; measurable.
|
||||
|
||||
### D3. Multi-view synthesis — the RIGHT fix for touching limbs
|
||||
- Single-view can't separate an arm fused to a torso in silhouette. Generate back + ¾ views (image
|
||||
model) → feed **multi-view reconstruction** (both models support it; MESHGOD already wires the MV
|
||||
path). The extra angle disambiguates. Prototype: photo → gen back view → MV recon → compare.
|
||||
|
||||
### D4. Input QA gate (save wasted gens)
|
||||
- Pre-flight heuristic/VLM (local Ollama VLM or DINOv3-embedding rules) flags too-small, busy-bg,
|
||||
cropped-limb, multi-subject BEFORE spending a generation. Cheap insurance.
|
||||
|
||||
### D5. T-pose normalization = POST-process via MIRPAMO (NOT a preprocess)
|
||||
- Reposing a 2D human hallucinates. Correct pipeline: reconstruct in-pose → **MIRPAMO auto-rig +
|
||||
retarget to T/A-pose**. Design-doc it; tie in MOCAPGOD/HSMR (SMPL fit, on m1) as the body prior.
|
||||
|
||||
---
|
||||
|
||||
## Suggested execution order for the long session
|
||||
1. **G1-C** close-out (read KDTree result, act) — unblocks the quality question. *(minutes)*
|
||||
2. **G1-B** Hunyuan tune — certain win, warms up the parity/bench rhythm. *(hours)*
|
||||
3. **G2-D1+D2** preprocessing front line — compounding ROI on every future gen. *(hours)*
|
||||
4. **G1-A0** scout MLX ports — the adopt-vs-build decision gate. *(hour)*
|
||||
5. **G1-A1→A3** the TRELLIS.2 MLX port + kernels — the multi-day main event. *(days)*
|
||||
6. **G2-D3/D4/D5** as reach goals.
|
||||
|
||||
## Definition of done
|
||||
- TRELLIS.2 runs MLX on BOTH Ultras, faster than today's 124.8s MPS, parity ≤1e-4, in `nodes.json`.
|
||||
- Hunyuan tuned with a measured fleet-matrix win.
|
||||
- Dark-patch quality gap root-caused and closed (or a clear upstream bug filed).
|
||||
- `prep_local()` live as MESHGOD's default local pre-pass; BiRefNet for commercial, multi-view
|
||||
path proven on a touching-limbs case.
|
||||
- Every step has a BENCHMARKS.md row. Upstreamable kernel work offered to PR #175.
|
||||
@ -1,106 +0,0 @@
|
||||
# TRELLIS.2 on Apple Silicon — recon (2026-07-18)
|
||||
|
||||
**Verdict up front:** we already run TRELLIS.2 locally — `vendor/trellis-mac`
|
||||
(torch-MPS + Metal kernels) IS TRELLIS.2-4B and is wired as the
|
||||
`trellis_mac` operator. There is no "MLX interface" shortcut for CUDA code
|
||||
(CUDA kernels must be *rewritten*, not wrapped), but ~90% of TRELLIS.2 is
|
||||
standard tensor math that MPS/MLX already runs; only four custom CUDA
|
||||
pieces matter, and the community has replaced all four. The open work is
|
||||
**speed** (unfused sparse ops ≈ 10× slower than CUDA) and **quality gaps**
|
||||
(hole-filling disabled, forced pre-simplification) — not feasibility.
|
||||
|
||||
## What we run today (`vendor/trellis-mac`, shivampkumar fork)
|
||||
|
||||
- torch-MPS + `PYTORCH_ENABLE_MPS_FALLBACK`, Metal kernels by @pedronaugusto
|
||||
(mtlgemm / mtldiffrast / mtlbvh / mtlmesh), SDPA attention.
|
||||
- Full inference: sparse structure → shape SLat → tex SLat → decode →
|
||||
dual-grid mesh extract → simplify → Metal PBR bake → GLB.
|
||||
- Our benchmark: **318 s/gen on m3ultra**, ~18 GB peak (M4 Pro 24GB: 5m13s
|
||||
cold at 512). H100 does 3–17 s — the gap is almost entirely the unfused
|
||||
sparse conv + padded attention.
|
||||
- Known gaps vs fal's trellis-2: CuMesh skipped → **no hole filling**,
|
||||
meshes pre-simplified ~858K→200K faces before baking, Metal BVH
|
||||
instability, macOS GPU-watchdog can kill long kernels (detected +
|
||||
workarounds printed by generate.py). macOS 26 needed for the metallib
|
||||
(fleet is on 26.5 ✓).
|
||||
- Licensing: DINOv3 is Meta-gated; **RMBG-2.0 preprocessing is CC BY-NC**.
|
||||
|
||||
## What TRELLIS.2 actually needs (upstream: Linux, CUDA 12.4, ≥24 GB)
|
||||
|
||||
| CUDA dep | Role | Mac status |
|
||||
|---|---|---|
|
||||
| o-voxel ext (hash/convert/rasterize) | the O-Voxel representation | pure-Python reimpl (trellis-mac `backends/`) + CPU fork |
|
||||
| FlexGEMM (**Triton**) | all sparse conv | **the crux** — Triton ≠ Metal; Metal `mtlgemm` or slow pure-torch gather/scatter |
|
||||
| flash-attn / xformers | 4B flow transformer attention | SDPA (padded → unfused, big cost) |
|
||||
| CuMesh | decimate/remesh/**hole-fill**/UV | skipped → `fast_simplification`; hole-fill lost |
|
||||
| nvdiffrast | texture bake | `mtldiffrast` (Metal) |
|
||||
| nvdiffrec | preview renders only | not needed |
|
||||
|
||||
## The MLX question, answered
|
||||
|
||||
- **No shim exists or can exist**: CUDA kernels are NVIDIA-machine code;
|
||||
"an MLX interface" means rewriting each custom op in MLX/Metal. MLX can
|
||||
express them (`mx.fast.metal_kernel` JIT-compiles Metal from Python) —
|
||||
but MLX has **no sparse-tensor type and no sparse-voxel precedent**;
|
||||
SparseTensor/varlen semantics must be hand-rolled. A TRELLIS.2-MLX would
|
||||
be a first.
|
||||
- **Our own playbook (proven 2×) is adopt-then-patch, not from-scratch**:
|
||||
hunyuan3d-mlx = dgrauet's 8.4k-LOC port + our thin packaging layer;
|
||||
corridorkey-mlx = cmoyates/Niko's 5.3k-LOC port + our **8-line**
|
||||
`mx.compile`-in-tiled-mode patch = the 1.47× win. From-scratch dense→MLX
|
||||
ports of this scale are months of solo work (mflux, mlx-video authors).
|
||||
- **Starting points already exist**: upstream **PR #175** (Jourloy,
|
||||
2026-07-17 — MPS + Metal + an *experimental `mlx` backend flag*, 28
|
||||
tests green on M4 Max) and **pedronaugusto/trellis2-apple** (an
|
||||
`mlx_backend/` dir, no benchmarks yet).
|
||||
|
||||
## Recommended ladder (effort-ordered)
|
||||
|
||||
1. **Hours — tune what we have**: benchmark `trellis_mac` 1024_cascade vs
|
||||
fal trellis-2 on identical inputs (BENCHMARKS.md format); route
|
||||
MESHGOD's batch/overnight work to the local lane (m3ultra clears 18 GB
|
||||
~14× over; fal stays for interactive one-offs). $93/mo → mostly $0.
|
||||
2. **Days — adopt + fleet-patch**: vendor PR #175 / trellis2-apple as
|
||||
`monster/trellis2-*` Gitea forks (house pattern); run the corridorkey
|
||||
ablation moves on them: attention head_dim → pad to 64 fast-path,
|
||||
`mx.compile` on fixed shapes, sdpa gating by GPU generation, tiled-vs-
|
||||
full sweeps. Prize: **M1 Ultra compatibility** (MLX-native, like
|
||||
hunyuan3d — today trellis is m3-only in practice) = 2nd free 3D box.
|
||||
3. **Weeks — the real kernel work** (only if we want fal-class speed):
|
||||
fused Metal sparse-conv (gather-GEMM-scatter) + varlen attention via
|
||||
`mx.fast.metal_kernel`; port CuMesh hole-filling. Closes most of the
|
||||
10× gap; genuinely novel, upstreamable to PR #175.
|
||||
4. **Quality parity misc**: raise simplification budget on 256 GB boxes,
|
||||
swap RMBG-2.0 → our licensed bg-remove lane, wire `trellis_mac` into
|
||||
MESHGOD's `local/` model list next to hunyuan3d-mlx.
|
||||
|
||||
## Sources
|
||||
|
||||
- github.com/microsoft/TRELLIS.2 (setup.sh = dep manifest) · PR #175 ·
|
||||
issue #74 · shivampkumar/trellis-mac · pedronaugusto/trellis2-apple
|
||||
- ml-explore.github.io/mlx custom-metal-kernels docs
|
||||
- Local: HANDOFF_HY3D_MLX.md, CORRIDORKEY.md, BENCHMARKS.md,
|
||||
vendor/trellis-mac/README, vendor/corridorkey-mlx/prompts/ (the 6-phase
|
||||
parity-first port template — the blueprint if we ever do ladder step 3)
|
||||
|
||||
## First benchmark (2026-07-19, m3ultra, anatomy écorché test image)
|
||||
|
||||
| run | tris | time | peak mem | cost |
|
||||
|---|---|---|---|---|
|
||||
| trellis_mac 1024-cascade, tex 2048 | 191,336 | **124.8 s** compute (248.8 s wall, 103 s load) | 20.8 GB | $0 |
|
||||
| fal trellis-2, same image | 468,049 | 298.6 s wall (queue incl.) | — | $0.30 |
|
||||
|
||||
**Local already beats fal on wall-clock.** Quality is the gap, not speed:
|
||||
the local master shows shattered dark patches — CuMesh hole-fill skipped +
|
||||
forced 858k→191k pre-simplify before baking (the un-simplified master IS
|
||||
saved as .obj alongside the GLB). Fix quality first, then fuse kernels.
|
||||
|
||||
Tri-budget ladder (Blender decimate on the fal master): 2k = confetti
|
||||
(ratio floors at 0.01 → 4,674 min), 8k = torn, 30k = good, 50k ≈ master.
|
||||
**Écorché-class meshes (many disconnected thin shells) cannot pure-decimate
|
||||
below ~30k** — game budgets need the MESHGOD solidify/remesh route.
|
||||
Discovered in passing: MESHGOD finish_glb.py's tri budget is silently
|
||||
ignored on Blender 5 (modifier_apply cancels — fix task spawned 2026-07-19).
|
||||
|
||||
Bench artifacts: m3ultra ~/Documents/trellis2-bench/ (masters, sweeps,
|
||||
renders, logs, sweep2.csv / sweep_fal.csv).
|
||||
@ -1,25 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Install the Qwen-Image-Layered MLX runtime (mflux PR#302 fork, vendored at
|
||||
# our Gitea) into vendor/mflux-qwen-layered + a uv venv. Idempotent.
|
||||
#
|
||||
# Model weights are NOT fetched here. run.py resolves, in order:
|
||||
# $QWEN_LAYERED_MODEL -> ~/qwen-layered/qwen-layered-q8 (m3)
|
||||
# -> ~/qwen-layered-staging/qwen-layered-q8 (m1)
|
||||
# -> HF Qwen/Qwen-Image-Layered with on-the-fly -q8 (54GB download).
|
||||
# Bake a local q8 once per box: .venv/bin/mflux-save \
|
||||
# --model Qwen/Qwen-Image-Layered --base-model qwen-image-layered \
|
||||
# --quantize 8 --path ~/qwen-layered/qwen-layered-q8
|
||||
set -euo pipefail
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
DEST="$ROOT/vendor/mflux-qwen-layered"
|
||||
REPO="ssh://git@100.71.119.27:222/monster/Qwen-Image-Layered-MRP-MLX.git"
|
||||
|
||||
if [ ! -d "$DEST/.git" ]; then
|
||||
git clone "$REPO" "$DEST"
|
||||
else
|
||||
echo "already cloned: $DEST"
|
||||
fi
|
||||
cd "$DEST"
|
||||
/opt/homebrew/bin/uv venv .venv --python 3.12
|
||||
/opt/homebrew/bin/uv pip install -q --python .venv/bin/python -e .
|
||||
.venv/bin/mflux-generate-qwen-layered --help >/dev/null && echo "qwen-layered OK: $DEST"
|
||||
@ -1,20 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Vendor the MRP MLX fork of TRELLIS.2 (monster/trellis-2-mrp-mlx) for the
|
||||
# trellis2_mlx operator. Idempotent. Existing per-box clones also work
|
||||
# (run.py resolves m3 staging / m1 ~/trellis2-mlx automatically) — this
|
||||
# script is for fresh boxes.
|
||||
# Weights: TRELLIS.2-4B + gated facebook/dinov3 + briaai/RMBG-2.0 — rsync
|
||||
# from m3ultra's HF cache (no HF login):
|
||||
# for M in models--facebook--dinov3-vitl16-pretrain-lvd1689m \
|
||||
# models--microsoft--TRELLIS.2-4B models--briaai--RMBG-2.0; do
|
||||
# rsync -a m3ultra@100.89.131.57:.cache/huggingface/hub/$M ~/.cache/huggingface/hub/
|
||||
# done
|
||||
set -euo pipefail
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
DEST="$ROOT/vendor/trellis2-mlx"
|
||||
REPO="ssh://git@100.71.119.27:222/monster/trellis-2-mrp-mlx.git"
|
||||
[ -d "$DEST/.git" ] && echo "already cloned: $DEST" || git clone "$REPO" "$DEST"
|
||||
cd "$DEST"
|
||||
PB="${PYTHON_BIN:-$(/opt/homebrew/bin/uv python find 3.11)}"
|
||||
PYTHON_BIN="$PB" bash scripts/setup_macos.sh
|
||||
.venv/bin/python -c "import mlx_backend" && echo "trellis2-mlx OK: $DEST"
|
||||
@ -1,90 +0,0 @@
|
||||
"""MLX fleet probe — one JSON line of capability + micro-bench per machine.
|
||||
|
||||
Benches (each ~seconds, sized for 16GB boxes):
|
||||
matmul fp16/bf16 4096^2 -> TFLOPS (bf16-on-M1 question)
|
||||
elementwise add 256MB -> GB/s (memory bandwidth proxy)
|
||||
sdpa head_dim 56 vs 64 -> ratio (fused fast-path check, corridorkey lesson)
|
||||
"""
|
||||
import json
|
||||
import platform
|
||||
import subprocess
|
||||
import time
|
||||
|
||||
R = {"host": platform.node().split(".")[0]}
|
||||
try:
|
||||
R["chip"] = subprocess.check_output(
|
||||
["sysctl", "-n", "machdep.cpu.brand_string"], text=True).strip()
|
||||
R["ram_gb"] = round(int(subprocess.check_output(
|
||||
["sysctl", "-n", "hw.memsize"], text=True)) / 2**30)
|
||||
R["macos"] = subprocess.check_output(
|
||||
["sw_vers", "-productVersion"], text=True).strip()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
import mlx.core as mx
|
||||
R["mlx"] = getattr(mx, "__version__", "?")
|
||||
except Exception as e:
|
||||
R["mlx"] = None
|
||||
R["error"] = f"mlx import failed: {e}"
|
||||
print(json.dumps(R))
|
||||
raise SystemExit(0)
|
||||
|
||||
try:
|
||||
di = mx.metal.device_info() if hasattr(mx, "metal") else {}
|
||||
R["gpu"] = {k: di[k] for k in ("architecture", "max_recommended_working_set_size",
|
||||
"memory_size") if k in di}
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def timeit(fn, warmup=3, iters=10):
|
||||
for _ in range(warmup):
|
||||
mx.eval(fn())
|
||||
t0 = time.perf_counter()
|
||||
for _ in range(iters):
|
||||
mx.eval(fn())
|
||||
return (time.perf_counter() - t0) / iters
|
||||
|
||||
|
||||
N = 4096
|
||||
flops = 2 * N * N * N
|
||||
for dt, name in ((mx.float16, "fp16"), (mx.bfloat16, "bf16")):
|
||||
try:
|
||||
a = mx.random.normal((N, N)).astype(dt)
|
||||
b = mx.random.normal((N, N)).astype(dt)
|
||||
mx.eval(a, b)
|
||||
t = timeit(lambda: a @ b)
|
||||
R[f"matmul_{name}_tflops"] = round(flops / t / 1e12, 2)
|
||||
except Exception as e:
|
||||
R[f"matmul_{name}_tflops"] = f"fail: {e}"
|
||||
|
||||
try:
|
||||
M = 64 * 1024 * 1024 # 64M floats = 256MB per array
|
||||
x = mx.random.normal((M,))
|
||||
y = mx.random.normal((M,))
|
||||
mx.eval(x, y)
|
||||
t = timeit(lambda: x + y, warmup=2, iters=8)
|
||||
R["bandwidth_gbs"] = round(3 * M * 4 / t / 1e9) # 2 reads + 1 write, fp32
|
||||
except Exception as e:
|
||||
R["bandwidth_gbs"] = f"fail: {e}"
|
||||
|
||||
try:
|
||||
B, H, L = 1, 8, 2048
|
||||
sdpa = {}
|
||||
for D in (56, 64):
|
||||
q = mx.random.normal((B, H, L, D)).astype(mx.float16)
|
||||
k = mx.random.normal((B, H, L, D)).astype(mx.float16)
|
||||
v = mx.random.normal((B, H, L, D)).astype(mx.float16)
|
||||
mx.eval(q, k, v)
|
||||
t = timeit(lambda: mx.fast.scaled_dot_product_attention(
|
||||
q, k, v, scale=D ** -0.5), warmup=3, iters=20)
|
||||
sdpa[f"d{D}_us"] = round(t * 1e6)
|
||||
sdpa["d56_vs_d64"] = round(sdpa["d56_us"] / max(sdpa["d64_us"], 1), 2)
|
||||
R["sdpa"] = sdpa
|
||||
except Exception as e:
|
||||
R["sdpa"] = f"fail: {e}"
|
||||
|
||||
R["peak_mem_gb"] = round(mx.get_peak_memory() / 2**30, 2) \
|
||||
if hasattr(mx, "get_peak_memory") else None
|
||||
print(json.dumps(R))
|
||||
23
server/operators/corridorkey_local/manifest.json
Normal file
23
server/operators/corridorkey_local/manifest.json
Normal file
@ -0,0 +1,23 @@
|
||||
{
|
||||
"id": "corridorkey_local",
|
||||
"name": "CorridorKey Green-Screen Unmix (local)",
|
||||
"category": "video-prep",
|
||||
"description": "Green/blue-screen clip or frame → true un-multiplied foreground color + linear alpha via Corridor's neural keyer (the corridorkey-mrp-mlx fork, MLX on Apple silicon). Preserves hair, motion blur and translucency — no binary roto masks. Input: a video shot on green/blue, or a single frame — NOT square (GVM's resize rejects smaller-edge ≥1024-after-scale square plates; 16:9/9:16 is fine). Output: zipped frame sequence (straight color + alpha) plus a comp preview when enabled. Free, fully local.",
|
||||
"accepts": ["video", "image"],
|
||||
"produces": ["archive"],
|
||||
"resources": "gpu",
|
||||
"entry": "run.py",
|
||||
"python": "vendor/corridorkey/.venv/bin/python",
|
||||
"params_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"screen_color": {"type": "string", "enum": ["auto", "green", "blue"], "default": "auto", "description": "Screen color (blue = torch backend only for now)"},
|
||||
"backend": {"type": "string", "enum": ["auto", "mlx", "torch"], "default": "auto", "description": "Inference backend"},
|
||||
"despill": {"type": "integer", "minimum": 0, "maximum": 10, "default": 5, "description": "Despill strength"},
|
||||
"refiner": {"type": "number", "default": 1.0, "description": "Refiner strength multiplier"},
|
||||
"comp": {"type": "boolean", "default": true, "description": "Also render a comp preview"},
|
||||
"max_frames": {"type": "integer", "description": "Limit frames (quick tests)"},
|
||||
"image_size": {"type": "integer", "description": "Inference size override (default: model native)"}
|
||||
}
|
||||
}
|
||||
}
|
||||
115
server/operators/corridorkey_local/run.py
Normal file
115
server/operators/corridorkey_local/run.py
Normal file
@ -0,0 +1,115 @@
|
||||
"""corridorkey_local — Corridor's neural green-screen unmixer, headless.
|
||||
|
||||
Shells out to the vendored CLI (vendor/corridorkey, the corridorkey-mrp-mlx fork):
|
||||
stage the input as a clip → generate-alphas (GVM coarse hint) → run-inference with
|
||||
every flag set (non-interactive) → zip Output/<clip> as the result.
|
||||
|
||||
Clip staging uses the job's outdir basename as the clip name, so concurrent jobs
|
||||
can't collide; the shared ClipsForInference/Output dirs are cleaned afterwards
|
||||
(win or lose) so the vendor tree doesn't accumulate gigabytes of frames.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--input", action="append", default=[])
|
||||
ap.add_argument("--outdir", required=True)
|
||||
ap.add_argument("--params", default="{}")
|
||||
a = ap.parse_args()
|
||||
p = json.loads(a.params)
|
||||
|
||||
if not a.input:
|
||||
print("ERROR: no input clip/frame")
|
||||
sys.exit(1)
|
||||
|
||||
VENDOR = Path(__file__).resolve().parents[3] / "vendor" / "corridorkey"
|
||||
CLI = VENDOR / ".venv" / "bin" / "corridorkey"
|
||||
if not CLI.exists():
|
||||
print(f"ERROR: {CLI} missing — run the corridorkey install script on this node")
|
||||
sys.exit(1)
|
||||
|
||||
src = Path(a.input[0])
|
||||
outdir = Path(a.outdir)
|
||||
clip_name = "ck_" + outdir.name # unique per job
|
||||
clip_dir = VENDOR / "ClipsForInference" / clip_name
|
||||
out_clip = VENDOR / "Output" / clip_name
|
||||
|
||||
VIDEO_EXTS = {".mp4", ".mov", ".mkv", ".avi", ".webm", ".mxf", ".m4v"}
|
||||
|
||||
|
||||
def run(cmd):
|
||||
print("+", " ".join(str(c) for c in cmd), flush=True)
|
||||
res = subprocess.run(cmd, cwd=str(VENDOR), stdout=sys.stdout, stderr=subprocess.STDOUT)
|
||||
if res.returncode != 0:
|
||||
cleanup()
|
||||
sys.exit(res.returncode)
|
||||
|
||||
|
||||
def cleanup():
|
||||
shutil.rmtree(clip_dir, ignore_errors=True)
|
||||
shutil.rmtree(out_clip, ignore_errors=True)
|
||||
|
||||
|
||||
try:
|
||||
clip_dir.mkdir(parents=True, exist_ok=True)
|
||||
if src.suffix.lower() in VIDEO_EXTS:
|
||||
shutil.copy(src, clip_dir / f"Input{src.suffix.lower()}")
|
||||
else: # single frame → 1-frame sequence
|
||||
(clip_dir / "Input").mkdir(exist_ok=True)
|
||||
shutil.copy(src, clip_dir / "Input" / src.name)
|
||||
|
||||
run([CLI, "generate-alphas"])
|
||||
hint = clip_dir / "AlphaHint"
|
||||
if not hint.is_dir() or not any(hint.iterdir()): # generate-alphas exits 0 even when GVM fails
|
||||
print("ERROR: no alpha hints generated — are the GVM weights installed? "
|
||||
"(vendor/corridorkey: uv run hf download geyongtao/gvm --local-dir gvm_core/weights)")
|
||||
cleanup()
|
||||
sys.exit(1)
|
||||
|
||||
cmd = [CLI, "run-inference",
|
||||
"--backend", str(p.get("backend", "auto")),
|
||||
"--srgb", # camera clips; EXR pipelines can re-run --linear
|
||||
"--despill", str(int(p.get("despill", 5))),
|
||||
"--despeckle",
|
||||
"--refiner", str(float(p.get("refiner", 1.0))),
|
||||
"--screen-color", str(p.get("screen_color", "auto")),
|
||||
"--comp" if p.get("comp", True) else "--no-comp",
|
||||
# cpu-post default: gpu post-processing dies with an MPSNDArray buffer assertion
|
||||
# on both backends (mac, 2026-07). Opt back in with {"gpu_post": true} to retest.
|
||||
"--gpu-post" if p.get("gpu_post") else "--cpu-post",
|
||||
"--skip-existing"]
|
||||
if p.get("max_frames"):
|
||||
cmd += ["--max-frames", str(int(p["max_frames"]))]
|
||||
if p.get("image_size"):
|
||||
cmd += ["--image-size", str(int(p["image_size"]))]
|
||||
run(cmd)
|
||||
|
||||
if not out_clip.is_dir() or not any(out_clip.rglob("*")):
|
||||
print("ERROR: inference produced no output")
|
||||
cleanup()
|
||||
sys.exit(1)
|
||||
|
||||
zpath = outdir / f"{src.stem}_corridorkey.zip"
|
||||
outputs = []
|
||||
with zipfile.ZipFile(zpath, "w", zipfile.ZIP_STORED) as z: # frames are already compressed
|
||||
for f in sorted(out_clip.rglob("*")):
|
||||
if f.is_file():
|
||||
z.write(f, f.relative_to(out_clip))
|
||||
outputs.append({"path": zpath.name, "meta": {"tool": "corridorkey-mrp-mlx"}})
|
||||
|
||||
previews = sorted(out_clip.rglob("*omp*.mp4")) or sorted(out_clip.rglob("*.mp4"))
|
||||
if previews: # comp preview as its own asset
|
||||
prev = outdir / f"{src.stem}_comp{previews[0].suffix}"
|
||||
shutil.copy(previews[0], prev)
|
||||
outputs.append({"path": prev.name, "meta": {"tool": "corridorkey-comp"}})
|
||||
|
||||
(outdir / "result.json").write_text(json.dumps({"outputs": outputs}))
|
||||
print(f"done: {', '.join(o['path'] for o in outputs)}", flush=True)
|
||||
finally:
|
||||
cleanup()
|
||||
@ -2,43 +2,19 @@
|
||||
"id": "mflux_image_edit",
|
||||
"name": "Edit Image (local, Qwen)",
|
||||
"category": "image-prep",
|
||||
"description": "1-3 images + instruction \u2192 edited image (multi-image composes: 'put the shirt from image 2 on the person in image 1') entirely on this Mac via mflux Qwen-Image-Edit (ungated/Apache-2.0). Free local alternative to fal_image_edit / nano-banana edit \u2014 'remove the sticker', 'make it studio-lit on white', material/lighting changes. First run downloads weights.",
|
||||
"accepts": [
|
||||
"image"
|
||||
],
|
||||
"produces": [
|
||||
"image"
|
||||
],
|
||||
"description": "Image + instruction → edited image entirely on this Mac via mflux Qwen-Image-Edit (ungated/Apache-2.0). Free local alternative to fal_image_edit / nano-banana edit — 'remove the sticker', 'make it studio-lit on white', material/lighting changes. First run downloads weights.",
|
||||
"accepts": ["image"],
|
||||
"produces": ["image"],
|
||||
"resources": "gpu",
|
||||
"entry": "run.py",
|
||||
"python": "venvs/mflux/bin/python",
|
||||
"params_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"prompt": {
|
||||
"type": "string",
|
||||
"default": "",
|
||||
"description": "Edit instruction (what to change)"
|
||||
},
|
||||
"steps": {
|
||||
"type": "integer",
|
||||
"default": 25,
|
||||
"minimum": 1,
|
||||
"maximum": 50,
|
||||
"description": "Inference steps"
|
||||
},
|
||||
"seed": {
|
||||
"type": "integer",
|
||||
"default": 42,
|
||||
"description": "Random seed"
|
||||
},
|
||||
"guidance": {
|
||||
"type": "number",
|
||||
"default": 4.0,
|
||||
"minimum": 0,
|
||||
"maximum": 10,
|
||||
"description": "Guidance scale"
|
||||
}
|
||||
"prompt": {"type": "string", "default": "", "description": "Edit instruction (what to change)"},
|
||||
"steps": {"type": "integer", "default": 25, "minimum": 1, "maximum": 50, "description": "Inference steps"},
|
||||
"seed": {"type": "integer", "default": 42, "description": "Random seed"},
|
||||
"guidance": {"type": "number", "default": 4.0, "minimum": 0, "maximum": 10, "description": "Guidance scale"}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@ -25,9 +25,7 @@ if not cli.exists():
|
||||
src = Path(a.input[0])
|
||||
outdir = Path(a.outdir)
|
||||
out = outdir / f"{src.stem}_edited.png"
|
||||
# ALL inputs go to Qwen (it composes up to ~3 images: "put the shirt from image 2 on the person
|
||||
# in image 1"). Single-image jobs behave exactly as before.
|
||||
cmd = [str(cli), "--image-paths", *[str(Path(x).resolve()) for x in a.input],
|
||||
cmd = [str(cli), "--image-paths", str(src.resolve()),
|
||||
"--prompt", p["prompt"],
|
||||
"--steps", str(p.get("steps", 25)),
|
||||
"--seed", str(p.get("seed", 42)),
|
||||
|
||||
@ -1,21 +0,0 @@
|
||||
{
|
||||
"id": "qwen_layered_local",
|
||||
"name": "Qwen-Image-Layered (local)",
|
||||
"category": "image-edit",
|
||||
"description": "Image → N editable RGBA layers (bg / subject / detail separation), fully local via the Qwen-Image-Layered 20B MLX port (mflux PR#302 fork). ~4 min on m3ultra, ~6 min on m1ultra at the tuned defaults (20 steps, 640, baked q8). The subject layer doubles as matting-grade background removal. Bake a local q8 model once per box (see scripts/install_qwen_layered.sh) or first run falls back to a 54GB HF download + on-the-fly quantization.",
|
||||
"accepts": ["image"],
|
||||
"produces": ["image"],
|
||||
"resources": "gpu",
|
||||
"entry": "run.py",
|
||||
"python": "vendor/mflux-qwen-layered/.venv/bin/python",
|
||||
"params_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"layers": {"type": "integer", "default": 4, "description": "Number of RGBA layers to decompose into"},
|
||||
"steps": {"type": "integer", "default": 20, "description": "Denoising steps (20 = tuned default; 50 = upstream default, ~2.9x slower, no visible gain on tested inputs)"},
|
||||
"resolution": {"type": "integer", "enum": [640, 1024], "default": 640, "description": "Working resolution"},
|
||||
"seed": {"type": "integer", "default": 0, "description": "Random seed"},
|
||||
"prompt": {"type": "string", "default": "", "description": "Optional prompt to guide the decomposition"}
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -1,71 +0,0 @@
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[3]
|
||||
VENDOR = ROOT / "vendor" / "mflux-qwen-layered"
|
||||
CLI = VENDOR / ".venv" / "bin" / "mflux-generate-qwen-layered"
|
||||
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--input", action="append", default=[])
|
||||
ap.add_argument("--outdir", required=True)
|
||||
ap.add_argument("--params", default="{}")
|
||||
a = ap.parse_args()
|
||||
p = json.loads(a.params)
|
||||
|
||||
if not a.input:
|
||||
print("ERROR: no input image")
|
||||
sys.exit(1)
|
||||
if not CLI.exists():
|
||||
print(f"ERROR: qwen-layered not installed at {VENDOR}. "
|
||||
"Run scripts/install_qwen_layered.sh")
|
||||
sys.exit(1)
|
||||
|
||||
# Baked-model resolution: env override, then per-box conventional paths,
|
||||
# else fall back to the HF repo with on-the-fly q8 (slow first run: 54GB).
|
||||
candidates = [os.environ.get("QWEN_LAYERED_MODEL", "")]
|
||||
candidates += [str(Path.home() / "qwen-layered" / "qwen-layered-q8"),
|
||||
str(Path.home() / "qwen-layered-staging" / "qwen-layered-q8")]
|
||||
model_path = next((c for c in candidates if c and Path(c).is_dir()), None)
|
||||
|
||||
outdir = Path(a.outdir)
|
||||
cmd = [
|
||||
str(CLI), "--image", str(Path(a.input[0]).resolve()),
|
||||
"--layers", str(p.get("layers", 4)),
|
||||
"--steps", str(p.get("steps", 20)),
|
||||
"--resolution", str(p.get("resolution", 640)),
|
||||
"--seed", str(p.get("seed", 0)),
|
||||
"--output-dir", str(outdir.resolve()),
|
||||
]
|
||||
if model_path:
|
||||
cmd += ["--model-path", model_path]
|
||||
print(f"using baked model: {model_path}", flush=True)
|
||||
else:
|
||||
cmd += ["-q", "8"]
|
||||
print("no baked model found — HF fallback with on-the-fly q8 "
|
||||
"(first run downloads 54GB)", flush=True)
|
||||
if p.get("prompt"):
|
||||
cmd += ["--prompt", str(p["prompt"])]
|
||||
|
||||
env = os.environ.copy()
|
||||
env["HF_HUB_DISABLE_XET"] = "1"
|
||||
|
||||
print("+", " ".join(cmd), flush=True)
|
||||
res = subprocess.run(cmd, cwd=str(VENDOR), stdout=sys.stdout,
|
||||
stderr=subprocess.STDOUT, env=env)
|
||||
if res.returncode != 0:
|
||||
sys.exit(res.returncode)
|
||||
|
||||
layers = sorted(outdir.glob("layer_*.png"))
|
||||
if not layers:
|
||||
print("ERROR: qwen-layered produced no layer PNGs")
|
||||
sys.exit(1)
|
||||
stem = Path(a.input[0]).stem
|
||||
outputs = [{"path": str(f), "name": f"{stem}_{f.stem}.png",
|
||||
"meta": {"tool": "qwen_layered_local", "layer": i}}
|
||||
for i, f in enumerate(layers)]
|
||||
(outdir / "result.json").write_text(json.dumps({"outputs": outputs}))
|
||||
print(f"done: {len(layers)} layers", flush=True)
|
||||
@ -1,26 +0,0 @@
|
||||
{
|
||||
"id": "sd_local",
|
||||
"name": "SD1.5 + anatomy LoRAs (local, primary-only)",
|
||||
"category": "generate",
|
||||
"description": "Prompt → image via the repose stack on the primary: SD1.5 Hyper_Realism + John's anatomy LoRAs (~/Documents/repose/models/lora) + optional OpenPose T-pose template (front/back/side) + optional IP-Adapter reference image. The no-cloud, no-content-filter lane FLUX can't cover — LoRA-styled anatomy, exact pose control, reference-guided identity. Primary-only: needs ~/Documents/repose (absolute venv path keeps it off remote dispatch by design).",
|
||||
"accepts": ["image"],
|
||||
"produces": ["image"],
|
||||
"resources": "gpu",
|
||||
"entry": "run.py",
|
||||
"python": "/Users/m3ultra/Documents/repose/venv/bin/python",
|
||||
"params_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"prompt": {"type": "string", "default": "", "description": "What to generate"},
|
||||
"negative": {"type": "string", "default": "deformed, extra limbs, extra fingers, mutated hands, missing limbs, cropped, blurry, lowres, watermark, text, multiple people, child", "description": "Negative prompt"},
|
||||
"loras": {"type": "array", "items": {"type": "string"}, "default": [], "description": "'stem=weight' from repose/models/lora, e.g. 'GodPussy1 v4=0.6'"},
|
||||
"pose": {"type": "string", "enum": ["none", "front", "back", "side"], "default": "none", "description": "OpenPose T-pose template via ControlNet"},
|
||||
"ip": {"type": "number", "default": 0.7, "description": "IP-Adapter strength when a reference image is attached"},
|
||||
"steps": {"type": "integer", "default": 28, "minimum": 1, "maximum": 60},
|
||||
"cfg": {"type": "number", "default": 7.0},
|
||||
"seed": {"type": "integer", "default": 7},
|
||||
"width": {"type": "integer", "default": 512, "description": "SD1.5 native ≈512-768; upscale after via seedvr2"},
|
||||
"height": {"type": "integer", "default": 768}
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -1,97 +0,0 @@
|
||||
"""sd_local — SD1.5 Hyper_Realism + anatomy LoRAs (+ optional pose ControlNet / IP-Adapter ref).
|
||||
|
||||
Runs UNDER the repose venv (manifest python is that venv, absolute → primary-only).
|
||||
Generalizes scripts/repose.py from MESHGOD: pose template optional, plain txt2img otherwise.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
os.environ.setdefault("PYTORCH_ENABLE_MPS_FALLBACK", "1")
|
||||
|
||||
import torch # noqa: E402
|
||||
from diffusers import DPMSolverMultistepScheduler # noqa: E402
|
||||
from PIL import Image # noqa: E402
|
||||
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--input", action="append", default=[])
|
||||
ap.add_argument("--outdir", required=True)
|
||||
ap.add_argument("--params", default="{}")
|
||||
a = ap.parse_args()
|
||||
p = json.loads(a.params)
|
||||
|
||||
HERE = os.path.expanduser("~/Documents/repose")
|
||||
BASE = os.path.join(HERE, "models", "Hyper_Realism_1.2_fp16.safetensors")
|
||||
# LoRA search path: John's master dir first, the repose copy second (SD_LORA_DIRS to override)
|
||||
LORA_DIRS = [os.path.expanduser(d) for d in os.environ.get(
|
||||
"SD_LORA_DIRS", "~/Documents/localmodels/Lora:~/Documents/repose/models/lora").split(":")]
|
||||
|
||||
|
||||
def find_lora(stem):
|
||||
for d in LORA_DIRS:
|
||||
cand = os.path.join(d, stem + ".safetensors")
|
||||
if os.path.exists(cand):
|
||||
return cand
|
||||
return None
|
||||
if not os.path.exists(BASE):
|
||||
print(f"ERROR: {BASE} missing — sd_local runs on the primary only")
|
||||
sys.exit(1)
|
||||
|
||||
dev = "mps" if torch.backends.mps.is_available() else "cpu"
|
||||
dtype = torch.float16 if dev == "mps" else torch.float32
|
||||
pose = p.get("pose", "none")
|
||||
kw = {}
|
||||
|
||||
if pose != "none":
|
||||
from diffusers import ControlNetModel, StableDiffusionControlNetPipeline
|
||||
tpl = os.path.join(HERE, f"tpose_{pose}.png")
|
||||
cn = ControlNetModel.from_pretrained("lllyasviel/control_v11p_sd15_openpose", torch_dtype=dtype)
|
||||
pipe = StableDiffusionControlNetPipeline.from_single_file(
|
||||
BASE, controlnet=cn, torch_dtype=dtype, safety_checker=None, requires_safety_checker=False)
|
||||
kw["image"] = Image.open(tpl).convert("RGB")
|
||||
else:
|
||||
from diffusers import StableDiffusionPipeline
|
||||
pipe = StableDiffusionPipeline.from_single_file(
|
||||
BASE, torch_dtype=dtype, safety_checker=None, requires_safety_checker=False)
|
||||
|
||||
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config, use_karras_sigmas=True)
|
||||
pipe = pipe.to(dev)
|
||||
# no attention slicing — it crashes IP-Adapter attn-processor injection (see repose.py)
|
||||
|
||||
if p.get("loras"):
|
||||
from safetensors.torch import load_file
|
||||
names, weights = [], []
|
||||
for spec in p["loras"]:
|
||||
stem, _, w = str(spec).partition("=")
|
||||
path = find_lora(stem.strip())
|
||||
if not path:
|
||||
print(f"[sd] LORA MISS '{stem}' — searched {LORA_DIRS}", flush=True)
|
||||
continue
|
||||
# UNet-only: this diffusers build trips an IndexError on these kohya files' text-encoder
|
||||
# half (empty rank_dict in load_lora_into_text_encoder). The UNet half carries the look.
|
||||
sd_l = load_file(path)
|
||||
unet_only = {k: v for k, v in sd_l.items() if not k.startswith("lora_te")}
|
||||
pipe.load_lora_weights(unet_only, adapter_name=stem.replace(" ", "_"))
|
||||
names.append(stem.replace(" ", "_"))
|
||||
weights.append(float(w or 0.6))
|
||||
if names:
|
||||
pipe.set_adapters(names, weights)
|
||||
print(f"[sd] loras: {dict(zip(names, weights))}", flush=True)
|
||||
|
||||
if a.input: # reference image → IP-Adapter
|
||||
pipe.load_ip_adapter("h94/IP-Adapter", subfolder="models", weight_name="ip-adapter_sd15.bin")
|
||||
pipe.set_ip_adapter_scale(float(p.get("ip", 0.7)))
|
||||
kw["ip_adapter_image"] = Image.open(a.input[0]).convert("RGB")
|
||||
print(f"[sd] ip-adapter ref @ {p.get('ip', 0.7)}", flush=True)
|
||||
|
||||
gen = torch.Generator("cpu").manual_seed(int(p.get("seed", 7)))
|
||||
img = pipe(prompt=p.get("prompt", ""), negative_prompt=p.get("negative", ""),
|
||||
num_inference_steps=int(p.get("steps", 28)), guidance_scale=float(p.get("cfg", 7.0)),
|
||||
generator=gen, width=int(p.get("width", 512)), height=int(p.get("height", 768)),
|
||||
**kw).images[0]
|
||||
out = os.path.join(a.outdir, "sd_local.png")
|
||||
img.save(out)
|
||||
json.dump({"outputs": [{"path": "sd_local.png", "meta": {"tool": "sd_local"}}]},
|
||||
open(os.path.join(a.outdir, "result.json"), "w"))
|
||||
print("done: sd_local.png", flush=True)
|
||||
@ -1,20 +0,0 @@
|
||||
{
|
||||
"id": "stereo_depth",
|
||||
"name": "Stereo SBS → depth + point cloud (local)",
|
||||
"category": "video-prep",
|
||||
"description": "Side-by-side stereoscopic video (or one SBS frame) → per-frame disparity/depth maps via OpenCV StereoSGBM + a colored .ply point cloud of the middle frame for Blender. TRUE geometric depth from the L/R eye shift — no AI hallucination. Uncalibrated SBS rips aren't rectified, so expect relief-map depth (great for Displace-modifier planes), not survey-grade geometry. Output: depth preview mp4 + mid-frame .ply + zipped depth PNGs.",
|
||||
"accepts": ["video", "image"],
|
||||
"produces": ["archive"],
|
||||
"resources": "cpu",
|
||||
"entry": "run.py",
|
||||
"python": "venvs/stereo/bin/python",
|
||||
"params_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"max_frames": {"type": "integer", "default": 120, "description": "Cap processed frames"},
|
||||
"num_disp": {"type": "integer", "default": 128, "description": "Disparity search range (multiple of 16; raise for close subjects)"},
|
||||
"block": {"type": "integer", "default": 5, "description": "SGBM block size (odd, 3-11)"},
|
||||
"layout": {"type": "string", "enum": ["sbs", "tab"], "default": "sbs", "description": "side-by-side or top-and-bottom stereo"}
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -1,112 +0,0 @@
|
||||
"""stereo_depth — SBS stereoscopic video/frame → SGBM disparity maps + mid-frame point cloud.
|
||||
|
||||
True geometric depth from the built-in L/R parallax (cv2.StereoSGBM). ffmpeg (system)
|
||||
decodes/splits; depth PNGs + preview mp4 + a colored .ply land as the outputs.
|
||||
ponytail: uncalibrated disparity (SBS rips have no camera intrinsics) — z is relative,
|
||||
perfect for Blender displace planes; not metric reconstruction.
|
||||
"""
|
||||
import argparse
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import zipfile
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
FFMPEG = shutil.which("ffmpeg") or "/opt/homebrew/bin/ffmpeg" # headless ssh has no brew PATH
|
||||
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--input", action="append", default=[])
|
||||
ap.add_argument("--outdir", required=True)
|
||||
ap.add_argument("--params", default="{}")
|
||||
a = ap.parse_args()
|
||||
p = json.loads(a.params)
|
||||
|
||||
if not a.input:
|
||||
print("ERROR: no input")
|
||||
sys.exit(1)
|
||||
|
||||
src = a.input[0]
|
||||
outdir = a.outdir
|
||||
max_frames = int(p.get("max_frames", 120))
|
||||
num_disp = max(16, (int(p.get("num_disp", 128)) // 16) * 16)
|
||||
block = int(p.get("block", 5)) | 1
|
||||
tab = p.get("layout", "sbs") == "tab"
|
||||
|
||||
work = tempfile.mkdtemp(prefix="stereo_")
|
||||
frames_dir = os.path.join(work, "frames")
|
||||
depth_dir = os.path.join(work, "depth")
|
||||
os.makedirs(frames_dir)
|
||||
os.makedirs(depth_dir)
|
||||
|
||||
# decode → frames (a single image just becomes frame 1)
|
||||
subprocess.run([FFMPEG, "-y", "-loglevel", "error", "-i", src,
|
||||
"-frames:v", str(max_frames), os.path.join(frames_dir, "f%05d.png")], check=True)
|
||||
frames = sorted(glob.glob(os.path.join(frames_dir, "*.png")))
|
||||
print(f"[stereo] {len(frames)} frames, num_disp={num_disp} block={block}", flush=True)
|
||||
|
||||
min_disp = -(num_disp // 2) # movie stereo converges on the screen plane — content
|
||||
sgbm = cv2.StereoSGBM_create( # lives BOTH sides of it, so the window must go negative
|
||||
minDisparity=min_disp, numDisparities=num_disp, blockSize=block,
|
||||
P1=8 * 3 * block * block, P2=32 * 3 * block * block,
|
||||
disp12MaxDiff=1, uniquenessRatio=10, speckleWindowSize=100, speckleRange=2,
|
||||
mode=cv2.STEREO_SGBM_MODE_SGBM_3WAY)
|
||||
|
||||
mid = frames[len(frames) // 2]
|
||||
for i, f in enumerate(frames):
|
||||
img = cv2.imread(f)
|
||||
h, w = img.shape[:2]
|
||||
if tab:
|
||||
L, R = img[: h // 2], img[h // 2:]
|
||||
else:
|
||||
L, R = img[:, : w // 2], img[:, w // 2:]
|
||||
gl, gr = cv2.cvtColor(L, cv2.COLOR_BGR2GRAY), cv2.cvtColor(R, cv2.COLOR_BGR2GRAY)
|
||||
disp = sgbm.compute(gl, gr).astype(np.float32) / 16.0
|
||||
valid = disp > (min_disp - 0.5) # cv2 marks unmatched pixels min_disp-1
|
||||
if valid.sum() > 100:
|
||||
lo, hi = np.percentile(disp[valid], [2, 98])
|
||||
else:
|
||||
lo, hi = 0.0, 1.0
|
||||
norm = np.zeros(disp.shape, np.uint8)
|
||||
norm[valid] = (np.clip((disp[valid] - lo) / max(1e-6, hi - lo), 0, 1) * 255).astype(np.uint8)
|
||||
cv2.imwrite(os.path.join(depth_dir, os.path.basename(f)), norm)
|
||||
if f == mid: # colored point cloud of the middle frame
|
||||
ys, xs = np.where(valid & (norm > 2))
|
||||
step = max(1, len(ys) // 400_000) # cap cloud size
|
||||
ys, xs = ys[::step], xs[::step]
|
||||
z = disp[ys, xs]
|
||||
rgb = L[ys, xs][:, ::-1] # BGR→RGB
|
||||
ply = os.path.join(outdir, "midframe_cloud.ply")
|
||||
with open(ply, "w") as fh:
|
||||
fh.write("ply\nformat ascii 1.0\n"
|
||||
f"element vertex {len(ys)}\n"
|
||||
"property float x\nproperty float y\nproperty float z\n"
|
||||
"property uchar red\nproperty uchar green\nproperty uchar blue\nend_header\n")
|
||||
for (yy, xx, zz, (r, g, b)) in zip(ys, xs, z, rgb):
|
||||
fh.write(f"{xx} {-yy} {zz:.1f} {r} {g} {b}\n")
|
||||
print(f"[stereo] wrote midframe_cloud.ply ({len(ys)} pts)", flush=True)
|
||||
if i % 24 == 0:
|
||||
print(f"[stereo] frame {i + 1}/{len(frames)}", flush=True)
|
||||
|
||||
outputs = [{"path": "midframe_cloud.ply", "meta": {"tool": "stereo_sgbm"}}]
|
||||
if len(frames) > 1: # depth preview video
|
||||
prev = os.path.join(outdir, "depth_preview.mp4")
|
||||
subprocess.run([FFMPEG, "-y", "-loglevel", "error", "-framerate", "24",
|
||||
"-i", os.path.join(depth_dir, "f%05d.png"),
|
||||
"-pix_fmt", "yuv420p", prev], check=True)
|
||||
outputs.append({"path": "depth_preview.mp4", "meta": {"tool": "stereo_sgbm"}})
|
||||
|
||||
zpath = os.path.join(outdir, "depth_frames.zip")
|
||||
with zipfile.ZipFile(zpath, "w", zipfile.ZIP_DEFLATED) as zf:
|
||||
for f in sorted(glob.glob(os.path.join(depth_dir, "*.png"))):
|
||||
zf.write(f, os.path.basename(f))
|
||||
outputs.append({"path": "depth_frames.zip", "meta": {"tool": "stereo_sgbm"}})
|
||||
|
||||
json.dump({"outputs": outputs}, open(os.path.join(outdir, "result.json"), "w"))
|
||||
shutil.rmtree(work, ignore_errors=True)
|
||||
print("done:", ", ".join(o["path"] for o in outputs), flush=True)
|
||||
@ -1,79 +0,0 @@
|
||||
{
|
||||
"id": "trellis2_mlx",
|
||||
"name": "TRELLIS.2 MLX (local, fastest)",
|
||||
"category": "mesh-gen",
|
||||
"description": "Image → PBR GLB via the MRP MLX fork of TRELLIS.2 (monster/trellis-2-mrp-mlx): pure-MLX sampler, Metal PBR bake with RAM-sized face cap + forced-opaque alpha (2026-07-22 quality fixes — fal-parity detail), kdtree/vertex fallbacks. ~72s vertex / ~2min metal-PBR on m3ultra — still the fastest local TRELLIS.2. baker=vertex for the 1s clean-albedo bake (no MR maps). Weights: TRELLIS.2-4B + gated DINOv3/RMBG (rsync from m3 HF cache to workers; no HF login needed).",
|
||||
"accepts": [
|
||||
"image"
|
||||
],
|
||||
"produces": [
|
||||
"model"
|
||||
],
|
||||
"resources": "gpu",
|
||||
"entry": "run.py",
|
||||
"params_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"pipeline_type": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"512",
|
||||
"1024",
|
||||
"1024_cascade"
|
||||
],
|
||||
"default": "1024_cascade",
|
||||
"description": "Resolution tier"
|
||||
},
|
||||
"seed": {
|
||||
"type": "integer",
|
||||
"default": 0,
|
||||
"description": "Random seed"
|
||||
},
|
||||
"steps": {
|
||||
"type": "integer",
|
||||
"default": 12,
|
||||
"description": "Sampler steps per stage (12 = upstream default)"
|
||||
},
|
||||
"baker": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"metal",
|
||||
"vertex",
|
||||
"kdtree"
|
||||
],
|
||||
"default": "metal",
|
||||
"description": "metal = UV PBR textures (meshgod-grade; minor dark-patch texels remain); vertex = 1s bake, cleanest colors, no MR maps"
|
||||
},
|
||||
"texture_size": {
|
||||
"type": "integer",
|
||||
"enum": [
|
||||
512,
|
||||
1024,
|
||||
2048,
|
||||
4096
|
||||
],
|
||||
"default": 2048,
|
||||
"description": "Baked texture resolution (metal/kdtree bakers)"
|
||||
},
|
||||
"max_bake_faces": {
|
||||
"type": "integer",
|
||||
"enum": [
|
||||
200000,
|
||||
500000,
|
||||
1000000
|
||||
],
|
||||
"default": 500000,
|
||||
"description": "Pre-bake face cap. 500k ≈ fal parity (M3 Ultra verified); node RAM sizes the default when unset — laptops get 200k"
|
||||
},
|
||||
"alpha_mode": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"opaque",
|
||||
"auto"
|
||||
],
|
||||
"default": "opaque",
|
||||
"description": "opaque = force alphaMode OPAQUE (fixes speckle-veil hair); auto = keep baked alpha for glassy subjects"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -1,86 +0,0 @@
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[3]
|
||||
|
||||
# fork checkout resolution: vendored path first, then the per-box conventional
|
||||
# clones (m3 staging, m1 home) so existing installs work without re-vendoring.
|
||||
CANDIDATES = [
|
||||
ROOT / "vendor" / "trellis2-mlx",
|
||||
Path.home() / "Documents" / "trellis2-mlx-staging" / "trellis2-jourloy",
|
||||
Path.home() / "trellis2-mlx",
|
||||
]
|
||||
FORK = next((c for c in CANDIDATES if (c / "scripts" / "generate_asset.py").exists()), None)
|
||||
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--input", action="append", default=[])
|
||||
ap.add_argument("--outdir", required=True)
|
||||
ap.add_argument("--params", default="{}")
|
||||
a = ap.parse_args()
|
||||
p = json.loads(a.params)
|
||||
|
||||
if not a.input:
|
||||
print("ERROR: no input image")
|
||||
sys.exit(1)
|
||||
if FORK is None:
|
||||
print("ERROR: trellis-2-mrp-mlx not found. Run scripts/install_trellis2_mlx.sh")
|
||||
sys.exit(1)
|
||||
|
||||
py = FORK / ".venv" / "bin" / "python"
|
||||
if not py.exists():
|
||||
print(f"ERROR: no venv at {FORK}/.venv — run its scripts/setup_macos.sh")
|
||||
sys.exit(1)
|
||||
|
||||
outdir = Path(a.outdir)
|
||||
|
||||
# Meshgod-grade PBR by default (metal baker + capped bake), matching the
|
||||
# trellis_mac lane's 2026-07-22 quality fixes. The fork's schedule bakes the
|
||||
# FULL-density mesh when no target is passed (75GB peak, 131MB GLB) — always
|
||||
# pass an explicit cap; RAM-sized default like trellis_mac (500k ≈ fal parity
|
||||
# on ≥96GB Studios, 200k mtlbvh-safe elsewhere).
|
||||
faces = p.get("max_bake_faces")
|
||||
if faces is None:
|
||||
mem_gb = os.sysconf("SC_PAGE_SIZE") * os.sysconf("SC_PHYS_PAGES") / 1e9
|
||||
faces = 500000 if mem_gb >= 96 else 200000
|
||||
|
||||
cmd = [
|
||||
str(py), str(FORK / "scripts" / "generate_asset.py"),
|
||||
str(Path(a.input[0]).resolve()),
|
||||
"--output-dir", str(outdir.resolve()),
|
||||
"--backend", "mlx-experimental",
|
||||
"--baker", str(p.get("baker", "metal")),
|
||||
"--pipeline-type", str(p.get("pipeline_type", "1024_cascade")),
|
||||
"--seed", str(p.get("seed", 0)),
|
||||
"--texture-size", str(p.get("texture_size", 2048)),
|
||||
"--pbr-decimation-target", str(faces),
|
||||
"--force",
|
||||
]
|
||||
if p.get("steps"):
|
||||
cmd += ["--steps", str(p["steps"])]
|
||||
|
||||
env = os.environ.copy()
|
||||
env["HF_HUB_DISABLE_XET"] = "1"
|
||||
# "auto" keeps o_voxel's baked-alpha BLEND detection (glass etc.); default
|
||||
# opaque — noisy baked alpha renders solid hair as a speckle veil.
|
||||
if p.get("alpha_mode"):
|
||||
env["TRELLIS2_ALPHA_MODE"] = str(p["alpha_mode"])
|
||||
|
||||
print("+", " ".join(cmd), flush=True)
|
||||
res = subprocess.run(cmd, cwd=str(FORK), stdout=sys.stdout,
|
||||
stderr=subprocess.STDOUT, env=env)
|
||||
if res.returncode != 0:
|
||||
sys.exit(res.returncode)
|
||||
|
||||
glbs = sorted(outdir.rglob("candidate_pbr.glb")) or sorted(outdir.rglob("*.glb"))
|
||||
if not glbs:
|
||||
print("ERROR: no GLB produced")
|
||||
sys.exit(1)
|
||||
stem = Path(a.input[0]).stem
|
||||
outputs = [{"path": str(glbs[0]), "name": f"trellis2mlx_{stem}.glb",
|
||||
"meta": {"tool": "trellis2_mlx", "baker": p.get("baker", "vertex")}}]
|
||||
(outdir / "result.json").write_text(json.dumps({"outputs": outputs}))
|
||||
print("done:", glbs[0], flush=True)
|
||||
@ -3,80 +3,18 @@
|
||||
"name": "TRELLIS.2 (local, SOTA)",
|
||||
"category": "mesh-gen",
|
||||
"description": "Image → GLB + baked PBR locally via the TRELLIS.2 MPS/Metal port (best open-weights quality on Apple Silicon, ~3-5 min/gen). Weights need a one-time HuggingFace login (facebook/dinov3-vitl16, briaai/RMBG-2.0). Note: 1536 is upstream-only; Mac port maxes at 1024_cascade.",
|
||||
"accepts": [
|
||||
"image"
|
||||
],
|
||||
"produces": [
|
||||
"model"
|
||||
],
|
||||
"accepts": ["image"],
|
||||
"produces": ["model"],
|
||||
"resources": "gpu",
|
||||
"entry": "run.py",
|
||||
"python": "vendor/trellis-mac/.venv/bin/python",
|
||||
"params_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"pipeline_type": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"512",
|
||||
"1024",
|
||||
"1024_cascade"
|
||||
],
|
||||
"default": "1024",
|
||||
"description": "Resolution tier (1024_cascade = highest quality on Mac)"
|
||||
},
|
||||
"texture_size": {
|
||||
"type": "integer",
|
||||
"enum": [
|
||||
512,
|
||||
1024,
|
||||
2048
|
||||
],
|
||||
"default": 2048,
|
||||
"description": "Baked texture resolution"
|
||||
},
|
||||
"seed": {
|
||||
"type": "integer",
|
||||
"default": 0,
|
||||
"description": "Random seed"
|
||||
},
|
||||
"no_texture": {
|
||||
"type": "boolean",
|
||||
"default": false,
|
||||
"description": "Geometry only (skip texture stage, much faster)"
|
||||
},
|
||||
"max_bake_faces": {
|
||||
"type": "integer",
|
||||
"enum": [
|
||||
200000,
|
||||
500000,
|
||||
1000000
|
||||
],
|
||||
"default": 500000,
|
||||
"description": "Pre-bake simplify cap. 500k ≈ fal parity, verified crash-free on the M3 Ultra (2026-07-22); drop to 200k on laptops (mtlbvh guard)"
|
||||
},
|
||||
"steps": {
|
||||
"type": "integer",
|
||||
"default": 0,
|
||||
"description": "Sampler steps override for all 3 flow phases (0 = pipeline default, 12)"
|
||||
},
|
||||
"baker": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"",
|
||||
"vertex"
|
||||
],
|
||||
"default": "",
|
||||
"description": "'vertex' = KDTree vertex-color bake (dark-patch workaround, no PBR maps)"
|
||||
},
|
||||
"alpha_mode": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"opaque",
|
||||
"auto"
|
||||
],
|
||||
"default": "opaque",
|
||||
"description": "opaque = force alphaMode OPAQUE in the GLB (fixes speckle-veil hair from noisy baked alpha); auto = keep o_voxel's BLEND detection for glassy subjects"
|
||||
}
|
||||
"pipeline_type": {"type": "string", "enum": ["512", "1024", "1024_cascade"], "default": "1024", "description": "Resolution tier (1024_cascade = highest quality on Mac)"},
|
||||
"texture_size": {"type": "integer", "enum": [512, 1024, 2048], "default": 2048, "description": "Baked texture resolution"},
|
||||
"seed": {"type": "integer", "default": 0, "description": "Random seed"},
|
||||
"no_texture": {"type": "boolean", "default": false, "description": "Geometry only (skip texture stage, much faster)"}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@ -21,18 +21,12 @@ if not a.input:
|
||||
if not (VENDOR / "generate.py").exists():
|
||||
print(f"ERROR: trellis-mac not installed at {VENDOR}. Run scripts/install_trellis_mac.sh")
|
||||
sys.exit(1)
|
||||
# manifest no longer pins "python" (broke remote dispatch when the venv path
|
||||
# didn't exist on the worker) — resolve the vendor venv here instead.
|
||||
VENV_PY = VENDOR / ".venv" / "bin" / "python"
|
||||
if not VENV_PY.exists():
|
||||
print(f"ERROR: no venv at {VENV_PY} — run scripts/install_trellis_mac.sh")
|
||||
sys.exit(1)
|
||||
|
||||
outdir = Path(a.outdir)
|
||||
out_stem = outdir / f"trellis2_{Path(a.input[0]).stem}"
|
||||
|
||||
cmd = [
|
||||
str(VENV_PY), "generate.py", str(Path(a.input[0]).resolve()),
|
||||
sys.executable, "generate.py", str(Path(a.input[0]).resolve()),
|
||||
"--pipeline-type", str(p.get("pipeline_type", "1024")),
|
||||
"--texture-size", str(p.get("texture_size", 2048)),
|
||||
"--seed", str(p.get("seed", 0)),
|
||||
@ -40,8 +34,6 @@ cmd = [
|
||||
]
|
||||
if p.get("no_texture"):
|
||||
cmd.append("--no-texture")
|
||||
if p.get("steps"):
|
||||
cmd += ["--steps", str(p["steps"])]
|
||||
|
||||
env = os.environ.copy()
|
||||
# xet chunked downloader fails on these BFL/Meta repos ("Unable to parse string
|
||||
@ -51,22 +43,6 @@ env["HF_HUB_DISABLE_XET"] = "1"
|
||||
env["KMP_DUPLICATE_LIB_OK"] = "TRUE"
|
||||
env.setdefault("OMP_NUM_THREADS", "1")
|
||||
env.setdefault("MKL_NUM_THREADS", "1")
|
||||
# Pre-bake simplify cap. The 200k generate.py default is a laptop-derived
|
||||
# mtlbvh guard that costs most of the mesh detail (2.2M-tri decode → 95k GLB
|
||||
# vs fal's 495k). 500k verified crash-free on the M3 Ultra Studio 2026-07-22.
|
||||
# The server passes only client-sent params (manifest defaults are UI-only),
|
||||
# so default here, sized by node RAM: big-memory Studios get the quality cap.
|
||||
if p.get("max_bake_faces") is not None:
|
||||
env["TRELLIS2_MAX_BAKE_FACES"] = str(p["max_bake_faces"])
|
||||
else:
|
||||
mem_gb = os.sysconf("SC_PAGE_SIZE") * os.sysconf("SC_PHYS_PAGES") / 1e9
|
||||
env.setdefault("TRELLIS2_MAX_BAKE_FACES", "500000" if mem_gb >= 96 else "200000")
|
||||
if p.get("baker"):
|
||||
env["TRELLIS2_BAKER"] = str(p["baker"])
|
||||
# "auto" keeps o_voxel's baked-alpha BLEND detection (glass etc.); default
|
||||
# opaque — the Metal baker's noisy alpha turns solid hair into speckles.
|
||||
if p.get("alpha_mode"):
|
||||
env["TRELLIS2_ALPHA_MODE"] = str(p["alpha_mode"])
|
||||
|
||||
print("+", " ".join(cmd), flush=True)
|
||||
print("(first run downloads TRELLIS.2-4B + dinov3 + RMBG-2.0 weights)", flush=True)
|
||||
|
||||
@ -341,25 +341,11 @@ class Runner:
|
||||
p = Path(out["path"])
|
||||
if not p.is_absolute():
|
||||
p = outdir / p
|
||||
if not p.exists():
|
||||
# a remote worker writes result.json with paths on ITS disk
|
||||
# (<root>/data/remote/<job>/out/...); after collect() those
|
||||
# files sit in the local outdir — remap by the out/-relative
|
||||
# tail, falling back to the basename
|
||||
marker = f"/data/remote/{job_id}/out/"
|
||||
_, _, tail = str(out["path"]).partition(marker)
|
||||
if tail and (outdir / tail).exists():
|
||||
p = outdir / tail
|
||||
else:
|
||||
p = outdir / Path(out["path"]).name
|
||||
if p.exists():
|
||||
a = store.register_file(con, p, name=out.get("name"),
|
||||
parent_job=job_id, move=True, meta=out.get("meta"),
|
||||
user_id=owner_id)
|
||||
registered.append(a["id"])
|
||||
else:
|
||||
print(f"[runner] {job_id}: result.json output not found after "
|
||||
f"collect, skipped: {out.get('path')}", flush=True)
|
||||
else:
|
||||
for p in sorted(outdir.iterdir()):
|
||||
if p.name == "result.json" or p.name.startswith("."):
|
||||
|
||||
Loading…
Reference in New Issue
Block a user