Compare commits

..

1 Commits

Author SHA1 Message Date
type-two
d5217575eb wip(corridorkey_local): operator scaffold — BLOCKED on GVM inference crash
Full staging/zip/cleanup contract works; generate-alphas dies in GVM with
'MPSNDArray buffer is not large enough. Must be 59768832 bytes' on BOTH
m1ultra and m3ultra, any input dims, torch or mlx backend, gpu or cpu post.
GVM --device cpu wrote 0 frames in 3.5h (hung). Weights installed both nodes
(gvm_core/weights, 6GB). Next: either debug GVM's torch-MPS path or replace
the alpha-hint stage with the corridorkey-mlx fork / bg_remove_local hints.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 22:30:39 +10:00
51 changed files with 378 additions and 4076 deletions

View File

@ -147,372 +147,3 @@ regression suite (12 checks, ~30s).
can make a node look 511x slower and it is invisible to load average (the M3 Ultra reads load can make a node look 511x slower and it is invisible to load average (the M3 Ultra reads load
2.45 with its GPU pinned). Anything measured here should use min-of-runs and check 2.45 with its GPU pinned). Anything measured here should use min-of-runs and check
`ioreg … IOAccelerator` "Device Utilization %" *before* touching the GPU. See PERFCHECK.md. `ioreg … IOAccelerator` "Device Utilization %" *before* touching the GPU. See PERFCHECK.md.
## 2026-07-19 — fleet MLX capability matrix (mlx 0.32.0 everywhere)
Probe: 4096² matmul TFLOPS (fp16/bf16), 256MB elementwise GB/s, sdpa head_dim 56 vs 64 (B1·H8·L2048).
| host | chip | RAM | fp16 TF | bf16 TF | ~GB/s | sdpa d56/d64 |
|---|---|---|---|---|---|---|
| m3ultra | M3 Ultra | 256 | 23.48 | 23.54 | 400 | 1.51× |
| ultra | M1 Ultra | 128 | 16.31 | 13.16 | 297 | 0.94× |
| JING5 | M5 | 16 | 8.61 | 8.99 | 43* | **4.72×** |
| m4pro | M4 Pro | 24 | 5.24 | 3.86 | 225 | 1.29× |
| m4mini | M4 | 16 | 3.65 | 3.65 | 98 | 1.55× |
| AIRPRO | M3 | 16 | 1.55 | 1.64 | 87 | 0.95× |
*JING5 bandwidth measured under interactive load — retest idle before trusting.
Verdicts:
- **bf16-on-M1 question answered**: works, ~20% slower than fp16 (16.3→13.2). Ship fp16 on M1-class, bf16 free on M3 Ultra/M4/M5.
- **head_dim pad-to-64 matters MOST on newest chips** (M5 4.7×!, M4-class ~1.3-1.6×, M3 Ultra 1.5×) and ~zero on M1-class (no fused fast path there anyway — matches the corridorkey ablation).
- Fleet total ≈ 59 TFLOPS fp16; the two Ultras are 2/3 of it. TRELLIS.2-MLX lane assignment: m3ultra primary, ultra second (fp16), M4 Pro light ops.
- Probe script: reusable one-liner via `uv run --with mlx` — mlxprobe.py (this bench).
Addendum (same day) — m1max studio joins the matrix (login `studio@100.92.78.24`;
the fleet doc's `m1max@` user and "disk-critical" note are both stale — 170GB free):
| M1MAX | M1 Max | 32 | 7.09 | 6.04 | 336 | 1.22× |
Notable: 2nd-best bandwidth in the fleet (336 GB/s > M4 Pro's 225) despite mid-pack
compute — good fit for memory-bound MLX work (VAE decode, big-tensor elementwise).
Fleet total ≈ 66 TFLOPS fp16 across 7 machines.
## 2026-07-19 — trellis_mac rung 1: sparse-conv backend verified (m3ultra, anatomy image, seed 42, 1024-cascade)
| SPARSE_CONV_BACKEND | total gen+bake | wall | peak RSS |
|---|---|---|---|
| flex_gemm (Metal) | **124.8 s** | 248.8 s | 20.8 GB |
| none (pure-torch) | 214.4 s | 337.6 s | 20.8 GB |
Verdicts:
- The 124.8s baseline WAS the Metal fast path (log line "[SPARSE] Conv backend: flex_gemm") — no hidden 10x jackpot; metallib loads fine on macOS 26.5.
- Metal sparse-conv is worth **1.72x end-to-end** (saves 89.6s/gen). Keep flex_gemm the default; the conv_none fallback is a real 90s regression if the metallib ever breaks silently — worth a GODBOARD/log assertion.
- Remaining fat in the fast path: ~46s diffusion sampling (padded SDPA attention — next kernel frontier), ~50s decode/extract (incl. aten::segment_reduce falling back to CPU — patchable), 19s Metal bake.
- Decode actually produces 2.79M faces before the forced ~200k simplify — the quality-recovery headroom (rung 2) is bigger than the recon estimated.
## 2026-07-19 — Hunyuan paint-UNet mx.compile (Target 1, branch mlx-tune)
Micro gate (real arch, random fp16 weights, 6-view 64² latents, m3ultra):
eager 270.5 ms/fwd → compiled 260.5 ms/fwd = **1.04×**; parity max|diff|
3.0e-05 (PASS); first-trace 1.2 s; peak 4.9 GB. Env-gated
`HY3D_MLX_COMPILE=1`, default off (numerics untouched).
Verdict: keep (free, safe) but the paint UNet is GEMM-bound — fusion alone
is small. The bigger Hunyuan levers are Target 3 (MoE `.item()` sync-tax,
4800 syncs/gen, unlocks whole-DiT compile) and Target 4 (manual→fused
SDPA in paint DINO). E2E flag-on-vs-off measurement pending after those.
## 2026-07-19 — A0 GATE: first end-to-end MLX TRELLIS.2 (Jourloy PR#175 fork, m3ultra)
Same anatomy image, seed 42, 1024-cascade, tex 2048:
| path | wall (incl. load) | bake | peak mem | output |
|---|---|---|---|---|
| torch-MPS (trellis-mac) | 248.8 s | 19 s @191K faces | 20.8 GB | 21 MB GLB |
| **mlx-experimental (Jourloy)** | **216.6 s** | 83 s @**2.74M faces** | 75.4 GB | 131 MB PBR GLB |
**VERDICT: ADOPT — the MLX path BEAT the MPS baseline on wall-clock while
baking 14× the faces.** Key discovery: `mlx-experimental` is a HYBRID —
resolver log shows `Conv backend: flex_gemm; Attention: flex_gemm_sparse_attn`,
i.e. sparse ops still run pedronaugusto's Metal kernels; MLX runs the dense
stages. The feared naive-MLX-sparse-conv path never executes. Model load is
lazy/instant on the MLX side (vs 103 s torch load).
Quality: geometry excellent at full density; the DARK-PATCH texture defect
REPRODUCES on this second, independent pipeline — both paths share the
pedronaugusto mtldiffrast/BVH texel sampling. Culprit narrowed to those
shared Metal kernels OR the decoded tex attrs; the pure-Python KDTree bake
control (running) decides. fal renders clean with the same weights, so the
attrs are likely fine → Metal sampling is the prime suspect.
Caveat: 75 GB peak fits both Ultras but check m1 headroom under load.
## 2026-07-19 — Hunyuan mlx-tune e2e (T1+T3+T4, HY3D_MLX_COMPILE=1, m3ultra)
anatomy bench image, defaults (50 steps, octree 256): shape 146s + tex 101s
= **246s total, 17.5GB peak** vs recorded defaults baseline 260s (149+112).
**1.06× e2e** — texture stage carries the win (112→101s = T4 fused DINO
+ T1 UNet compile); shape delta noise-level. Caveat: baseline row used a
different input image; same settings though. Next lever: Target 2
whole-DiT compile (unblocked by T3).
Addendum — with T2 (whole-DiT compile) included: shape 144s + tex 99s =
**243s total**. Full mlx-tune vs 260s baseline ≈ **1.07× e2e**. Verdict:
on M3 Ultra these 2B GEMM-bound models get ~1% from mx.compile fusion;
the tune's real wins were T4 fused attention (texture stage) + T3
compile-safety. Branch mlx-tune @ 5a32240; flag HY3D_MLX_COMPILE=1.
Worth re-benching on M1 Ultra where launch overhead is proportionally larger.
## 10 · DARK-PATCH VERDICT: Metal texel-sampling convicted (2026-07-19)
Decisive experiment: dumped the decoded tex voxels (E2 fixture,
1,398,769 voxels × 6 PBR attrs) and rendered base_color as a direct
front-projection — `trellis2-bench/attrs_front.png`. **The attrs are
near-fal quality** (clean anatomy palette; dark voxels only 5.6%, all
interior). Both local pipelines (trellis-mac MPS and Jourloy MLX) produce
the same mottled dark bake from these clean attrs, and both share the
pedronaugusto mtldiffrast/mtlbvh texel→voxel sampling. ⇒ **the bake's
attr-sampling is the sole quality gap.** The 3h+ KDTree run is now merely
confirmatory (left running).
### Designed fix (next session's opening move)
The attrs live on a REGULAR voxel grid — no BVH/KDTree needed at all:
texel → (xatlas UV → triangle barycentric → world pos) →
`grid[(pos-origin)/voxel_size]` direct index (+8-neighbor occupancy
search). One vectorized gather over all texels; replaces the suspect
ray-cast sampling entirely; implementable in torch-MPS or MLX in ~a page.
Validate against `attrs_front.png` colors + the fal render. If it works,
BOTH local paths get fal-class texture quality at zero bake-speed risk.
### Parity fixture set: COMPLETE
All 8 fixtures captured at `trellis2-bench/parity_fixtures/` (cond_512/
1024, ss_coords, shape_slat, shape_dec, tex_slat, tex_voxels, mesh_final;
seed 42, 1024-cascade). Hook env needed: PYTORCH_ENABLE_MPS_FALLBACK=1
ATTN_BACKEND=sdpa SPARSE_ATTN_BACKEND=sdpa SPARSE_CONV_BACKEND=flex_gemm.
## 2026-07-19 — KDTree bake control: post-mortem (confirmatory run, superseded)
The TRELLIS2_FORCE_KDTREE control finally finished: **bake 21,063s (5.85h)**
and the exported GLB is GARBAGE geometry (mangled 395K-tri blob) — the
fork's KDTree export path has its own vertex-mapping bug on top of being
unusably slow (per-triangle Python rasterizer loop). Verdict unchanged —
the attrs-projection had already convicted the Metal texel-sampling — but
this rules out "just use the Python baker" as shipped: it needs the
fast-rasterizer rewrite + export fix (in progress as fast_bake_test.py).
## 11 · BAKE FIX PROVEN (2026-07-19 evening) — dark patches eliminated
`trellis2-bench/render_vbake5_front.png`: full anatomy figure, correct
colors, ZERO dark patches — from the same decoder attrs the Metal baker
mangles. Method: cKDTree IDW sampling of the voxel grid at mesh vertices
(k=4, distance² weights, 1.5-voxel radius, single-nearest fallback) →
vertex-colored GLB. **Sampling cost: 0.3s** for 270K verts / 1.4M voxels.
Two integration gotchas (cost hours; do not rediscover):
1. **Axis spaces.** Voxel grid + raw pipeline mesh = Z-up-long-axis in
[-0.5,0.5]³; GLB-loaded meshes = glTF Y-up. Transform before sampling:
`(x,y,z)_gltf → (x,z,y)_voxel`. The "mangled blob" renders were ONLY
this (figure viewed down its own axis) — no data corruption anywhere.
2. **glTF vertex colors are LINEAR** — do not pre-gamma (double-gamma =
washed out). Store linear, let the renderer do the transfer.
Also learned: raw `MeshWithVoxel.vertices/faces` need to_glb's weld/init
before use as a plain triangle soup; `output_3d.obj` inherits the raw
convention. And xatlas hangs (2h+) on écorché-topology meshes — the UV
path needs chart budgeting or per-island parallel unwrap.
### Production path (next session)
- Option A (fast to ship): full-density vertex-colored GLB as the raw
output; MESHGOD's Blender finish farm already decimates + can bake
verts→texture (standard Blender bake) — sidesteps xatlas entirely.
- Option B (fal-parity): fast bucket rasterizer (written, in
fast_bake_test.py) + xatlas-alternative UV → 2048² texture bake with
this sampler. Needs the écorché-safe unwrap.
- Wire as `--baker python-fast` in the Jourloy fork + trellis-mac, make it
the default over the convicted Metal texel path; port sampler to MLX
(trivial — gather + weighted sum) for the pure-MLX lane.
### §11 addendum — vertex baker PRODUCTIONIZED (2026-07-19 night)
Wired and validated in BOTH pipelines:
- `vendor/trellis-mac` @ 55fdd7b: `TRELLIS2_BAKER=vertex` env mode in
generate.py (Metal + kdtree paths untouched; sentinel-gated).
- Jourloy fork @ 746e727 (pushed to monster/trellis-2-mrp-mlx): CLI
`--baker vertex`.
E2E validation (anatomy, seed 42, 1024-cascade, uncapped density):
**130.0s total, bake 1s**, 232s wall, 20.8GB peak — full 2.79M-face
vertex-colored GLB, ZERO dark patches, best local quality yet
(`render_vprod_front.png`). vs Metal-bake baseline: same speed envelope,
quality gap closed for albedo. Not yet carried: metallic/roughness maps
(vertex colors are albedo-only) — MR needs the UV path (Option B) or a
COLOR_1 convention. MESHGOD finish-farm verts→texture bake = the
remaining Option-A step for game-budget assets.
## 2026-07-19 — Qwen-Image-Layered on MLX (m1ultra) — WORKING
New capability: image → N editable RGBA layers, fully local. Model:
Qwen/Qwen-Image-Layered (20B, BF16, Apache 2.0, 54GB in m1 HF cache).
Runtime: mflux PR#302 fork (unmerged upstream; vendored at
monster/Qwen-Image-Layered-MRP-MLX, pin tag pr302-head-a255e4f, isolated
venv ~/qwen-layered-staging/mflux-layered/.venv — production mflux venv
untouched). CLI: `mflux-generate-qwen-layered`.
Smoke (anatomy test image, 4 layers, 640, q8-on-the-fly): **1180.8s wall
first-run** (dominated by 20B load + live quantization), 27.1GB peak.
Output verified: 4 true RGBA PNGs — bg layer, atmosphere layer, clean
figure cutout (74.5% transparent, crisp to the fingers), detail layer.
The figure-cutout layer doubles as a high-end matting result — relevant
to the MESHGOD prep frontline for hero assets.
Optimization in progress: `mflux-save --quantize 8` pre-baked model →
skips load+quant cost; timed gen-only rerun to follow. Next: 1024-res,
q4/q6 quality ladder, m3ultra cross-bench.
### Qwen-Image-Layered optimization ladder (m1ultra, 640, q8, 4 layers)
| config | wall | note |
|---|---|---|
| cold: HF bf16 + on-the-fly q8, 50 steps | 1180.8 s | first smoke |
| baked q8 model, 50 steps | 1075.9 s | load cost was NOT dominant |
| **baked q8, 20 steps** | **365.6 s** | **quality held — figure cutout indistinguishable from 50-step. RECOMMENDED DEFAULT.** |
3.2× end-to-end. Peak 36.6GB. Baked model: ~/qwen-layered-staging/qwen-layered-q8 (34GB).
Remaining ladder (untested): q6/q4 quality/speed, 1024-res, fewer-steps floor (10?),
m3ultra cross-bench (~1.4× compute but needs 34GB disk — m3 disk currently tight),
MODELBEAST operator wrapper (qwen_layered_local) once defaults settle.
### Cross-bench addendum: m3ultra joins (same baked q8, 20 steps, 640, 4 layers)
| box | wall | note |
|---|---|---|
| m3ultra | **243.0 s** | 1.50× vs m1 — matches fleet-matrix TFLOPS ratio (1.44×) |
| m1ultra | 365.6 s | |
Both Ultras now serve Qwen-Image-Layered from the same 34GB baked-q8
artifact (m3: ~/qwen-layered/, m1: ~/qwen-layered-staging/) + the same
Gitea fork clone. Peak 36.7GB either box. m3 disk after install: ~82GB free.
### Qwen-Image-Layered — COMPLETE ladder + verdicts (2026-07-19/20)
| config | box | wall | verdict |
|---|---|---|---|
| q8 · 20 steps · 640 | m3ultra | **243.0 s** | **volume default** |
| q8 · 20 steps · 640 | m1ultra | 365.6 s | second lane |
| q8 · 20 steps · **1024** | m3ultra | 857.6 s | **hero tier** — superb detail, true RGBA (68% transp) |
| q6 · 20 steps · 640 | m1ultra | 366.4 s | ❌ REJECTED — decomposition semantics break (silhouette, no cutout) |
| q4 · 20 steps · 640 | m1ultra | 350.1 s | ❌ REJECTED — pure noise |
Quant findings: speed is FLAT across q4/q6/q8 (compute-bound at these
settings) — quantizing below q8 buys only disk/RAM and DESTROYS quality.
**q8 is the floor for this model.** q6/q4 bakes deleted (m1 back to 3.4T).
Operator `qwen_layered_local` shipped: vendored fork at
vendor/mflux-qwen-layered (install_qwen_layered.sh), tuned defaults
(20/640/baked-q8, resolution 1024 opt-in), m1 registered in nodes.json
(machine-local), e2e run.py contract smoke passed on m3.
Also audited flux_local (mflux 0.18.0): DEFAULT PATH HEALTHY — flux2-klein
family has mx.compile on denoise + compiled scheduler (the 9s/image path).
GAP: FLUX.1 family (dev/schnell/krea) has ZERO mx.compile — backport =
same env-gated pattern as Hunyuan T1/T2, modest expected win (1-4% M3U,
more on small boxes), upstreamable to mflux. Queued, not urgent.
## 12 · RUNG-3 KERNEL DAY — RESULTS (2026-07-20)
**Headline: 216.6s → 71.96s e2e (3.0×), peak 75.4GB → 26.5GB (2.8×),
parity held (raw mesh Δ0.03%), full gate EXIT=0.** All on the fork
(monster/trellis-2-mrp-mlx): fa972aa + 7860148 + 0db816b.
What actually moved the needle (in order of impact):
1. **Vertex baker properly wired into generate_asset** (scheduler
fallthrough fixed — `--baker vertex` was silently running the 5h
pure-python kdtree path): "other" bucket 69.4s → 8.3s, and the
75GB peak turned out to be the METAL BAKE's, not the conv's.
2. **Pure-MLX sampler loop** (mlx_samplers.py): torch CPU loop + per-
forward host bounces → one conversion per stage, batched dense CFG,
once-per-stage concat_cond. Sampling 56.8→54.4s (~4% — forwards are
compute-bound) but it is the correctness-proven, compile-ready
architecture, and removes ~72 host round-trips/gen.
3. **Validator taught the vertex-colored asset class** (COLOR_0 without
UVs = legit; normals materialized at export).
Honest negatives (documented so nobody re-chases them):
- mx.compile on the step forward: ~0 on M3 Ultra (GEMM-bound; matches
Hunyuan T1/T2 findings).
- Fused Metal sparse-conv kernel: built, parity 8e-4, but stock chunking
already bounds memory at decoder scale and beats the scalar kernel on
speed. Kept as TRELLIS2_METAL_SPCONV=1 opt-in reference (simdgroup
tiling = the future version if decoders ever matter for time).
- Remaining gap to H100 (~17s at 1024): inside the DiT forwards
(54s sampling ≈ 2.3× ideal FLOPs) — attention/MLP internals, a
future deep-dive, not orchestration.
Env knobs (all default-sane): TRELLIS2_MLX_SAMPLER=0 (revert sampler),
TRELLIS2_MLX_COMPILE=1, TRELLIS2_METAL_SPCONV=1.
M1 Ultra validation: in flight (26.5GB peak fits with 100GB headroom).
### Rung-3 addendum — M1 Ultra validated (2026-07-20)
| box | e2e wall | peak mem |
|---|---|---|
| m3ultra | 71.96 s | 26.5 GB |
| **m1ultra** | **181.5 s** | **17.2 GB** |
The M1 Ultra — which could not run TRELLIS.2 at all before this fork
(torch-MPS path was m3-only in practice) — now generates full-density
vertex-baked assets in ~3 min at 17GB. Install: clone
monster/trellis-2-mrp-mlx to ~/trellis2-mlx + scripts/setup_macos.sh
(PYTHON_BIN=uv 3.11) + gated weights rsynced from m3's HF cache
(no HF login needed on workers). Both Ultras now serve the fastest
local TRELLIS.2 in existence. TRELLIS.2 fleet: m3 primary (72s),
m1 second lane (182s), zero dollars per asset.
## 2026-07-22 — trellis_mac closes the fal quality gap (m3ultra, black-model image, seed 0, pipeline 1024)
Root-caused why local GLBs looked far worse than fal's TRELLIS.2 despite the
same 4B weights. Two independent bugs, neither in the diffusion (local decode
was already 2.23M tris / 101s):
1. **Decimation ceiling.** generate.py's `TRELLIS2_MAX_BAKE_FACES=200000`
laptop mtlbvh guard + o_voxel's cleanup pass (dup/degenerate removal +
small-component pruning) crushed every model to exactly **94,535 faces**
(fal ships ~495k). The 23× pre-bake simplify also shredded thin shells —
hair fragments then got deleted by `remove_small_connected_components`.
Fix: cap raised to **500k → 351,898-face GLB**, bake 7s→14s, no Metal
watchdog crash on the M3 Ultra Studio. Now the operator default
(`max_bake_faces` param; drop to 200k on laptops).
2. **Speckle-veil alpha.** o_voxel's Metal baker samples the decoder's alpha
attr noisily (same family as the 07-19 dark-patch bug); any texel <250
flips the material to BLEND → solid hair renders as a transparent speckle
veil, "broken face". Fix: generate.py now rewrites the exported GLB to
`alphaMode=OPAQUE` by default (`TRELLIS2_ALPHA_MODE=auto` keeps baked
alpha for glassy subjects; `alpha_mode` operator param).
Verified in Blender side-by-side (old 95k / new 352k opaque / fal 495k):
face + hair now read at fal level. Remaining deltas vs fal: slightly fuller
fal hair (try `max_bake_faces=1000000`), and the known Metal dark-patch
texels (workaround: `baker=vertex`). Also wired `steps` param passthrough.
### Addendum — trellis2_mlx gains the metal PBR lane (2026-07-22, m3ultra)
Item 1 of the post-quality-fix roadmap: the fast MLX fork lane now ships
meshgod-grade UV-PBR by default (baker=metal, RAM-sized cap, forced-opaque
alpha — fork commit ba4e2ad, operator ac817a9). Same image/seed as the
trellis_mac runs: **156s e2e, 26.2GB peak, candidate_pbr 391,782 tris +
2048² basecolor+MR, raw master 2.19M tris kept** (vs trellis_mac ~250s
wall for 351,898 tris; fal 495,176 @ $0.30). Farm-path e2e verified with
zero client params (defaults gap closed in run.py — server forwards only
client-sent params). Blender side-by-side vs fal: parity; local keeps
more hair curl detail. baker=vertex remains the 1s clean-albedo option.
1M-cap probe (same image/seed, farm path): candidate 832,883 tris, 43MB
GLB, OPAQUE+PBR — hair marginally fuller than 500k, subtle at viewing
distance. Verdict: 500k stays the default; 1M = hero-asset tier.
### wan_video verified on two machines — RAM + speed + visual (2026-08-04)
Post-rewrite verification of the mlx-video operator path (5B q8), identical
settings both boxes: 832x480, 49f, 20 steps, cfg 5.0, seed 1234, prompt
"a red fox walking through fresh snow in a pine forest, golden hour".
| box | total | T5 | denoise | VAE | max RSS |
|---|---|---|---|---|---|
| **M3 Ultra 256GB** (80-core) | **148.7s** | 12.4s | 108.5s (5.4s/step) | 20.2s | **9.19 GB** |
| **M2 Max 96GB** (38-core) | **503.7s** | 5.9s (warm) | 450.4s (22.5s/step) | 46.5s | **9.19 GB** |
Findings:
1. **Output visually verified clean on both** (frames eyeballed) — and
*frame-identical across machines at the same seed*. Deterministic
cross-node reproduction; safe to route wan jobs to any capable node.
2. **True working set is 9.2 GB** for the default job — identical RSS on
both boxes. macOS `peak memory footprint` read ~81 GB on the M2 Max:
that is MLX's opportunistic Metal cache filling free RAM, not demand.
Don't size nodes off footprint.
3. Denoise scales with GPU cores almost exactly (4.15x for 80 vs 38);
VAE decode only 2.3x — small-job latency gap shrinks a little.
4. m2max now holds the 5B q8 weights (`~/MODELBEAST/vendor/mlx-video-models/`)
next to its A14B set — it can serve wan_video once a `venvs/mlxvideo`
is installed there (`scripts/install_mlx_video.sh`; note the numba pin).

View File

@ -12,8 +12,7 @@ The order of preference for any task:
|---|---|---|---| |---|---|---|---|
| **M3 Ultra** (primary) | `m3ultra.local` · 100.89.131.57 | 256 GB | Runs the server + the central queue on :8777. Handles everything, especially the heavy 3D models (`hunyuan3d_mlx`, `trellis_mac`) and FLUX. | | **M3 Ultra** (primary) | `m3ultra.local` · 100.89.131.57 | 256 GB | Runs the server + the central queue on :8777. Handles everything, especially the heavy 3D models (`hunyuan3d_mlx`, `trellis_mac`) and FLUX. |
| **M1 Ultra** (worker) | `ultra.local` · 100.91.239.7 | 128 GB | GPU worker in the pool **and** its own standalone instance. MLX-native — the right node for `hunyuan3d_mlx` (trellis's torch-MPS is unverified here). Also FLUX, `sf3d`, `bg_remove_local`. | | **M1 Ultra** (worker) | `ultra.local` · 100.91.239.7 | 128 GB | GPU worker in the pool **and** its own standalone instance. MLX-native — the right node for `hunyuan3d_mlx` (trellis's torch-MPS is unverified here). Also FLUX, `sf3d`, `bg_remove_local`. |
| **M2 Max** (m2max) | `m2max@100.120.83.110` | 96 GB | Second full gen node (added 2026-08-03), 4 TB SSD — the fleet's spare-disk box. Allowlisted: `flux_local`, `hunyuan3d_mlx`, `trellis_mac`, `trellis2_mlx`, `sf3d`, `seedvr2_upscale`, `mflux_image_edit`, `bg_remove_local`, `ffmpeg`/`ffprobe`. M2 does bf16, so unlike the M1s it **can** run trellis. Laptop — often asleep; treat as overflow, not latency-critical. If it silently drops out of the pool, check its `authorized_keys` first (wiped once, 7/31→8/3). | | **M4 Pro** (helper) | `m4pro.local` · 100.69.21.128 | 24 GB | Helper for light gpu ops (`bg_remove_local`) + **cpu ops** (`ffmpeg_frames`, `ffprobe`; 2 slots) + a standalone **Ollama LLM** (`qwen2.5:7b` at `http://100.69.21.128:11434`). **Not** the big 3D/diffusion models — too little RAM. |
| **M4 Pro** (helper) | `m4pro@100.69.21.128` | 26 GB | Helper for light gpu ops (`bg_remove_local`) + **cpu ops** (`ffmpeg_frames`, `ffprobe`; 2 slots) + a standalone **Ollama LLM** (`qwen2.5:7b` at `http://100.69.21.128:11434`). Too little RAM for the big 3D/diffusion models. |
| **M1 Max** (studio) | `studio@100.92.78.24` | 32 GB | Second full gen node (added 2026-07-17). `flux_local` (Klein 4B, ~38s/image), `hunyuan3d_mlx` (fp16, ~2× M3 time), `bg_remove_local`, `ffmpeg`/`ffprobe`. **Not** trellis (M1 bf16). Shared with audio work (stupendo) → treat as overflow, not latency-critical. | | **M1 Max** (studio) | `studio@100.92.78.24` | 32 GB | Second full gen node (added 2026-07-17). `flux_local` (Klein 4B, ~38s/image), `hunyuan3d_mlx` (fp16, ~2× M3 time), `bg_remove_local`, `ffmpeg`/`ffprobe`. **Not** trellis (M1 bf16). Shared with audio work (stupendo) → treat as overflow, not latency-critical. |
| **M4 mini** | `m4mini@100.124.220.31` | 16 GB | Light helper (added 2026-07-17): `bg_remove_local` + cpu ops (`ffmpeg_frames`, `ffprobe`; 2 slots). Also the always-on godcheck node. Too small for FLUX/fp16-hunyuan. | | **M4 mini** | `m4mini@100.124.220.31` | 16 GB | Light helper (added 2026-07-17): `bg_remove_local` + cpu ops (`ffmpeg_frames`, `ffprobe`; 2 slots). Also the always-on godcheck node. Too small for FLUX/fp16-hunyuan. |
| **M1 mini** | `mini@100.79.229.50` | 8 GB | Light node (added 2026-07-17): `bg_remove_local` + `net`-class overflow. Baseline/clean-room test box. | | **M1 mini** | `mini@100.79.229.50` | 8 GB | Light node (added 2026-07-17): `bg_remove_local` + `net`-class overflow. Baseline/clean-room test box. |

View File

@ -27,8 +27,6 @@ Which Apple Silicon Mac can run which operator, by **unified memory (RAM)**. Num
| `flux_local` schnell-4bit | gpu | 8.9 GB | **19.1 GB** | 24 GB | 32 GB | fast FLUX.1 draft | | `flux_local` schnell-4bit | gpu | 8.9 GB | **19.1 GB** | 24 GB | 32 GB | fast FLUX.1 draft |
| `hunyuan3d_mlx` fp16 *(our default, 4096/120k)* | gpu | 13 GB | **20.2 GB** | 24 GB *(tight)* | 32 GB | Stage 1 alone ~10 GB; both stages ~20 GB | | `hunyuan3d_mlx` fp16 *(our default, 4096/120k)* | gpu | 13 GB | **20.2 GB** | 24 GB *(tight)* | 32 GB | Stage 1 alone ~10 GB; both stages ~20 GB |
| `trellis_mac` | gpu | 14 GB | ~1518 GB | 24 GB | 32 GB | **M3+ (bf16)**; SOTA local mesh | | `trellis_mac` | gpu | 14 GB | ~1518 GB | 24 GB | 32 GB | **M3+ (bf16)**; SOTA local mesh |
| `wan_video` 5B q8 (832×480×49f) | gpu | 18 GB | **9.2 GB** RSS | 16 GB *(tight)* | 32 GB | measured on M3 Ultra AND M2 Max 2026-08-04, identical RSS; scales with res/frames — use `--tiling aggressive` on big jobs. MLX cache will *opportunistically* balloon into free RAM (81 GB footprint seen on 96 GB box) — that's cache, not demand |
| `ardy_motion` (text→motion) | gpu | 15.6 GB | **~22 GB** | 32 GB | 48 GB | Llama-3-8B text encoder is ~14.5 GB of it; motion model sub-GB. First job per boot pays ~12 min load |
| `flux_local` FLUX.1 dev | gpu | 31 GB | **25.1 GB** | 32 GB | 48 GB | cinematic, ~2 min | | `flux_local` FLUX.1 dev | gpu | 31 GB | **25.1 GB** | 32 GB | 48 GB | cinematic, ~2 min |
| `flux_local` Klein 9B | gpu | 32 GB | **28.4 GB** | 36 GB | 48 GB | best object accuracy in the lineup | | `flux_local` Klein 9B | gpu | 32 GB | **28.4 GB** | 36 GB | 48 GB | best object accuracy in the lineup |
| `mflux_image_edit` (Qwen-Image-Edit) | gpu | 54 GB | ~3040 GB *(est.)* | 48 GB | **64 GB+** | the heavyweight local editor | | `mflux_image_edit` (Qwen-Image-Edit) | gpu | 54 GB | ~3040 GB *(est.)* | 48 GB | **64 GB+** | the heavyweight local editor |
@ -53,9 +51,8 @@ Which Apple Silicon Mac can run which operator, by **unified memory (RAM)**. Num
|---|---|---|---| |---|---|---|---|
| **M3 Ultra** | 256 GB | the flex | everything + primary/queue | | **M3 Ultra** | 256 GB | the flex | everything + primary/queue |
| **M1 Ultra** | 128 GB | multi-model | `hunyuan3d_mlx` (MLX-native), FLUX, sf3d, RMBG, cpu ops — *not* trellis (bf16 unverified) | | **M1 Ultra** | 128 GB | multi-model | `hunyuan3d_mlx` (MLX-native), FLUX, sf3d, RMBG, cpu ops — *not* trellis (bf16 unverified) |
| **M2 Max** (m2max) | 96 GB | multi-model | `flux_local`, `hunyuan3d_mlx`, **`trellis_mac`** + **`trellis2_mlx`**, `sf3d`, `seedvr2_upscale`, `mflux_image_edit`, `bg_remove_local`, cpu ops. M2 = real bf16, so it *can* do trellis (the M1s can't). 4 TB disk — the fleet's model-storage box. Laptop, often asleep |
| **M1 Max** (studio) | 32 GB | second gen node | `flux_local` (Klein 4B), `hunyuan3d_mlx` (fp16), `bg_remove_local`, `ffmpeg`/`ffprobe` — *not* trellis (M1 bf16); ~2× M3 time. Shared audio box | | **M1 Max** (studio) | 32 GB | second gen node | `flux_local` (Klein 4B), `hunyuan3d_mlx` (fp16), `bg_remove_local`, `ffmpeg`/`ffprobe` — *not* trellis (M1 bf16); ~2× M3 time. Shared audio box |
| **M4 Pro** | 26 GB | light helper | `bg_remove_local`, `ffmpeg_frames`/`ffprobe`, Ollama 7B — too small for the 1828 GB gen models | | **M4 Pro** | 24 GB | light helper | `bg_remove_local`, `ffmpeg_frames`/`ffprobe`, Ollama 7B — too small for the 1828 GB gen models |
| **M4 mini** | 16 GB | light helper | `bg_remove_local` + cpu ops — too small for FLUX/fp16-hunyuan/trellis | | **M4 mini** | 16 GB | light helper | `bg_remove_local` + cpu ops — too small for FLUX/fp16-hunyuan/trellis |
| **M1 mini** | 8 GB | cloud+light node | `bg_remove_local`, `ffmpeg`/`ffprobe`-class, `net` ops — the cheapest-useful tier | | **M1 mini** | 8 GB | cloud+light node | `bg_remove_local`, `ffmpeg`/`ffprobe`-class, `net` ops — the cheapest-useful tier |

View File

@ -1,518 +0,0 @@
# FABLE STARTUP PACKET — TRELLIS.2 + Hunyuan MLX build
> Read this once, top to bottom, before touching anything. It consolidates five prep-only recon
> passes (all inspection, **no GPU job launched, no installs, no production code touched**). Every
> path here is on **m3ultra** (`m3ultra@100.89.131.57`) unless noted. Where a recon was uncertain,
> it says **(UNVERIFIED)** — do not treat those as facts.
>
> Two baselines you will see quoted, do **not** conflate them:
> - **124.8 s** = our current torch-MPS *production* path on the anatomy bench (the number to beat).
> - **1004.6 s / 3.09M tris / 17.82 GB** = PR#175's reported **MPS** E2E on an M4 Max 36 GB
> (different hardware, different mesh size, and NOT the MLX path). Not comparable to 124.8 s.
---
## 1 · STATE ON DISK — what prep already put on the box
Everything below already exists. **Do not redo it.**
### Staging dir: `~/Documents/trellis2-mlx-staging/`
| Path | What it is | Status |
|---|---|---|
| `trellis2-apple/` | Full clone of `pedronaugusto/trellis2-apple`, HEAD `6055b86` (2026-04-22), full 17-commit history deepened, MIT. 39 MB. **Nothing installed.** | read-only clone |
| `prep_local_draft.py` | 9638 B preprocessing draft (§4). `py_compile` OK under rmbg venv. Not wired in. | draft, do-not-run-yet-in-prod |
| `convert_weights.py` | Draft torch→MLX weight converter (§5). Loads ~14 GB — **do not run now.** | draft |
| `parity_dump_hook.py` | Draft monkeypatch fixture dumper (§5). Loads models+GPU — **do not run now.** | draft |
| `inspect_safetensors.py` | Stdlib safetensors header parser (used to produce the §5 layout table). | utility, safe |
| `__pycache__/` | Compile artifacts from the py_compile checks. | ignore |
### Venvs & operators already on the box (do not reinstall)
- **rmbg venv** `~/Documents/MODELBEAST/venvs/rmbg/` (uv, CPython 3.11). Has torch 2.13.0, torchvision 0.28.0, transformers 5.13.1, timm, kornia, einops, safetensors, hf_hub, Pillow, numpy, scipy, scikit-image. **No** cv2 / rembg / birefnet-pkg / upscaler. Installed by `~/Documents/MODELBEAST/scripts/install_rmbg.sh`.
- **`bg_remove_local` operator** `~/Documents/MODELBEAST/server/operators/bg_remove_local/run.py` — runs RMBG-2.0, pins `venvs/rmbg/bin/python`.
### Weights already in HF cache
- `microsoft/TRELLIS.2-4B``~/.cache/huggingface/hub/models--microsoft--TRELLIS.2-4B/snapshots/af44b45.../ckpts/` (7 of 8 pipeline modules).
- `sparse_structure_decoder` (8th module) comes from `microsoft/TRELLIS-image-large` snapshot `25e0d31.../ckpts/ss_dec_conv3d_16l8_fp16.safetensors`.
- `briaai/RMBG-2.0` — 844 MB, present.
- Hunyuan MLX converted weights — `~/.cache/huggingface/hub/models--dgrauet--hunyuan3d-2.1-mlx/`.
- **NOT present:** `ZhengPeng7/BiRefNet` (Apache/MIT, ~900 MB) — must be fetched once for the commercial preprocessing path.
---
## 2 · TRELLIS.2 MLX — ADOPT-vs-BUILD DECISION
### VERDICT: **ADOPT** the Jourloy PR#175 `mlx_backend/` tree as the MLX vendor base. Do **not** build from scratch, and do **not** wholesale-swap the 124.8 s production path — gate any promotion on a measured A0 run.
**Why PR#175 and not the already-cloned `trellis2-apple`:** the two share the *identical* 16-file / ~3,114-LOC `mlx_backend/` (same author — `pedronaugusto` co-authors PR#175). PR#175 (`Jourloy/TRELLIS.2`, opened 2026-07-17, +7,419/258 across 62 files) is a **strict superset**: same MLX modules **plus** the correctness oracle we lack (`tests/test_mlx_parity.py`, 4 unit parities at rtol 1e-63e-5), a backend probe/fallback resolver (`trellis2/backends.py`), a full CLI (`scripts/generate_asset.py`), macOS setup/probe/weight tooling, and the `--backend mlx-experimental` flag so optimized modules drop in behind a flag while MPS keeps E2E green. So PR#175 dominates `trellis2-apple` on every axis. Keep `vendor/trellis-mac` (shivampkumar, no MLX) only as the torch-MPS E2E oracle.
**Why NOT a blind swap (both recons agree):** the stock MLX path will very likely **regress** as-is, for two structural reasons:
- **Un-fused sparse conv.** `mlx_backend/sparse_conv.py` is the naive gather→matmul→sum, 512 MB-chunked — it does **not** use the `flex_gemm` Metal fast path, which our recon measured at **1.72× faster** on this shape. This is the ~10× crux, reimplemented slow in MLX.
- **CPU host bounce.** `mlx_backend/adapters.py` does `torch→numpy→mx→numpy→torch` on **every** model call — per sampler step (default ~12 steps × 3 stages × 2 for CFG) and at every sparse-conv boundary. The upstream torch pipeline still owns sampling orchestration + mesh extraction; MLX models are injected as adapters.
- **Zero benchmarks in either repo.** No evidence the MLX path beats — or matches — 124.8 s. Do **not** benchmark the stock `--backend mlx-experimental` path as representative; it's a correct scaffold, not a fast path.
- **Parity gap in `trellis2-apple`, closed by PR#175.** `trellis2-apple` has NO MLX parity test; PR#175 adds `test_mlx_parity.py`. Adopt PR#175 specifically to get that oracle.
**What is safe to reuse directly (low risk, no sparse-conv tax):** `dinov3.py`, `norm.py` (two-pass LayerNorm32 PT-parity), `rope.py` (interleaved complex-mul parity), `transformer_block.py`, `structure_decoder.py` (dense `mx.conv_general`). **Rewrite/replace first:** `sparse_conv.py` (fused Metal gather-GEMM-scatter), the `adapters.py` boundary (move the Euler/CFG loop *inside* MLX to kill the bounce), `sparse_ops.py` (numpy `np.where` host sync in upsample).
**Known smells to verify before trusting a module (UNVERIFIED in recon):** (1) `structure_decoder.py` `_remap_structure_decoder_weights` is a **no-op** feeding dict-stored conv weights — untested whether `nn.Module.load_weights` populates plain-dict children; (2) hand-rolled softplus/quad_lerp in `vae_decoders.py:1204` FlexiDualGrid decode; (3) full **non-windowed** attention (`attention.py`) may diverge from upstream windowed sparse attn; (4) docstrings claim "custom Metal kernel" but code is plain MLX built-in SDPA — **misleading comments, no hand-written kernel** anywhere in `mlx_backend/`. `mx.compile` *is* already applied to the DiT block loop and structure-decoder graph (with try/except fallback).
**Upstreamability: LOW — plan to fork, not to depend on merge.** PR#175 is unmerged, CLA-gated, 0 reviews, only the CLA bot has commented. Issue #74 (non-CUDA backends) has no maintainer response. Vendor it and **pin the commit**; basing our kernels on #175's layout keeps our work a clean, eventually-upstreamable diff.
### First 3 concrete commands
```bash
# 1. Get the superset fork (has test_mlx_parity.py; the cloned trellis2-apple does not)
cd ~/Documents/trellis2-mlx-staging && \
git clone https://github.com/Jourloy/TRELLIS.2 trellis2-jourloy && cd trellis2-jourloy
# 2. Run the correctness ORACLE first — cheap, no heavy GPU E2E, validates
# LayerNorm32 / apply_rope / SDPA / MlxSparseConv3d vs torch before any gen.
# (Needs an isolated venv with mlx + the deps; test no-ops off-Mac via importorskip.)
pytest tests/test_mlx_parity.py -v
# 3. The single GPU job — the measured A0 gate. Isolated venv, the 3 pedronaugusto
# Metal pkgs --no-build-isolation, reuse the existing TRELLIS.2-4B weights.
# ONE gen, anatomy bench, seed 42, 1024-cascade; compare wall-clock to 124.8 s
# and quality to the fal renders in ~/Documents/trellis2-bench/.
pip install -r requirements_macos.txt --no-build-isolation # torch>=2.11, mlx>=0.31, mtlgemm/mtldiffrast/cumesh
# then: scripts/generate_asset.py --backend mlx-experimental --seed 42 --pipeline-type 1024_cascade <anatomy_img>
```
**Decision rule after A0:** within ~1.5× of 124.8 s AND parity holds → patch-and-tune, adopt as production path behind the flag. If 35× slower (the likely outcome given un-fused conv + host bounce) → keep torch-MPS in `nodes.json` for production, lift `mlx_backend/` as the module-by-module scaffold, and land the kernel rewrites (fused Metal sparse conv → in-MLX sampler loop → parity fixtures) in that order. Also: fork to Gitea `monster/trellis2-mrp-mlx`, keep `origin` upstream for rebases (house pattern).
---
## 3 · HUNYUAN TUNE — edit-target list (ready to apply)
All paths under `~/Documents/MODELBEAST/vendor/hunyuan3d-mlx/`. Static inspection only (GPU was busy). Ship each edit with a PT-parity test (≤1e-4 rel) + a BENCHMARKS.md row.
**Headline: `head_dim` pad-to-64 does NOT apply** — every attention site is already a 64-multiple (64 or 128). The corridorkey padding win is **out of scope**; do not build pad wrappers. `mx.compile` count in the whole MLX port = **0** — that is the entire opportunity surface (corridorkey precedent: fixed-shape compile = 1.47× M3 / 1.11× M1, bit-identical, an 8-line change `vendor/corridorkey-mlx@4c660df`).
| # | Target | file:line | Change | Expected | Risk |
|---|---|---|---|---|---|
| 1 | **Paint UNet forward** (called 3×/step, ~15 steps) | `hy3dpaint/hunyuanpaintpbr_mlx/unet/unet_mlx.py:158` (`__call__`); loop `inference.py:254`, calls `:314,328,339`; steps default `inference.py:72` | Wrap forward in `mx.compile` (add `compile=True` ctor flag, `self._fwd = mx.compile(self._forward_impl)`). No MoE / no `.item()` → compile-safe. | ~1.11.3× on ~112 s paint | low; last chunk may have fewer views → 1 retrace (pad final chunk to fixed `n_chunk`) |
| 2 | **DiT dense blocks 014** (100 forwards = 50 steps × CFG 2) | `hy3dshape/.../denoisers/hunyuandit_mlx.py:379` (`HunYuanDiTBlock.__call__`); driver `pipeline_mlx.py:160`, loop `:143`, steps `:95` | `mx.compile` the dense-block forward (fixed shape `(B∈{1,2},4097,2048)`). Leave 6 MoE blocks (1520) eager until #3. | ~1.11.3× on ~149 s shape (dense = 15/21 depth) | low |
| 3 | **MoE `.item()` sync tax** (PREREQUISITE for full-DiT compile) | `mlx_arsenal/moe/moe.py:110``if not mx.any(w>0).item(): continue` | Remove the `.item()` early-skip; always run all 8 experts, let `w*out` (w=0) zero them. Deletes ~4800 syncs/gen AND makes the block static → unlocks compiling whole `HunYuanDiTPlain.__call__` (`hunyuandit_mlx.py:514`). | 4800 syncs/gen + unblocks #2 to 21/21 blocks | **`mlx_arsenal` is a SHARED dep** — do NOT edit in place. Vendor a local `MoELayer` subclass override, or upstream to arsenal. `MoEGate` routing is already static/compile-clean. |
| 4 | **Paint DINOv2-giant attn** (manual→fused, once/gen ×40 layers) | `hy3dpaint/hunyuanpaintpbr_mlx/dino_mlx.py:5860` (manual `q@kᵀ` softmax, fp32) | Replace with `mx.fast.scaled_dot_product_attention(q,k,v,scale=self.scale)` + existing transpose/reshape. `head_dim=64` hits fast path; fp32 supported. | one-shot; kills N² materialization (fused d64 = 1.51× on M3) | low |
| 5 | **Paint VAE decode** (once/gen) | `hy3dpaint/hunyuanpaintpbr_mlx/vae_mlx.py` decode; manual softmax `:8184` (`head_dim=512`) | `mx.compile` the decode (fuses the softmax). Prefer compile over `mx.fast` swap — `head_dim=512` likely **misses** MLX fused sdpa (kernel ≤256), would fall back anyway. | one-shot, minor | low |
| 6 | **(CONDITIONAL) Shape-VAE geo cross-attn** | `hy3dshape/.../autoencoders/model_mlx.py:206` (`CrossAttentionBlock`); driver `_query_sdf_volume` `:506`, chunk `:510` | `mx.compile` per-chunk cross-attn (fixed 10000-query × 4096-latent). Only final ragged chunk retraces. | smallmoderate | **Verify** decode isn't dominated by the Metal SDF/marching-cubes step before investing |
**Rasterizer: NO opportunity.** `hy3dpaint/DifferentiableRenderer/mesh_render_mlx.py` is a thin adapter over the `mlx_arsenal.rasterize` Metal kernel (already fused); Python side is glue + numpy round-trips. Skip for both levers.
**DTYPE: keep fp16 as the global default.** Port is fp16 everywhere (one deliberate exception: paint DINOv2-giant upcast to **fp32** in `load_model.py` — accumulation-depth fix, not a format preference). bf16 buys **zero** speed on M3/M4/M5 (equal TFLOPS) and is **~20% slower on M1** — and M1 Ultra (`johnking@100.91.239.7`) is the second production lane and hunyuan's headline "runs on M1" selling point. Do NOT globally switch to bf16. (This is the opposite of the TRELLIS.2 port, where bf16 range matters.)
**Execution order:** 1 → 3 → 2(dense, then whole-forward once 3 lands) → 4/5/6 (one-shot cleanups last).
---
## 4 · PREPROCESSING — draft, plug-in point, license, TODOs
**North star** (`meshgod/config.py:65-70`, `STYLE_3D`): a single isolated object, centered, fully visible (no crop); plain flat neutral light-gray bg; even soft shadowless studio light; neutral symmetrical A-pose.
**Draft: `~/Documents/trellis2-mlx-staging/prep_local_draft.py`** (9638 B, `py_compile` OK under rmbg venv, uses only PIL/numpy/torch/torchvision/transformers, not wired in). Signature `prep_local(in_path, out_path, commercial=False)`:
1. `_maybe_upscale` — if `min(w,h) < 1024`, Lanczos up to short-edge 1024, on RGB, **BEFORE** matte.
2. `_matte` — BiRefNet if `commercial` else RMBG-2.0; real inference mirroring `bg_remove_local/run.py` exactly (`Resize((1024,1024))→ToTensor→Normalize(imagenet)`; `model(inp)[-1].sigmoid()`; → `L` mask); model cached per-process.
37. `_compose_center_square` — composite onto `NEUTRAL_GRAY=(200,200,200)` RGB (not alpha) via mask, tight bbox crop from thresholded mask (`MASK_THRESH=8`), center, square-pad to `SUBJECT_FRAC=0.88`, resize 1024 Lanczos.
Run (GPU-free-time session): `venvs/rmbg/bin/python prep_local_draft.py --in X --out Y [--commercial]`.
**Order lesson (verified live 2026-07-13):** upscale FIRST, matte LAST — SeedVR strips alpha (RGBA→RGB), so matting first gets flattened. Draft honors this.
**Plug-in point:** `~/Documents/meshgod/meshgod/server.py:350` (local lane, `server.py:346-350`):
```python
paths[i] = stage3_reconstruct.mb_bg_remove(p, pp) # <- replace with prep_local(p, pp, commercial=...)
```
`prep_local` supersedes `mb_bg_remove` (does the full north-star transform in-process on rmbg venv, not just a farm-round-trip cutout to RMBG-2.0). Secondary optional precede-point: fal lane `server.py:359` (run `prep_local` locally, then skip fal `clean_bg`). Companion anchors to update: `config.py:143` (`BG_REMOVE_MODEL`), `web/index.html:229` label.
**LICENSE RULE (hard):**
| Model | HF id | License | Commercial ship? | Local? |
|---|---|---|---|---|
| RMBG-2.0 | `briaai/RMBG-2.0` | CC-BY-NC 4.0 | **NO** — non-commercial / personal one-offs only | Yes (844 MB) |
| BiRefNet | `ZhengPeng7/BiRefNet` | MIT | **YES** — safe for game/product assets | **No — fetch once** |
Default for anything game-bound = **BiRefNet**. RMBG-2.0 stays default only for local/personal one-offs. (Note: RMBG-2.0 *is itself* a BiRefNet-architecture checkpoint — same code, differs only in weights + license.)
**Open TODOs (in draft):**
1. **BiRefNet weights fetch** — first `--commercial` run must `huggingface-cli download ZhengPeng7/BiRefNet` (~900 MB, MIT, ungated). Until then only RMBG-2.0 works.
2. **Better upscaler** — PIL Lanczos is the zero-dep default (never invents detail). Swap for local faithful SR: MODELBEAST `seedvr2_upscale` operator (weights present: `models--numz--SeedVR2_comfyUI`) or Real-ESRGAN. Must stay BEFORE `_matte`.
3. **Lighting/exposure normalization (D2)** — optional auto-levels + gray-world WB on RGB before matte (fights TRELLIS dark-patch mode). Stubbed so v0 silhouette stays identical to the fal path for A/B.
4. **Tunables to A/B once GPU free:** `NEUTRAL_GRAY=(200,200,200)`, `SUBJECT_FRAC=0.88`, `MASK_THRESH=8` (all chosen from north-star text).
5. **Wire-in / flag plumbing** — add `req.commercial`, replace `server.py:350`, expose in `web/index.html`, decide local-lane upscale policy (`server.py:511` currently forces `upscale=False`).
---
## 5 · WEIGHT CONVERSION + PARITY
Drafts in staging: `convert_weights.py`, `parity_dump_hook.py`, `inspect_safetensors.py`. Both loaders touch ~14 GB / GPU — **run in a later session, not now.**
### Conversion pattern (from Hunyuan's `convert_realesrgan.py:34-58`)
`torch.load(weights_only=True)`**keys pass through unchanged** (module-tree names already match MLX list indexing, no remap dict) → **the only tensor mutation is conv transpose** (PyTorch `(O,I,H,W)`→MLX channels-last `(O,H,W,I)`, detected by `ndim==4`; generalizes to 5D Conv3d as `(0,2,3,4,1)`) → `mx.save_safetensors`.
### TRELLIS.2-4B layout — the transpose decision (resolved from source, not guessed)
There are **two Conv3d conventions on disk; the correct action is opposite for each:**
| pipeline key | file (ckpts/) | dtype | conv action |
|---|---|---|---|
| sparse_structure_flow_model | ss_flow_img_dit_1_3B_64_bf16 (640 t, 2.58 GB) | BF16 | none (pure Linear DiT) |
| **sparse_structure_decoder** (from TRELLIS-image-large) | ss_dec_conv3d_16l8_fp16 (74 t) | F32/F16 | **20 dense Conv3d → TRANSPOSE `(0,2,3,4,1)`** |
| shape_slat_flow_model_512 / _1024 | slat_flow_img2shape_dit_1_3B_{512,1024}_bf16 | BF16 | none |
| shape_slat_decoder | shape_dec_next_dc_f16c32_fp16 (292 t, 948 MB) | F16 | **40 sparse Conv3d → NO transpose (already channels-last)** |
| tex_slat_flow_model_512 / _1024 | slat_flow_imgshape2tex_dit_1_3B_{512,1024}_bf16 | BF16 | none |
| tex_slat_decoder | tex_dec_next_dc_f16c32_fp16 (284 t, 948 MB) | F16 | **40 sparse Conv3d → NO transpose** |
- **`ss_dec_conv3d` = dense `nn.Conv3d`**, on-disk `(Co,Ci,Kd,Kh,Kw)` → needs the 5D transpose. All 20 5-D tensors.
- **`*_next_dc` = o-voxel `SparseConv3d`**, on-disk **already** `(Co,Kd,Kh,Kw,Ci)`**load verbatim, NO transpose.** Proven by source: `conv_flex_gemm.py:33-34` and `conv_none.py:40-41` both `permute(0,2,3,4,1)` at construction, and the checkpoint was saved from that path. MLX forward must reshape `(Co,Kd,Kh,Kw,Ci)→(Kvol,Ci,Co)` for the per-offset gather-GEMM (contract at `conv_none.py:106`).
- **All 5 DiTs: pure `nn.Linear` (ndim 2) + attention + RMSNorm** — zero conv, zero transpose; only a BF16→target-dtype cast (**fp16 on M1** per fleet matrix). Block arch (30 blocks, identical across all 5): hidden 1536, 12 heads × **head_dim 128**, MLP 8192, cross-ctx 1024, adaLN-zero `modulation [9216]`. **head_dim 128 ≥ 64 → fused-SDPA fast path applies, NO padding.** Keep fused `to_qkv [4608,1536]` / `to_kv [3072,1024]` packed and slice in-kernel.
`convert_weights.py` encodes this per-file policy (dense-transpose for `ss_dec_conv3d`, keep for `*_next_dc`, passthrough for DiTs) with a kernel-dims sanity-assert. Uses `safetensors.torch.load_file` + `mx.save_safetensors`. Publish converted safetensors to Gitea/HF so re-runs skip conversion.
### Parity fixture plan
Stage boundaries all live in `run()` of `trellis2/pipelines/trellis2_image_to_3d.py:488-596`; each fixture = the **return value of a sub-method** → clean non-invasive monkeypatch, zero vendor edits. `parity_dump_hook.py` monkeypatches the 7 sub-methods, `torch.save`s each return to `$TRELLIS2_DUMP_DIR/<name>.pt`, then `runpy`-delegates to vendor `generate.py`. Unset `TRELLIS2_DUMP_DIR` → patches are no-ops.
| # | Fixture | Return / hook (file:line in trellis2_image_to_3d.py) | Shape/notes |
|---|---|---|---|
| A/A | Image-cond embed 512 / 1024 | `get_cond` return `:177` (called `:539`/`:540`; disambiguate by `resolution` arg → `cond_512.pt`/`cond_1024.pt`) | `[1,N,1024]` DINOv3 ViT-L/16 tokens; neg = zeros_like |
| B | Sparse-structure coords | `sample_sparse_structure` return `:235` (call `:542`) | int `[M,4]` (b,x,y,z); subs: `z_s [1,C,64³]` `:219`, occupancy `:227` |
| C | Shape-SLat latent | `sample_shape_slat_cascade` `:364` (`:567`) / non-cascade `:275` | SparseTensor `feats [K,32]`+`coords [K,4]`, **un-normalized** |
| D | Tex-SLat latent | `sample_tex_slat` `:432` (`:574`) | SparseTensor `feats [K,32]`, un-normalized, sampled w/ `concat_cond=shape_slat` |
| E1 | Decoded shape mesh + subs | `decode_shape_slat` `:389` (`:470`) | `List[Mesh]` + `List[SparseTensor]` |
| E2 | Decoded tex voxels | `decode_tex_slat` `:453` (`:471`) | SparseTensor `feats [K,6]` PBR (base_color 0:3, metallic 3:4, roughness 4:5, alpha 5:6) |
| E3 | Final mesh | `decode_latent` `:486` (`:592`) | `MeshWithVoxel`: `.vertices`,`.faces`,`.coords`,`.attrs`,`.voxel_size`,`.origin` |
Invocation (later, GPU): `TRELLIS2_DUMP_DIR=~/Documents/trellis2-bench/parity_fixtures python parity_dump_hook.py <anatomy_img> --pipeline-type 1024_cascade --seed 42 --texture-size 2048`. Fix `seed=42` (`torch.manual_seed` `:538`). Parity target **≤1e-4 rel**. Order cheap→expensive: A/A → B → C (the FlexGEMM sparse-conv crux) → D → E.
**GOTCHA:** C/D fixtures are captured **after** the std/mean de-normalization inside each sampler — the MLX port must apply the same `shape_slat_normalization`/`tex_slat_normalization` (32-dim mean/std, verbatim in `pipeline.json`) at the identical point or they diverge by a constant affine. `sample_tex_slat` also **re-normalizes** `shape_slat` back down (`:407-409`) before concatenating as `concat_cond` — replicate that round-trip.
---
## 6 · RECOMMENDED FIRST HOUR
1. **Read this packet + skim the 3 staging drafts** (`prep_local_draft.py`, `convert_weights.py`, `parity_dump_hook.py`) so you know what already exists. Do NOT re-clone `trellis2-apple` or re-inventory the rmbg venv.
2. **Clone the Jourloy superset** (`git clone https://github.com/Jourloy/TRELLIS.2 ~/Documents/trellis2-mlx-staging/trellis2-jourloy`) — §2 command 1. This is the vendor base; `trellis2-apple` lacks the parity oracle.
3. **Run the parity oracle** `pytest tests/test_mlx_parity.py -v` in an isolated venv (no heavy GPU E2E). Confirms LayerNorm32/RoPE/SDPA/SparseConv parity before trusting any module. This is the cheapest signal available and needs no A0.
4. **Fork to Gitea** `monster/trellis2-mrp-mlx`, keep `origin` upstream, **pin the PR#175 head commit** (it's unmerged/CLA-gated — do not depend on merge).
5. **Queue the single A0 GPU gate** for when the box is free — the ONE gen (anatomy bench, seed 42, 1024-cascade), compare to 124.8 s and the `~/Documents/trellis2-bench/` fal renders. Do not benchmark the stock MLX path as representative before the sparse-conv + adapter rewrites; expect a regression. Apply the §2 decision rule to the result.
6. **In parallel while GPU is busy** (all CPU/inspection, no contention): start Hunyuan **Target 1** (paint UNet `mx.compile`, §3) — cleanest, no blockers — with its PT-parity test; and/or dry-run `prep_local_draft.py` on a sample image under the rmbg venv (§4). Both are safe without the A0 result.
7. **Before any commercial preprocessing run:** `huggingface-cli download ZhengPeng7/BiRefNet` (§4 TODO 1) — otherwise only the non-commercial RMBG-2.0 path works.
**Prime directives:** parity-first (every kernel/edit ships a ≤1e-4-rel PT-parity test + a BENCHMARKS.md row); do NOT edit shared `mlx_arsenal` in place (§3 Target 3); default BiRefNet for game-bound assets (§4). The GPU A0 gen is the ONLY job that must wait for a free box.
---
## 7 · PREP ROUND 2 — VERIFICATION RESULTS (2026-07-19)
> Round-2 deep-verify of §1§6 against **live source + on-disk safetensors headers**. Still
> inspection-only: **no GPU gen, no weights loaded, no prod code touched** (`py_compile` /
> header-parse / `pytest --collect-only`). Where §7 says CONFIRMED / CORRECTION / CONTRADICTION,
> **trust §7 over the original section.**
### 7.1 · ADOPT confidence — §2 stance → **CONFIRMED (HIGH)**
- **Clone DONE & pinnable.** `~/Documents/trellis2-mlx-staging/trellis2-jourloy/`, HEAD `754d403`
(2026-07-17, Jourloy — matches "PR#175"), branch `main` **is** the PR head, tree clean.
**Pin `754d403`** when forking.
- **Superset CONFIRMED — stronger than §2 claimed.** All 4 named artifacts present
(`tests/test_mlx_parity.py`, `trellis2/backends.py`, `scripts/generate_asset.py`,
`requirements_macos.txt`) **plus 4 extra test files** (`test_backend_fallbacks`,
`test_background_preprocess`, `test_generate_asset`, `test_model_revisions`) **and** a new
`trellis2/model_revisions.py` (HF revision pins). `mlx_backend` = 3172 LOC / 16 files, all
`py_compile` clean.
- **`mlx_backend` numerics = `trellis2-apple` byte-for-byte.** `diff -rq`: 14/16 files identical;
the only 2 that differ (`dinov3.py`, `pipeline.py`) add **HF revision-pinning / offline kwargs
only — zero algorithm change.** §2's "identical mlx_backend" holds for numerics; jourloy is a
strict superset (adds reproducibility plumbing on top of the same math).
- **`--backend mlx-experimental` cleanly wired** (`generate_asset.py:333`→`create_mlx_pipeline`,
`:343`) behind a real **Metal-completeness probe gate** (`:170-187` → `backends.probe_metal_backends()`)
that refuses a half-built stack instead of silently degrading. MPS stays the E2E path.
- **§2's 4 UNVERIFIED smells — now resolved:**
| # | smell | round-2 disposition |
|---|---|---|
| 1 | structure_decoder no-op remap + dict-child conv load | **BENIGN by construction** (leaves pre-alloc'd `mx.zeros`; MLX param tree recurses dict/list; `load_weights(strict=True)` replaces leaves). Residual = exact key-name match only, **not** exercised by parity test → run a 2-sec non-GPU load smoke on ss_dec. |
| 2 | vae_decoders softplus/quad_lerp | **BENIGN** — it's the numerically-stable softplus identity `max(x,0)+log1p(exp(-|x|))` = torch softplus. Real line `vae_decoders.py:196` (§2's `:1204` misattributed). Needs one decode-boundary fixture (E1/E2). |
| 3 | non-windowed global attention | **NEEDS-RUNTIME-CHECK / POTENTIALLY REAL — highest-value item.** `_sdpa` does full global SDPA, **no window mask**. If upstream sparse flow is windowed → numeric divergence **and** O(N²) blowup at 1024-cascade. Tiny SDPA parity test validates the primitive, not the windowing semantics. **Verify vs upstream sparse-attn config before trusting shape/tex output.** |
| 4 | "custom Metal kernel" docstrings | **CONFIRMED BENIGN (cosmetic)** — grep `metal_kernel|@mx.custom|.metal|kernel_source` over `mlx_backend/` = zero matches; code calls MLX built-in `mx.fast.scaled_dot_product_attention`. No hand-written kernel anywhere. |
- **Bonus (confirms §2):** sparse-conv perf crux is **REAL**`sparse_conv.py` is naive
gather→matmul→sum, 512 MB-chunked with a per-chunk `mx.eval` host sync, O(batch·D³) dense LUT,
**no `flex_gemm`.** Matches the ~10× claim.
- **Confidence HIGH, not lowered:** nothing contradicted the ADOPT case; several findings
strengthened it. Two items still need a *runtime* signal (smell 3 windowed-attn, smell 1
dict-child load) — **both cheap, non-GPU; run before the A0 gate** (see 7.6).
### 7.2 · WEIGHT-CONVERT — transpose policy **VERIFIED vs disk**; converter has 2 bugs + 1 double-transpose contradiction
All 8 modules parsed **header-only** (no tensor bytes). Tensor counts match §5 **verbatim**
(640/640/640/640/640/292/284/74). Classifier found **ZERO ambiguous 5D tensors** — conventions
separate cleanly by shape.
| module(s) | dtype on disk | 5D convs | §5 policy | matches disk |
|---|---|---|---|---|
| 5 DiTs | **BF16** ×640 ea | 0 (215 Linear + 120 RMSNorm γ + 305 1D) | passthrough + dtype cast | ✅ |
| shape_dec / tex_dec | **F16** | 40 sparse ea, `(Co,K,K,K,Ci)` | **NO transpose** | ✅ |
| ss_dec | **F32×38 + F16×36** (2 F32 convs = `input_layer` + `out_layer.2`) | 20 dense, `(Co,Ci,K,K,K)` | **transpose `(0,2,3,4,1)`** | ✅ |
Transpose/convention classification in §5 holds on disk with **zero contradictions**. The
converter's `else: raise RuntimeError` on any unclassified 5D fails **loud**, so no silent
transpose-convention corruption is possible. **But three items must be fixed before any convert run:**
- 🔴 **BUG 1 (BLOCKING)** `convert_weights.py:59` `v.numpy()` on `torch.bfloat16`
`TypeError: unsupported ScalarType BFloat16`. **Crashes on all 5 BF16 DiTs** (the bulk of the
pipeline); only survives on F16/F32 decoders. Loud, not silent. Fix: `v.float().numpy()`, or
bit-reinterpret via `uint16`, or drop torch and use `mx.load`/`mx.save_safetensors`.
- 🟠 **BUG 2 (SILENT precision loss)** uniform `target_dtype` default `"bf16"` (`:46,81-83,94`)
downcasts the decoders under default args: shape_dec/tex_dec **F16→bf16** (lose 3 mantissa bits,
strictly worse + ~20% slower on M1); ss_dec's **2 F32 convs → bf16** (lose 16 bits on the most
sensitive first/last SDF layers). **Contradicts §5's own dtype column (decoders = F16).** Fix:
per-file dtype policy — DiTs→bf16 (fp16 on M1), decoders→preserve on-disk dtype.
- ⚠️ **CONTRADICTION — ss_dec DOUBLE-TRANSPOSE (silent-corruption risk).** §5/converter transpose
ss_dec `(0,2,3,4,1)` **at load**. But the live MLX `structure_decoder.py` stores PT-format
`(Co,Ci,kD,kH,kW)` **verbatim** and transposes **at forward-call time** (`conv3d`,
`structure_decoder.py:14`). If the converter *also* transposes ss_dec, the decoder
**double-transposes → garbage, silently.** **Resolve before converting:** convert ss_dec to
PT-format on disk (NO load transpose) to match this file, OR move the transpose — pick exactly
one, not both.
- **Port dependency (not the converter):** the MLX sparse-conv module MUST do the
`(Co,K,K,K,Ci)→(Kvol,Ci,Co)` reshape at load (`conv_none.py:106`) or the `*_next_dc` decoders
emit garbage. Converter correctly leaves this to the module.
### 7.3 · PARITY HOOKS — all 8 targets match live source, **ZERO corrections to the hook**
`parity_dump_hook.py` needs no edits. Source `trellis2_image_to_3d.py` is frozen at upstream
`75fbf01` (2026-06-05) + 1 benign local Mac cuda-guard (`:590-591`, downstream of every hook →
shifts no fixture line). **No post-packet drift.** Verified def/return/call (corrections to §5's
fixture table where it drifted):
| fixture | def | return | call-site(s) |
|---|---|---|---|
| get_cond (A/A) | 164 | **181** (no-neg) / **183-186** (dict) | 539, 540 |
| sample_sparse_structure (B) | 188 | 235 | 542 |
| sample_shape_slat (C, non-cascade) | 237 | 275 | 547, 557 |
| sample_shape_slat_cascade (C) | 277 | 364 (tuple `slat,hr_res`; `transform` takes `r[0]`) | 567, 579 |
| sample_tex_slat (D) | 391 | 432 | 551, 561, 574, 586 |
| decode_shape_slat (E1) | 366 | 389 | 470 |
| decode_tex_slat (E2) | 434 | 453 | 471 |
| decode_latent (E3) | 456 (`@torch.no_grad` :455) | 486 | 592 |
- **§5 CORRECTION:** the fixture table cites get_cond "return **:177**" — that line is the model
call `cond = self.image_cond_model(image)`, **not** the return. Real returns are **:181 / :183-186**.
Cosmetic (hook wraps by method name), but corrected.
- **Normalization gotcha CONFIRMED:** `shape_slat.pt` / `tex_slat.pt` captured **POST-denorm**
(`:271-273`, `:360-362`, `:428-430`); tex sampler **re-normalizes** shape_slat back down at
`:407-409` before `concat_cond` (:419). Port must replicate **both** the on-return de-norm and the
denorm→renorm round-trip or fixtures diverge by a constant affine (silent ≤1e-4 failure).
- **UNCERTAIN — flag for exec session:** the hook's `__main__` `runpy`-delegates to vendor
`generate.py`; that file's existence + CLI contract (`--pipeline-type` / `--seed` /
`--texture-size`) was **not** checked. Verify before the dump run — a missing/renamed delegate
means the hook installs but never fires.
### 7.4 · PREPROCESSING — BiRefNet **PRESENT**; D2 **IMPLEMENTED**
- **BiRefNet fetched (§1 "NOT present" TODO → DONE).** Cache:
`/Users/m3ultra/.cache/huggingface/hub/models--ZhengPeng7--BiRefNet/snapshots/e2bf8e4460fc8fa32bba5ea4d94b3233d367b0e4/`.
Size **424 MB** (`model.safetensors` = 444,473,596 B) — **§1/§4's "~900 MB" was high.**
License **MIT** (README `license: mit`) — **§1/§4's "Apache-2.0" label was wrong.** Ungated, no login.
- **BiRefNet branch VERIFIED CORRECT (not stubbed):** RMBG-2.0 *is* a BiRefNet checkpoint (identical
`auto_map`/`architectures`); forward `model(inp)[-1].sigmoid()` holds (`birefnet.py:2087/2090`);
deps satisfied by the rmbg venv — **no cv2 needed** (`birefnet.py` has zero opencv refs). Only
metadata was hardened; no code change.
- **D2 (light normalization) IMPLEMENTED** behind `normalize_light=False` (default **OFF** → v0
silhouette stays A/B-identical to fal). New `_normalize_light` (`prep_local_draft.py:84`, PIL/numpy
only): 1/99-pct auto-levels (flat-channel guarded) + gray-world WB; wired at step **1b** (after
upscale, before matte — honors "matte LAST"). CLI `--normalize-light`. Functional smoke passed on a
synthetic cast image.
- **Draft status:** `~/Documents/trellis2-mlx-staging/prep_local_draft.py` now 236 lines,
`py_compile` OK + imports clean under rmbg venv. Signature
`prep_local(in_path, out_path, commercial=False, normalize_light=False)`. **Staging only — still
NOT wired into prod** (§4.5 wire-in + tunable A/B still open; need a GPU-free box for real A/B).
### 7.5 · MLX TEST VENV — oracle is **one command from running**
- **Venv READY:** `/Users/m3ultra/Documents/trellis2-mlx-staging/mlx-test-venv` (CPython **3.12.13**,
uv). `mlx==0.32.0` (+`mlx-metal` 0.32.0), `torch==2.13.0` (prebuilt wheel, no compile),
`numpy` 2.5.1, `pytest` 9.1.1. (safetensors/transformers/einops also installed but the oracle uses
**none** of them — droppable for a leaner venv.)
- **Collects clean:** `--collect-only`**4 tests collected, 0 import errors, 0 skips** (0 skips
proves both `importorskip("mlx.core")` and `importorskip("torch")` succeeded on the box). **Not yet
executed** — a real pass is still pending (low-risk caveat: `np.asarray(mlx_array)` under numpy
2.5.1 surfaces only on the real run, not collection).
- **GPU/Metal?** 4 tests run microsecond ops on tiny tensors. MLX's default device is Metal (GPU), so
they *touch* the GPU — but this is trivial library compute, **NOT a generation/inference job**, uses
**no Metal build packages** (mtlgemm/mtldiffrast/cumesh) and no compile. Won't meaningfully contend
for the box; force CPU with `mx.set_default_device(mx.cpu)` if ever desired.
- **EXACT run command** (use the venv python — a bare `pytest` uses the wrong interpreter, correcting
§6/§2):
```
ssh -n m3ultra@100.89.131.57 'cd ~/Documents/trellis2-mlx-staging/trellis2-jourloy && \
../mlx-test-venv/bin/python -m pytest tests/test_mlx_parity.py -v'
```
Expected: **4 passed.** Baked tolerances: LayerNorm `2e-5`, RoPE `1e-6`, SDPA `3e-5`,
SparseConv `2e-5` (all tighter than the ≤1e-4 target).
### 7.6 · UPDATED FIRST MOVES — deltas to §6
- **§6 step 2 (clone Jourloy): DONE** — `trellis2-jourloy/` @ `754d403`, tree clean. Skip the clone;
just `git checkout 754d403` / pin it when forking.
- **§6 step 3 (run parity oracle): venv READY, oracle collects clean.** Now a one-command move — run
the exact command in **7.5**. **Do NOT use the bare `pytest tests/...` from §6/§2** (wrong
interpreter).
- **§6 step 7 / §4 TODO 1 (BiRefNet fetch): DONE** — skip. Present at the 7.4 path (424 MB, MIT).
- **NEW — before any weight-convert run (§5):** fix converter **Bug 1** (blocking bf16 crash) +
**Bug 2** (silent decoder downcast), AND resolve the **ss_dec double-transpose** contradiction (7.2).
Else DiT convert crashes, decoders lose precision, or ss_dec silently corrupts.
- **NEW — before the A0 GPU gate:** run the two cheap **non-GPU** runtime checks — (a) ss_dec
dict-child weight-load smoke, and especially **(c) global-vs-windowed attention vs the upstream
sparse-attn config** (the one smell that can be a true correctness bug, not just slow). Both fit
CPU/inspection while the box is busy.
- **Unchanged:** §6 step 5 (single A0 GPU gate — seed 42, 1024-cascade, compare 124.8 s) is still the
**only** job that must wait for a free box; §6 step 6 (Hunyuan Target 1) still safe in parallel.
## 8 · CORRECTION — the adopt path needs NO weight conversion (2026-07-19, verified in source)
Round-2 flagged three bugs in `convert_weights.py` (bf16 crash, decoder downcast, ss_dec double-transpose). **All three are moot: the converter is not needed for the adopted Jourloy base, and using it would corrupt.** Verified by reading Jourloy `mlx_backend/`:
- **Weights load directly from the original TRELLIS.2-4B safetensors.** `pipeline.py:116/141/162``load_safetensors("<path>.safetensors")``model.load_weights(...)`. `load_safetensors` (`__init__.py:26`) is just `mx.load(path)`, keeping native bf16/fp16; an optional `dtype=` casts float weights (this is the M1 fp16 path — **not** a separate converter).
- **Remap = key-rename ONLY, zero tensor transform.** `remap_flow_model_weights` / `remap_vae_decoder_weights` only rewrite key strings (nn.Sequential `.N.`→`.layers.N.`; MlxSparseLinear `.linear.` wrap). No transpose, no dtype change to any tensor.
- **All conv transposing is IN-MODULE at forward.** `structure_decoder.py:14` transposes the dense ss_dec (Co,Ci,K,K,K)→(Co,K,K,K,Ci) every forward; `sparse_conv.py:130` `w.transpose(1,2,0)` for the sparse GEMM. So the modules EXPECT PyTorch-format weights.
- **Why the converter is a trap:** it transposes ss_dec at load; the module transposes again at forward → double-transpose. For conv1/conv2 (Ci==Co==channels) the SHAPE still matches, so `load_weights` succeeds silently with wrong values; for input_layer (Ci=latent≠Co) it would shape-mismatch and error. Mixed silent/loud corruption.
**ACTION for Fable:** do **not** run `convert_weights.py` for the Jourloy path (it's now banner-warned in staging). For M1 fp16, pass `dtype=mx.float16` to `load_safetensors`. §5's transpose table remains correct as a *description of the on-disk layout*, but the CONVERSION step it implied is unnecessary — Jourloy consumes originals directly. A converter is only relevant if you ever build a from-scratch decoder that does not transpose at forward.
## 9 · EXECUTION LOG — Fable session 1 (2026-07-19)
Done this session (all committed; BENCHMARKS.md has the rows):
1. **Parity oracle: 4/4 PASS** (2.55s, CPU-only, mlx-test-venv).
2. **Jourloy venv built** via scripts/setup_macos.sh (PYTHON_BIN=uv 3.11; Metal pkgs compiled).
3. **A0 GATE PASSED — ADOPT CONFIRMED**: mlx-experimental beat MPS (216.6s vs 248.8s wall) at 14× bake density. It's a HYBRID (Metal flex_gemm sparse ops + MLX dense); naive MLX sparse-conv never runs. 75.4GB peak.
4. **Hunyuan mlx-tune branch @ 4b368ee** (pushed to partly): T1 UNet mx.compile (env-gated HY3D_MLX_COMPILE, parity 3e-5, 1.04× micro), T3 StaticMoELayer (bit-identical, kills ~4800 syncs, compile-unlocked), T4 fused DINO SDPA (1.41× micro, parity 1e-7). E2E flag-on run in flight vs the 260s defaults baseline.
5. **Dark-patch culprit narrowed**: reproduces on BOTH independent pipelines (trellis-mac MPS + Jourloy MLX) → shared pedronaugusto Metal texel-sampling (mtldiffrast/BVH) or decoded attrs. fal is clean on the same weights → attrs likely fine → **Metal sampling = prime suspect**. KDTree pure-Python bake control still grinding (2h+, alive, 300% CPU) — its render decides.
NEXT (in order): read KDTree verdict when it lands → Hunyuan e2e A/B number → wire prep_local into MESHGOD behind a flag → Target 2 (whole-DiT compile now that T3 unblocked it) → promote mlx-experimental into MODELBEAST as a `trellis2_mlx` operator behind a flag once the bake-quality question is settled.
## 10 · DARK-PATCH VERDICT: Metal texel-sampling convicted (2026-07-19)
Decisive experiment: dumped the decoded tex voxels (E2 fixture,
1,398,769 voxels × 6 PBR attrs) and rendered base_color as a direct
front-projection — `trellis2-bench/attrs_front.png`. **The attrs are
near-fal quality** (clean anatomy palette; dark voxels only 5.6%, all
interior). Both local pipelines (trellis-mac MPS and Jourloy MLX) produce
the same mottled dark bake from these clean attrs, and both share the
pedronaugusto mtldiffrast/mtlbvh texel→voxel sampling. ⇒ **the bake's
attr-sampling is the sole quality gap.** The 3h+ KDTree run is now merely
confirmatory (left running).
### Designed fix (next session's opening move)
The attrs live on a REGULAR voxel grid — no BVH/KDTree needed at all:
texel → (xatlas UV → triangle barycentric → world pos) →
`grid[(pos-origin)/voxel_size]` direct index (+8-neighbor occupancy
search). One vectorized gather over all texels; replaces the suspect
ray-cast sampling entirely; implementable in torch-MPS or MLX in ~a page.
Validate against `attrs_front.png` colors + the fal render. If it works,
BOTH local paths get fal-class texture quality at zero bake-speed risk.
### Parity fixture set: COMPLETE
All 8 fixtures captured at `trellis2-bench/parity_fixtures/` (cond_512/
1024, ss_coords, shape_slat, shape_dec, tex_slat, tex_voxels, mesh_final;
seed 42, 1024-cascade). Hook env needed: PYTORCH_ENABLE_MPS_FALLBACK=1
ATTN_BACKEND=sdpa SPARSE_ATTN_BACKEND=sdpa SPARSE_CONV_BACKEND=flex_gemm.
### Fork home (added 2026-07-19, post-§10)
The vendored Jourloy/PR#175 tree now lives on our Gitea:
`ssh://git@100.71.119.27:222/monster/trellis-2-mrp-mlx.git` (note the
hyphens), branch `main`, pin tag `pr175-head-754d403`. m3 staging clone has
it as remote `partly`; `origin` stays github/Jourloy for rebases. All our
kernel/bake work lands as commits on this fork.
## 11 · BAKE FIX PROVEN (2026-07-19 evening) — dark patches eliminated
`trellis2-bench/render_vbake5_front.png`: full anatomy figure, correct
colors, ZERO dark patches — from the same decoder attrs the Metal baker
mangles. Method: cKDTree IDW sampling of the voxel grid at mesh vertices
(k=4, distance² weights, 1.5-voxel radius, single-nearest fallback) →
vertex-colored GLB. **Sampling cost: 0.3s** for 270K verts / 1.4M voxels.
Two integration gotchas (cost hours; do not rediscover):
1. **Axis spaces.** Voxel grid + raw pipeline mesh = Z-up-long-axis in
[-0.5,0.5]³; GLB-loaded meshes = glTF Y-up. Transform before sampling:
`(x,y,z)_gltf → (x,z,y)_voxel`. The "mangled blob" renders were ONLY
this (figure viewed down its own axis) — no data corruption anywhere.
2. **glTF vertex colors are LINEAR** — do not pre-gamma (double-gamma =
washed out). Store linear, let the renderer do the transfer.
Also learned: raw `MeshWithVoxel.vertices/faces` need to_glb's weld/init
before use as a plain triangle soup; `output_3d.obj` inherits the raw
convention. And xatlas hangs (2h+) on écorché-topology meshes — the UV
path needs chart budgeting or per-island parallel unwrap.
### Production path (next session)
- Option A (fast to ship): full-density vertex-colored GLB as the raw
output; MESHGOD's Blender finish farm already decimates + can bake
verts→texture (standard Blender bake) — sidesteps xatlas entirely.
- Option B (fal-parity): fast bucket rasterizer (written, in
fast_bake_test.py) + xatlas-alternative UV → 2048² texture bake with
this sampler. Needs the écorché-safe unwrap.
- Wire as `--baker python-fast` in the Jourloy fork + trellis-mac, make it
the default over the convicted Metal texel path; port sampler to MLX
(trivial — gather + weighted sum) for the pure-MLX lane.
### §11 addendum — vertex baker PRODUCTIONIZED (2026-07-19 night)
Wired and validated in BOTH pipelines:
- `vendor/trellis-mac` @ 55fdd7b: `TRELLIS2_BAKER=vertex` env mode in
generate.py (Metal + kdtree paths untouched; sentinel-gated).
- Jourloy fork @ 746e727 (pushed to monster/trellis-2-mrp-mlx): CLI
`--baker vertex`.
E2E validation (anatomy, seed 42, 1024-cascade, uncapped density):
**130.0s total, bake 1s**, 232s wall, 20.8GB peak — full 2.79M-face
vertex-colored GLB, ZERO dark patches, best local quality yet
(`render_vprod_front.png`). vs Metal-bake baseline: same speed envelope,
quality gap closed for albedo. Not yet carried: metallic/roughness maps
(vertex colors are albedo-only) — MR needs the UV path (Option B) or a
COLOR_1 convention. MESHGOD finish-farm verts→texture bake = the
remaining Option-A step for game-budget assets.
## 12 · RUNG-3 KERNEL DAY — RESULTS (2026-07-20)
**Headline: 216.6s → 71.96s e2e (3.0×), peak 75.4GB → 26.5GB (2.8×),
parity held (raw mesh Δ0.03%), full gate EXIT=0.** All on the fork
(monster/trellis-2-mrp-mlx): fa972aa + 7860148 + 0db816b.
What actually moved the needle (in order of impact):
1. **Vertex baker properly wired into generate_asset** (scheduler
fallthrough fixed — `--baker vertex` was silently running the 5h
pure-python kdtree path): "other" bucket 69.4s → 8.3s, and the
75GB peak turned out to be the METAL BAKE's, not the conv's.
2. **Pure-MLX sampler loop** (mlx_samplers.py): torch CPU loop + per-
forward host bounces → one conversion per stage, batched dense CFG,
once-per-stage concat_cond. Sampling 56.8→54.4s (~4% — forwards are
compute-bound) but it is the correctness-proven, compile-ready
architecture, and removes ~72 host round-trips/gen.
3. **Validator taught the vertex-colored asset class** (COLOR_0 without
UVs = legit; normals materialized at export).
Honest negatives (documented so nobody re-chases them):
- mx.compile on the step forward: ~0 on M3 Ultra (GEMM-bound; matches
Hunyuan T1/T2 findings).
- Fused Metal sparse-conv kernel: built, parity 8e-4, but stock chunking
already bounds memory at decoder scale and beats the scalar kernel on
speed. Kept as TRELLIS2_METAL_SPCONV=1 opt-in reference (simdgroup
tiling = the future version if decoders ever matter for time).
- Remaining gap to H100 (~17s at 1024): inside the DiT forwards
(54s sampling ≈ 2.3× ideal FLOPs) — attention/MLP internals, a
future deep-dive, not orchestration.
Env knobs (all default-sane): TRELLIS2_MLX_SAMPLER=0 (revert sampler),
TRELLIS2_MLX_COMPILE=1, TRELLIS2_METAL_SPCONV=1.
M1 Ultra validation: in flight (26.5GB peak fits with 100GB headroom).
### Rung-3 addendum — M1 Ultra validated (2026-07-20)
| box | e2e wall | peak mem |
|---|---|---|
| m3ultra | 71.96 s | 26.5 GB |
| **m1ultra** | **181.5 s** | **17.2 GB** |
The M1 Ultra — which could not run TRELLIS.2 at all before this fork
(torch-MPS path was m3-only in practice) — now generates full-density
vertex-baked assets in ~3 min at 17GB. Install: clone
monster/trellis-2-mrp-mlx to ~/trellis2-mlx + scripts/setup_macos.sh
(PYTHON_BIN=uv 3.11) + gated weights rsynced from m3's HF cache
(no HF login needed on workers). Both Ultras now serve the fastest
local TRELLIS.2 in existence. TRELLIS.2 fleet: m3 primary (72s),
m1 second lane (182s), zero dollars per asset.

View File

@ -1,212 +0,0 @@
# MRP-MLX — the MonsterRobotParty MLX stack
**The short version:** every serious generative model we run — image, layered
image, image→3D, video keying — runs *locally, on Apple Silicon, in MLX or
Metal, from forks we control, at $0 per generation.* This document is the
canonical reference: what each piece is, where it lives, how fast it goes,
how to run it, and the hard-won rules that keep it fast. Written 2026-07-20
after the rung-3 kernel push; every number here was measured, not estimated.
---
## 1 · Philosophy (why this stack wins)
1. **Local-first.** Hosted GPU (fal) is for interactive one-offs and A/B
references only. Volume work runs on the fleet. (July fal bill was
~$93/mo before this stack; the same workload is now ≈ $0.)
2. **Adopt-then-patch, never from-scratch.** Every port here started as the
best community MLX effort, vendored to our Gitea, then patched where
measurement said to. Proven 5×: hunyuan3d (dgrauet), corridorkey
(cmoyates/Niko), trellis-2 (Jourloy PR#175), qwen-layered (mflux PR#302),
flux (mflux upstream).
3. **Forks live on our Gitea** (`ssh://git@100.71.119.27:222/monster/…`)
with upstream kept as `origin` for rebases and a pin tag on the adopted
commit. Our infra, our history, our rollback.
4. **Parity-first.** No optimization ships without a numeric gate against
the reference implementation (tolerances recorded), and e2e output
stats compared at fixed seed. Fixture sets are kept for regression.
5. **Bench-everything, keep the negatives.** BENCHMARKS.md records failures
(q4 = noise, mx.compile ≈ 0 on big GEMMs) with the same care as wins, so
no future session re-chases a ghost.
6. **Env-gated changes.** Every behavioral change has an env knob and a
default that preserves upstream behavior (or our validated best). Prod
rollback is an environment variable, not a revert.
## 2 · The fleet it runs on
| box | chip | RAM | fp16 TFLOPS | bandwidth | role |
|---|---|---|---|---|---|
| m3ultra | M3 Ultra | 256GB | 23.5 | 400 GB/s | primary: TRELLIS.2, queue, heavy gen |
| m1ultra | M1 Ultra | 128GB | 16.3 | 297 GB/s | second lane: everything (post rung-3) |
| m1max | M1 Max | 32GB | 7.1 | 336 GB/s | bandwidth-friendly light ops |
| m4pro | M4 Pro | 24GB | 5.2 | 225 GB/s | MB light ops + Ollama (24GB caps it) |
Full 7-machine MLX matrix + probe script: `BENCHMARKS.md` (2026-07-19) +
`scripts/mlxprobe.py`. Rules from it: **fp16 on M1-class** (bf16 20%),
bf16 free on M3/M4/M5; pad attention head_dim→64 on M4/M5-class.
## 3 · The pillars
### 3.1 TRELLIS.2 — image → full-density 3D · `monster/trellis-2-mrp-mlx`
- **Base:** Jourloy/TRELLIS.2 (upstream PR#175 head, pin `pr175-head-754d403`)
— hybrid MLX (dense stages) + pedronaugusto Metal kernels (sparse ops).
- **Our commits:** `fa972aa` pure-MLX sampler loop (batched dense CFG,
once-per-stage conversions, one eval/step; parity Δ0.03% e2e) ·
`7860148` opt-in Metal spconv kernel + vertex-baker scheduler fix ·
`0db816b` vertex-normal export + COLOR_0 asset class in the validator.
- **Weights:** original `microsoft/TRELLIS.2-4B` safetensors, loaded
directly (`mx.load` + key renames; **no conversion step exists or should**
— a converter would double-transpose ss_dec into silent garbage).
- **Performance (anatomy bench, seed 42, 1024-cascade, vertex baker):**
| box | e2e | peak mem |
|---|---|---|
| m3ultra | **72.0 s** | 26.5 GB |
| m1ultra | **181.5 s** | 17.2 GB |
| (old torch-MPS `trellis_mac`) | 124.8 s gen (248.8 wall) | 20.8 GB |
| (fal, same image) | 298.6 s wall | $0.30 |
- **Run:** `cd <fork> && .venv/bin/python scripts/generate_asset.py IMG
--output-dir OUT --backend mlx-experimental --baker vertex
--pipeline-type 1024_cascade --seed N` · Farm: operator `trellis2_mlx`.
- **Knobs:** `TRELLIS2_MLX_SAMPLER=0` (revert to torch sampler) ·
`TRELLIS2_MLX_COMPILE=1` (≈0 on M3U; try on small boxes) ·
`TRELLIS2_METAL_SPCONV=1` (opt-in kernel; slower-but-leaner, reference).
- **Output class:** vertex-colored full-density GLB (albedo only). MR maps
need the UV path (`--baker metal` — has the dark-patch sampling bug) or
the future fast-UV bake. Game budgets: MESHGOD finish farm downstream.
- **Checkouts:** m3 `~/Documents/trellis2-mlx-staging/trellis2-jourloy` ·
m1 `~/trellis2-mlx` · fresh boxes: `scripts/install_trellis2_mlx.sh`.
### 3.2 Hunyuan3D 2.1 — image → textured 3D · `monster/Hunyuan3D-2.2-mrp-MLX`
- **Base:** dgrauet/Hunyuan3D-2.1-mlx (pure MLX, both stages; 13GB
pre-converted MLX weights on HF). *Terminology note: this IS the
v2-generation Hunyuan; fal's `/v2`,`/v21`,`/turbo` are hosted variants.*
- **Our branch `mlx-tune` @ `5a32240`** (deployed both Ultras): T1 UNet
mx.compile + T2 whole-DiT compile (env `HY3D_MLX_COMPILE=1`; parity
3e-5/7e-6) · T3 StaticMoELayer (bit-identical, kills ~4800 host syncs,
makes the DiT compile-legal) · T4 fused-SDPA DINO attention (1.41× its
stage, parity 1e-7).
- **Performance:** 260 s baseline → **243 s** tuned (m3ultra, defaults).
Runs on every box M1-and-up (fp16). ~20GB peak.
- **Run:** `generate_e2e.py IMG --output OUT` in `vendor/hunyuan3d-mlx`
(venv `.venv`) · Farm: operator `hunyuan3d_mlx`.
### 3.3 FLUX / image generation — `mflux` (upstream; not forked)
- mflux 0.18.0 (latest) in `venvs/mflux`, both Ultras + m4pro. Written
from scratch in MLX by Filip Strand — no port debt to carry.
- **Audit (2026-07-19):** our default path (`flux2-klein-4b`, 4 steps,
~9 s/image) has mx.compile on the denoise step + a shapeless-compiled
scheduler — healthy, keep. **Known gap:** the FLUX.1 family
(dev/schnell/krea) has zero mx.compile — a backport is the same 10-line
pattern as Hunyuan T1, expected win 14% on Ultras (more on small
boxes), and belongs UPSTREAM as an mflux PR rather than a site-packages
patch. Queued, not urgent.
- Operator `flux_local` (klein default) + `mflux_image_edit`.
### 3.4 CorridorKey — neural green-screen · `monster/corridorkey-mrp-mlx`
- **Base:** nikopueringer/corridorkey-mlx (cmoyates' phased parity-first
port — its `prompts/` folder is our template for any future from-scratch
port). License CC BY-NC-SA — non-commercial only.
- **Our patch (`4c660df`, branch `modelbeast`):** honor the compile flag in
tiled mode — 8 lines, **1.47×** on m3ultra (3.64→2.48 s/frame), output
bit-identical. Plus the 8-machine ablation that found the head_dim→64
fast-path rule and the "tiled beats full-frame on quality AND memory"
result (2.3GB vs 28GB). Nightly perfcheck watches the fast path.
### 3.5 Qwen-Image-Layered — image → editable RGBA layers · `monster/Qwen-Image-Layered-MRP-MLX`
- **Base:** mflux PR#302 (ZimengXiong; unmerged upstream), pin
`pr302-head-a255e4f`. 20B, Apache 2.0 (ship-commercial OK).
- **Validated ladder:** **q8 is the quality FLOOR** — q6 breaks the layer
semantics (silhouettes), q4 is noise; speed is FLAT across quants
(compute-bound), so never quantize below q8. 20 steps ≈ 50 steps
visually (2.9× faster). 1024-res = hero tier.
- **Performance (4 layers, 640, baked q8, 20 steps):** m3ultra **243 s** ·
m1ultra 366 s · 1024-res hero: 858 s (m3). Peak ~37GB.
- **Baked models:** m3 `~/qwen-layered/qwen-layered-q8` · m1
`~/qwen-layered-staging/qwen-layered-q8` (34GB; rebake via `mflux-save`).
- **Run:** `mflux-generate-qwen-layered --image IMG --layers 4
--resolution 640 --steps 20 --model-path <baked>` · Farm: operator
`qwen_layered_local`. Bonus: the subject layer is matting-grade — the
premium tier of image preprocessing for hero assets.
## 4 · MODELBEAST integration
| operator | backs | speed (m3 / m1) | notes |
|---|---|---|---|
| `trellis2_mlx` | 3.1 | 72 s / 182 s | vertex baker default; **preferred 3D lane** |
| `trellis_mac` | torch-MPS port | 125 s gen / n/a | legacy; UV-textured (dark-patch bug) |
| `hunyuan3d_mlx` | 3.2 | 243 s / ~5-6 min | textured GLB; runs everywhere |
| `flux_local` | 3.3 | ~9 s/img | klein-4b 4-step default |
| `qwen_layered_local` | 3.5 | 243 s / 366 s | RGBA layer decomposition |
Routing lives in `nodes.json` (machine-local, gitignored — edit on the
m3 primary). MESHGOD's `local/` lane calls these via the MB REST API.
## 5 · The lessons ledger (do not re-learn these)
1. **Profile before optimizing — every time.** The rung-3 3× came from a
scheduler bug fix + baker swap the profiler exposed; the glamorous
kernel work measured ≈ 0. The 75GB "sparse conv" memory monster was
actually the Metal texture bake.
2. **mx.compile ≈ 0 on GEMM-bound 2B+ models on M3 Ultra** (measured 4×:
Hunyuan T1, T2, TRELLIS step, sampler). It's free insurance, not a win.
Compile wins live on small models (corridorkey 1.47×) and maybe small
chips.
3. **Quantization floors are model-specific and cliff-shaped.** Qwen-
Layered: q8 fine, q6 semantically broken, q4 noise — and quant does
NOT buy speed on compute-bound models, only footprint.
4. **glTF axis + gamma:** pipeline space is Z-up in [-0.5,0.5]³; glTF is
Y-up (`(x,y,z)→(x,z,y)`); COLOR_0 is LINEAR (never pre-gamma).
A "mangled blob" render is usually just the figure viewed down its own
axis after a convention miss.
5. **xatlas hangs (hours) on écorché-class meshes** (thousands of
disconnected shells). Vertex bake sidesteps UV unwrap entirely.
6. **macOS TCC blocks launchd from ~/Documents** — services live in `~/`.
7. **Gated HF weights don't need logins on workers** — rsync the model
dirs from a cache that has them.
8. **The fal-outage class of bug:** any `while True` polling a paid API
needs a deadline + a cancel; an in-memory queue wedges silently.
9. **Adapters that convert per-call hide O(steps) waste** — convert at
stage boundaries, hold state in the fast runtime.
10. **Sentinel fallthroughs bite:** a baker/scheduler that silently falls
back can burn 5 hours before anyone notices. Fail loudly instead.
## 6 · Runbooks
**New box bring-up (any Apple Silicon, macOS 26+):**
1. Keys: box's SSH key added to Gitea (`monster` account).
2. `git clone` the MODELBEAST repo; run the relevant
`scripts/install_*.sh` (trellis2_mlx, qwen_layered, hunyuan…) — each is
idempotent, uses uv, and states its weight needs.
3. Weights: rsync the HF cache model dirs from m3ultra (see §5.7).
4. Bench: `uv run --with mlx python scripts/mlxprobe.py` → add the row to
BENCHMARKS fleet matrix; set fp16/bf16 per §2 rules.
5. Register the box + operator list in `nodes.json` on the m3 primary.
**Rollback:** every MRP patch is env-gated (§3 knobs) or a pinned-tag fork
`git checkout <pin-tag>` restores the adopted upstream exactly.
**Where results go:** every experiment appends to `BENCHMARKS.md` (wall
time, peak mem via `mx.get_peak_memory()`, parity numbers, and a verdict
line). Session-scale context lives in `docs/FABLE_STARTUP_PACKET.md`.
## 7 · Roadmap (scoped, in value order)
1. **DiT-internals deep-dive** — TRELLIS.2 sampling runs ≈2.3× above ideal
FLOPs (54 s of the 72). Attention/MLP kernel work inside the forward;
the corridorkey `prompts/` phased method is the template.
2. **Fast UV bake** — bucket rasterizer (written, in fast_bake_test.py) +
an écorché-safe unwrap → texture+MR parity with fal on the vertex
baker's quality. Unlocks `--baker vertex`-quality WITH UV maps.
3. **simdgroup spconv kernel** — upgrade the opt-in scalar kernel if the
decoders ever matter for time (they're 10 s today).
4. **Upstreaming** — our sampler + baker + validator to Jourloy PR#175;
FLUX.1 compile backport as an mflux PR; spconv kernel as reference.
5. **MESHGOD default flip** — route its `local/` 3D lane to `trellis2_mlx`
once finish-farm verts→texture baking lands for game budgets.

View File

@ -1,143 +0,0 @@
# Local 3D → full MLX + preprocessing frontend — master build plan
**For:** a fresh long-running Fable session on m3ultra (no execution-time cap).
**Author:** Opus planning pass, 2026-07-19. **Read first:** `docs/TRELLIS2_MLX_RECON.md`,
`BENCHMARKS.md` (fleet MLX matrix + rung 1/2), `HANDOFF_HY3D_MLX.md`, `CORRIDORKEY.md`.
**Prime directive:** parity-first. Every kernel/port change ships with a PT-parity test
(≤1e-4 rel) + a BENCHMARKS.md row (per-stage time + `mx.get_peak_memory()`). No silent
regressions — that discipline is why hunyuan3d-mlx and corridorkey-mlx worked.
**Two goals, independent, run in parallel:**
- **G1 — the models go fully MLX + fast** (TRELLIS.2 port from scratch-ish; Hunyuan tune).
- **G2 — a local preprocessing frontend** that turns any messy photo into the input the
models want. Model-agnostic; helps every generation from here on.
---
## Ground truth (measured, don't re-derive)
- Local TRELLIS.2 = `vendor/trellis-mac` (shivampkumar fork), **torch-MPS, NOT MLX**.
124.8s/gen on m3ultra (1024-cascade, tex2048), 20.8GB peak. Weights `TRELLIS.2-4B` 14GB.
- Stage breakdown of the 124.8s: **~46s diffusion sampling** (padded SDPA attention),
**~50s decode + mesh extract** (incl. `aten::segment_reduce`**CPU fallback**),
**19s Metal PBR bake**. Decoder emits **2.79M faces** pre-simplify.
- Sparse-conv backend already on the Metal fast path (`flex_gemm`); worth **1.72×** vs the
pure-torch fallback (rung 1). Metallib loads on macOS 26.5.
- Hunyuan3D-2.1 = `vendor/hunyuan3d-mlx` (dgrauet), **already MLX**, ~260s m3ultra, runs on M1.
- Quality gap vs fal = dark shattered patches; density is NOT the cause (rung 2: 200K/800K/2.79M
all show it) → **texture bake or decoder attrs**. KDTree-baker A/B pending (`TRELLIS2_FORCE_KDTREE`).
- Fleet MLX matrix (BENCHMARKS.md): bf16 fine except M1 (~20% slower → ship fp16 on M1);
**pad attention head_dim→64** (up to 4.7× M5, 1.5× M3U, ~0 on M1); `mx.compile` on fixed shapes.
- Knobs already added to `vendor/trellis-mac/generate.py`: `TRELLIS2_MAX_BAKE_FACES` (0=uncapped),
`TRELLIS2_FORCE_KDTREE` (A/B the baker). Bench artifacts in `~/Documents/trellis2-bench/`.
---
## G1 · WORKSTREAM A — TRELLIS.2 → MLX
### A0. Scout the existing MLX efforts (do FIRST — decides adopt vs build)
- Vendor `pedronaugusto/trellis2-apple` (has an `mlx_backend/`) and upstream **PR #175**
(MPS+Metal + experimental `mlx` flag, 28 tests green on M4 Max). Fork both to Gitea as
`monster/trellis2-mrp-*` (house pattern; keep upstream remote for rebases).
- Install each, run ONE gen on the anatomy bench image, seed 42, 1024-cascade. Compare to the
124.8s MPS baseline **and** to fal quality (renders already in `trellis2-bench/`).
- **Decision gate:** if trellis2-apple's MLX path is correct + faster → we ADOPT and jump to A3
(patch, not rebuild). If it's partial/slower/buggy → we BUILD (A1→A2), using it + the
corridorkey `prompts/` 6-phase template as reference.
### A1. Scaffold the MLX port (if building)
- New repo `monster/trellis2-mlx`. Mirror the corridorkey-mlx phased layout: `prompts/` phase docs,
`src/`, `tests/test_parity.py`, `scripts/convert_weights.py`.
- Weight conversion: TRELLIS.2-4B safetensors → MLX. Reuse the hunyuan pattern
(`hy3dpaint/utils/convert_realesrgan.py`): torch load → key remap → **transpose Conv (O,I,H,W)→(O,H,W,I)** → save. fp16 (M1-safe); publish converted weights to HF/Gitea so re-runs skip conversion.
- Build a PT reference harness first (dump per-stage activations from the working MPS build as
fixtures) — every MLX module validates against these.
### A2. The kernel rewrites (the real work — this is where the days go)
Priority = by measured cost. Each = an `mx.fast.metal_kernel` + parity test + bench row.
1. **Sparse-conv (FlexGEMM → fused Metal).** THE crux and the ~10× lever. Triton has no Metal
target; implement gather→GEMM→scatter as a fused Metal kernel. Reference: the existing
`mtlgemm` Metal port in trellis-mac/deps and `backends/conv_none.py` (the slow pure-torch
version) for correct semantics. MLX has no sparse tensor type — hand-roll SparseTensor as
coord-list + hash (same as CUDA/MPS side).
2. **Varlen/sparse attention (padded SDPA → fused).** ~46s of the run. Kill the pad-to-maxlen
waste; pad head_dim→64 for the fused fast path (fleet matrix). This + #1 close most of the gap.
3. **`segment_reduce` MPS→CPU fallback → MLX-native.** Currently silently on CPU each step;
pure MLX scatter-add removes a real tax in the decode path.
4. **o-voxel hashmap + dual-grid mesh extract.** Keep the Metal/CPU fork if it's not hot; port to
MLX only if profiling says so.
Leave the PBR bake on the existing Metal path (`mtldiffrast`) unless G1-C convicts it.
### A3. Fleet-tune + the prize
- Apply corridorkey ablation moves across the port: head_dim→64, `mx.compile` fixed shapes,
sdpa gating by GPU generation, tiled-vs-full sweeps. Nightly perfcheck like corridorkey.
- **Prize: M1 Ultra compatibility.** MLX-native runs where torch-MPS/bf16 is shaky → a *second*
free TRELLIS box (297 GB/s, 128GB). Validate a full gen on `johnking@100.91.239.7`.
- Wire `trellis_mac`→MLX into MODELBEAST `nodes.json` (m3ultra primary, m1ultra fp16 second) and
add `trellis2` to MESHGOD's `local/` model list beside `hunyuan3d_mlx`.
## G1 · WORKSTREAM B — Hunyuan3D-2.1-MLX optimization (already MLX; just tune)
- It's a *tune*, not a port. Apply the same corridorkey wins to `vendor/hunyuan3d-mlx`:
pad attention head_dim→64, `mx.compile` on fixed-shape blocks (DiT denoiser, VAE), sdpa gating.
- Check `mlx_arsenal` rasterizer (`mesh_render_mlx.py`, 1137 LOC) for the same fixed-shape
compile opportunity. Bench each on the fleet matrix; expect corridorkey-scale wins (1.11.5×).
- Low risk, high certainty — good "warm-up" task while A0 scouting runs.
## G1 · WORKSTREAM C — quality (rung 2 close-out; feeds BOTH MPS + MLX)
- **KDTree-baker A/B** (`TRELLIS2_FORCE_KDTREE=1`, running now). If its surface is clean → the
**Metal baker (mtldiffrast/mtlbvh texel→voxel sampling) is the dark-patch culprit** → fix or
default-swap the baker. If still dirty → decoder voxel attrs are suspect → investigate upstream.
- Keep `TRELLIS2_MAX_BAKE_FACES` (geometry win, free).
- Add a **hole-fill pass** (CuMesh replacement) pre-bake — Blender fill or a voxel-remesh.
- Loud assertion if `flex_gemm` metallib ever fails to load (silent → +90s/gen forever).
---
## G2 · WORKSTREAM D — image preprocessing frontend ("best chance of success")
North star already written in `meshgod/config.py`: *centered · plain neutral light-gray bg ·
even shadowless studio light · neutral symmetric A-pose*. The job = transform any user photo → that.
Model-agnostic; helps TRELLIS.2 AND Hunyuan equally; no fal/network.
### D1. `prep_local()` — the front line (highest ROI, pure glue)
- RMBG-2.0 (local `rmbg` venv) or BiRefNet matte → **composite on neutral gray (NOT alpha**
config's own spec) → bbox crop → center → square-pad → upscale to ≥1024.
- Wire as a LOCAL replacement for MESHGOD's current `clean_bg` (today it's a fal call). Respect the
learned order: **upscale FIRST, matte LAST** (SeedVR strips alpha).
- ⚠️ **Licensing:** RMBG-2.0 is CC-BY-NC → local/personal only. Ship-commercial path = **BiRefNet
(Apache)**. Expose both; default BiRefNet for anything headed to a game.
### D2. Lighting/exposure normalization
- Auto-levels + gray-world WB before recon. TRELLIS bakes light into albedo → even input = cleaner
texture, and directly fights the dark-patch failure mode. Trivial PIL; measurable.
### D3. Multi-view synthesis — the RIGHT fix for touching limbs
- Single-view can't separate an arm fused to a torso in silhouette. Generate back + ¾ views (image
model) → feed **multi-view reconstruction** (both models support it; MESHGOD already wires the MV
path). The extra angle disambiguates. Prototype: photo → gen back view → MV recon → compare.
### D4. Input QA gate (save wasted gens)
- Pre-flight heuristic/VLM (local Ollama VLM or DINOv3-embedding rules) flags too-small, busy-bg,
cropped-limb, multi-subject BEFORE spending a generation. Cheap insurance.
### D5. T-pose normalization = POST-process via MIRPAMO (NOT a preprocess)
- Reposing a 2D human hallucinates. Correct pipeline: reconstruct in-pose → **MIRPAMO auto-rig +
retarget to T/A-pose**. Design-doc it; tie in MOCAPGOD/HSMR (SMPL fit, on m1) as the body prior.
---
## Suggested execution order for the long session
1. **G1-C** close-out (read KDTree result, act) — unblocks the quality question. *(minutes)*
2. **G1-B** Hunyuan tune — certain win, warms up the parity/bench rhythm. *(hours)*
3. **G2-D1+D2** preprocessing front line — compounding ROI on every future gen. *(hours)*
4. **G1-A0** scout MLX ports — the adopt-vs-build decision gate. *(hour)*
5. **G1-A1→A3** the TRELLIS.2 MLX port + kernels — the multi-day main event. *(days)*
6. **G2-D3/D4/D5** as reach goals.
## Definition of done
- TRELLIS.2 runs MLX on BOTH Ultras, faster than today's 124.8s MPS, parity ≤1e-4, in `nodes.json`.
- Hunyuan tuned with a measured fleet-matrix win.
- Dark-patch quality gap root-caused and closed (or a clear upstream bug filed).
- `prep_local()` live as MESHGOD's default local pre-pass; BiRefNet for commercial, multi-view
path proven on a touching-limbs case.
- Every step has a BENCHMARKS.md row. Upstreamable kernel work offered to PR #175.

View File

@ -1,106 +0,0 @@
# TRELLIS.2 on Apple Silicon — recon (2026-07-18)
**Verdict up front:** we already run TRELLIS.2 locally — `vendor/trellis-mac`
(torch-MPS + Metal kernels) IS TRELLIS.2-4B and is wired as the
`trellis_mac` operator. There is no "MLX interface" shortcut for CUDA code
(CUDA kernels must be *rewritten*, not wrapped), but ~90% of TRELLIS.2 is
standard tensor math that MPS/MLX already runs; only four custom CUDA
pieces matter, and the community has replaced all four. The open work is
**speed** (unfused sparse ops ≈ 10× slower than CUDA) and **quality gaps**
(hole-filling disabled, forced pre-simplification) — not feasibility.
## What we run today (`vendor/trellis-mac`, shivampkumar fork)
- torch-MPS + `PYTORCH_ENABLE_MPS_FALLBACK`, Metal kernels by @pedronaugusto
(mtlgemm / mtldiffrast / mtlbvh / mtlmesh), SDPA attention.
- Full inference: sparse structure → shape SLat → tex SLat → decode →
dual-grid mesh extract → simplify → Metal PBR bake → GLB.
- Our benchmark: **318 s/gen on m3ultra**, ~18 GB peak (M4 Pro 24GB: 5m13s
cold at 512). H100 does 317 s — the gap is almost entirely the unfused
sparse conv + padded attention.
- Known gaps vs fal's trellis-2: CuMesh skipped → **no hole filling**,
meshes pre-simplified ~858K→200K faces before baking, Metal BVH
instability, macOS GPU-watchdog can kill long kernels (detected +
workarounds printed by generate.py). macOS 26 needed for the metallib
(fleet is on 26.5 ✓).
- Licensing: DINOv3 is Meta-gated; **RMBG-2.0 preprocessing is CC BY-NC**.
## What TRELLIS.2 actually needs (upstream: Linux, CUDA 12.4, ≥24 GB)
| CUDA dep | Role | Mac status |
|---|---|---|
| o-voxel ext (hash/convert/rasterize) | the O-Voxel representation | pure-Python reimpl (trellis-mac `backends/`) + CPU fork |
| FlexGEMM (**Triton**) | all sparse conv | **the crux** — Triton ≠ Metal; Metal `mtlgemm` or slow pure-torch gather/scatter |
| flash-attn / xformers | 4B flow transformer attention | SDPA (padded → unfused, big cost) |
| CuMesh | decimate/remesh/**hole-fill**/UV | skipped → `fast_simplification`; hole-fill lost |
| nvdiffrast | texture bake | `mtldiffrast` (Metal) |
| nvdiffrec | preview renders only | not needed |
## The MLX question, answered
- **No shim exists or can exist**: CUDA kernels are NVIDIA-machine code;
"an MLX interface" means rewriting each custom op in MLX/Metal. MLX can
express them (`mx.fast.metal_kernel` JIT-compiles Metal from Python) —
but MLX has **no sparse-tensor type and no sparse-voxel precedent**;
SparseTensor/varlen semantics must be hand-rolled. A TRELLIS.2-MLX would
be a first.
- **Our own playbook (proven 2×) is adopt-then-patch, not from-scratch**:
hunyuan3d-mlx = dgrauet's 8.4k-LOC port + our thin packaging layer;
corridorkey-mlx = cmoyates/Niko's 5.3k-LOC port + our **8-line**
`mx.compile`-in-tiled-mode patch = the 1.47× win. From-scratch dense→MLX
ports of this scale are months of solo work (mflux, mlx-video authors).
- **Starting points already exist**: upstream **PR #175** (Jourloy,
2026-07-17 — MPS + Metal + an *experimental `mlx` backend flag*, 28
tests green on M4 Max) and **pedronaugusto/trellis2-apple** (an
`mlx_backend/` dir, no benchmarks yet).
## Recommended ladder (effort-ordered)
1. **Hours — tune what we have**: benchmark `trellis_mac` 1024_cascade vs
fal trellis-2 on identical inputs (BENCHMARKS.md format); route
MESHGOD's batch/overnight work to the local lane (m3ultra clears 18 GB
~14× over; fal stays for interactive one-offs). $93/mo → mostly $0.
2. **Days — adopt + fleet-patch**: vendor PR #175 / trellis2-apple as
`monster/trellis2-*` Gitea forks (house pattern); run the corridorkey
ablation moves on them: attention head_dim → pad to 64 fast-path,
`mx.compile` on fixed shapes, sdpa gating by GPU generation, tiled-vs-
full sweeps. Prize: **M1 Ultra compatibility** (MLX-native, like
hunyuan3d — today trellis is m3-only in practice) = 2nd free 3D box.
3. **Weeks — the real kernel work** (only if we want fal-class speed):
fused Metal sparse-conv (gather-GEMM-scatter) + varlen attention via
`mx.fast.metal_kernel`; port CuMesh hole-filling. Closes most of the
10× gap; genuinely novel, upstreamable to PR #175.
4. **Quality parity misc**: raise simplification budget on 256 GB boxes,
swap RMBG-2.0 → our licensed bg-remove lane, wire `trellis_mac` into
MESHGOD's `local/` model list next to hunyuan3d-mlx.
## Sources
- github.com/microsoft/TRELLIS.2 (setup.sh = dep manifest) · PR #175 ·
issue #74 · shivampkumar/trellis-mac · pedronaugusto/trellis2-apple
- ml-explore.github.io/mlx custom-metal-kernels docs
- Local: HANDOFF_HY3D_MLX.md, CORRIDORKEY.md, BENCHMARKS.md,
vendor/trellis-mac/README, vendor/corridorkey-mlx/prompts/ (the 6-phase
parity-first port template — the blueprint if we ever do ladder step 3)
## First benchmark (2026-07-19, m3ultra, anatomy écorché test image)
| run | tris | time | peak mem | cost |
|---|---|---|---|---|
| trellis_mac 1024-cascade, tex 2048 | 191,336 | **124.8 s** compute (248.8 s wall, 103 s load) | 20.8 GB | $0 |
| fal trellis-2, same image | 468,049 | 298.6 s wall (queue incl.) | — | $0.30 |
**Local already beats fal on wall-clock.** Quality is the gap, not speed:
the local master shows shattered dark patches — CuMesh hole-fill skipped +
forced 858k→191k pre-simplify before baking (the un-simplified master IS
saved as .obj alongside the GLB). Fix quality first, then fuse kernels.
Tri-budget ladder (Blender decimate on the fal master): 2k = confetti
(ratio floors at 0.01 → 4,674 min), 8k = torn, 30k = good, 50k ≈ master.
**Écorché-class meshes (many disconnected thin shells) cannot pure-decimate
below ~30k** — game budgets need the MESHGOD solidify/remesh route.
Discovered in passing: MESHGOD finish_glb.py's tri budget is silently
ignored on Blender 5 (modifier_apply cancels — fix task spawned 2026-07-19).
Bench artifacts: m3ultra ~/Documents/trellis2-bench/ (masters, sweeps,
renders, logs, sweep2.csv / sweep_fal.csv).

View File

@ -1,25 +0,0 @@
#!/bin/bash
# Install the Qwen-Image-Layered MLX runtime (mflux PR#302 fork, vendored at
# our Gitea) into vendor/mflux-qwen-layered + a uv venv. Idempotent.
#
# Model weights are NOT fetched here. run.py resolves, in order:
# $QWEN_LAYERED_MODEL -> ~/qwen-layered/qwen-layered-q8 (m3)
# -> ~/qwen-layered-staging/qwen-layered-q8 (m1)
# -> HF Qwen/Qwen-Image-Layered with on-the-fly -q8 (54GB download).
# Bake a local q8 once per box: .venv/bin/mflux-save \
# --model Qwen/Qwen-Image-Layered --base-model qwen-image-layered \
# --quantize 8 --path ~/qwen-layered/qwen-layered-q8
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
DEST="$ROOT/vendor/mflux-qwen-layered"
REPO="ssh://git@100.71.119.27:222/monster/Qwen-Image-Layered-MRP-MLX.git"
if [ ! -d "$DEST/.git" ]; then
git clone "$REPO" "$DEST"
else
echo "already cloned: $DEST"
fi
cd "$DEST"
/opt/homebrew/bin/uv venv .venv --python 3.12
/opt/homebrew/bin/uv pip install -q --python .venv/bin/python -e .
.venv/bin/mflux-generate-qwen-layered --help >/dev/null && echo "qwen-layered OK: $DEST"

View File

@ -1,20 +0,0 @@
#!/bin/bash
# Vendor the MRP MLX fork of TRELLIS.2 (monster/trellis-2-mrp-mlx) for the
# trellis2_mlx operator. Idempotent. Existing per-box clones also work
# (run.py resolves m3 staging / m1 ~/trellis2-mlx automatically) — this
# script is for fresh boxes.
# Weights: TRELLIS.2-4B + gated facebook/dinov3 + briaai/RMBG-2.0 — rsync
# from m3ultra's HF cache (no HF login):
# for M in models--facebook--dinov3-vitl16-pretrain-lvd1689m \
# models--microsoft--TRELLIS.2-4B models--briaai--RMBG-2.0; do
# rsync -a m3ultra@100.89.131.57:.cache/huggingface/hub/$M ~/.cache/huggingface/hub/
# done
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
DEST="$ROOT/vendor/trellis2-mlx"
REPO="ssh://git@100.71.119.27:222/monster/trellis-2-mrp-mlx.git"
[ -d "$DEST/.git" ] && echo "already cloned: $DEST" || git clone "$REPO" "$DEST"
cd "$DEST"
PB="${PYTHON_BIN:-$(/opt/homebrew/bin/uv python find 3.11)}"
PYTHON_BIN="$PB" bash scripts/setup_macos.sh
.venv/bin/python -c "import mlx_backend" && echo "trellis2-mlx OK: $DEST"

View File

@ -1,90 +0,0 @@
"""MLX fleet probe — one JSON line of capability + micro-bench per machine.
Benches (each ~seconds, sized for 16GB boxes):
matmul fp16/bf16 4096^2 -> TFLOPS (bf16-on-M1 question)
elementwise add 256MB -> GB/s (memory bandwidth proxy)
sdpa head_dim 56 vs 64 -> ratio (fused fast-path check, corridorkey lesson)
"""
import json
import platform
import subprocess
import time
R = {"host": platform.node().split(".")[0]}
try:
R["chip"] = subprocess.check_output(
["sysctl", "-n", "machdep.cpu.brand_string"], text=True).strip()
R["ram_gb"] = round(int(subprocess.check_output(
["sysctl", "-n", "hw.memsize"], text=True)) / 2**30)
R["macos"] = subprocess.check_output(
["sw_vers", "-productVersion"], text=True).strip()
except Exception:
pass
try:
import mlx.core as mx
R["mlx"] = getattr(mx, "__version__", "?")
except Exception as e:
R["mlx"] = None
R["error"] = f"mlx import failed: {e}"
print(json.dumps(R))
raise SystemExit(0)
try:
di = mx.metal.device_info() if hasattr(mx, "metal") else {}
R["gpu"] = {k: di[k] for k in ("architecture", "max_recommended_working_set_size",
"memory_size") if k in di}
except Exception:
pass
def timeit(fn, warmup=3, iters=10):
for _ in range(warmup):
mx.eval(fn())
t0 = time.perf_counter()
for _ in range(iters):
mx.eval(fn())
return (time.perf_counter() - t0) / iters
N = 4096
flops = 2 * N * N * N
for dt, name in ((mx.float16, "fp16"), (mx.bfloat16, "bf16")):
try:
a = mx.random.normal((N, N)).astype(dt)
b = mx.random.normal((N, N)).astype(dt)
mx.eval(a, b)
t = timeit(lambda: a @ b)
R[f"matmul_{name}_tflops"] = round(flops / t / 1e12, 2)
except Exception as e:
R[f"matmul_{name}_tflops"] = f"fail: {e}"
try:
M = 64 * 1024 * 1024 # 64M floats = 256MB per array
x = mx.random.normal((M,))
y = mx.random.normal((M,))
mx.eval(x, y)
t = timeit(lambda: x + y, warmup=2, iters=8)
R["bandwidth_gbs"] = round(3 * M * 4 / t / 1e9) # 2 reads + 1 write, fp32
except Exception as e:
R["bandwidth_gbs"] = f"fail: {e}"
try:
B, H, L = 1, 8, 2048
sdpa = {}
for D in (56, 64):
q = mx.random.normal((B, H, L, D)).astype(mx.float16)
k = mx.random.normal((B, H, L, D)).astype(mx.float16)
v = mx.random.normal((B, H, L, D)).astype(mx.float16)
mx.eval(q, k, v)
t = timeit(lambda: mx.fast.scaled_dot_product_attention(
q, k, v, scale=D ** -0.5), warmup=3, iters=20)
sdpa[f"d{D}_us"] = round(t * 1e6)
sdpa["d56_vs_d64"] = round(sdpa["d56_us"] / max(sdpa["d64_us"], 1), 2)
R["sdpa"] = sdpa
except Exception as e:
R["sdpa"] = f"fail: {e}"
R["peak_mem_gb"] = round(mx.get_peak_memory() / 2**30, 2) \
if hasattr(mx, "get_peak_memory") else None
print(json.dumps(R))

View File

@ -62,7 +62,6 @@ MIGRATIONS = [
("jobs", "user_id", "TEXT"), # nullable; legacy rows = owner-era ("jobs", "user_id", "TEXT"), # nullable; legacy rows = owner-era
("assets", "user_id", "TEXT"), ("assets", "user_id", "TEXT"),
("users", "session_epoch", "INTEGER NOT NULL DEFAULT 0"), # bump to revoke sessions ("users", "session_epoch", "INTEGER NOT NULL DEFAULT 0"), # bump to revoke sessions
("jobs", "node", "TEXT"), # which pool node ran it; NULL = pre-2026-08-05 rows
] ]

View File

@ -204,20 +204,9 @@ def _guest_may_run(user, op) -> bool:
@app.get("/api/assets") @app.get("/api/assets")
def list_assets(parent_job: str = None, user=Depends(auth.current_user), def list_assets(user=Depends(auth.current_user), request: Request = None):
request: Request = None):
con = _con(request) con = _con(request)
rows = [a for a in store.list_assets(con) if _can_see(user, a)] rows = [a for a in store.list_assets(con) if _can_see(user, a)]
# ?parent_job= used to be accepted and silently ignored, so callers filtering on it
# received the whole table and typically took rows[0] — correct only by accident of
# newest-first ordering, and wrong the moment two jobs run at once.
if parent_job:
def _pj(a):
try:
return a["parent_job"]
except Exception:
return None
rows = [a for a in rows if str(_pj(a)) == str(parent_job)]
return _with_usernames(con, rows) return _with_usernames(con, rows)
@ -369,13 +358,6 @@ def system(user=Depends(auth.current_user), request: Request = None):
return sysinfo.snapshot(_con(request), runner) return sysinfo.snapshot(_con(request), runner)
@app.get("/api/nodes")
def nodes(user=Depends(auth.current_user), request: Request = None):
"""Per-node capability + live work. /api/system only reports name/remote/busy,
and its busy flag is never updated this is the real per-machine view."""
return sysinfo.nodes_detail(_con(request), runner)
# -- inbox watch folder -------------------------------------------------------- # -- inbox watch folder --------------------------------------------------------
async def watch_inbox(): async def watch_inbox():
INBOX.mkdir(parents=True, exist_ok=True) INBOX.mkdir(parents=True, exist_ok=True)

View File

@ -1,22 +0,0 @@
{
"id": "ardy_motion",
"name": "Motion (local, ARDY/MPS)",
"category": "motion",
"description": "Text → character motion as BVH (+ npz) via NVIDIA ARDY on MPS. Streaming-class text-to-motion trained on 630+ hrs of studio mocap — the Mixamo-replacement upgrade over motion_local/MoMask. Mixamo-convention bone names: output BVH retargets straight onto character_kit rigs in Blender. Strong on locomotion/gesture/combat/dance; DJ-specific hand work is out of distribution. First job after a restart pays ~1-2 min model load (Llama-3-8B text encoder); generation itself is faster than real-time.",
"accepts": [],
"produces": ["motion"],
"resources": "gpu",
"entry": "run.py",
"python": "venvs/ardy/bin/python",
"params_schema": {
"type": "object",
"properties": {
"text": {"type": "string", "default": "", "description": "The action: 'a person crouches at a crate, flips through records, stands up holding one'"},
"duration": {"type": "number", "default": 5.0, "minimum": 0.5, "maximum": 30, "description": "Seconds of motion at 20fps"},
"seed": {"type": "integer", "default": -1, "description": "Random seed; -1 = random"},
"num_samples": {"type": "integer", "default": 1, "minimum": 1, "maximum": 4, "description": "Variations to generate"},
"model": {"type": "string", "default": "core", "enum": ["core", "core8"], "description": "core = Horizon40 (best quality), core8 = short-horizon (fastest)"},
"no_postprocess": {"type": "boolean", "default": false, "description": "Skip the foot-skate IK cleanup pass"}
}
}
}

View File

@ -1,89 +0,0 @@
import argparse
import json
import os
import shutil
import subprocess
import sys
import time
import uuid
from pathlib import Path
os.environ["PATH"] = "/opt/homebrew/bin:/usr/local/bin:" + os.environ.get("PATH", "")
ap = argparse.ArgumentParser()
ap.add_argument("--input", action="append", default=[])
ap.add_argument("--outdir", required=True)
ap.add_argument("--params", default="{}")
a = ap.parse_args()
p = json.loads(a.params)
text = (p.get("text") or "").strip()
if not text:
print("ERROR: text is required")
sys.exit(1)
ROOT = Path(__file__).resolve().parents[3]
ARDY = Path.home() / "Documents" / "ardy"
ARDY2BVH = Path.home() / "Documents" / "ardy2bvh" / "ardy2bvh.py"
if not (ARDY / "scripts" / "generate.py").exists():
print("ERROR: ardy repo not found at ~/Documents/ardy")
sys.exit(1)
env = dict(os.environ)
env["LOCAL_CACHE"] = "true"
env["TEXT_ENCODERS_DIR"] = str(ROOT / "data" / "text_encoders")
# The adapters in TEXT_ENCODERS_DIR pin the ungated Llama-3 mirror; never let a
# stale ~/.cache/huggingface/token break those public downloads.
env["HF_HUB_DISABLE_IMPLICIT_TOKEN"] = "1"
env["TOKENIZERS_PARALLELISM"] = "false"
stem = f"mb_{uuid.uuid4().hex[:10]}"
workdir = Path(a.outdir) / "_work"
workdir.mkdir(parents=True, exist_ok=True)
cmd = [
sys.executable, "scripts/generate.py", text,
"--model", p.get("model", "core"),
"--duration", str(float(p.get("duration", 5.0))),
"--num_samples", str(int(p.get("num_samples", 1))),
"--device", "mps",
"--output", str(workdir / stem),
]
seed = int(p.get("seed", -1))
if seed >= 0:
cmd += ["--seed", str(seed)]
if p.get("no_postprocess"):
cmd += ["--no-postprocess"]
t0 = time.time()
r = subprocess.run(cmd, cwd=str(ARDY), capture_output=True, text=True, env=env)
npz = sorted(workdir.rglob(f"{stem}*.npz"))
if r.returncode != 0 or not npz:
print(r.stdout[-1500:])
print(r.stderr[-2000:], file=sys.stderr)
print("ERROR: ardy generation failed")
sys.exit(1)
t_gen = time.time() - t0
outdir = Path(a.outdir)
slug = "".join(c if c.isalnum() else "_" for c in text[:40]).strip("_")
n = 0
for i, f in enumerate(npz):
suffix = f"_{i:02d}" if len(npz) > 1 else ""
npz_out = outdir / f"motion_{slug}{suffix}.npz"
shutil.copy(f, npz_out)
n += 1
rb = subprocess.run(
[sys.executable, str(ARDY2BVH), str(f), "-o", str(outdir / f"motion_{slug}{suffix}.bvh")],
capture_output=True, text=True, env=env,
)
if rb.returncode == 0:
n += 1
else:
print(f"WARN: bvh conversion failed for {f.name}: {rb.stderr[-500:]}", file=sys.stderr)
shutil.rmtree(workdir, ignore_errors=True)
if not any(outdir.glob("*.bvh")):
print("ERROR: no BVH produced")
sys.exit(1)
print(f"done: gen {t_gen:.0f}s, {n} files for '{text[:60]}'", flush=True)

View File

@ -2,128 +2,28 @@
"id": "comfyui_sd", "id": "comfyui_sd",
"name": "Stable Diffusion + LoRA (local, ComfyUI)", "name": "Stable Diffusion + LoRA (local, ComfyUI)",
"category": "generate", "category": "generate",
"description": "Prompt -> image via local SD/SDXL checkpoints + LoRAs (ComfyUI on Metal). Stacks multiple LoRAs and supports ControlNet (recolour a garment into N colourways with an identical silhouette). Cannot load FLUX/Z-Image LoRAs \u2014 those go to flux_local.", "description": "Prompt → image via local SD/SDXL checkpoints + LoRAs (ComfyUI on Metal). Complements flux_local, which runs mflux and CANNOT load SD/SDXL. Talks to a resident ComfyUI on 127.0.0.1:8188 (auto-starts it), so models stay cached between jobs. NOTE: the SD1.5 LoRAs only work with Hyper_Realism_1.2_fp16 — an SDXL checkpoint + SD1.5 LoRA silently does nothing. Full guide: localmodels/README.md",
"accepts": [ "accepts": [],
"image" "produces": ["image"],
],
"produces": [
"image"
],
"resources": "gpu", "resources": "gpu",
"entry": "run.py", "entry": "run.py",
"params_schema": { "params_schema": {
"type": "object", "type": "object",
"properties": { "properties": {
"prompt": { "prompt": {"type": "string", "default": "", "description": "Positive prompt (put LoRA trigger words early)"},
"type": "string", "negative": {"type": "string", "default": "(worst quality, low quality:1.4), blurry, jpeg artifacts, watermark, text, deformed, bad anatomy, extra fingers", "description": "Negative prompt (ignored at cfg 1.0 on Turbo/Lightning)"},
"default": "", "checkpoint": {"type": "string", "default": "Hyper_Realism_1.2_fp16.safetensors", "description": "SD1.5 = Hyper_Realism_1.2_fp16 (the only one LoRAs work with); SDXL = bigLust_v16 / sd_xl_base_1.0"},
"description": "Positive prompt (put LoRA trigger words early)" "lora": {"type": "string", "default": "", "description": "LoRA filename, blank for none (SD1.5 LoRAs need an SD1.5 checkpoint)"},
}, "lora_weight": {"type": "number", "default": 0.8, "minimum": 0, "maximum": 2, "description": "0.5 for breastinclassBetter; 0.65-0.9 vector; 0.5-1.0 GodPussy1"},
"negative": { "clip_skip": {"type": "integer", "enum": [1, 2], "default": 2, "description": "2 for all the local SD1.5 LoRAs (their training config)"},
"type": "string", "width": {"type": "integer", "default": 512, "description": "SD1.5: 512 (768 max). SDXL: 1024. Turbo: 512"},
"default": "(worst quality, low quality:1.4), blurry, jpeg artifacts, watermark, text, deformed, bad anatomy, extra fingers", "height": {"type": "integer", "default": 512, "description": "SD1.5: 512/768. SDXL: 1024"},
"description": "Negative prompt (ignored at cfg 1.0 on Turbo/Lightning)" "steps": {"type": "integer", "default": 25, "description": "SD1.5/SDXL: 25-30. Turbo: 1-4. Lightning: 8"},
}, "cfg": {"type": "number", "default": 7.0, "description": "SD1.5/SDXL: 5-8. Turbo: 1.0. Lightning: 1-2 (wrong cfg = fried image)"},
"checkpoint": { "sampler": {"type": "string", "default": "dpmpp_2m", "description": "dpmpp_2m (+karras) is the workhorse; euler for Turbo/Lightning"},
"type": "string", "scheduler": {"type": "string", "default": "karras", "description": "karras normally; sgm_uniform for Lightning"},
"default": "Hyper_Realism_1.2_fp16.safetensors", "seed": {"type": "integer", "default": -1, "description": "-1 = random. Fix it when tuning LoRA weight."},
"description": "SD1.5 = Hyper_Realism_1.2_fp16 (the only one LoRAs work with); SDXL = bigLust_v16 / sd_xl_base_1.0" "batch": {"type": "integer", "default": 1, "minimum": 1, "maximum": 16, "description": "Images per job — brute-force seeds and pick"}
},
"lora": {
"type": [
"string",
"array"
],
"default": "",
"description": "LoRA filename, or a LIST to stack them (style + texture together). Each entry may carry its own weight as 'name=0.6'. A LoRA on the wrong base architecture is a SILENT no-op \u2014 filenames lie, check metadata."
},
"lora_weight": {
"type": [
"number",
"array"
],
"default": 0.8,
"minimum": 0,
"maximum": 2,
"description": "Weight, or a list matching `lora` order. Per-entry 'name=0.6' wins."
},
"clip_skip": {
"type": "integer",
"enum": [
1,
2
],
"default": 2,
"description": "2 for all the local SD1.5 LoRAs (their training config)"
},
"width": {
"type": "integer",
"default": 512,
"description": "SD1.5: 512 (768 max). SDXL: 1024. Turbo: 512"
},
"height": {
"type": "integer",
"default": 512,
"description": "SD1.5: 512/768. SDXL: 1024"
},
"steps": {
"type": "integer",
"default": 25,
"description": "SD1.5/SDXL: 25-30. Turbo: 1-4. Lightning: 8"
},
"cfg": {
"type": "number",
"default": 7.0,
"description": "SD1.5/SDXL: 5-8. Turbo: 1.0. Lightning: 1-2 (wrong cfg = fried image)"
},
"sampler": {
"type": "string",
"default": "dpmpp_2m",
"description": "dpmpp_2m (+karras) is the workhorse; euler for Turbo/Lightning"
},
"scheduler": {
"type": "string",
"default": "karras",
"description": "karras normally; sgm_uniform for Lightning"
},
"seed": {
"type": "integer",
"default": -1,
"description": "-1 = random. Fix it when tuning LoRA weight."
},
"batch": {
"type": "integer",
"default": 1,
"minimum": 1,
"maximum": 16,
"description": "Images per job \u2014 brute-force seeds and pick"
},
"controlnet": {
"type": "string",
"default": "",
"description": "ControlNet model filename. Needs a control image passed as the job's input asset. Recolor holds the silhouette fixed while changing colour \u2014 N colourways of one garment sharing a byte-identical cut-out alpha."
},
"controlnet_strength": {
"type": "number",
"default": 1.0,
"minimum": 0,
"maximum": 2,
"description": "ControlNet conditioning strength"
},
"controlnet_start": {
"type": "number",
"default": 0.0,
"minimum": 0,
"maximum": 1,
"description": "Fraction of sampling at which conditioning starts"
},
"controlnet_end": {
"type": "number",
"default": 1.0,
"minimum": 0,
"maximum": 1,
"description": "Fraction of sampling at which conditioning ends"
}
} }
} }
} }

View File

@ -1,29 +0,0 @@
{
"id": "comfyui_sd",
"name": "Stable Diffusion + LoRA (local, ComfyUI)",
"category": "generate",
"description": "Prompt → image via local SD/SDXL checkpoints + LoRAs (ComfyUI on Metal). Complements flux_local, which runs mflux and CANNOT load SD/SDXL. Talks to a resident ComfyUI on 127.0.0.1:8188 (auto-starts it), so models stay cached between jobs. NOTE: the SD1.5 LoRAs only work with Hyper_Realism_1.2_fp16 — an SDXL checkpoint + SD1.5 LoRA silently does nothing. Full guide: localmodels/README.md",
"accepts": [],
"produces": ["image"],
"resources": "gpu",
"entry": "run.py",
"params_schema": {
"type": "object",
"properties": {
"prompt": {"type": "string", "default": "", "description": "Positive prompt (put LoRA trigger words early)"},
"negative": {"type": "string", "default": "(worst quality, low quality:1.4), blurry, jpeg artifacts, watermark, text, deformed, bad anatomy, extra fingers", "description": "Negative prompt (ignored at cfg 1.0 on Turbo/Lightning)"},
"checkpoint": {"type": "string", "default": "Hyper_Realism_1.2_fp16.safetensors", "description": "SD1.5 = Hyper_Realism_1.2_fp16 (the only one LoRAs work with); SDXL = bigLust_v16 / sd_xl_base_1.0"},
"lora": {"type": "string", "default": "", "description": "LoRA filename, blank for none (SD1.5 LoRAs need an SD1.5 checkpoint)"},
"lora_weight": {"type": "number", "default": 0.8, "minimum": 0, "maximum": 2, "description": "0.5 for breastinclassBetter; 0.65-0.9 vector; 0.5-1.0 GodPussy1"},
"clip_skip": {"type": "integer", "enum": [1, 2], "default": 2, "description": "2 for all the local SD1.5 LoRAs (their training config)"},
"width": {"type": "integer", "default": 512, "description": "SD1.5: 512 (768 max). SDXL: 1024. Turbo: 512"},
"height": {"type": "integer", "default": 512, "description": "SD1.5: 512/768. SDXL: 1024"},
"steps": {"type": "integer", "default": 25, "description": "SD1.5/SDXL: 25-30. Turbo: 1-4. Lightning: 8"},
"cfg": {"type": "number", "default": 7.0, "description": "SD1.5/SDXL: 5-8. Turbo: 1.0. Lightning: 1-2 (wrong cfg = fried image)"},
"sampler": {"type": "string", "default": "dpmpp_2m", "description": "dpmpp_2m (+karras) is the workhorse; euler for Turbo/Lightning"},
"scheduler": {"type": "string", "default": "karras", "description": "karras normally; sgm_uniform for Lightning"},
"seed": {"type": "integer", "default": -1, "description": "-1 = random. Fix it when tuning LoRA weight."},
"batch": {"type": "integer", "default": 1, "minimum": 1, "maximum": 16, "description": "Images per job — brute-force seeds and pick"}
}
}
}

View File

@ -4,18 +4,11 @@ Pure stdlib — no venv needed (the runner falls back to the node's python3), be
all the heavy lifting happens inside ComfyUI's own venv. Keeps ComfyUI resident so all the heavy lifting happens inside ComfyUI's own venv. Keeps ComfyUI resident so
checkpoints stay cached in RAM between jobs (a cold load costs seconds; a warm one checkpoints stay cached in RAM between jobs (a cold load costs seconds; a warm one
doesn't). doesn't).
`lora` takes one name or a LIST, each optionally `name=weight`, and they are chained
style + texture together, which a single loader could never do. `controlnet` +
`control_image` add structural conditioning: the headline use is recolouring one garment
into N colourways that share a byte-identical silhouette, so a single cut-out alpha is
reusable across every variant.
""" """
import argparse import argparse
import json import json
import os import os
import random import random
import shutil
import subprocess import subprocess
import sys import sys
import time import time
@ -44,46 +37,6 @@ if not (COMFY / "main.py").exists():
sys.exit(1) sys.exit(1)
def parse_loras():
"""-> [(name, weight)]. Accepts 'a', 'a=0.6', ['a','b=0.4'], and a list of weights.
A LoRA on the wrong base is a SILENT no-op no error, the image just comes back
unchanged and you blame the LoRA. So everything here fails loudly instead.
"""
raw = p.get("lora") or p.get("loras") or []
if isinstance(raw, str):
raw = [s for s in (x.strip() for x in raw.split(",")) if s]
elif not isinstance(raw, list):
raw = []
w_raw = p.get("lora_weight", 0.8)
weights = w_raw if isinstance(w_raw, list) else [w_raw] * len(raw)
out = []
for i, item in enumerate(raw):
item = str(item).strip()
if not item:
continue
if "=" in item:
name, _, w = item.rpartition("=")
try:
out.append((name.strip(), float(w)))
continue
except ValueError:
pass # '=' was part of the filename, not a weight
try:
w = float(weights[i]) if i < len(weights) else 0.8
except (TypeError, ValueError):
w = 0.8
out.append((item, w))
return out
LORAS = parse_loras()
CKPT = p.get("checkpoint", "Hyper_Realism_1.2_fp16.safetensors")
CTRL = (p.get("controlnet") or "").strip()
CTRL_STRENGTH = float(p.get("controlnet_strength", 1.0))
ctrl_file = None # basename inside ComfyUI's input dir
def alive(): def alive():
try: try:
urllib.request.urlopen(f"{API}/", timeout=3) urllib.request.urlopen(f"{API}/", timeout=3)
@ -98,12 +51,8 @@ def ensure_server():
return return
print("comfyui: starting resident server ...", flush=True) print("comfyui: starting resident server ...", flush=True)
log = open("/tmp/comfyui.log", "ab") log = open("/tmp/comfyui.log", "ab")
# --listen 0.0.0.0: tailnet-only box; lets John drive the same resident
# instance interactively at http://100.89.131.57:8188 (FLUX park lives in
# models/ now) while farm jobs keep using it via localhost.
subprocess.Popen( subprocess.Popen(
[str(COMFY / ".venv" / "bin" / "python"), "main.py", "--port", "8188", [str(COMFY / ".venv" / "bin" / "python"), "main.py", "--port", "8188"],
"--listen", "0.0.0.0"],
cwd=str(COMFY), stdout=log, stderr=log, start_new_session=True) cwd=str(COMFY), stdout=log, stderr=log, start_new_session=True)
for _ in range(90): for _ in range(90):
if alive(): if alive():
@ -114,18 +63,11 @@ def ensure_server():
sys.exit(1) sys.exit(1)
def node_options(cls, field):
"""What this node actually has installed — used to fail fast with a useful list."""
try:
info = json.load(urllib.request.urlopen(f"{API}/object_info/{cls}"))
return info[cls]["input"]["required"][field][0]
except Exception:
return None
def build(seed): def build(seed):
ckpt = p.get("checkpoint", "Hyper_Realism_1.2_fp16.safetensors")
lora = (p.get("lora") or "").strip()
g = { g = {
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": CKPT}}, "1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": ckpt}},
"5": {"class_type": "EmptyLatentImage", "inputs": { "5": {"class_type": "EmptyLatentImage", "inputs": {
"width": int(p.get("width", 512)), "height": int(p.get("height", 512)), "width": int(p.get("width", 512)), "height": int(p.get("height", 512)),
"batch_size": int(p.get("batch", 1))}}, "batch_size": int(p.get("batch", 1))}},
@ -133,34 +75,23 @@ def build(seed):
"8": {"class_type": "SaveImage", "inputs": {"filename_prefix": "mb_sd", "images": ["7", 0]}}, "8": {"class_type": "SaveImage", "inputs": {"filename_prefix": "mb_sd", "images": ["7", 0]}},
} }
msrc, csrc = ["1", 0], ["1", 1] msrc, csrc = ["1", 0], ["1", 1]
# Chain one LoraLoader per entry so stacking works (style + texture together). if lora:
nid = 100 g["2"] = {"class_type": "LoraLoader", "inputs": {
for name, w in LORAS: "lora_name": lora,
g[str(nid)] = {"class_type": "LoraLoader", "inputs": { "strength_model": float(p.get("lora_weight", 0.8)),
"lora_name": name, "strength_model": w, "strength_clip": w, "strength_clip": float(p.get("lora_weight", 0.8)),
"model": msrc, "clip": csrc}} "model": msrc, "clip": csrc}}
msrc, csrc = [str(nid), 0], [str(nid), 1] msrc, csrc = ["2", 0], ["2", 1]
nid += 1
# clip_skip 2 == CLIPSetLastLayer -2 (what the local SD1.5 LoRAs were trained at) # clip_skip 2 == CLIPSetLastLayer -2 (what the local SD1.5 LoRAs were trained at)
if int(p.get("clip_skip", 2)) == 2: if int(p.get("clip_skip", 2)) == 2:
g["9"] = {"class_type": "CLIPSetLastLayer", "inputs": {"clip": csrc, "stop_at_clip_layer": -2}} g["9"] = {"class_type": "CLIPSetLastLayer", "inputs": {"clip": csrc, "stop_at_clip_layer": -2}}
csrc = ["9", 0] csrc = ["9", 0]
g["3"] = {"class_type": "CLIPTextEncode", "inputs": {"text": prompt, "clip": csrc}} g["3"] = {"class_type": "CLIPTextEncode", "inputs": {"text": prompt, "clip": csrc}}
g["4"] = {"class_type": "CLIPTextEncode", "inputs": {"text": p.get("negative", ""), "clip": csrc}} g["4"] = {"class_type": "CLIPTextEncode", "inputs": {"text": p.get("negative", ""), "clip": csrc}}
pos, neg = ["3", 0], ["4", 0]
if CTRL and ctrl_file:
g["20"] = {"class_type": "LoadImage", "inputs": {"image": ctrl_file}}
g["21"] = {"class_type": "ControlNetLoader", "inputs": {"control_net_name": CTRL}}
g["22"] = {"class_type": "ControlNetApplyAdvanced", "inputs": {
"positive": pos, "negative": neg, "control_net": ["21", 0], "image": ["20", 0],
"strength": CTRL_STRENGTH,
"start_percent": float(p.get("controlnet_start", 0.0)),
"end_percent": float(p.get("controlnet_end", 1.0))}}
pos, neg = ["22", 0], ["22", 1]
g["6"] = {"class_type": "KSampler", "inputs": { g["6"] = {"class_type": "KSampler", "inputs": {
"seed": seed, "steps": int(p.get("steps", 25)), "cfg": float(p.get("cfg", 7.0)), "seed": seed, "steps": int(p.get("steps", 25)), "cfg": float(p.get("cfg", 7.0)),
"sampler_name": p.get("sampler", "dpmpp_2m"), "scheduler": p.get("scheduler", "karras"), "sampler_name": p.get("sampler", "dpmpp_2m"), "scheduler": p.get("scheduler", "karras"),
"denoise": 1.0, "model": msrc, "positive": pos, "negative": neg, "denoise": 1.0, "model": msrc, "positive": ["3", 0], "negative": ["4", 0],
"latent_image": ["5", 0]}} "latent_image": ["5", 0]}}
return g return g
@ -169,62 +100,37 @@ ensure_server()
seed = int(p.get("seed", -1)) seed = int(p.get("seed", -1))
if seed < 0: if seed < 0:
seed = random.randint(0, 2**31 - 1) seed = random.randint(0, 2**31 - 1)
lora_desc = ", ".join(f"{n}@{w}" for n, w in LORAS) or "none" CKPT = p.get("checkpoint", "Hyper_Realism_1.2_fp16.safetensors")
print(f"seed={seed} ckpt={CKPT} lora={lora_desc}" LORA = (p.get("lora") or "").strip()
+ (f" controlnet={CTRL}@{CTRL_STRENGTH}" if CTRL else ""), flush=True) print(f"seed={seed} ckpt={CKPT} "
f"lora={LORA or 'none'}" + (f"@{p.get('lora_weight', 0.8)}" if LORA else ""), flush=True)
if LORA and ("xl" in CKPT.lower() or "biglust" in CKPT.lower() or "v2-1" in CKPT.lower()):
print(f"WARNING: {LORA} is SD1.5 but {CKPT} is not — the LoRA will silently do nothing. "
f"Use Hyper_Realism_1.2_fp16.safetensors (see localmodels/README.md)", flush=True)
# A ControlNet needs its conditioning image inside ComfyUI's own input dir — LoadImage # Checkpoints are NOT identical across nodes (the M3 keeps only the LoRA-compatible
# resolves by basename, not path. # SD1.5 model; the M1 holds the full SDXL archive). Ask this node's ComfyUI what it
if CTRL: # actually has and fail fast with a useful message rather than a cryptic 400.
if not a.input: try:
print("ERROR: controlnet requested but no input image was supplied " info = json.load(urllib.request.urlopen(f"{API}/object_info/CheckpointLoaderSimple"))
"(pass the control image as the job's asset)", flush=True) have = info["CheckpointLoaderSimple"]["input"]["required"]["ckpt_name"][0]
sys.exit(1) if CKPT not in have:
src = Path(a.input[0])
if not src.is_file():
print(f"ERROR: control image not found: {src}", flush=True)
sys.exit(1)
indir = COMFY / "input"
indir.mkdir(parents=True, exist_ok=True)
ctrl_file = f"mb_ctrl_{seed}{src.suffix or '.png'}"
shutil.copyfile(src, indir / ctrl_file)
print(f"controlnet: staged {src.name} -> input/{ctrl_file}", flush=True)
# Checkpoints, LoRAs and ControlNets are NOT identical across nodes. Ask this node's
# ComfyUI what it actually has and fail fast with a useful message rather than a
# cryptic 400 — or worse, a silent no-op.
have = node_options("CheckpointLoaderSimple", "ckpt_name")
if have is not None and CKPT not in have:
print(f"ERROR: '{CKPT}' is not on this node.\n" print(f"ERROR: '{CKPT}' is not on this node.\n"
f" available here: {', '.join(have) or '(none)'}\n" f" available here: {', '.join(have) or '(none)'}\n"
f" Fix: use a checkpoint listed above, or rsync it to " f" The full SDXL set lives on the m1 node; every node has "
f"~/Documents/localmodels/Stable-diffusion/ on this node.", flush=True) f"Hyper_Realism_1.2_fp16.safetensors (the only LoRA-compatible checkpoint).\n"
f" Fix: use a checkpoint listed above, or rsync it from m1.", flush=True)
sys.exit(1) sys.exit(1)
if LORAS: if LORA:
have_l = node_options("LoraLoader", "lora_name") have_l = json.load(urllib.request.urlopen(f"{API}/object_info/LoraLoader"))
if have_l is not None: have_l = have_l["LoraLoader"]["input"]["required"]["lora_name"][0]
missing = [n for n, _ in LORAS if n not in have_l] if LORA not in have_l:
if missing: print(f"ERROR: LoRA '{LORA}' is not on this node. available: {', '.join(have_l) or '(none)'}", flush=True)
print(f"ERROR: LoRA(s) not on this node: {', '.join(missing)}\n"
f" available here: {', '.join(have_l) or '(none)'}", flush=True)
sys.exit(1) sys.exit(1)
if CTRL: except SystemExit:
have_c = node_options("ControlNetLoader", "control_net_name") raise
if have_c is not None and CTRL not in have_c: except Exception as e:
print(f"ERROR: ControlNet '{CTRL}' is not on this node.\n" print(f"(could not pre-check model availability: {e}) — continuing", flush=True)
f" available here: {', '.join(have_c) or '(none)'}", flush=True)
sys.exit(1)
# Architecture mismatch is the classic silent failure: an SD1.5 LoRA on an SDXL base
# loads without complaint and does nothing at all.
xl_ckpt = any(t in CKPT.lower() for t in ("xl", "juggernaut"))
for n, _ in LORAS:
xl_lora = "xl" in n.lower()
if xl_ckpt != xl_lora:
print(f"WARNING: '{n}' looks {'SDXL' if xl_lora else 'SD1.5'} but the checkpoint "
f"'{CKPT}' looks {'SDXL' if xl_ckpt else 'SD1.5'} — a wrong-base LoRA is a "
f"SILENT no-op (no error, no effect). Filenames lie; check the metadata.",
flush=True)
body = json.dumps({"prompt": build(seed)}).encode() body = json.dumps({"prompt": build(seed)}).encode()
req = urllib.request.Request(f"{API}/prompt", data=body, headers={"Content-Type": "application/json"}) req = urllib.request.Request(f"{API}/prompt", data=body, headers={"Content-Type": "application/json"})
@ -263,15 +169,8 @@ for i, im in enumerate(images):
dest = outdir / name dest = outdir / name
dest.write_bytes(data) dest.write_bytes(data)
outputs.append({"path": str(dest), "name": name, outputs.append({"path": str(dest), "name": name,
"meta": {"tool": "comfyui_sd", "seed": seed, "checkpoint": CKPT, "meta": {"tool": "comfyui_sd", "seed": seed,
"lora": lora_desc if LORAS else None, "checkpoint": p.get("checkpoint"), "lora": p.get("lora") or None}})
"controlnet": CTRL or None}})
if ctrl_file:
try:
(COMFY / "input" / ctrl_file).unlink()
except OSError:
pass
(outdir / "result.json").write_text(json.dumps({"outputs": outputs})) (outdir / "result.json").write_text(json.dumps({"outputs": outputs}))
print(f"done: {len(outputs)} image(s) in {time.time() - t0:.1f}s -> {outputs[0]['path']}", flush=True) print(f"done: {len(outputs)} image(s) in {time.time() - t0:.1f}s -> {outputs[0]['path']}", flush=True)

View File

@ -1,176 +0,0 @@
"""SD/SDXL + LoRA generation via a resident ComfyUI (Metal).
Pure stdlib — no venv needed (the runner falls back to the node's python3), because
all the heavy lifting happens inside ComfyUI's own venv. Keeps ComfyUI resident so
checkpoints stay cached in RAM between jobs (a cold load costs seconds; a warm one
doesn't).
"""
import argparse
import json
import os
import random
import subprocess
import sys
import time
import urllib.parse
import urllib.request
from pathlib import Path
ROOT = Path(__file__).resolve().parents[3]
COMFY = ROOT / "vendor" / "comfyui"
API = "http://127.0.0.1:8188"
ap = argparse.ArgumentParser()
ap.add_argument("--input", action="append", default=[])
ap.add_argument("--outdir", required=True)
ap.add_argument("--params", default="{}")
a = ap.parse_args()
p = json.loads(a.params)
outdir = Path(a.outdir)
prompt = (p.get("prompt") or "").strip()
if not prompt:
print("ERROR: prompt is required")
sys.exit(1)
if not (COMFY / "main.py").exists():
print(f"ERROR: ComfyUI not installed at {COMFY}. Run scripts/install_comfyui.sh")
sys.exit(1)
def alive():
try:
urllib.request.urlopen(f"{API}/", timeout=3)
return True
except Exception:
return False
def ensure_server():
if alive():
print("comfyui: already resident (models stay cached)", flush=True)
return
print("comfyui: starting resident server ...", flush=True)
log = open("/tmp/comfyui.log", "ab")
subprocess.Popen(
[str(COMFY / ".venv" / "bin" / "python"), "main.py", "--port", "8188"],
cwd=str(COMFY), stdout=log, stderr=log, start_new_session=True)
for _ in range(90):
if alive():
print("comfyui: up", flush=True)
return
time.sleep(2)
print("ERROR: ComfyUI did not come up — see /tmp/comfyui.log")
sys.exit(1)
def build(seed):
ckpt = p.get("checkpoint", "Hyper_Realism_1.2_fp16.safetensors")
lora = (p.get("lora") or "").strip()
g = {
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": ckpt}},
"5": {"class_type": "EmptyLatentImage", "inputs": {
"width": int(p.get("width", 512)), "height": int(p.get("height", 512)),
"batch_size": int(p.get("batch", 1))}},
"7": {"class_type": "VAEDecode", "inputs": {"samples": ["6", 0], "vae": ["1", 2]}},
"8": {"class_type": "SaveImage", "inputs": {"filename_prefix": "mb_sd", "images": ["7", 0]}},
}
msrc, csrc = ["1", 0], ["1", 1]
if lora:
g["2"] = {"class_type": "LoraLoader", "inputs": {
"lora_name": lora,
"strength_model": float(p.get("lora_weight", 0.8)),
"strength_clip": float(p.get("lora_weight", 0.8)),
"model": msrc, "clip": csrc}}
msrc, csrc = ["2", 0], ["2", 1]
# clip_skip 2 == CLIPSetLastLayer -2 (what the local SD1.5 LoRAs were trained at)
if int(p.get("clip_skip", 2)) == 2:
g["9"] = {"class_type": "CLIPSetLastLayer", "inputs": {"clip": csrc, "stop_at_clip_layer": -2}}
csrc = ["9", 0]
g["3"] = {"class_type": "CLIPTextEncode", "inputs": {"text": prompt, "clip": csrc}}
g["4"] = {"class_type": "CLIPTextEncode", "inputs": {"text": p.get("negative", ""), "clip": csrc}}
g["6"] = {"class_type": "KSampler", "inputs": {
"seed": seed, "steps": int(p.get("steps", 25)), "cfg": float(p.get("cfg", 7.0)),
"sampler_name": p.get("sampler", "dpmpp_2m"), "scheduler": p.get("scheduler", "karras"),
"denoise": 1.0, "model": msrc, "positive": ["3", 0], "negative": ["4", 0],
"latent_image": ["5", 0]}}
return g
ensure_server()
seed = int(p.get("seed", -1))
if seed < 0:
seed = random.randint(0, 2**31 - 1)
CKPT = p.get("checkpoint", "Hyper_Realism_1.2_fp16.safetensors")
LORA = (p.get("lora") or "").strip()
print(f"seed={seed} ckpt={CKPT} "
f"lora={LORA or 'none'}" + (f"@{p.get('lora_weight', 0.8)}" if LORA else ""), flush=True)
if LORA and ("xl" in CKPT.lower() or "biglust" in CKPT.lower() or "v2-1" in CKPT.lower()):
print(f"WARNING: {LORA} is SD1.5 but {CKPT} is not — the LoRA will silently do nothing. "
f"Use Hyper_Realism_1.2_fp16.safetensors (see localmodels/README.md)", flush=True)
# Checkpoints are NOT identical across nodes (the M3 keeps only the LoRA-compatible
# SD1.5 model; the M1 holds the full SDXL archive). Ask this node's ComfyUI what it
# actually has and fail fast with a useful message rather than a cryptic 400.
try:
info = json.load(urllib.request.urlopen(f"{API}/object_info/CheckpointLoaderSimple"))
have = info["CheckpointLoaderSimple"]["input"]["required"]["ckpt_name"][0]
if CKPT not in have:
print(f"ERROR: '{CKPT}' is not on this node.\n"
f" available here: {', '.join(have) or '(none)'}\n"
f" The full SDXL set lives on the m1 node; every node has "
f"Hyper_Realism_1.2_fp16.safetensors (the only LoRA-compatible checkpoint).\n"
f" Fix: use a checkpoint listed above, or rsync it from m1.", flush=True)
sys.exit(1)
if LORA:
have_l = json.load(urllib.request.urlopen(f"{API}/object_info/LoraLoader"))
have_l = have_l["LoraLoader"]["input"]["required"]["lora_name"][0]
if LORA not in have_l:
print(f"ERROR: LoRA '{LORA}' is not on this node. available: {', '.join(have_l) or '(none)'}", flush=True)
sys.exit(1)
except SystemExit:
raise
except Exception as e:
print(f"(could not pre-check model availability: {e}) — continuing", flush=True)
body = json.dumps({"prompt": build(seed)}).encode()
req = urllib.request.Request(f"{API}/prompt", data=body, headers={"Content-Type": "application/json"})
try:
pid = json.load(urllib.request.urlopen(req))["prompt_id"]
except Exception as e:
print(f"ERROR: ComfyUI rejected the workflow: {e}")
sys.exit(1)
t0 = time.time()
images = []
while time.time() - t0 < 900:
h = json.load(urllib.request.urlopen(f"{API}/history/{pid}"))
if pid in h:
st = h[pid].get("status", {})
if st.get("status_str") == "error":
print("ERROR: generation failed — check /tmp/comfyui.log")
print(json.dumps(st)[:400])
sys.exit(1)
if h[pid].get("outputs"):
for node in h[pid]["outputs"].values():
images += node.get("images", [])
break
time.sleep(2)
if not images:
print("ERROR: no image produced (timeout)")
sys.exit(1)
outputs = []
for i, im in enumerate(images):
q = urllib.parse.urlencode({"filename": im["filename"], "subfolder": im.get("subfolder", ""),
"type": im.get("type", "output")})
data = urllib.request.urlopen(f"{API}/view?{q}").read()
name = f"sd_{seed}_{i}.png" if len(images) > 1 else f"sd_{seed}.png"
dest = outdir / name
dest.write_bytes(data)
outputs.append({"path": str(dest), "name": name,
"meta": {"tool": "comfyui_sd", "seed": seed,
"checkpoint": p.get("checkpoint"), "lora": p.get("lora") or None}})
(outdir / "result.json").write_text(json.dumps({"outputs": outputs}))
print(f"done: {len(outputs)} image(s) in {time.time() - t0:.1f}s -> {outputs[0]['path']}", flush=True)

View File

@ -0,0 +1,23 @@
{
"id": "corridorkey_local",
"name": "CorridorKey Green-Screen Unmix (local)",
"category": "video-prep",
"description": "Green/blue-screen clip or frame → true un-multiplied foreground color + linear alpha via Corridor's neural keyer (the corridorkey-mrp-mlx fork, MLX on Apple silicon). Preserves hair, motion blur and translucency — no binary roto masks. Input: a video shot on green/blue, or a single frame — NOT square (GVM's resize rejects smaller-edge ≥1024-after-scale square plates; 16:9/9:16 is fine). Output: zipped frame sequence (straight color + alpha) plus a comp preview when enabled. Free, fully local.",
"accepts": ["video", "image"],
"produces": ["archive"],
"resources": "gpu",
"entry": "run.py",
"python": "vendor/corridorkey/.venv/bin/python",
"params_schema": {
"type": "object",
"properties": {
"screen_color": {"type": "string", "enum": ["auto", "green", "blue"], "default": "auto", "description": "Screen color (blue = torch backend only for now)"},
"backend": {"type": "string", "enum": ["auto", "mlx", "torch"], "default": "auto", "description": "Inference backend"},
"despill": {"type": "integer", "minimum": 0, "maximum": 10, "default": 5, "description": "Despill strength"},
"refiner": {"type": "number", "default": 1.0, "description": "Refiner strength multiplier"},
"comp": {"type": "boolean", "default": true, "description": "Also render a comp preview"},
"max_frames": {"type": "integer", "description": "Limit frames (quick tests)"},
"image_size": {"type": "integer", "description": "Inference size override (default: model native)"}
}
}
}

View File

@ -0,0 +1,115 @@
"""corridorkey_local — Corridor's neural green-screen unmixer, headless.
Shells out to the vendored CLI (vendor/corridorkey, the corridorkey-mrp-mlx fork):
stage the input as a clip generate-alphas (GVM coarse hint) run-inference with
every flag set (non-interactive) zip Output/<clip> as the result.
Clip staging uses the job's outdir basename as the clip name, so concurrent jobs
can't collide; the shared ClipsForInference/Output dirs are cleaned afterwards
(win or lose) so the vendor tree doesn't accumulate gigabytes of frames.
"""
import argparse
import json
import os
import shutil
import subprocess
import sys
import zipfile
from pathlib import Path
ap = argparse.ArgumentParser()
ap.add_argument("--input", action="append", default=[])
ap.add_argument("--outdir", required=True)
ap.add_argument("--params", default="{}")
a = ap.parse_args()
p = json.loads(a.params)
if not a.input:
print("ERROR: no input clip/frame")
sys.exit(1)
VENDOR = Path(__file__).resolve().parents[3] / "vendor" / "corridorkey"
CLI = VENDOR / ".venv" / "bin" / "corridorkey"
if not CLI.exists():
print(f"ERROR: {CLI} missing — run the corridorkey install script on this node")
sys.exit(1)
src = Path(a.input[0])
outdir = Path(a.outdir)
clip_name = "ck_" + outdir.name # unique per job
clip_dir = VENDOR / "ClipsForInference" / clip_name
out_clip = VENDOR / "Output" / clip_name
VIDEO_EXTS = {".mp4", ".mov", ".mkv", ".avi", ".webm", ".mxf", ".m4v"}
def run(cmd):
print("+", " ".join(str(c) for c in cmd), flush=True)
res = subprocess.run(cmd, cwd=str(VENDOR), stdout=sys.stdout, stderr=subprocess.STDOUT)
if res.returncode != 0:
cleanup()
sys.exit(res.returncode)
def cleanup():
shutil.rmtree(clip_dir, ignore_errors=True)
shutil.rmtree(out_clip, ignore_errors=True)
try:
clip_dir.mkdir(parents=True, exist_ok=True)
if src.suffix.lower() in VIDEO_EXTS:
shutil.copy(src, clip_dir / f"Input{src.suffix.lower()}")
else: # single frame → 1-frame sequence
(clip_dir / "Input").mkdir(exist_ok=True)
shutil.copy(src, clip_dir / "Input" / src.name)
run([CLI, "generate-alphas"])
hint = clip_dir / "AlphaHint"
if not hint.is_dir() or not any(hint.iterdir()): # generate-alphas exits 0 even when GVM fails
print("ERROR: no alpha hints generated — are the GVM weights installed? "
"(vendor/corridorkey: uv run hf download geyongtao/gvm --local-dir gvm_core/weights)")
cleanup()
sys.exit(1)
cmd = [CLI, "run-inference",
"--backend", str(p.get("backend", "auto")),
"--srgb", # camera clips; EXR pipelines can re-run --linear
"--despill", str(int(p.get("despill", 5))),
"--despeckle",
"--refiner", str(float(p.get("refiner", 1.0))),
"--screen-color", str(p.get("screen_color", "auto")),
"--comp" if p.get("comp", True) else "--no-comp",
# cpu-post default: gpu post-processing dies with an MPSNDArray buffer assertion
# on both backends (mac, 2026-07). Opt back in with {"gpu_post": true} to retest.
"--gpu-post" if p.get("gpu_post") else "--cpu-post",
"--skip-existing"]
if p.get("max_frames"):
cmd += ["--max-frames", str(int(p["max_frames"]))]
if p.get("image_size"):
cmd += ["--image-size", str(int(p["image_size"]))]
run(cmd)
if not out_clip.is_dir() or not any(out_clip.rglob("*")):
print("ERROR: inference produced no output")
cleanup()
sys.exit(1)
zpath = outdir / f"{src.stem}_corridorkey.zip"
outputs = []
with zipfile.ZipFile(zpath, "w", zipfile.ZIP_STORED) as z: # frames are already compressed
for f in sorted(out_clip.rglob("*")):
if f.is_file():
z.write(f, f.relative_to(out_clip))
outputs.append({"path": zpath.name, "meta": {"tool": "corridorkey-mrp-mlx"}})
previews = sorted(out_clip.rglob("*omp*.mp4")) or sorted(out_clip.rglob("*.mp4"))
if previews: # comp preview as its own asset
prev = outdir / f"{src.stem}_comp{previews[0].suffix}"
shutil.copy(previews[0], prev)
outputs.append({"path": prev.name, "meta": {"tool": "corridorkey-comp"}})
(outdir / "result.json").write_text(json.dumps({"outputs": outputs}))
print(f"done: {', '.join(o['path'] for o in outputs)}", flush=True)
finally:
cleanup()

View File

@ -2,97 +2,23 @@
"id": "flux_local", "id": "flux_local",
"name": "FLUX (local, MLX)", "name": "FLUX (local, MLX)",
"category": "generate", "category": "generate",
"description": "Prompt -> image entirely on this Mac via mflux (MLX). Covers FLUX.1, FLUX.2 klein, Z-Image Turbo and Qwen-Image, with LoRA support. flux2-klein-4b/schnell-4bit/z-image-turbo are ungated. Z-Image Turbo is the pick for anatomy (correct hands at baseline). Cannot load SD/SDXL LoRAs - those need ComfyUI (comfyui_sd).", "description": "Prompt → image entirely on this Mac via mflux 0.18 (MLX). flux2-klein-4b = Apache/UNGATED, works out of the box, better than FLUX.1-schnell (~15GB first download). schnell-4bit = ungated community quant. schnell/dev/krea-dev/klein-9b are HF-gated (license accept + HF token in Settings). No API cost, no cloud. Elo context: klein ~1083-1119, FLUX.1-dev ~1027, nano-banana ~1154.",
"accepts": [], "accepts": [],
"produces": [ "produces": ["image"],
"image"
],
"resources": "gpu", "resources": "gpu",
"entry": "run.py", "entry": "run.py",
"python": "venvs/mflux/bin/python", "python": "venvs/mflux/bin/python",
"params_schema": { "params_schema": {
"type": "object", "type": "object",
"properties": { "properties": {
"prompt": { "prompt": {"type": "string", "default": "", "description": "What to generate"},
"type": "string", "model": {"type": "string", "enum": ["flux2-klein-4b", "flux2-klein-9b", "schnell", "schnell-4bit", "dev", "krea-dev"], "default": "flux2-klein-4b", "description": "klein-4b + schnell-4bit are ungated; the rest need the HF token"},
"default": "", "steps": {"type": "integer", "default": 4, "minimum": 1, "maximum": 50, "description": "klein/schnell: 2-4; dev/krea-dev: 20-28"},
"description": "What to generate" "width": {"type": "integer", "default": 1024, "minimum": 256, "maximum": 2048},
}, "height": {"type": "integer", "default": 1024, "minimum": 256, "maximum": 2048},
"model": { "seed": {"type": "integer", "default": 42, "description": "Fixed seed = reproducible A/B tests"},
"type": "string", "quantize": {"type": "string", "enum": ["none", "8", "4"], "default": "8", "description": "FLUX.1 models only: 8-bit ≈ full quality at half memory"},
"enum": [ "guidance": {"type": "number", "default": 3.5, "minimum": 0, "maximum": 10, "description": "dev/krea-dev only"}
"flux2-klein-4b",
"flux2-klein-9b",
"schnell",
"schnell-4bit",
"dev",
"krea-dev",
"z-image-turbo",
"z-image",
"qwen"
],
"default": "flux2-klein-4b",
"description": "flux2-klein-4b + schnell-4bit + z-image-turbo are UNGATED (work out of the box). schnell/dev/krea-dev/klein-9b need an HF license accept + token. z-image-turbo = Alibaba Z-Image (DiT): best anatomy/hands, few-step (6-10), 4-bit ~6GB. qwen = Qwen-Image. NOTE these are DIFFERENT ARCHITECTURES \u2014 a LoRA only works on its own."
},
"steps": {
"type": "integer",
"default": 4,
"minimum": 1,
"maximum": 50,
"description": "klein/schnell: 2-4; dev/krea-dev: 20-28"
},
"width": {
"type": "integer",
"default": 1024,
"minimum": 256,
"maximum": 2048
},
"height": {
"type": "integer",
"default": 1024,
"minimum": 256,
"maximum": 2048
},
"seed": {
"type": "integer",
"default": 42,
"description": "Fixed seed = reproducible A/B tests"
},
"quantize": {
"type": "string",
"enum": [
"none",
"8",
"4"
],
"default": "8",
"description": "FLUX.1: 4 or 8. Z-Image/Qwen: 3-8 (4 recommended, ~6GB). Ignored for klein."
},
"guidance": {
"type": "number",
"default": 3.5,
"minimum": 0,
"maximum": 10,
"description": "dev/krea-dev only"
},
"lora": {
"type": [
"string",
"array"
],
"default": "",
"description": "LoRA file(s): full path, or a bare name resolved against MB_LORA_DIRS (localmodels/Lora, civit-lib/{sd15,sdxl}/Lora). Accepts 'name=0.7' shorthand. MUST match the model's architecture \u2014 a mismatched LoRA is a SILENT no-op, not an error."
},
"lora_weight": {
"type": [
"number",
"array"
],
"default": 0.8,
"minimum": 0,
"maximum": 2,
"description": "LoRA strength, scalar or per-LoRA list. 0.6-1.0 is the useful range."
}
} }
} }
} }

View File

@ -1,24 +0,0 @@
{
"id": "flux_local",
"name": "FLUX (local, MLX)",
"category": "generate",
"description": "Prompt → image entirely on this Mac via mflux 0.18 (MLX). flux2-klein-4b = Apache/UNGATED, works out of the box, better than FLUX.1-schnell (~15GB first download). schnell-4bit = ungated community quant. schnell/dev/krea-dev/klein-9b are HF-gated (license accept + HF token in Settings). No API cost, no cloud. Elo context: klein ~1083-1119, FLUX.1-dev ~1027, nano-banana ~1154.",
"accepts": [],
"produces": ["image"],
"resources": "gpu",
"entry": "run.py",
"python": "venvs/mflux/bin/python",
"params_schema": {
"type": "object",
"properties": {
"prompt": {"type": "string", "default": "", "description": "What to generate"},
"model": {"type": "string", "enum": ["flux2-klein-4b", "flux2-klein-9b", "schnell", "schnell-4bit", "dev", "krea-dev"], "default": "flux2-klein-4b", "description": "klein-4b + schnell-4bit are ungated; the rest need the HF token"},
"steps": {"type": "integer", "default": 4, "minimum": 1, "maximum": 50, "description": "klein/schnell: 2-4; dev/krea-dev: 20-28"},
"width": {"type": "integer", "default": 1024, "minimum": 256, "maximum": 2048},
"height": {"type": "integer", "default": 1024, "minimum": 256, "maximum": 2048},
"seed": {"type": "integer", "default": 42, "description": "Fixed seed = reproducible A/B tests"},
"quantize": {"type": "string", "enum": ["none", "8", "4"], "default": "8", "description": "FLUX.1 models only: 8-bit ≈ full quality at half memory"},
"guidance": {"type": "number", "default": 3.5, "minimum": 0, "maximum": 10, "description": "dev/krea-dev only"}
}
}
}

View File

@ -1,7 +1,6 @@
import argparse import argparse
import json import json
import os import os
import shlex
import subprocess import subprocess
import sys import sys
import time import time
@ -24,30 +23,11 @@ outdir = Path(a.outdir)
out_png = outdir / f"flux_{model.replace('/', '_')}_s{p.get('seed', 42)}.png" out_png = outdir / f"flux_{model.replace('/', '_')}_s{p.get('seed', 42)}.png"
bindir = Path(sys.executable).parent bindir = Path(sys.executable).parent
# Each family needs a different mflux CLI, and they do NOT accept each other's flags.
# z-image/qwen are separate architectures from FLUX despite living in the same venv.
if model.startswith("flux2-"): if model.startswith("flux2-"):
cli = bindir / "mflux-generate-flux2" cli = bindir / "mflux-generate-flux2"
model_args = ["--model", model] model_args = ["--model", model]
quant_args = [] # klein is small; skip quantization flags quant_args = [] # klein is small; skip quantization flags
guidance_args = [] # distilled klein: guidance fixed at 1.0 guidance_args = [] # distilled klein: guidance fixed at 1.0
elif model in ("z-image", "z-image-turbo"):
# Alibaba Z-Image (DiT). filipstrand's 4-bit MLX build runs in ~6GB, which is what
# makes it viable on the 16-32GB boxes. Turbo is few-step: 6-10, not 25.
cli = bindir / ("mflux-generate-z-image-turbo" if model.endswith("turbo")
else "mflux-generate-z-image")
model_args = ["--base-model", model]
if model == "z-image-turbo":
model_args += ["-m", p.get("hf_repo", "filipstrand/Z-Image-Turbo-mflux-4bit")]
quant = str(p.get("quantize", "4"))
quant_args = ["-q", quant] if quant in ("3", "4", "5", "6", "8") else []
guidance_args = []
elif model == "qwen":
cli = bindir / "mflux-generate-qwen"
model_args = ["--base-model", "qwen"]
quant = str(p.get("quantize", "4"))
quant_args = ["-q", quant] if quant in ("3", "4", "5", "6", "8") else []
guidance_args = []
elif model == "schnell-4bit": elif model == "schnell-4bit":
cli = bindir / "mflux-generate" cli = bindir / "mflux-generate"
# ungated community pre-quantized weights (~10GB) — no HF license wall # ungated community pre-quantized weights (~10GB) — no HF license wall
@ -64,66 +44,9 @@ else:
if model in ("dev", "krea-dev") else []) if model in ("dev", "krea-dev") else [])
if not cli.exists(): if not cli.exists():
print(f"ERROR: mflux CLI not installed ({cli} missing). Run scripts/install_mflux.sh") print(f"ERROR: mflux not installed ({cli} missing). Run scripts/install_mflux.sh")
sys.exit(1) sys.exit(1)
# ---- LoRA -------------------------------------------------------------------
# THE RULE: a LoRA only works on its own base architecture. Loading an SD1.5 LoRA on
# FLUX, or a FLUX.1 LoRA on FLUX.2, is a SILENT no-op — mflux does not error, the image
# just comes back unchanged and you blame the LoRA. So we fail loudly on a missing file
# and log exactly what was applied.
#
# `lora` accepts a path, a bare filename resolved against LORA_DIRS, or a list. Weights
# come from `lora_weight` (scalar or list, default 0.8).
lora_dirs = [Path(d).expanduser() for d in
os.environ.get("MB_LORA_DIRS",
"~/Documents/localmodels/Lora:~/Documents/civit-lib/sd15/Lora:"
"~/Documents/civit-lib/sdxl/Lora:~/Documents/loras").split(":")]
def resolve_lora(name):
q = Path(name).expanduser()
if q.is_file():
return q
for d in lora_dirs:
c = d / name
if c.is_file():
return c
if not name.endswith(".safetensors"):
c = d / f"{name}.safetensors"
if c.is_file():
return c
return None
lora_args = []
raw = p.get("lora") or p.get("loras") or []
if isinstance(raw, str):
raw = [raw] if raw.strip() else []
if raw:
weights = p.get("lora_weight", p.get("lora_scales", 0.8))
if not isinstance(weights, list):
weights = [weights] * len(raw)
paths, scales = [], []
for i, nm in enumerate(raw):
# allow "name=0.7" shorthand
if isinstance(nm, str) and "=" in nm and not Path(nm).exists():
nm, _, w = nm.rpartition("=")
try:
weights[i] = float(w)
except ValueError:
pass
hit = resolve_lora(nm)
if not hit:
print(f"ERROR: LoRA not found: {nm}")
print(f" searched: {', '.join(str(d) for d in lora_dirs)}")
sys.exit(1)
paths.append(str(hit))
scales.append(str(weights[i] if i < len(weights) else 0.8))
lora_args = ["--lora-paths", *paths, "--lora-scales", *scales]
for pth, sc in zip(paths, scales):
print(f"[lora] {Path(pth).name} @ {sc}", flush=True)
cmd = [str(cli), *model_args, cmd = [str(cli), *model_args,
"--prompt", p["prompt"], "--prompt", p["prompt"],
"--steps", str(steps), "--steps", str(steps),
@ -131,9 +54,9 @@ cmd = [str(cli), *model_args,
"--height", str(p.get("height", 1024)), "--height", str(p.get("height", 1024)),
"--seed", str(p.get("seed", 42)), "--seed", str(p.get("seed", 42)),
"--output", str(out_png), "--output", str(out_png),
*quant_args, *guidance_args, *lora_args] *quant_args, *guidance_args]
print("+", " ".join(shlex.quote(c) for c in cmd), flush=True) print("+", " ".join(cmd), flush=True)
print("(first run per model downloads weights from HuggingFace)", flush=True) print("(first run per model downloads weights from HuggingFace)", flush=True)
env = os.environ.copy() env = os.environ.copy()
# hf_xet's chunked downloader intermittently fails ("Unable to parse string as # hf_xet's chunked downloader intermittently fails ("Unable to parse string as
@ -143,8 +66,8 @@ t0 = time.time()
res = subprocess.run(cmd, stdout=sys.stdout, stderr=subprocess.STDOUT, env=env) res = subprocess.run(cmd, stdout=sys.stdout, stderr=subprocess.STDOUT, env=env)
if res.returncode != 0: if res.returncode != 0:
print("HINT: gated-repo/auth errors mean this model needs an HF license accept " print("HINT: gated-repo/auth errors mean this model needs an HF license accept "
"+ token (Settings → HuggingFace token). flux2-klein-4b, schnell-4bit and " "+ token (Settings → HuggingFace token). flux2-klein-4b and schnell-4bit "
"z-image-turbo are ungated and need nothing.") "are ungated and need nothing.")
sys.exit(res.returncode) sys.exit(res.returncode)
elapsed = round(time.time() - t0, 1) elapsed = round(time.time() - t0, 1)
@ -158,6 +81,5 @@ if not out_png.exists():
(outdir / "result.json").write_text(json.dumps({"outputs": [ (outdir / "result.json").write_text(json.dumps({"outputs": [
{"path": out_png.name, {"path": out_png.name,
"meta": {"tool": "mflux", "model": model, "steps": steps, "meta": {"tool": "mflux", "model": model, "steps": steps,
"seed": p.get("seed", 42), "seconds": elapsed, "seed": p.get("seed", 42), "seconds": elapsed}}]}))
"loras": [Path(x).name for x in lora_args[1:1 + len(raw)]] if raw else []}}]}))
print(f"done in {elapsed}s: {out_png.name}", flush=True) print(f"done in {elapsed}s: {out_png.name}", flush=True)

View File

@ -1,85 +0,0 @@
import argparse
import json
import os
import subprocess
import sys
import time
from pathlib import Path
ap = argparse.ArgumentParser()
ap.add_argument("--input", action="append", default=[])
ap.add_argument("--outdir", required=True)
ap.add_argument("--params", default="{}")
a = ap.parse_args()
p = json.loads(a.params)
if not p.get("prompt"):
print("ERROR: prompt is required")
sys.exit(1)
model = p.get("model", "flux2-klein-4b")
steps = int(p.get("steps", 4))
outdir = Path(a.outdir)
out_png = outdir / f"flux_{model.replace('/', '_')}_s{p.get('seed', 42)}.png"
bindir = Path(sys.executable).parent
if model.startswith("flux2-"):
cli = bindir / "mflux-generate-flux2"
model_args = ["--model", model]
quant_args = [] # klein is small; skip quantization flags
guidance_args = [] # distilled klein: guidance fixed at 1.0
elif model == "schnell-4bit":
cli = bindir / "mflux-generate"
# ungated community pre-quantized weights (~10GB) — no HF license wall
model_args = ["--model", "dhairyashil/FLUX.1-schnell-mflux-4bit",
"--base-model", "schnell"]
quant_args = []
guidance_args = []
else:
cli = bindir / "mflux-generate"
model_args = ["--model", model]
quant = str(p.get("quantize", "8"))
quant_args = ["--quantize", quant] if quant in ("4", "8") else []
guidance_args = (["--guidance", str(p.get("guidance", 3.5))]
if model in ("dev", "krea-dev") else [])
if not cli.exists():
print(f"ERROR: mflux not installed ({cli} missing). Run scripts/install_mflux.sh")
sys.exit(1)
cmd = [str(cli), *model_args,
"--prompt", p["prompt"],
"--steps", str(steps),
"--width", str(p.get("width", 1024)),
"--height", str(p.get("height", 1024)),
"--seed", str(p.get("seed", 42)),
"--output", str(out_png),
*quant_args, *guidance_args]
print("+", " ".join(cmd), flush=True)
print("(first run per model downloads weights from HuggingFace)", flush=True)
env = os.environ.copy()
# hf_xet's chunked downloader intermittently fails ("Unable to parse string as
# hex hash value"); the plain HTTP path is reliable.
env["HF_HUB_DISABLE_XET"] = "1"
t0 = time.time()
res = subprocess.run(cmd, stdout=sys.stdout, stderr=subprocess.STDOUT, env=env)
if res.returncode != 0:
print("HINT: gated-repo/auth errors mean this model needs an HF license accept "
"+ token (Settings → HuggingFace token). flux2-klein-4b and schnell-4bit "
"are ungated and need nothing.")
sys.exit(res.returncode)
elapsed = round(time.time() - t0, 1)
if not out_png.exists():
pngs = sorted(outdir.glob("*.png"))
if not pngs:
print("ERROR: no image produced")
sys.exit(1)
out_png = pngs[-1]
(outdir / "result.json").write_text(json.dumps({"outputs": [
{"path": out_png.name,
"meta": {"tool": "mflux", "model": model, "steps": steps,
"seed": p.get("seed", 42), "seconds": elapsed}}]}))
print(f"done in {elapsed}s: {out_png.name}", flush=True)

View File

@ -1,19 +0,0 @@
{
"id": "llm_v4",
"name": "LLM (DeepSeek-V4-Flash 304B, local)",
"category": "text",
"description": "Prompt → text via DeepSeek-V4-Flash-0731 (304B MoE, 13B active) served by oMLX on :8020. ~45 tok/s warm with DSpark speculative decode, 1M context capable. The heavyweight for hard reasoning/code; for quick/cheap calls use the m4pro Ollama endpoint, for mid-tier use llm_local. Primary-node only — the server holds ~150GB wired. gpu lane so the queue serializes it against mesh/image gen on this box.",
"accepts": [],
"produces": ["text"],
"resources": "gpu",
"entry": "run.py",
"params_schema": {
"type": "object",
"properties": {
"prompt": {"type": "string", "default": "", "description": "The user prompt"},
"system": {"type": "string", "default": "", "description": "Optional system prompt"},
"max_tokens": {"type": "integer", "default": 1024, "minimum": 16, "maximum": 32768},
"temperature": {"type": "number", "default": 0.7, "minimum": 0, "maximum": 2}
}
}
}

View File

@ -1,91 +0,0 @@
import argparse
import json
import subprocess
import sys
import time
import urllib.error
import urllib.request
from pathlib import Path
# DeepSeek-V4-Flash-0731 via the oMLX server (repo: monster/deepseek_v4_flash_0731_mrp_mlx).
# The server is a resident service (launchd party.monster.deepseek-v4) holding ~150GB
# wired when warm; this operator is a thin HTTP client. It runs on the gpu lane so the
# queue serializes it against trellis/flux on this node — that serialization, not this
# script, is what stops the LLM and mesh gen fighting over Metal.
BASE = "http://127.0.0.1:8020"
MODEL = "DeepSeek-V4-Flash-0731-MXFP4-MLX"
LAUNCHD_LABEL = "party.monster.deepseek-v4"
ap = argparse.ArgumentParser()
ap.add_argument("--input", action="append", default=[])
ap.add_argument("--outdir", required=True)
ap.add_argument("--params", default="{}")
a = ap.parse_args()
p = json.loads(a.params)
prompt = (p.get("prompt") or "").strip()
if not prompt:
print("ERROR: prompt is required")
sys.exit(1)
def server_up() -> bool:
try:
with urllib.request.urlopen(f"{BASE}/health", timeout=5) as r:
return r.status == 200
except (urllib.error.URLError, OSError):
return False
if not server_up():
# Cold path: ask launchd to (re)start the service, then wait for readiness.
# First token after a cold start also pays the ~30s weight load.
print("oMLX server down — kickstarting launchd service", flush=True)
uid = subprocess.run(["id", "-u"], capture_output=True, text=True).stdout.strip()
subprocess.run(["launchctl", "kickstart", f"gui/{uid}/{LAUNCHD_LABEL}"],
capture_output=True)
deadline = time.time() + 180
while time.time() < deadline and not server_up():
time.sleep(3)
if not server_up():
print("ERROR: oMLX server did not come up on :8020. Install the service with "
"~/Documents/deepseek_v4_flash_0731_mrp_mlx/scripts/install_launchagent.sh")
sys.exit(1)
messages = []
if (p.get("system") or "").strip():
messages.append({"role": "system", "content": p["system"].strip()})
messages.append({"role": "user", "content": prompt})
body = json.dumps({
"model": MODEL,
"messages": messages,
"max_tokens": int(p.get("max_tokens", 1024)),
"temperature": float(p.get("temperature", 0.7)),
}).encode()
t0 = time.time()
req = urllib.request.Request(f"{BASE}/v1/chat/completions", data=body,
headers={"Content-Type": "application/json"})
try:
# generous timeout: cold start pays ~30s model load before first token
with urllib.request.urlopen(req, timeout=600) as r:
d = json.loads(r.read())
except (urllib.error.URLError, OSError) as e:
print(f"ERROR: request failed: {e}")
sys.exit(1)
if "error" in d:
print(f"ERROR: {d['error']}")
sys.exit(1)
msg = d["choices"][0]["message"]
text = (msg.get("content") or "").strip()
usage = d.get("usage", {})
el = time.time() - t0
ct = usage.get("completion_tokens") or 0
out = Path(a.outdir) / "llm_output.txt"
out.write_text(text + "\n")
tps = f", {ct / el:.1f} tok/s" if ct and el > 0 else ""
print(f"done in {el:.1f}s ({ct} tokens{tps}), {len(text)} chars:\n{text[:400]}", flush=True)

View File

@ -2,48 +2,19 @@
"id": "mflux_image_edit", "id": "mflux_image_edit",
"name": "Edit Image (local, Qwen)", "name": "Edit Image (local, Qwen)",
"category": "image-prep", "category": "image-prep",
"description": "1-3 images + instruction → edited image (multi-image composes: 'put the shirt from image 2 on the person in image 1') entirely on this Mac via mflux Qwen-Image-Edit (ungated/Apache-2.0). Free local alternative to fal_image_edit / nano-banana edit — 'remove the sticker', 'make it studio-lit on white', material/lighting changes. First run downloads weights.", "description": "Image + instruction → edited image entirely on this Mac via mflux Qwen-Image-Edit (ungated/Apache-2.0). Free local alternative to fal_image_edit / nano-banana edit — 'remove the sticker', 'make it studio-lit on white', material/lighting changes. First run downloads weights.",
"accepts": [ "accepts": ["image"],
"image" "produces": ["image"],
],
"produces": [
"image"
],
"resources": "gpu", "resources": "gpu",
"entry": "run.py", "entry": "run.py",
"python": "venvs/mflux/bin/python", "python": "venvs/mflux/bin/python",
"params_schema": { "params_schema": {
"type": "object", "type": "object",
"properties": { "properties": {
"prompt": { "prompt": {"type": "string", "default": "", "description": "Edit instruction (what to change)"},
"type": "string", "steps": {"type": "integer", "default": 25, "minimum": 1, "maximum": 50, "description": "Inference steps"},
"default": "", "seed": {"type": "integer", "default": 42, "description": "Random seed"},
"description": "Edit instruction (what to change)" "guidance": {"type": "number", "default": 4.0, "minimum": 0, "maximum": 10, "description": "Guidance scale"}
},
"steps": {
"type": "integer",
"default": 25,
"minimum": 1,
"maximum": 50,
"description": "Inference steps"
},
"seed": {
"type": "integer",
"default": 42,
"description": "Random seed"
},
"guidance": {
"type": "number",
"default": 4.0,
"minimum": 0,
"maximum": 10,
"description": "Guidance scale"
},
"loras": {
"type": "string",
"default": "",
"description": "Qwen-Edit LoRAs, comma-separated 'name' or 'name:scale' (e.g. 'qwen-studio-realism:0.8,gymnast'). Resolved from ~/Documents/localmodels/QwenLora on both mflux nodes (QWEN_LORA_DIRS overrides). Wrong-base LoRAs are silent no-ops."
}
} }
} }
} }

View File

@ -1,6 +1,5 @@
import argparse import argparse
import json import json
import os
import subprocess import subprocess
import sys import sys
from pathlib import Path from pathlib import Path
@ -26,47 +25,13 @@ if not cli.exists():
src = Path(a.input[0]) src = Path(a.input[0])
outdir = Path(a.outdir) outdir = Path(a.outdir)
out = outdir / f"{src.stem}_edited.png" out = outdir / f"{src.stem}_edited.png"
# ALL inputs go to Qwen (it composes up to ~3 images: "put the shirt from image 2 on the person cmd = [str(cli), "--image-paths", str(src.resolve()),
# in image 1"). Single-image jobs behave exactly as before.
cmd = [str(cli), "--image-paths", *[str(Path(x).resolve()) for x in a.input],
"--prompt", p["prompt"], "--prompt", p["prompt"],
"--steps", str(p.get("steps", 25)), "--steps", str(p.get("steps", 25)),
"--seed", str(p.get("seed", 42)), "--seed", str(p.get("seed", 42)),
"--guidance", str(p.get("guidance", 4.0)), "--guidance", str(p.get("guidance", 4.0)),
"--output", str(out.resolve())] "--output", str(out.resolve())]
# Qwen-Edit LoRAs: "name" or "name:scale", comma-separated. Names resolve
# (case-insensitive prefix match) against the fleet-standard stash, which is
# present on BOTH mflux nodes (m3ultra + ultra) — a job can dispatch to either,
# so a node-local dir would resolve on one and fail on the other.
# mflux natively takes --lora-paths/--lora-scales; a LoRA for the wrong base
# model is a SILENT no-op (no error, the edit just ignores it).
LORA_DIRS = [Path(d).expanduser() for d in
(os.environ.get("QWEN_LORA_DIRS") or "").split(os.pathsep) if d.strip()] \
or [Path.home() / "Documents" / "localmodels" / "QwenLora"]
if p.get("loras"):
paths, scales = [], []
avail = {}
for d in LORA_DIRS:
for f in d.rglob("*.safetensors"):
avail.setdefault(f.name.lower(), f)
for item in str(p["loras"]).split(","):
item = item.strip()
if not item:
continue
name, _, scale = item.partition(":")
name = name.strip().lower()
hit = avail.get(name) or avail.get(name + ".safetensors") or next(
(f for k, f in sorted(avail.items()) if k.startswith(name)), None)
if not hit:
print(f"ERROR: no LoRA matching '{name}' under {LORA_DIR}")
print("available:", ", ".join(sorted(f.stem for f in avail.values())[:40]))
sys.exit(1)
paths.append(str(hit))
scales.append(str(float(scale) if scale.strip() else 1.0))
if paths:
cmd += ["--lora-paths", *paths, "--lora-scales", *scales]
print("+", " ".join(cmd), flush=True) print("+", " ".join(cmd), flush=True)
print("(first run downloads Qwen-Image-Edit weights)", flush=True) print("(first run downloads Qwen-Image-Edit weights)", flush=True)
res = subprocess.run(cmd, cwd=str(outdir), stdout=sys.stdout, stderr=subprocess.STDOUT) res = subprocess.run(cmd, cwd=str(outdir), stdout=sys.stdout, stderr=subprocess.STDOUT)

View File

@ -1,21 +0,0 @@
{
"id": "qwen_layered_local",
"name": "Qwen-Image-Layered (local)",
"category": "image-edit",
"description": "Image → N editable RGBA layers (bg / subject / detail separation), fully local via the Qwen-Image-Layered 20B MLX port (mflux PR#302 fork). ~4 min on m3ultra, ~6 min on m1ultra at the tuned defaults (20 steps, 640, baked q8). The subject layer doubles as matting-grade background removal. Bake a local q8 model once per box (see scripts/install_qwen_layered.sh) or first run falls back to a 54GB HF download + on-the-fly quantization.",
"accepts": ["image"],
"produces": ["image"],
"resources": "gpu",
"entry": "run.py",
"python": "vendor/mflux-qwen-layered/.venv/bin/python",
"params_schema": {
"type": "object",
"properties": {
"layers": {"type": "integer", "default": 4, "description": "Number of RGBA layers to decompose into"},
"steps": {"type": "integer", "default": 20, "description": "Denoising steps (20 = tuned default; 50 = upstream default, ~2.9x slower, no visible gain on tested inputs)"},
"resolution": {"type": "integer", "enum": [640, 1024], "default": 640, "description": "Working resolution"},
"seed": {"type": "integer", "default": 0, "description": "Random seed"},
"prompt": {"type": "string", "default": "", "description": "Optional prompt to guide the decomposition"}
}
}
}

View File

@ -1,71 +0,0 @@
import argparse
import json
import os
import subprocess
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[3]
VENDOR = ROOT / "vendor" / "mflux-qwen-layered"
CLI = VENDOR / ".venv" / "bin" / "mflux-generate-qwen-layered"
ap = argparse.ArgumentParser()
ap.add_argument("--input", action="append", default=[])
ap.add_argument("--outdir", required=True)
ap.add_argument("--params", default="{}")
a = ap.parse_args()
p = json.loads(a.params)
if not a.input:
print("ERROR: no input image")
sys.exit(1)
if not CLI.exists():
print(f"ERROR: qwen-layered not installed at {VENDOR}. "
"Run scripts/install_qwen_layered.sh")
sys.exit(1)
# Baked-model resolution: env override, then per-box conventional paths,
# else fall back to the HF repo with on-the-fly q8 (slow first run: 54GB).
candidates = [os.environ.get("QWEN_LAYERED_MODEL", "")]
candidates += [str(Path.home() / "qwen-layered" / "qwen-layered-q8"),
str(Path.home() / "qwen-layered-staging" / "qwen-layered-q8")]
model_path = next((c for c in candidates if c and Path(c).is_dir()), None)
outdir = Path(a.outdir)
cmd = [
str(CLI), "--image", str(Path(a.input[0]).resolve()),
"--layers", str(p.get("layers", 4)),
"--steps", str(p.get("steps", 20)),
"--resolution", str(p.get("resolution", 640)),
"--seed", str(p.get("seed", 0)),
"--output-dir", str(outdir.resolve()),
]
if model_path:
cmd += ["--model-path", model_path]
print(f"using baked model: {model_path}", flush=True)
else:
cmd += ["-q", "8"]
print("no baked model found — HF fallback with on-the-fly q8 "
"(first run downloads 54GB)", flush=True)
if p.get("prompt"):
cmd += ["--prompt", str(p["prompt"])]
env = os.environ.copy()
env["HF_HUB_DISABLE_XET"] = "1"
print("+", " ".join(cmd), flush=True)
res = subprocess.run(cmd, cwd=str(VENDOR), stdout=sys.stdout,
stderr=subprocess.STDOUT, env=env)
if res.returncode != 0:
sys.exit(res.returncode)
layers = sorted(outdir.glob("layer_*.png"))
if not layers:
print("ERROR: qwen-layered produced no layer PNGs")
sys.exit(1)
stem = Path(a.input[0]).stem
outputs = [{"path": str(f), "name": f"{stem}_{f.stem}.png",
"meta": {"tool": "qwen_layered_local", "layer": i}}
for i, f in enumerate(layers)]
(outdir / "result.json").write_text(json.dumps({"outputs": outputs}))
print(f"done: {len(layers)} layers", flush=True)

View File

@ -1,26 +0,0 @@
{
"id": "sd_local",
"name": "SD1.5 + anatomy LoRAs (local, primary-only)",
"category": "generate",
"description": "Prompt → image via the repose stack on the primary: SD1.5 Hyper_Realism + John's anatomy LoRAs (~/Documents/repose/models/lora) + optional OpenPose T-pose template (front/back/side) + optional IP-Adapter reference image. The no-cloud, no-content-filter lane FLUX can't cover — LoRA-styled anatomy, exact pose control, reference-guided identity. Primary-only: needs ~/Documents/repose (absolute venv path keeps it off remote dispatch by design).",
"accepts": ["image"],
"produces": ["image"],
"resources": "gpu",
"entry": "run.py",
"python": "/Users/m3ultra/Documents/repose/venv/bin/python",
"params_schema": {
"type": "object",
"properties": {
"prompt": {"type": "string", "default": "", "description": "What to generate"},
"negative": {"type": "string", "default": "deformed, extra limbs, extra fingers, mutated hands, missing limbs, cropped, blurry, lowres, watermark, text, multiple people, child", "description": "Negative prompt"},
"loras": {"type": "array", "items": {"type": "string"}, "default": [], "description": "'stem=weight' from repose/models/lora, e.g. 'GodPussy1 v4=0.6'"},
"pose": {"type": "string", "enum": ["none", "front", "back", "side"], "default": "none", "description": "OpenPose T-pose template via ControlNet"},
"ip": {"type": "number", "default": 0.7, "description": "IP-Adapter strength when a reference image is attached"},
"steps": {"type": "integer", "default": 28, "minimum": 1, "maximum": 60},
"cfg": {"type": "number", "default": 7.0},
"seed": {"type": "integer", "default": 7},
"width": {"type": "integer", "default": 512, "description": "SD1.5 native ≈512-768; upscale after via seedvr2"},
"height": {"type": "integer", "default": 768}
}
}
}

View File

@ -1,97 +0,0 @@
"""sd_local — SD1.5 Hyper_Realism + anatomy LoRAs (+ optional pose ControlNet / IP-Adapter ref).
Runs UNDER the repose venv (manifest python is that venv, absolute primary-only).
Generalizes scripts/repose.py from MESHGOD: pose template optional, plain txt2img otherwise.
"""
import argparse
import json
import os
import sys
os.environ.setdefault("PYTORCH_ENABLE_MPS_FALLBACK", "1")
import torch # noqa: E402
from diffusers import DPMSolverMultistepScheduler # noqa: E402
from PIL import Image # noqa: E402
ap = argparse.ArgumentParser()
ap.add_argument("--input", action="append", default=[])
ap.add_argument("--outdir", required=True)
ap.add_argument("--params", default="{}")
a = ap.parse_args()
p = json.loads(a.params)
HERE = os.path.expanduser("~/Documents/repose")
BASE = os.path.join(HERE, "models", "Hyper_Realism_1.2_fp16.safetensors")
# LoRA search path: John's master dir first, the repose copy second (SD_LORA_DIRS to override)
LORA_DIRS = [os.path.expanduser(d) for d in os.environ.get(
"SD_LORA_DIRS", "~/Documents/localmodels/Lora:~/Documents/repose/models/lora").split(":")]
def find_lora(stem):
for d in LORA_DIRS:
cand = os.path.join(d, stem + ".safetensors")
if os.path.exists(cand):
return cand
return None
if not os.path.exists(BASE):
print(f"ERROR: {BASE} missing — sd_local runs on the primary only")
sys.exit(1)
dev = "mps" if torch.backends.mps.is_available() else "cpu"
dtype = torch.float16 if dev == "mps" else torch.float32
pose = p.get("pose", "none")
kw = {}
if pose != "none":
from diffusers import ControlNetModel, StableDiffusionControlNetPipeline
tpl = os.path.join(HERE, f"tpose_{pose}.png")
cn = ControlNetModel.from_pretrained("lllyasviel/control_v11p_sd15_openpose", torch_dtype=dtype)
pipe = StableDiffusionControlNetPipeline.from_single_file(
BASE, controlnet=cn, torch_dtype=dtype, safety_checker=None, requires_safety_checker=False)
kw["image"] = Image.open(tpl).convert("RGB")
else:
from diffusers import StableDiffusionPipeline
pipe = StableDiffusionPipeline.from_single_file(
BASE, torch_dtype=dtype, safety_checker=None, requires_safety_checker=False)
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config, use_karras_sigmas=True)
pipe = pipe.to(dev)
# no attention slicing — it crashes IP-Adapter attn-processor injection (see repose.py)
if p.get("loras"):
from safetensors.torch import load_file
names, weights = [], []
for spec in p["loras"]:
stem, _, w = str(spec).partition("=")
path = find_lora(stem.strip())
if not path:
print(f"[sd] LORA MISS '{stem}' — searched {LORA_DIRS}", flush=True)
continue
# UNet-only: this diffusers build trips an IndexError on these kohya files' text-encoder
# half (empty rank_dict in load_lora_into_text_encoder). The UNet half carries the look.
sd_l = load_file(path)
unet_only = {k: v for k, v in sd_l.items() if not k.startswith("lora_te")}
pipe.load_lora_weights(unet_only, adapter_name=stem.replace(" ", "_"))
names.append(stem.replace(" ", "_"))
weights.append(float(w or 0.6))
if names:
pipe.set_adapters(names, weights)
print(f"[sd] loras: {dict(zip(names, weights))}", flush=True)
if a.input: # reference image → IP-Adapter
pipe.load_ip_adapter("h94/IP-Adapter", subfolder="models", weight_name="ip-adapter_sd15.bin")
pipe.set_ip_adapter_scale(float(p.get("ip", 0.7)))
kw["ip_adapter_image"] = Image.open(a.input[0]).convert("RGB")
print(f"[sd] ip-adapter ref @ {p.get('ip', 0.7)}", flush=True)
gen = torch.Generator("cpu").manual_seed(int(p.get("seed", 7)))
img = pipe(prompt=p.get("prompt", ""), negative_prompt=p.get("negative", ""),
num_inference_steps=int(p.get("steps", 28)), guidance_scale=float(p.get("cfg", 7.0)),
generator=gen, width=int(p.get("width", 512)), height=int(p.get("height", 768)),
**kw).images[0]
out = os.path.join(a.outdir, "sd_local.png")
img.save(out)
json.dump({"outputs": [{"path": "sd_local.png", "meta": {"tool": "sd_local"}}]},
open(os.path.join(a.outdir, "result.json"), "w"))
print("done: sd_local.png", flush=True)

View File

@ -1,20 +0,0 @@
{
"id": "stereo_depth",
"name": "Stereo SBS → depth + point cloud (local)",
"category": "video-prep",
"description": "Side-by-side stereoscopic video (or one SBS frame) → per-frame disparity/depth maps via OpenCV StereoSGBM + a colored .ply point cloud of the middle frame for Blender. TRUE geometric depth from the L/R eye shift — no AI hallucination. Uncalibrated SBS rips aren't rectified, so expect relief-map depth (great for Displace-modifier planes), not survey-grade geometry. Output: depth preview mp4 + mid-frame .ply + zipped depth PNGs.",
"accepts": ["video", "image"],
"produces": ["archive"],
"resources": "cpu",
"entry": "run.py",
"python": "venvs/stereo/bin/python",
"params_schema": {
"type": "object",
"properties": {
"max_frames": {"type": "integer", "default": 120, "description": "Cap processed frames"},
"num_disp": {"type": "integer", "default": 128, "description": "Disparity search range (multiple of 16; raise for close subjects)"},
"block": {"type": "integer", "default": 5, "description": "SGBM block size (odd, 3-11)"},
"layout": {"type": "string", "enum": ["sbs", "tab"], "default": "sbs", "description": "side-by-side or top-and-bottom stereo"}
}
}
}

View File

@ -1,112 +0,0 @@
"""stereo_depth — SBS stereoscopic video/frame → SGBM disparity maps + mid-frame point cloud.
True geometric depth from the built-in L/R parallax (cv2.StereoSGBM). ffmpeg (system)
decodes/splits; depth PNGs + preview mp4 + a colored .ply land as the outputs.
ponytail: uncalibrated disparity (SBS rips have no camera intrinsics) z is relative,
perfect for Blender displace planes; not metric reconstruction.
"""
import argparse
import glob
import json
import os
import shutil
import subprocess
import sys
import tempfile
import zipfile
import cv2
import numpy as np
FFMPEG = shutil.which("ffmpeg") or "/opt/homebrew/bin/ffmpeg" # headless ssh has no brew PATH
ap = argparse.ArgumentParser()
ap.add_argument("--input", action="append", default=[])
ap.add_argument("--outdir", required=True)
ap.add_argument("--params", default="{}")
a = ap.parse_args()
p = json.loads(a.params)
if not a.input:
print("ERROR: no input")
sys.exit(1)
src = a.input[0]
outdir = a.outdir
max_frames = int(p.get("max_frames", 120))
num_disp = max(16, (int(p.get("num_disp", 128)) // 16) * 16)
block = int(p.get("block", 5)) | 1
tab = p.get("layout", "sbs") == "tab"
work = tempfile.mkdtemp(prefix="stereo_")
frames_dir = os.path.join(work, "frames")
depth_dir = os.path.join(work, "depth")
os.makedirs(frames_dir)
os.makedirs(depth_dir)
# decode → frames (a single image just becomes frame 1)
subprocess.run([FFMPEG, "-y", "-loglevel", "error", "-i", src,
"-frames:v", str(max_frames), os.path.join(frames_dir, "f%05d.png")], check=True)
frames = sorted(glob.glob(os.path.join(frames_dir, "*.png")))
print(f"[stereo] {len(frames)} frames, num_disp={num_disp} block={block}", flush=True)
min_disp = -(num_disp // 2) # movie stereo converges on the screen plane — content
sgbm = cv2.StereoSGBM_create( # lives BOTH sides of it, so the window must go negative
minDisparity=min_disp, numDisparities=num_disp, blockSize=block,
P1=8 * 3 * block * block, P2=32 * 3 * block * block,
disp12MaxDiff=1, uniquenessRatio=10, speckleWindowSize=100, speckleRange=2,
mode=cv2.STEREO_SGBM_MODE_SGBM_3WAY)
mid = frames[len(frames) // 2]
for i, f in enumerate(frames):
img = cv2.imread(f)
h, w = img.shape[:2]
if tab:
L, R = img[: h // 2], img[h // 2:]
else:
L, R = img[:, : w // 2], img[:, w // 2:]
gl, gr = cv2.cvtColor(L, cv2.COLOR_BGR2GRAY), cv2.cvtColor(R, cv2.COLOR_BGR2GRAY)
disp = sgbm.compute(gl, gr).astype(np.float32) / 16.0
valid = disp > (min_disp - 0.5) # cv2 marks unmatched pixels min_disp-1
if valid.sum() > 100:
lo, hi = np.percentile(disp[valid], [2, 98])
else:
lo, hi = 0.0, 1.0
norm = np.zeros(disp.shape, np.uint8)
norm[valid] = (np.clip((disp[valid] - lo) / max(1e-6, hi - lo), 0, 1) * 255).astype(np.uint8)
cv2.imwrite(os.path.join(depth_dir, os.path.basename(f)), norm)
if f == mid: # colored point cloud of the middle frame
ys, xs = np.where(valid & (norm > 2))
step = max(1, len(ys) // 400_000) # cap cloud size
ys, xs = ys[::step], xs[::step]
z = disp[ys, xs]
rgb = L[ys, xs][:, ::-1] # BGR→RGB
ply = os.path.join(outdir, "midframe_cloud.ply")
with open(ply, "w") as fh:
fh.write("ply\nformat ascii 1.0\n"
f"element vertex {len(ys)}\n"
"property float x\nproperty float y\nproperty float z\n"
"property uchar red\nproperty uchar green\nproperty uchar blue\nend_header\n")
for (yy, xx, zz, (r, g, b)) in zip(ys, xs, z, rgb):
fh.write(f"{xx} {-yy} {zz:.1f} {r} {g} {b}\n")
print(f"[stereo] wrote midframe_cloud.ply ({len(ys)} pts)", flush=True)
if i % 24 == 0:
print(f"[stereo] frame {i + 1}/{len(frames)}", flush=True)
outputs = [{"path": "midframe_cloud.ply", "meta": {"tool": "stereo_sgbm"}}]
if len(frames) > 1: # depth preview video
prev = os.path.join(outdir, "depth_preview.mp4")
subprocess.run([FFMPEG, "-y", "-loglevel", "error", "-framerate", "24",
"-i", os.path.join(depth_dir, "f%05d.png"),
"-pix_fmt", "yuv420p", prev], check=True)
outputs.append({"path": "depth_preview.mp4", "meta": {"tool": "stereo_sgbm"}})
zpath = os.path.join(outdir, "depth_frames.zip")
with zipfile.ZipFile(zpath, "w", zipfile.ZIP_DEFLATED) as zf:
for f in sorted(glob.glob(os.path.join(depth_dir, "*.png"))):
zf.write(f, os.path.basename(f))
outputs.append({"path": "depth_frames.zip", "meta": {"tool": "stereo_sgbm"}})
json.dump({"outputs": outputs}, open(os.path.join(outdir, "result.json"), "w"))
shutil.rmtree(work, ignore_errors=True)
print("done:", ", ".join(o["path"] for o in outputs), flush=True)

View File

@ -1,79 +0,0 @@
{
"id": "trellis2_mlx",
"name": "TRELLIS.2 MLX (local, fastest)",
"category": "mesh-gen",
"description": "Image → PBR GLB via the MRP MLX fork of TRELLIS.2 (monster/trellis-2-mrp-mlx): pure-MLX sampler, Metal PBR bake with RAM-sized face cap + forced-opaque alpha (2026-07-22 quality fixes — fal-parity detail), kdtree/vertex fallbacks. ~72s vertex / ~2min metal-PBR on m3ultra — still the fastest local TRELLIS.2. baker=vertex for the 1s clean-albedo bake (no MR maps). Weights: TRELLIS.2-4B + gated DINOv3/RMBG (rsync from m3 HF cache to workers; no HF login needed).",
"accepts": [
"image"
],
"produces": [
"model"
],
"resources": "gpu",
"entry": "run.py",
"params_schema": {
"type": "object",
"properties": {
"pipeline_type": {
"type": "string",
"enum": [
"512",
"1024",
"1024_cascade"
],
"default": "1024_cascade",
"description": "Resolution tier"
},
"seed": {
"type": "integer",
"default": 0,
"description": "Random seed"
},
"steps": {
"type": "integer",
"default": 12,
"description": "Sampler steps per stage (12 = upstream default)"
},
"baker": {
"type": "string",
"enum": [
"metal",
"vertex",
"kdtree"
],
"default": "metal",
"description": "metal = UV PBR textures (meshgod-grade; minor dark-patch texels remain); vertex = 1s bake, cleanest colors, no MR maps"
},
"texture_size": {
"type": "integer",
"enum": [
512,
1024,
2048,
4096
],
"default": 2048,
"description": "Baked texture resolution (metal/kdtree bakers)"
},
"max_bake_faces": {
"type": "integer",
"enum": [
200000,
500000,
1000000
],
"default": 500000,
"description": "Pre-bake face cap. 500k ≈ fal parity (M3 Ultra verified); node RAM sizes the default when unset — laptops get 200k"
},
"alpha_mode": {
"type": "string",
"enum": [
"opaque",
"auto"
],
"default": "opaque",
"description": "opaque = force alphaMode OPAQUE (fixes speckle-veil hair); auto = keep baked alpha for glassy subjects"
}
}
}
}

View File

@ -1,86 +0,0 @@
import argparse
import json
import os
import subprocess
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[3]
# fork checkout resolution: vendored path first, then the per-box conventional
# clones (m3 staging, m1 home) so existing installs work without re-vendoring.
CANDIDATES = [
ROOT / "vendor" / "trellis2-mlx",
Path.home() / "Documents" / "trellis2-mlx-staging" / "trellis2-jourloy",
Path.home() / "trellis2-mlx",
]
FORK = next((c for c in CANDIDATES if (c / "scripts" / "generate_asset.py").exists()), None)
ap = argparse.ArgumentParser()
ap.add_argument("--input", action="append", default=[])
ap.add_argument("--outdir", required=True)
ap.add_argument("--params", default="{}")
a = ap.parse_args()
p = json.loads(a.params)
if not a.input:
print("ERROR: no input image")
sys.exit(1)
if FORK is None:
print("ERROR: trellis-2-mrp-mlx not found. Run scripts/install_trellis2_mlx.sh")
sys.exit(1)
py = FORK / ".venv" / "bin" / "python"
if not py.exists():
print(f"ERROR: no venv at {FORK}/.venv — run its scripts/setup_macos.sh")
sys.exit(1)
outdir = Path(a.outdir)
# Meshgod-grade PBR by default (metal baker + capped bake), matching the
# trellis_mac lane's 2026-07-22 quality fixes. The fork's schedule bakes the
# FULL-density mesh when no target is passed (75GB peak, 131MB GLB) — always
# pass an explicit cap; RAM-sized default like trellis_mac (500k ≈ fal parity
# on ≥96GB Studios, 200k mtlbvh-safe elsewhere).
faces = p.get("max_bake_faces")
if faces is None:
mem_gb = os.sysconf("SC_PAGE_SIZE") * os.sysconf("SC_PHYS_PAGES") / 1e9
faces = 500000 if mem_gb >= 96 else 200000
cmd = [
str(py), str(FORK / "scripts" / "generate_asset.py"),
str(Path(a.input[0]).resolve()),
"--output-dir", str(outdir.resolve()),
"--backend", "mlx-experimental",
"--baker", str(p.get("baker", "metal")),
"--pipeline-type", str(p.get("pipeline_type", "1024_cascade")),
"--seed", str(p.get("seed", 0)),
"--texture-size", str(p.get("texture_size", 2048)),
"--pbr-decimation-target", str(faces),
"--force",
]
if p.get("steps"):
cmd += ["--steps", str(p["steps"])]
env = os.environ.copy()
env["HF_HUB_DISABLE_XET"] = "1"
# "auto" keeps o_voxel's baked-alpha BLEND detection (glass etc.); default
# opaque — noisy baked alpha renders solid hair as a speckle veil.
if p.get("alpha_mode"):
env["TRELLIS2_ALPHA_MODE"] = str(p["alpha_mode"])
print("+", " ".join(cmd), flush=True)
res = subprocess.run(cmd, cwd=str(FORK), stdout=sys.stdout,
stderr=subprocess.STDOUT, env=env)
if res.returncode != 0:
sys.exit(res.returncode)
glbs = sorted(outdir.rglob("candidate_pbr.glb")) or sorted(outdir.rglob("*.glb"))
if not glbs:
print("ERROR: no GLB produced")
sys.exit(1)
stem = Path(a.input[0]).stem
outputs = [{"path": str(glbs[0]), "name": f"trellis2mlx_{stem}.glb",
"meta": {"tool": "trellis2_mlx", "baker": p.get("baker", "vertex")}}]
(outdir / "result.json").write_text(json.dumps({"outputs": outputs}))
print("done:", glbs[0], flush=True)

View File

@ -1,30 +0,0 @@
{
"id": "trellis2cpp",
"name": "TRELLIS.2 C++ (local, Metal, no venv)",
"category": "mesh-gen",
"description": "Image → PBR GLB via the C++/ggml port of TRELLIS.2 (RobertBeckebans/AI_trellis2cpp) running on Metal. Self-contained binary + GGUFs — NO Python and NO venv, so it cannot hit the 'operator registered, venv absent' failure that takes out the MLX lane on unprepared nodes. Measured on m3ultra vs the same image: 512 fine 42.3s (1.55M tris), 1024 cascade 108.6s (3.78M tris) — about 3.7x faster than trellis2_mlx's 156s for 392k tris, with more geometry. Peak RSS only 9-12 GB. Laptops work too: m2max 99.5s, m4probook 118.0s @512. Keeps one resident server per node (-unload-idle frees the ~15 GB of models when quiet).",
"accepts": [
"image"
],
"produces": [
"model"
],
"resources": "gpu",
"entry": "run.py",
"params_schema": {
"type": "object",
"properties": {
"quality": {
"type": "string",
"enum": [
"coarse",
"512",
"1024",
"1536"
],
"default": "1024",
"description": "coarse = 64^3 marching-cubes preview (seconds). 512 = fine, ~1.5M tris (42s on m3ultra). 1024 = cascade, ~3.8M tris (109s on m3ultra, 301s on an M4 Pro laptop). 1536 = highest tier, loaded but UNTESTED here."
}
}
}
}

View File

@ -1,171 +0,0 @@
"""TRELLIS.2 via the C++/ggml port, on Metal. No Python, no venv.
Why this lane exists alongside trellis2_mlx: that one runs a Python venv per node, and has
repeatedly failed when the farm routed a job to a node whose venv was missing. This is a
self-contained binary + GGUF files, so "operator registered, venv absent" cannot happen if
the install dir is there it runs, and it says so loudly if it is not.
The fine/cascade path is only reachable through the bundled Go server (the CLI examples cover
the coarse path plus mesh utilities), so we keep ONE resident server per node and POST to it.
Resident means the ~15 GB of GGUFs stay cached between jobs; -unload-idle frees them when the
node goes quiet, so a shared box is not permanently down 15 GB.
Measured on the same image (trellis2-bench/anatomy1.jpeg), 512 fine:
m3ultra 42.3s · m2max 99.5s · m4probook 118.0s (peak RSS 9-12 GB)
1024 cascade: m3ultra 108.6s · m4probook 300.7s
vs the trellis2_mlx lane's 156s for 392k tris — this is ~3.7x faster and produces more geometry.
"""
import argparse
import json
import mimetypes
import os
import subprocess
import sys
import time
import urllib.error
import urllib.parse
import urllib.request
import uuid
from pathlib import Path
ROOT = Path(__file__).resolve().parents[3]
PORT = int(os.environ.get("TRELLIS2CPP_PORT", "8743"))
API = f"http://127.0.0.1:{PORT}"
# Install-dir resolution, same spirit as trellis2_mlx: vendored first, then the conventional
# per-box locations, so an existing manual install keeps working without re-vendoring.
CANDIDATES = [
ROOT / "vendor" / "trellis2cpp",
Path.home() / "trellis2cpp",
Path.home() / "trellis2cpp-test" / "src",
]
def resolve_install():
for c in CANDIDATES:
lib = c / "build-shared" / "libtrellis2.dylib"
srv = c / "server" / "trellis2-server"
if lib.exists() and srv.exists() and (c / "ggufs").is_dir():
return c
return None
ap = argparse.ArgumentParser()
ap.add_argument("--input", action="append", default=[])
ap.add_argument("--outdir", required=True)
ap.add_argument("--params", default="{}")
a = ap.parse_args()
p = json.loads(a.params)
outdir = Path(a.outdir)
if not a.input:
print("ERROR: no input image")
sys.exit(1)
INSTALL = resolve_install()
if INSTALL is None:
print("ERROR: trellis2cpp not installed on this node. Expected one of:")
for c in CANDIDATES:
print(f" {c}/ (needs build-shared/libtrellis2.dylib, server/trellis2-server, ggufs/)")
print(" Deploy with wardrobegod tools/deploy_trellis2cpp.sh <user@host>")
sys.exit(1)
quality = str(p.get("quality", "1024"))
if quality not in ("coarse", "512", "1024", "1536"):
print(f"ERROR: quality must be coarse|512|1024|1536, got {quality!r}")
sys.exit(1)
def alive():
try:
urllib.request.urlopen(f"{API}/api/info", timeout=5)
return True
except Exception:
return False
def ensure_server():
if alive():
print("trellis2cpp: server already resident (models stay cached)", flush=True)
return
print(f"trellis2cpp: starting resident server on :{PORT} ...", flush=True)
log = open("/tmp/trellis2cpp-server.log", "ab")
subprocess.Popen(
[str(INSTALL / "server" / "trellis2-server"),
"-lib", str(INSTALL / "build-shared" / "libtrellis2.dylib"),
"-ggufs", str(INSTALL / "ggufs"),
"-store", str(Path.home() / "trellis2cpp" / "generations"),
"-unload-idle", # free ~15 GB when the node goes quiet
"-addr", f"127.0.0.1:{PORT}"],
cwd=str(INSTALL / "server"), stdout=log, stderr=log, start_new_session=True)
for _ in range(120): # model load is slow on a cold start
if alive():
print("trellis2cpp: server up", flush=True)
return
time.sleep(2)
print("ERROR: server did not come up — see /tmp/trellis2cpp-server.log")
sys.exit(1)
def post_image(path):
boundary = uuid.uuid4().hex
ctype = mimetypes.guess_type(path)[0] or "application/octet-stream"
body = (
f"--{boundary}\r\nContent-Disposition: form-data; name=\"quality\"\r\n\r\n{quality}\r\n"
f"--{boundary}\r\nContent-Disposition: form-data; name=\"image\"; "
f"filename=\"{os.path.basename(path)}\"\r\nContent-Type: {ctype}\r\n\r\n"
).encode() + open(path, "rb").read() + f"\r\n--{boundary}--\r\n".encode()
req = urllib.request.Request(
f"{API}/api/generate", data=body,
headers={"Content-Type": f"multipart/form-data; boundary={boundary}"})
with urllib.request.urlopen(req, timeout=120) as r:
return json.load(r)["job"]
ensure_server()
src = str(Path(a.input[0]).resolve())
print(f"+ trellis2cpp quality={quality} <- {src}", flush=True)
job = post_image(src)
print(f"trellis2cpp: job {job}", flush=True)
t0 = time.time()
last = ""
state, info = "running", {}
while time.time() - t0 < 3600:
time.sleep(5)
try:
with urllib.request.urlopen(f"{API}/api/job/{job}", timeout=30) as r:
info = json.load(r)
except Exception:
continue
state = info.get("state", "?")
stages = info.get("stageTimings") or []
if stages and stages[-1]["stage"] != last:
last = stages[-1]["stage"]
print(f" ... {last}", flush=True)
if state in ("done", "error", "failed", "cancelled"):
break
if state != "done":
print(f"ERROR: job {job} ended {state}: {str(info.get('error'))[:300]}")
sys.exit(1)
outdir.mkdir(parents=True, exist_ok=True)
stem = Path(a.input[0]).stem
dest = outdir / f"trellis2cpp_{stem}_{quality}.glb"
with urllib.request.urlopen(f"{API}/api/glb/{job}", timeout=1800) as r:
data = r.read()
if len(data) < 1024 or data[:4] != b"glTF":
print(f"ERROR: server returned {len(data)} bytes that are not a GLB")
sys.exit(1)
dest.write_bytes(data)
secs = (info.get("durationMs") or 0) / 1000.0
print(f"trellis2cpp: {secs:.1f}s on {info.get('device')} -> {dest} ({len(data)/1e6:.1f} MB)",
flush=True)
outputs = [{"path": str(dest), "name": dest.name,
"meta": {"tool": "trellis2cpp", "quality": quality,
"device": info.get("device"), "seconds": round(secs, 1),
"stages": {t["stage"]: t["milliseconds"] for t in (info.get("stageTimings") or [])}}}]
(outdir / "result.json").write_text(json.dumps({"outputs": outputs}))
print("done:", dest, flush=True)

View File

@ -3,80 +3,18 @@
"name": "TRELLIS.2 (local, SOTA)", "name": "TRELLIS.2 (local, SOTA)",
"category": "mesh-gen", "category": "mesh-gen",
"description": "Image → GLB + baked PBR locally via the TRELLIS.2 MPS/Metal port (best open-weights quality on Apple Silicon, ~3-5 min/gen). Weights need a one-time HuggingFace login (facebook/dinov3-vitl16, briaai/RMBG-2.0). Note: 1536 is upstream-only; Mac port maxes at 1024_cascade.", "description": "Image → GLB + baked PBR locally via the TRELLIS.2 MPS/Metal port (best open-weights quality on Apple Silicon, ~3-5 min/gen). Weights need a one-time HuggingFace login (facebook/dinov3-vitl16, briaai/RMBG-2.0). Note: 1536 is upstream-only; Mac port maxes at 1024_cascade.",
"accepts": [ "accepts": ["image"],
"image" "produces": ["model"],
],
"produces": [
"model"
],
"resources": "gpu", "resources": "gpu",
"entry": "run.py", "entry": "run.py",
"python": "vendor/trellis-mac/.venv/bin/python",
"params_schema": { "params_schema": {
"type": "object", "type": "object",
"properties": { "properties": {
"pipeline_type": { "pipeline_type": {"type": "string", "enum": ["512", "1024", "1024_cascade"], "default": "1024", "description": "Resolution tier (1024_cascade = highest quality on Mac)"},
"type": "string", "texture_size": {"type": "integer", "enum": [512, 1024, 2048], "default": 2048, "description": "Baked texture resolution"},
"enum": [ "seed": {"type": "integer", "default": 0, "description": "Random seed"},
"512", "no_texture": {"type": "boolean", "default": false, "description": "Geometry only (skip texture stage, much faster)"}
"1024",
"1024_cascade"
],
"default": "1024",
"description": "Resolution tier (1024_cascade = highest quality on Mac)"
},
"texture_size": {
"type": "integer",
"enum": [
512,
1024,
2048
],
"default": 2048,
"description": "Baked texture resolution"
},
"seed": {
"type": "integer",
"default": 0,
"description": "Random seed"
},
"no_texture": {
"type": "boolean",
"default": false,
"description": "Geometry only (skip texture stage, much faster)"
},
"max_bake_faces": {
"type": "integer",
"enum": [
200000,
500000,
1000000
],
"default": 500000,
"description": "Pre-bake simplify cap. 500k ≈ fal parity, verified crash-free on the M3 Ultra (2026-07-22); drop to 200k on laptops (mtlbvh guard)"
},
"steps": {
"type": "integer",
"default": 0,
"description": "Sampler steps override for all 3 flow phases (0 = pipeline default, 12)"
},
"baker": {
"type": "string",
"enum": [
"",
"vertex"
],
"default": "",
"description": "'vertex' = KDTree vertex-color bake (dark-patch workaround, no PBR maps)"
},
"alpha_mode": {
"type": "string",
"enum": [
"opaque",
"auto"
],
"default": "opaque",
"description": "opaque = force alphaMode OPAQUE in the GLB (fixes speckle-veil hair from noisy baked alpha); auto = keep o_voxel's BLEND detection for glassy subjects"
}
} }
} }
} }

View File

@ -21,18 +21,12 @@ if not a.input:
if not (VENDOR / "generate.py").exists(): if not (VENDOR / "generate.py").exists():
print(f"ERROR: trellis-mac not installed at {VENDOR}. Run scripts/install_trellis_mac.sh") print(f"ERROR: trellis-mac not installed at {VENDOR}. Run scripts/install_trellis_mac.sh")
sys.exit(1) sys.exit(1)
# manifest no longer pins "python" (broke remote dispatch when the venv path
# didn't exist on the worker) — resolve the vendor venv here instead.
VENV_PY = VENDOR / ".venv" / "bin" / "python"
if not VENV_PY.exists():
print(f"ERROR: no venv at {VENV_PY} — run scripts/install_trellis_mac.sh")
sys.exit(1)
outdir = Path(a.outdir) outdir = Path(a.outdir)
out_stem = outdir / f"trellis2_{Path(a.input[0]).stem}" out_stem = outdir / f"trellis2_{Path(a.input[0]).stem}"
cmd = [ cmd = [
str(VENV_PY), "generate.py", str(Path(a.input[0]).resolve()), sys.executable, "generate.py", str(Path(a.input[0]).resolve()),
"--pipeline-type", str(p.get("pipeline_type", "1024")), "--pipeline-type", str(p.get("pipeline_type", "1024")),
"--texture-size", str(p.get("texture_size", 2048)), "--texture-size", str(p.get("texture_size", 2048)),
"--seed", str(p.get("seed", 0)), "--seed", str(p.get("seed", 0)),
@ -40,8 +34,6 @@ cmd = [
] ]
if p.get("no_texture"): if p.get("no_texture"):
cmd.append("--no-texture") cmd.append("--no-texture")
if p.get("steps"):
cmd += ["--steps", str(p["steps"])]
env = os.environ.copy() env = os.environ.copy()
# xet chunked downloader fails on these BFL/Meta repos ("Unable to parse string # xet chunked downloader fails on these BFL/Meta repos ("Unable to parse string
@ -51,22 +43,6 @@ env["HF_HUB_DISABLE_XET"] = "1"
env["KMP_DUPLICATE_LIB_OK"] = "TRUE" env["KMP_DUPLICATE_LIB_OK"] = "TRUE"
env.setdefault("OMP_NUM_THREADS", "1") env.setdefault("OMP_NUM_THREADS", "1")
env.setdefault("MKL_NUM_THREADS", "1") env.setdefault("MKL_NUM_THREADS", "1")
# Pre-bake simplify cap. The 200k generate.py default is a laptop-derived
# mtlbvh guard that costs most of the mesh detail (2.2M-tri decode → 95k GLB
# vs fal's 495k). 500k verified crash-free on the M3 Ultra Studio 2026-07-22.
# The server passes only client-sent params (manifest defaults are UI-only),
# so default here, sized by node RAM: big-memory Studios get the quality cap.
if p.get("max_bake_faces") is not None:
env["TRELLIS2_MAX_BAKE_FACES"] = str(p["max_bake_faces"])
else:
mem_gb = os.sysconf("SC_PAGE_SIZE") * os.sysconf("SC_PHYS_PAGES") / 1e9
env.setdefault("TRELLIS2_MAX_BAKE_FACES", "500000" if mem_gb >= 96 else "200000")
if p.get("baker"):
env["TRELLIS2_BAKER"] = str(p["baker"])
# "auto" keeps o_voxel's baked-alpha BLEND detection (glass etc.); default
# opaque — the Metal baker's noisy alpha turns solid hair into speckles.
if p.get("alpha_mode"):
env["TRELLIS2_ALPHA_MODE"] = str(p["alpha_mode"])
print("+", " ".join(cmd), flush=True) print("+", " ".join(cmd), flush=True)
print("(first run downloads TRELLIS.2-4B + dinov3 + RMBG-2.0 weights)", flush=True) print("(first run downloads TRELLIS.2-4B + dinov3 + RMBG-2.0 weights)", flush=True)

View File

@ -1,19 +0,0 @@
{
"id": "vidgod_index",
"name": "Clip Library Index (VIDGOD)",
"category": "video",
"description": "Index a video into the node's VIDGOD clip library: PySceneDetect shot split + mlx-whisper transcript + thumbnails into library/clips.sqlite. Library lives on the node that runs the job (canonical: ultra) — search it there with vg-find. Returns an index summary JSON.",
"accepts": ["video"],
"produces": ["json"],
"resources": "cpu",
"entry": "run.py",
"requires_path": "~/Documents/VIDGOD/bin/vg-index",
"params_schema": {
"type": "object",
"properties": {
"no_whisper": {"type": "boolean", "default": false, "description": "Skip transcription (shots + thumbnails only)"},
"min_shot": {"type": "number", "default": 0.5, "description": "Minimum shot length, seconds"},
"keep_copy": {"type": "boolean", "default": true, "description": "Keep a copy of the video in VIDGOD/library/ingest/ so library entries outlive the job workspace"}
}
}
}

View File

@ -1,61 +0,0 @@
"""vidgod_index — MODELBEAST wrapper around VIDGOD's vg-index.
Stdlib-only: runs under system python3, shells out to ~/Documents/VIDGOD/bin/vg-index
(which brings its own venv). By default the video is copied into
VIDGOD/library/ingest/ first so library entries point at a path that outlives
the job workspace.
"""
import argparse
import json
import shutil
import sqlite3
import subprocess
import sys
from pathlib import Path
ap = argparse.ArgumentParser()
ap.add_argument("--input", required=True)
ap.add_argument("--outdir", required=True)
ap.add_argument("--params", default="{}")
args = ap.parse_args()
p = json.loads(args.params)
vg = Path.home() / "Documents/VIDGOD"
vg_index = vg / "bin/vg-index"
if not vg_index.exists():
sys.exit("VIDGOD not installed on this node — clone gitea monster/vidgod to "
"~/Documents/VIDGOD and run setup/setup_venvs.sh")
src = Path(args.input)
if p.get("keep_copy", True):
ingest = vg / "library/ingest"
ingest.mkdir(parents=True, exist_ok=True)
target = ingest / src.name
if not (target.exists() and target.stat().st_size == src.stat().st_size):
shutil.copy2(src, target)
src = target
cmd = [str(vg_index), str(src), "--min-shot", str(p.get("min_shot", 0.5))]
if p.get("no_whisper", False):
cmd.append("--no-whisper")
print("+", " ".join(cmd), flush=True)
subprocess.run(cmd, check=True, stderr=sys.stdout)
db = sqlite3.connect(vg / "library/clips.sqlite", timeout=30)
row = db.execute(
"""SELECT v.id, v.duration, COUNT(DISTINCT s.id), COUNT(DISTINCT g.id)
FROM videos v LEFT JOIN shots s ON s.video_id=v.id
LEFT JOIN segs g ON g.video_id=v.id WHERE v.path=? GROUP BY v.id""",
(str(src),)).fetchone()
if not row:
sys.exit(f"indexing ran but {src} not found in library db")
summary = {"indexed_path": str(src), "duration_s": row[1],
"shots": row[2], "dialogue_segments": row[3],
"search_hint": "vg-find <words> on this node"}
out = Path(args.outdir) / "index_summary.json"
out.write_text(json.dumps(summary, indent=2))
(Path(args.outdir) / "result.json").write_text(json.dumps(
{"outputs": [{"path": "index_summary.json", "name": "index_summary.json"}],
"summary": summary}))
print("done:", json.dumps(summary), flush=True)

View File

@ -1,22 +0,0 @@
{
"id": "vidgod_roto",
"name": "Actor Cutout (VIDGOD)",
"category": "video",
"description": "Video + click point → tracked actor cutout via SAM 2.1 + MatAnyone (VIDGOD stack). Returns ProRes 4444 alpha, grayscale matte, and a green-screen preview mp4. Needs the vidgod repo set up on the node (~/Documents/VIDGOD, gitea monster/vidgod). Runs on the node's own VIDGOD venvs — no MODELBEAST venv.",
"accepts": ["video"],
"produces": ["video"],
"resources": "gpu",
"entry": "run.py",
"requires_path": "~/Documents/VIDGOD/bin/vg-roto",
"params_schema": {
"type": "object",
"properties": {
"points": {"type": "string", "default": "", "description": "Positive clicks 'x,y' or 'x,y;x,y' on the prompt frame"},
"neg_points": {"type": "string", "default": "", "description": "Negative clicks 'x,y;x,y' (regions to exclude)"},
"box": {"type": "string", "default": "", "description": "Alternative to points: 'x1,y1,x2,y2' around the target"},
"frame": {"type": "integer", "default": 0, "description": "Frame index to prompt on (clip is trimmed from here in matte mode)"},
"mode": {"type": "string", "enum": ["matte", "mask"], "default": "matte", "description": "matte = soft alpha (MatAnyone), mask = hard binary (SAM2 propagation)"},
"max_size": {"type": "integer", "default": -1, "description": "Downscale min side for inference (-1 = native, 720 = faster)"}
}
}
}

View File

@ -1,58 +0,0 @@
"""vidgod_roto — MODELBEAST wrapper around VIDGOD's vg-roto.
Stdlib-only on purpose: runs under the node's system python3 and shells out to
~/Documents/VIDGOD/bin/vg-roto, which brings its own venv (SAM 2.1 + MatAnyone).
"""
import argparse
import json
import shutil
import subprocess
import sys
from pathlib import Path
ap = argparse.ArgumentParser()
ap.add_argument("--input", required=True)
ap.add_argument("--outdir", required=True)
ap.add_argument("--params", default="{}")
args = ap.parse_args()
p = json.loads(args.params)
vg_roto = Path.home() / "Documents/VIDGOD/bin/vg-roto"
if not vg_roto.exists():
sys.exit("VIDGOD not installed on this node — clone gitea monster/vidgod to "
"~/Documents/VIDGOD and run setup/setup_venvs.sh")
outdir = Path(args.outdir)
workdir = outdir / "roto"
cmd = [str(vg_roto), args.input, "--out", str(workdir),
"--frame", str(int(p.get("frame", 0))),
"--mode", p.get("mode", "matte"),
"--max-size", str(int(p.get("max_size", -1)))]
for pt in str(p.get("points", "")).split(";"):
if pt.strip():
cmd += ["--point", pt.strip()]
for pt in str(p.get("neg_points", "")).split(";"):
if pt.strip():
cmd += ["--neg", pt.strip()]
if str(p.get("box", "")).strip():
cmd += ["--box", str(p["box"]).strip()]
print("+", " ".join(cmd), flush=True)
subprocess.run(cmd, check=True, stderr=sys.stdout)
# vg-roto writes <stem>_{alpha.mov,matte.mov,green.mp4} into workdir; lift them
# to outdir top-level so the runner registers them as job outputs.
outputs = []
for f in sorted(workdir.glob("*_*.m*")):
dest = outdir / f.name
shutil.move(str(f), dest)
outputs.append({"path": f.name, "name": f.name})
if not outputs:
sys.exit("vg-roto produced no outputs")
shutil.rmtree(workdir, ignore_errors=True)
(outdir / "result.json").write_text(json.dumps({
"outputs": outputs,
"summary": {"mode": p.get("mode", "matte"), "files": [o["name"] for o in outputs]},
}))
print("done:", ", ".join(o["name"] for o in outputs), flush=True)

View File

@ -1,8 +1,8 @@
{ {
"id": "wan_video", "id": "wan_video",
"name": "Wan 2.2 Video (local, MLX)", "name": "Wan 2.2 Video (local, ComfyUI)",
"category": "video-gen", "category": "video-gen",
"description": "Prompt → MP4 (text-to-video), or image + prompt → MP4 (image-to-video), via Wan 2.2 TI2V-5B on mlx-video (native MLX/Metal). 24fps, native 1280x704. Defaults are deliberately small (832x480, 49 frames ≈ 2s ≈ 130s to render) — scale up once you know the node's speed. Weights: vendor/mlx-video-models/Wan2.2-TI2V-5B-mlx-q8 (~18GB, 8-bit transformer + bf16 UMT5 + VAE). Replaced the ComfyUI/PyTorch-MPS backend on 2026-08-02: that path was 1.6x slower and emitted garbage frames while reporting success.", "description": "Prompt → MP4 (text-to-video), or image + prompt → MP4 (image-to-video), via Wan 2.2 TI2V-5B on the resident ComfyUI (Metal). 24fps, native 1280x704. Defaults are deliberately small (832x480, 49 frames ≈ 2s) — scale up once you know the node's speed. Weights: vendor/comfyui/models (wan2.2_ti2v_5B_fp16 + umt5_xxl_fp8 + wan2.2_vae, ~18GB).",
"accepts": ["image"], "accepts": ["image"],
"produces": ["video"], "produces": ["video"],
"resources": "gpu", "resources": "gpu",

View File

@ -1,31 +1,25 @@
"""Wan 2.2 TI2V-5B text/image→video via mlx-video (native MLX, Apple Silicon). """Wan 2.2 TI2V-5B text/image→video via the resident ComfyUI (same pattern as
comfyui_sd: pure stdlib, ComfyUI's venv does the heavy lifting, model stays
Replaces the previous ComfyUI/PyTorch-MPS path, which produced structurally-valid cached in RAM between jobs). Frames come back as PNGs; ffmpeg muxes the MP4."""
but visually garbage MP4s on every job from 2026-07-17 onward and still reported
success. Benchmarked 2026-08-02 at identical settings (832x480, 49f, 20 steps,
cfg 5.0, seed 1234): MLX q8 132s and clean, ComfyUI fp16 216s and garbage.
Same operator id, same params, same output contract (outdir/wan_{seed}.mp4) the
queue and any downstream consumer see no change. Weights are the q8 MLX conversion
in vendor/mlx-video-models; the generator holds them in unified memory itself, so
there is no resident-server handshake and no PYTORCH_MPS_HIGH_WATERMARK_RATIO
tuning (MLX manages its own allocation).
"""
import argparse import argparse
import json import json
import mimetypes
import os import os
import random import random
import subprocess import subprocess
import sys import sys
import time import time
import urllib.parse
import urllib.request
import uuid
from pathlib import Path from pathlib import Path
# job env has a minimal PATH; we shell out to ffprobe for output validation # job env has a minimal PATH; we shell out to ffmpeg for the mux
os.environ["PATH"] = "/opt/homebrew/bin:/usr/local/bin:" + os.environ.get("PATH", "") os.environ["PATH"] = "/opt/homebrew/bin:/usr/local/bin:" + os.environ.get("PATH", "")
ROOT = Path(__file__).resolve().parents[3] ROOT = Path(__file__).resolve().parents[3]
GEN = ROOT / "venvs" / "mlxvideo" / "bin" / "mlx_video.wan_2.generate" COMFY = ROOT / "vendor" / "comfyui"
MODEL_DIR = ROOT / "vendor" / "mlx-video-models" / "Wan2.2-TI2V-5B-mlx-q8" API = "http://127.0.0.1:8188"
FPS = 24 # Wan 2.2 FPS = 24 # Wan 2.2
ap = argparse.ArgumentParser() ap = argparse.ArgumentParser()
@ -40,101 +34,141 @@ prompt = (p.get("prompt") or "").strip()
if not prompt: if not prompt:
print("ERROR: prompt is required") print("ERROR: prompt is required")
sys.exit(1) sys.exit(1)
if not GEN.exists(): if not (COMFY / "main.py").exists():
print(f"ERROR: mlx-video not installed at {GEN}. Run scripts/install_mlx_video.sh") print(f"ERROR: ComfyUI not installed at {COMFY}. Run scripts/install_comfyui.sh")
sys.exit(1) sys.exit(1)
if not (MODEL_DIR / "model.safetensors").exists(): model_file = COMFY / "models" / "diffusion_models" / "wan2.2_ti2v_5B_fp16.safetensors"
print(f"ERROR: MLX weights missing at {MODEL_DIR} — see manifest description") if not model_file.exists():
print("ERROR: Wan weights missing — see manifest description for the three files")
sys.exit(1) sys.exit(1)
def alive():
try:
urllib.request.urlopen(f"{API}/", timeout=3)
return True
except Exception:
return False
def ensure_server():
if alive():
print("comfyui: already resident", flush=True)
return
print("comfyui: starting resident server ...", flush=True)
log = open("/tmp/comfyui.log", "ab")
subprocess.Popen(
[str(COMFY / ".venv" / "bin" / "python"), "main.py", "--port", "8188"],
cwd=str(COMFY), stdout=log, stderr=log, start_new_session=True)
for _ in range(90):
if alive():
print("comfyui: up", flush=True)
return
time.sleep(2)
print("ERROR: ComfyUI did not come up — see /tmp/comfyui.log")
sys.exit(1)
def upload_image(path):
"""POST multipart to /upload/image, return server-side filename."""
boundary = uuid.uuid4().hex
fname = Path(path).name
ctype = mimetypes.guess_type(fname)[0] or "application/octet-stream"
body = (f"--{boundary}\r\nContent-Disposition: form-data; name=\"image\"; "
f"filename=\"{fname}\"\r\nContent-Type: {ctype}\r\n\r\n").encode()
body += Path(path).read_bytes()
body += f"\r\n--{boundary}--\r\n".encode()
req = urllib.request.Request(f"{API}/upload/image", data=body, headers={
"Content-Type": f"multipart/form-data; boundary={boundary}"})
return json.load(urllib.request.urlopen(req))["name"]
def build(seed, start_image):
length = int(p.get("length", 49)) length = int(p.get("length", 49))
if length % 4 != 1: if length % 4 != 1:
length = (length // 4) * 4 + 1 # model requires 4n+1 frames length = (length // 4) * 4 + 1 # model requires 4n+1 frames
width, height = int(p.get("width", 832)), int(p.get("height", 480)) lat = {"vae": ["v", 0], "width": int(p.get("width", 832)),
"height": int(p.get("height", 480)), "length": length, "batch_size": 1}
g = {
"u": {"class_type": "UNETLoader", "inputs": {
"unet_name": "wan2.2_ti2v_5B_fp16.safetensors", "weight_dtype": "default"}},
"c": {"class_type": "CLIPLoader", "inputs": {
"clip_name": "umt5_xxl_fp8_e4m3fn_scaled.safetensors", "type": "wan",
"device": "default"}},
"v": {"class_type": "VAELoader", "inputs": {"vae_name": "wan2.2_vae.safetensors"}},
"mo": {"class_type": "ModelSamplingSD3", "inputs": {"model": ["u", 0], "shift": 8.0}},
"pos": {"class_type": "CLIPTextEncode", "inputs": {"text": prompt, "clip": ["c", 0]}},
"neg": {"class_type": "CLIPTextEncode", "inputs": {
"text": p.get("negative", "blurry, distorted, low quality, static image, watermark, text"),
"clip": ["c", 0]}},
"lat": {"class_type": "Wan22ImageToVideoLatent", "inputs": lat},
"ks": {"class_type": "KSampler", "inputs": {
"seed": seed, "steps": int(p.get("steps", 20)), "cfg": float(p.get("cfg", 5.0)),
"sampler_name": "uni_pc", "scheduler": "simple", "denoise": 1.0,
"model": ["mo", 0], "positive": ["pos", 0], "negative": ["neg", 0],
"latent_image": ["lat", 0]}},
"dec": {"class_type": "VAEDecode", "inputs": {"samples": ["ks", 0], "vae": ["v", 0]}},
"sav": {"class_type": "SaveImage", "inputs": {
"filename_prefix": "mb_wan", "images": ["dec", 0]}},
}
if start_image:
g["img"] = {"class_type": "LoadImage", "inputs": {"image": start_image}}
lat["start_image"] = ["img", 0]
return g
ensure_server()
seed = int(p.get("seed", -1)) seed = int(p.get("seed", -1))
if seed < 0: if seed < 0:
seed = random.randint(0, 2**31 - 1) seed = random.randint(0, 2**31 - 1)
start_image = upload_image(a.input[0]) if a.input else None
start_image = a.input[0] if a.input else None
if start_image and not Path(start_image).exists():
print(f"ERROR: input image not found: {start_image}")
sys.exit(1)
out_mp4 = outdir / f"wan_{seed}.mp4"
cmd = [
str(GEN),
"--model-dir", str(MODEL_DIR),
"--prompt", prompt,
"--negative-prompt", p.get(
"negative", "blurry, distorted, low quality, static image, watermark, text"),
"--width", str(width),
"--height", str(height),
"--num-frames", str(length),
"--steps", str(int(p.get("steps", 20))),
"--guide-scale", str(float(p.get("cfg", 5.0))),
"--seed", str(seed),
"--output-path", str(out_mp4),
]
if start_image:
cmd += ["--image", start_image]
print(f"seed={seed} mode={'i2v' if start_image else 't2v'} " print(f"seed={seed} mode={'i2v' if start_image else 't2v'} "
f"{width}x{height}x{length}f backend=mlx-q8", flush=True) f"{p.get('width', 832)}x{p.get('height', 480)}x{p.get('length', 49)}f", flush=True)
body = json.dumps({"prompt": build(seed, start_image)}).encode()
req = urllib.request.Request(f"{API}/prompt", data=body,
headers={"Content-Type": "application/json"})
try:
pid = json.load(urllib.request.urlopen(req))["prompt_id"]
except urllib.error.HTTPError as e:
print(f"ERROR: ComfyUI rejected the workflow: {e.read().decode()[:600]}")
sys.exit(1)
t0 = time.time() t0 = time.time()
r = subprocess.run(cmd, capture_output=True, text=True) images = []
while time.time() - t0 < 5400:
h = json.load(urllib.request.urlopen(f"{API}/history/{pid}"))
if pid in h:
st = h[pid].get("status", {})
if st.get("status_str") == "error":
print("ERROR: generation failed — check /tmp/comfyui.log")
print(json.dumps(st)[:400])
sys.exit(1)
if h[pid].get("outputs"):
for node in h[pid]["outputs"].values():
images += node.get("images", [])
break
time.sleep(5)
# NOTE: do not gate on the exit code alone. Under memory pressure the generator if not images:
# has been observed finishing the render, writing a complete and valid MP4, and print("ERROR: no frames produced (timeout after 90min)")
# only then being SIGKILLed during interpreter teardown (rc=-9, "leaked
# semaphore" warning). Judge the artefact on disk first; the exit code is only
# authoritative when there is nothing usable to inspect.
# Echo the generator's own stage timings (T5 / denoise / VAE) into the job log.
import re
ANSI = re.compile(r"\x1b\[[0-9;]*m") # generator colourises; job logs are plain text
for line in (r.stdout or "").splitlines():
s = ANSI.sub("", line).strip()
if any(k in s for k in ("T5 encoding", "Models loaded", "Denoising:", "VAE decode", "Total time")):
if "it/s" not in s and "s/it" not in s: # skip the tqdm bar
print(f" {s}", flush=True)
# --- output validation ---------------------------------------------------
# The old ComfyUI path marked six garbage jobs "done" because nothing here ever
# checked the result. These are structural checks only (a real content-quality
# check would need a perceptual model and would risk false failures on
# legitimately flat footage) — but they do catch the silent-empty-output case.
if not out_mp4.exists() or out_mp4.stat().st_size < 50_000:
print(f"ERROR: no usable video produced at {out_mp4}")
if r.returncode != 0:
print(f" mlx-video exited {r.returncode}")
print((r.stderr or r.stdout)[-1200:])
sys.exit(1) sys.exit(1)
probe = subprocess.run( import tempfile
["ffprobe", "-v", "error", "-select_streams", "v:0", "-count_packets", frames_dir = Path(tempfile.mkdtemp(prefix="wan_frames_")) # outside outdir so frames don't register as assets
"-show_entries", "stream=width,height,nb_read_packets", "-of", "csv=p=0", str(out_mp4)], for i, im in enumerate(images):
capture_output=True, text=True) q = urllib.parse.urlencode({"filename": im["filename"],
if probe.returncode != 0: "subfolder": im.get("subfolder", ""),
print(f"ERROR: output is not a decodable video: {probe.stderr[-400:]}") "type": im.get("type", "output")})
sys.exit(1) (frames_dir / f"f{i:05d}.png").write_bytes(
urllib.request.urlopen(f"{API}/view?{q}").read())
try: out_mp4 = outdir / f"wan_{seed}.mp4"
gw, gh, gframes = (int(x) for x in probe.stdout.strip().split(",")[:3]) r = subprocess.run(["ffmpeg", "-y", "-framerate", str(FPS),
except ValueError: "-i", str(frames_dir / "f%05d.png"),
print(f"ERROR: could not parse ffprobe output: {probe.stdout!r}") "-c:v", "libx264", "-pix_fmt", "yuv420p", "-crf", "18",
str(out_mp4)], capture_output=True, text=True)
if r.returncode != 0 or not out_mp4.exists():
print("ERROR: ffmpeg mux failed:", r.stderr[-800:])
sys.exit(1) sys.exit(1)
print(f"done: {len(images)} frames → {out_mp4.name} in {time.time() - t0:.1f}s", flush=True)
if (gw, gh) != (width, height):
print(f"ERROR: dimension mismatch — asked {width}x{height}, got {gw}x{gh}")
sys.exit(1)
if gframes < length:
print(f"ERROR: truncated output — asked {length} frames, got {gframes}")
sys.exit(1)
if r.returncode != 0:
print(f"note: output validated OK despite generator exit {r.returncode} "
f"(killed at teardown, render was complete)", flush=True)
print(f"done: {gframes} frames @ {gw}x{gh}{out_mp4.name} "
f"in {time.time() - t0:.1f}s", flush=True)

View File

@ -1,174 +0,0 @@
"""Wan 2.2 TI2V-5B text/image→video via the resident ComfyUI (same pattern as
comfyui_sd: pure stdlib, ComfyUI's venv does the heavy lifting, model stays
cached in RAM between jobs). Frames come back as PNGs; ffmpeg muxes the MP4."""
import argparse
import json
import mimetypes
import os
import random
import subprocess
import sys
import time
import urllib.parse
import urllib.request
import uuid
from pathlib import Path
# job env has a minimal PATH; we shell out to ffmpeg for the mux
os.environ["PATH"] = "/opt/homebrew/bin:/usr/local/bin:" + os.environ.get("PATH", "")
ROOT = Path(__file__).resolve().parents[3]
COMFY = ROOT / "vendor" / "comfyui"
API = "http://127.0.0.1:8188"
FPS = 24 # Wan 2.2
ap = argparse.ArgumentParser()
ap.add_argument("--input", action="append", default=[])
ap.add_argument("--outdir", required=True)
ap.add_argument("--params", default="{}")
a = ap.parse_args()
p = json.loads(a.params)
outdir = Path(a.outdir)
prompt = (p.get("prompt") or "").strip()
if not prompt:
print("ERROR: prompt is required")
sys.exit(1)
if not (COMFY / "main.py").exists():
print(f"ERROR: ComfyUI not installed at {COMFY}. Run scripts/install_comfyui.sh")
sys.exit(1)
model_file = COMFY / "models" / "diffusion_models" / "wan2.2_ti2v_5B_fp16.safetensors"
if not model_file.exists():
print("ERROR: Wan weights missing — see manifest description for the three files")
sys.exit(1)
def alive():
try:
urllib.request.urlopen(f"{API}/", timeout=3)
return True
except Exception:
return False
def ensure_server():
if alive():
print("comfyui: already resident", flush=True)
return
print("comfyui: starting resident server ...", flush=True)
log = open("/tmp/comfyui.log", "ab")
subprocess.Popen(
[str(COMFY / ".venv" / "bin" / "python"), "main.py", "--port", "8188"],
cwd=str(COMFY), stdout=log, stderr=log, start_new_session=True)
for _ in range(90):
if alive():
print("comfyui: up", flush=True)
return
time.sleep(2)
print("ERROR: ComfyUI did not come up — see /tmp/comfyui.log")
sys.exit(1)
def upload_image(path):
"""POST multipart to /upload/image, return server-side filename."""
boundary = uuid.uuid4().hex
fname = Path(path).name
ctype = mimetypes.guess_type(fname)[0] or "application/octet-stream"
body = (f"--{boundary}\r\nContent-Disposition: form-data; name=\"image\"; "
f"filename=\"{fname}\"\r\nContent-Type: {ctype}\r\n\r\n").encode()
body += Path(path).read_bytes()
body += f"\r\n--{boundary}--\r\n".encode()
req = urllib.request.Request(f"{API}/upload/image", data=body, headers={
"Content-Type": f"multipart/form-data; boundary={boundary}"})
return json.load(urllib.request.urlopen(req))["name"]
def build(seed, start_image):
length = int(p.get("length", 49))
if length % 4 != 1:
length = (length // 4) * 4 + 1 # model requires 4n+1 frames
lat = {"vae": ["v", 0], "width": int(p.get("width", 832)),
"height": int(p.get("height", 480)), "length": length, "batch_size": 1}
g = {
"u": {"class_type": "UNETLoader", "inputs": {
"unet_name": "wan2.2_ti2v_5B_fp16.safetensors", "weight_dtype": "default"}},
"c": {"class_type": "CLIPLoader", "inputs": {
"clip_name": "umt5_xxl_fp8_e4m3fn_scaled.safetensors", "type": "wan",
"device": "default"}},
"v": {"class_type": "VAELoader", "inputs": {"vae_name": "wan2.2_vae.safetensors"}},
"mo": {"class_type": "ModelSamplingSD3", "inputs": {"model": ["u", 0], "shift": 8.0}},
"pos": {"class_type": "CLIPTextEncode", "inputs": {"text": prompt, "clip": ["c", 0]}},
"neg": {"class_type": "CLIPTextEncode", "inputs": {
"text": p.get("negative", "blurry, distorted, low quality, static image, watermark, text"),
"clip": ["c", 0]}},
"lat": {"class_type": "Wan22ImageToVideoLatent", "inputs": lat},
"ks": {"class_type": "KSampler", "inputs": {
"seed": seed, "steps": int(p.get("steps", 20)), "cfg": float(p.get("cfg", 5.0)),
"sampler_name": "uni_pc", "scheduler": "simple", "denoise": 1.0,
"model": ["mo", 0], "positive": ["pos", 0], "negative": ["neg", 0],
"latent_image": ["lat", 0]}},
"dec": {"class_type": "VAEDecode", "inputs": {"samples": ["ks", 0], "vae": ["v", 0]}},
"sav": {"class_type": "SaveImage", "inputs": {
"filename_prefix": "mb_wan", "images": ["dec", 0]}},
}
if start_image:
g["img"] = {"class_type": "LoadImage", "inputs": {"image": start_image}}
lat["start_image"] = ["img", 0]
return g
ensure_server()
seed = int(p.get("seed", -1))
if seed < 0:
seed = random.randint(0, 2**31 - 1)
start_image = upload_image(a.input[0]) if a.input else None
print(f"seed={seed} mode={'i2v' if start_image else 't2v'} "
f"{p.get('width', 832)}x{p.get('height', 480)}x{p.get('length', 49)}f", flush=True)
body = json.dumps({"prompt": build(seed, start_image)}).encode()
req = urllib.request.Request(f"{API}/prompt", data=body,
headers={"Content-Type": "application/json"})
try:
pid = json.load(urllib.request.urlopen(req))["prompt_id"]
except urllib.error.HTTPError as e:
print(f"ERROR: ComfyUI rejected the workflow: {e.read().decode()[:600]}")
sys.exit(1)
t0 = time.time()
images = []
while time.time() - t0 < 5400:
h = json.load(urllib.request.urlopen(f"{API}/history/{pid}"))
if pid in h:
st = h[pid].get("status", {})
if st.get("status_str") == "error":
print("ERROR: generation failed — check /tmp/comfyui.log")
print(json.dumps(st)[:400])
sys.exit(1)
if h[pid].get("outputs"):
for node in h[pid]["outputs"].values():
images += node.get("images", [])
break
time.sleep(5)
if not images:
print("ERROR: no frames produced (timeout after 90min)")
sys.exit(1)
import tempfile
frames_dir = Path(tempfile.mkdtemp(prefix="wan_frames_")) # outside outdir so frames don't register as assets
for i, im in enumerate(images):
q = urllib.parse.urlencode({"filename": im["filename"],
"subfolder": im.get("subfolder", ""),
"type": im.get("type", "output")})
(frames_dir / f"f{i:05d}.png").write_bytes(
urllib.request.urlopen(f"{API}/view?{q}").read())
out_mp4 = outdir / f"wan_{seed}.mp4"
r = subprocess.run(["ffmpeg", "-y", "-framerate", str(FPS),
"-i", str(frames_dir / "f%05d.png"),
"-c:v", "libx264", "-pix_fmt", "yuv420p", "-crf", "18",
str(out_mp4)], capture_output=True, text=True)
if r.returncode != 0 or not out_mp4.exists():
print("ERROR: ffmpeg mux failed:", r.stderr[-800:])
sys.exit(1)
print(f"done: {len(images)} frames → {out_mp4.name} in {time.time() - t0:.1f}s", flush=True)

View File

@ -14,7 +14,6 @@ Omit "operators" to allow every gpu op. The implicit local node is always presen
import asyncio import asyncio
import json import json
import shlex import shlex
import socket
import time import time
from pathlib import Path from pathlib import Path
@ -27,11 +26,7 @@ _HEALTH_TTL = 30.0
def load_gpu_nodes() -> list[dict]: def load_gpu_nodes() -> list[dict]:
"""Local node first, then remote workers from nodes.json.""" """Local node first, then remote workers from nodes.json."""
# Name the local node after the host, not "local" — job rows and the queue UI nodes = [{"name": "local", "local": True, "busy": False}]
# should say "m3ultra", not a placeholder that looks like every other box.
# Only the "local" flag is load-bearing; nothing keys off the name.
nodes = [{"name": socket.gethostname().split(".")[0] or "local",
"local": True, "busy": False}]
f = db.ROOT / "nodes.json" f = db.ROOT / "nodes.json"
if f.exists(): if f.exists():
try: try:
@ -44,14 +39,8 @@ def load_gpu_nodes() -> list[dict]:
return nodes return nodes
def node_supports(node: dict, op_id: str, op: dict | None = None) -> bool: def node_supports(node: dict, op_id: str) -> bool:
if node.get("local"): if node.get("local"):
# manifest "requires_path" pins an op to nodes that actually carry its
# stack (e.g. vidgod_* need ~/Documents/VIDGOD) — gates the local node;
# remotes are gated by their nodes.json allowlist as before.
req = (op or {}).get("requires_path")
if req and not Path(req).expanduser().exists():
return False
return True return True
allow = node.get("operators") allow = node.get("operators")
return (not allow) or (op_id in allow) return (not allow) or (op_id in allow)

View File

@ -28,18 +28,6 @@ JOBS_DIR = db.DATA / "jobs"
# primary-only semaphore because the cloud API keys live only on the primary. # primary-only semaphore because the cloud API keys live only on the primary.
LANE_LIMITS = {"cpu": 3, "net": 6} LANE_LIMITS = {"cpu": 3, "net": 6}
# Operators cheap enough (CPU-light, small assets) to keep OFF the big GPUs so
# qwen-image-edit / trellis / seedvr2 always find an ultra free. For these the
# dispatcher prefers the home helper on a direct link; every other operator keeps
# the default local-first order (i.e. still prefers the fast ultras).
LIGHT_OPS = {"bg_remove_local", "ffmpeg_frames", "ffprobe"}
# Lower rank = tried first for a LIGHT_OP. m4 (m4pro) is the home helper on a direct
# ~8ms link; the two ultras (local=m3ultra, m1=ultra) come next so a busy m4pro still
# falls back fast; the work-site machines are Sydney-DERP-relayed from the farm
# controller, so they are the last resort.
# the primary is named by hostname since 2026-08-05 (was "local"); keep both keys
_LIGHT_ORDER = {"m4": 0, "m1": 1, "m3ultra": 2, "local": 2, "studio": 3, "m4mini": 4, "mini": 5}
class Runner: class Runner:
def __init__(self): def __init__(self):
@ -183,10 +171,6 @@ class Runner:
lane = self.lane_of(job["operator"]) lane = self.lane_of(job["operator"])
if lane in ("gpu", "cpu"): # pooled lanes — distribute across the node pool if lane in ("gpu", "cpu"): # pooled lanes — distribute across the node pool
node = await self._acquire_node(job["operator"], lane) node = await self._acquire_node(job["operator"], lane)
# Persist placement: the pool picks a node in memory, so without this the
# job row has no record of where it ran and the queue UI attributes
# everything to the primary.
await self._update(con, job_id, node=node.get("name") or node.get("ssh"))
try: try:
if job_id in self.cancelled: if job_id in self.cancelled:
self.cancelled.discard(job_id) self.cancelled.discard(job_id)
@ -222,14 +206,10 @@ class Runner:
async def _acquire_node(self, op_id: str, lane: str) -> dict: async def _acquire_node(self, op_id: str, lane: str) -> dict:
"""Return the first node with free capacity in this lane that supports the """Return the first node with free capacity in this lane that supports the
operator (local always does; a remote must list it + be reachable). Polls. operator (local always does; a remote must list it + be reachable). Polls."""
LIGHT_OPS prefer the home helper (m4pro) so the ultras stay free for heavy work."""
order = self.gpu_nodes
if op_id in LIGHT_OPS:
order = sorted(order, key=lambda n: _LIGHT_ORDER.get(n.get("name", ""), 9))
while True: while True:
for n in order: for n in self.gpu_nodes:
if not remote.node_supports(n, op_id, self.operators.get(op_id)): if not remote.node_supports(n, op_id):
continue continue
cap = self._node_capacity(n, lane) cap = self._node_capacity(n, lane)
if cap <= 0 or n["inflight"].get(lane, 0) >= cap: if cap <= 0 or n["inflight"].get(lane, 0) >= cap:
@ -361,25 +341,11 @@ class Runner:
p = Path(out["path"]) p = Path(out["path"])
if not p.is_absolute(): if not p.is_absolute():
p = outdir / p p = outdir / p
if not p.exists():
# a remote worker writes result.json with paths on ITS disk
# (<root>/data/remote/<job>/out/...); after collect() those
# files sit in the local outdir — remap by the out/-relative
# tail, falling back to the basename
marker = f"/data/remote/{job_id}/out/"
_, _, tail = str(out["path"]).partition(marker)
if tail and (outdir / tail).exists():
p = outdir / tail
else:
p = outdir / Path(out["path"]).name
if p.exists(): if p.exists():
a = store.register_file(con, p, name=out.get("name"), a = store.register_file(con, p, name=out.get("name"),
parent_job=job_id, move=True, meta=out.get("meta"), parent_job=job_id, move=True, meta=out.get("meta"),
user_id=owner_id) user_id=owner_id)
registered.append(a["id"]) registered.append(a["id"])
else:
print(f"[runner] {job_id}: result.json output not found after "
f"collect, skipped: {out.get('path')}", flush=True)
else: else:
for p in sorted(outdir.iterdir()): for p in sorted(outdir.iterdir()):
if p.name == "result.json" or p.name.startswith("."): if p.name == "result.json" or p.name.startswith("."):

View File

@ -68,97 +68,10 @@ def _lanes(con, runner):
def gpu_pool(runner) -> list[dict]: def gpu_pool(runner) -> list[dict]:
"""Per-node status for the dashboard: name, remote?, live occupancy + free slots. """Per-node status for the dashboard: name, remote?, currently busy."""
return [{"name": n.get("name", n.get("ssh", "?")),
Occupancy is read from the runner's `inflight` counters. It used to read a
`busy` key, but nothing ever wrote to that after load_gpu_nodes() set it False
so every node reported idle even while running jobs. Don't reintroduce it.
"""
pool = []
for n in getattr(runner, "gpu_nodes", []):
inflight = n.get("inflight") or {}
gpu, cpu = int(inflight.get("gpu", 0)), int(inflight.get("cpu", 0))
gpu_cap, cpu_cap = runner._node_capacity(n, "gpu"), runner._node_capacity(n, "cpu")
ops = n.get("operators")
pool.append({
"name": n.get("name", n.get("ssh", "?")),
"remote": not n.get("local", False), "remote": not n.get("local", False),
"busy": (gpu + cpu) > 0, "busy": n.get("busy", False)} for n in getattr(runner, "gpu_nodes", [])]
"gpu": {"running": gpu, "slots": gpu_cap, "free": max(0, gpu_cap - gpu)},
"cpu": {"running": cpu, "slots": cpu_cap, "free": max(0, cpu_cap - cpu)},
"operators": len(ops) if ops else "all",
})
return pool
def nodes_detail(con, runner) -> dict:
"""Per-node capability + live work — the real per-machine view.
/api/system's gpu_pool only carries name/remote/busy. This joins the runner's
live inflight counters with the job table so the UI can show who is running
what, and why a heavy job waits while a light one flies.
`running` is attributed via jobs.node, written at dispatch (2026-08-05).
Before that column existed this had to infer placement, and every job was
credited to the primary while busy remotes rendered as idle.
"""
nodes = getattr(runner, "gpu_nodes", [])
local_name = next((n.get("name") for n in nodes if n.get("local")), "local")
rows = con.execute(
"SELECT id, operator, status, node, created_at, started_at FROM jobs "
"WHERE status IN ('running','queued') ORDER BY created_at"
).fetchall()
running_by_node: dict[str, list] = {}
queued: list[dict] = []
for r in rows:
rec = {"id": r["id"], "operator": r["operator"],
"created_at": r["created_at"], "started_at": r["started_at"]}
if r["status"] == "running":
# legacy rows predate jobs.node; credit them to the primary as before
running_by_node.setdefault(r["node"] or local_name, []).append(rec)
else:
queued.append(rec)
out, by_operator = [], {}
for n in nodes:
name = n.get("name", n.get("ssh", "?"))
ops = n.get("operators")
explicit = bool(ops)
op_ids = list(ops) if explicit else sorted(runner.operators or {})
for op in op_ids:
by_operator.setdefault(op, []).append(name)
gpu_cap = runner._node_capacity(n, "gpu")
inflight = n.get("inflight") or {}
# a queued job this node could pick up if it had a free slot
eligible = [q for q in queued
if (not explicit or q["operator"] in op_ids)
and runner.lane_of(q["operator"]) == "gpu"]
out.append({
"name": name,
"remote": not n.get("local", False),
"ssh": n.get("ssh"),
"gpu_capacity": gpu_cap,
"cpu_capacity": runner._node_capacity(n, "cpu"),
"gpu_inflight": int(inflight.get("gpu", 0)),
"cpu_inflight": int(inflight.get("cpu", 0)),
"operators_explicit": explicit,
"operators": op_ids,
"operator_count": len(op_ids),
"running": running_by_node.get(name, []),
"eligible_queued": eligible,
"notes": n.get("_notes") or [],
})
placed = {j["id"] for v in running_by_node.values() for j in v}
gpu_ops = sorted(op for op in (runner.operators or {})
if runner.lane_of(op) == "gpu")
return {"nodes": out, "by_operator": by_operator,
"queued_unassigned": [q for q in queued if q["id"] not in placed],
"gpu_ops": gpu_ops}
def _jobs_24h(con): def _jobs_24h(con):

View File

@ -1,147 +0,0 @@
"""Node-placement reporting: gpu_pool must reflect live occupancy.
The bug this guards: gpu_pool() read a `busy` key that load_gpu_nodes() set to
False once and nothing ever updated, so every node reported idle while running
jobs. Occupancy must come from the runner's `inflight` counters.
Run: python3 tests/test_node_reporting.py
"""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from server import sysinfo
class FakeRunner:
"""Minimal stand-in — gpu_pool only needs gpu_nodes + _node_capacity."""
def __init__(self, nodes):
self.gpu_nodes = nodes
def _node_capacity(self, node, lane):
if lane == "gpu":
return 1
if lane == "cpu":
return int(node.get("cpu_slots", 3 if node.get("local") else 2))
return 0
def test_busy_reflects_inflight():
runner = FakeRunner([
{"name": "m3ultra", "local": True, "busy": False,
"inflight": {"gpu": 1, "cpu": 0}},
{"name": "m1", "ssh": "x@y", "busy": False,
"inflight": {"gpu": 0, "cpu": 0}, "operators": ["flux_local"]},
])
pool = {n["name"]: n for n in sysinfo.gpu_pool(runner)}
# the stale `busy: False` on the node dict must NOT win
assert pool["m3ultra"]["busy"] is True, "node running a gpu job reported idle"
assert pool["m3ultra"]["gpu"] == {"running": 1, "slots": 1, "free": 0}
assert pool["m1"]["busy"] is False
assert pool["m1"]["gpu"] == {"running": 0, "slots": 1, "free": 1}
def test_local_is_not_named_local():
"""load_gpu_nodes names the local node after the host, so the UI can tell
the primary apart from the placeholder it used to show."""
from server import remote
nodes = remote.load_gpu_nodes()
assert nodes[0]["local"] is True
assert nodes[0]["name"] != "local", "local node should carry the hostname"
def test_operator_count_reported():
runner = FakeRunner([
{"name": "all-ops", "local": True, "inflight": {"gpu": 0, "cpu": 0}},
{"name": "fenced", "ssh": "x@y", "inflight": {"gpu": 0, "cpu": 0},
"operators": ["a", "b", "c"]},
])
pool = {n["name"]: n for n in sysinfo.gpu_pool(runner)}
assert pool["all-ops"]["operators"] == "all" # no allowlist == allow-all
assert pool["fenced"]["operators"] == 3
def test_node_column_migrated():
from server import db
assert ("jobs", "node", "TEXT") in db.MIGRATIONS, "jobs.node migration missing"
# --- nodes_detail: the /api/nodes contract fluxgod consumes -------------------
class FakeCon:
def __init__(self, rows): self._rows = rows
def execute(self, *a, **k): return self
def fetchall(self): return self._rows
def _runner_with_ops():
r = FakeRunner([
{"name": "m3ultra", "local": True, "inflight": {"gpu": 1, "cpu": 0}},
{"name": "m1", "ssh": "u@h", "inflight": {"gpu": 1, "cpu": 0},
"operators": ["mflux_image_edit", "flux_local"], "_notes": ["a note"]},
{"name": "m2max", "ssh": "u@h2", "inflight": {"gpu": 0, "cpu": 0},
"operators": ["flux_local"]},
])
r.operators = {"mflux_image_edit": {"resources": "gpu"},
"flux_local": {"resources": "gpu"},
"ffprobe": {"resources": "cpu"}}
r.lane_of = lambda op: r.operators.get(op, {}).get("resources", "cpu")
return r
def test_running_attributed_by_node_column():
"""The whole point: a job that ran on m1 must show under m1, not the primary."""
rows = [
{"id": "j1", "operator": "mflux_image_edit", "status": "running",
"node": "m1", "created_at": 1.0, "started_at": 1.1},
{"id": "j2", "operator": "mflux_image_edit", "status": "running",
"node": "m3ultra", "created_at": 2.0, "started_at": 2.1},
]
d = sysinfo.nodes_detail(FakeCon(rows), _runner_with_ops())
byname = {n["name"]: n for n in d["nodes"]}
assert [j["id"] for j in byname["m1"]["running"]] == ["j1"], "m1's job misattributed"
assert [j["id"] for j in byname["m3ultra"]["running"]] == ["j2"]
assert byname["m2max"]["running"] == []
def test_legacy_null_node_credited_to_primary():
rows = [{"id": "old", "operator": "flux_local", "status": "running",
"node": None, "created_at": 1.0, "started_at": 1.1}]
d = sysinfo.nodes_detail(FakeCon(rows), _runner_with_ops())
byname = {n["name"]: n for n in d["nodes"]}
assert [j["id"] for j in byname["m3ultra"]["running"]] == ["old"]
def test_eligible_queued_respects_allowlist():
rows = [{"id": "q1", "operator": "mflux_image_edit", "status": "queued",
"node": None, "created_at": 3.0, "started_at": None}]
d = sysinfo.nodes_detail(FakeCon(rows), _runner_with_ops())
byname = {n["name"]: n for n in d["nodes"]}
assert len(byname["m1"]["eligible_queued"]) == 1 # allowlists it
assert byname["m2max"]["eligible_queued"] == [] # does not
assert len(byname["m3ultra"]["eligible_queued"]) == 1 # no allowlist == all
def test_contract_keys_match_captured_api():
"""Field names fluxgod's server.py reads — captured from the live API."""
d = sysinfo.nodes_detail(FakeCon([]), _runner_with_ops())
assert set(d) == {"nodes", "by_operator", "queued_unassigned", "gpu_ops"}
assert set(d["nodes"][0]) == {
"name", "remote", "ssh", "gpu_capacity", "cpu_capacity", "gpu_inflight",
"cpu_inflight", "operators_explicit", "operators", "operator_count",
"running", "eligible_queued", "notes"}
assert d["by_operator"]["flux_local"] == ["m3ultra", "m1", "m2max"]
assert d["gpu_ops"] == ["flux_local", "mflux_image_edit"]
if __name__ == "__main__":
for name, fn in sorted(globals().items()):
if name.startswith("test_") and callable(fn):
fn()
print(f" ok {name}")
print("all node-reporting checks passed")