modelbeast/HANDOFF_HY3D_MLX.md

HANDOFF: hunyuan3d_mlx operator — fully-local Hunyuan3D 2.1 via MLX

For: Opus 4.8 executing autonomously in this repo. Goal: Add a second fully-local image→3D generator (Hunyuan3D 2.1, native MLX) alongside trellis_mac, benchmark it on both nodes, and (stretch) fork-improve it to exploit Studio-class GPUs (M3 Ultra 256GB / 819GB/s, M1 Ultra 128GB). Written 2026-07-16 after code-level assessment of the upstream repo.

---

1. Context & assessment (read first, don't re-derive)

Upstream: https://github.com/dgrauet/Hunyuan3D-2.1-mlx — fork of Tencent's Hunyuan3D-2.1 adding native MLX inference for BOTH stages:

• Stage 1 (image → mesh): hy3dshape/hy3dshape/pipeline_mlx.py → ShapePipeline.from_pretrained("dgrauet/hunyuan3d-2.1-mlx"). Validated numerically vs PyTorch within 1e-5.
• Stage 2 (mesh + reference image → PBR-textured GLB): hy3dpaint/textureGenPipeline_mlx.py → Hunyuan3DPaintPipelineMLX. UNet match 1.17e-5 vs PyTorch. Includes MLX RealESRGAN x4 (512²→2048² per view), cosine-weighted bake blend, EDT UV-gutter fill, proper glTF PBR export (baseColorTexture + metallicRoughnessTexture, doubleSided).

Weights: HF dgrauet/hunyuan3d-2.1-mlx, ~15.1GB total, auto-downloaded on first use. Override dir via env HUNYUAN3D_MLX_WEIGHTS_DIR. FP16 DiT is 5.68GB (dit.safetensors); paint stack adds paint_unet 3.66GB + paint_dino 2.12GB + paint_clip 1.18GB + VAEs + realesrgan_x4plus_mlx. Peak memory ~10GB stage 1 fp16 + ~6GB stage 2 → trivial on both our nodes. Use FP16, not INT8 — both Ultras have memory to burn and the owner wants best quality; INT8/INT4 exist only for 16GB Macs.

Port quality verdict: high for a 4-star repo. Disciplined single-author work: docs/forward_pass.md documents porting principles, docs/adr/ has architecture decisions, tests/ includes an end-to-end stage1→stage2 chain and unit tests. Deviations from the PyTorch reference are documented, not silent. Rough edges to expect: single author, ~188 weight downloads, no community battle-testing — treat every first run as a probe.

Reference timings (M2 Pro): ~9 min for 6-view texture synthesis at 512px. Our Ultras should beat this substantially; M1 Ultra runs MLX natively (unlike TRELLIS.2's torch-MPS bf16 question flagged in AGENTS.md).

The 4096-texture situation (this is the fork opportunity)

The README claims texture_size=2048 vs PyTorch's 4096 because "the MLX Metal rasterizer has no tiling." The code is ahead of the README:

• hy3dpaint/textureGenPipeline_mlx.py:82 — config default is ALREADY texture_size = 1024 * 4 (4096).
• hy3dpaint/DifferentiableRenderer/mesh_render_mlx.py:267 — _rasterize_tiled(pos_clip, resolution, tiles=4) exists, verified against single-dispatch at 2048².
• mesh_render_mlx.py:647-658 — extract_textiles() (the main UV-space raster feeding back_project(method="back_sample")) already auto-tiles: 2 tiles at ≥3072, 4 tiles at ≥4096.
• Remaining untiled gaps: uv_feature_map() at mesh_render_mlx.py:728 calls plain _rasterize(vtx_uv_clip, self.texture_size) — this will blow the Metal command-buffer budget at 4096 if it's on the hot path. (mesh_render_mlx.py:815 is camera-view raster at render_size=2048, fine as-is.) The e2e test (hy3dpaint/tests/test_e2e_mesh.py:49-51) pins 2048 "because we don't tile the bake."
• The command-buffer error (kIOGPUCommandBufferCallbackErrorImpactingInteractivity) is a watchdog concern documented on laptop GPUs. An M3 Ultra Studio may simply not hit it untiled — test before patching.

Other Studio-class headroom knobs (all currently tuned for laptops):

• Remesh budget: tests/test_stage1_to_stage2.py:82-91 decimates Stage 1's ~500k-face mesh to ~40-60k before texturing (rasterizer budget again). On Ultra, try 100-200k for bake fidelity.
• Stage 1 octree_resolution: default 256; try 384 for finer geometry (memory is not our constraint).
• max_num_view: default 6; the candidate camera list supports many more views.

---

2. Phase A — Fork + vendor + install (M3 first)

1. Fork dgrauet/Hunyuan3D-2.1-mlx to the owner's GitHub account with gh repo fork dgrauet/Hunyuan3D-2.1-mlx --clone=false (confirm the account gh auth status shows; do NOT create the fork under an org without asking). All our patches go to the fork's modelbeast branch; keep main tracking upstream for rebases.
2. Vendor: clone the fork to vendor/hunyuan3d-mlx (repo convention: vendor/trellis-mac, vendor/stable-fast-3d).
3. Install script scripts/install_hunyuan3d_mlx.sh, modeled on scripts/install_trellis_mac.sh:
   • Create vendor/hunyuan3d-mlx/.venv (python3.11, same as trellis-mac).
   • pip install mlx mlx-arsenal safetensors Pillow trimesh scikit-image PyMCubes scipy huggingface_hub xatlas opencv-python (README's list) plus pymeshlab (needed for use_remesh=True — see textureGenPipeline_mlx.py:39-41, it soft-fails to None without it).
   • NOTE: Stage 2's Hunyuan3DPaintConfigMLX references torch-era paths (device="cuda", multiview_pretrained_path) but with use_mlx_diffusion=True (the default) the PyTorch multiview model is skipped — verify at runtime whether torch is still imported anywhere on the MLX path before adding it to the venv. Prefer NOT installing torch if the MLX path doesn't need it.
   • Verify: .venv/bin/python -c "import mlx.core as mx; print(mx.default_device())".
4. Env guards: reuse the HF_HUB_DISABLE_XET=1 guard from trellis_mac/run.py (xet downloader breaks on some repos); the OMP guards likely unneeded without torch but harmless.

3. Phase B — CLI wrapper + operator

The upstream repo has no single-command CLI for the full chain — tests/test_stage1_to_stage2.py is the reference implementation of image→textured-GLB. Build:

1. In the fork (so it's upstreamable): generate_e2e.py at repo root — python generate_e2e.py input.png --output out_stem [--steps 50] [--guidance 7.5] [--octree-resolution 256] [--max-num-view 6] [--view-resolution 512] [--texture-size 2048|4096] [--seed 42] [--no-texture] [--remesh-faces 40000]
   • Stage 1: ShapePipeline.from_pretrained(...), export <stem>_shape.glb.
   • If --no-texture, stop there. Else Stage 2 per the test file: remesh → Hunyuan3DPaintPipelineMLX → <stem>.glb.
   • Print stage timings and peak memory (mx.get_peak_memory()) — we want these in job logs.
2. Operator server/operators/hunyuan3d_mlx/ mirroring trellis_mac exactly:
   • manifest.json: id hunyuan3d_mlx, name "Hunyuan3D 2.1 (local MLX)", category mesh-gen, accepts ["image"], produces ["model"], resources "gpu", python vendor/hunyuan3d-mlx/.venv/bin/python. Params: steps (default 50), guidance_scale (7.5), octree_resolution (enum 256/384, default 256), texture_size (enum 1024/2048/4096, default 2048 until Phase D proves 4096), max_num_view (default 6), seed, no_texture.
   • run.py: copy trellis_mac/run.py shape — subprocess generate_e2e.py, cwd vendor dir, collect *.glb (prefer the textured one over *_shape.glb), write result.json with {"outputs": [...], "meta": {"tool": "hunyuan3d_mlx"}}.
3. Smoke test on scratch DB (per repo rule: never test against the running gens DB): ./mb run hunyuan3d_mlx --file <test image> --wait --download out/. First run downloads ~15GB — job log shows progress, don't kill it (AGENTS.md warns 10-30+ min).
4. Best-practice chain: fal_bg_remove (or local bg_remove_local) → cutout → hunyuan3d_mlx, same as the fal chain in AGENTS.md.

4. Phase C — M1 Ultra node + benchmarks

1. Install on the M1 node (http://100.91.239.7:8777, see AGENTS.md) — same scripts; MLX is architecture-native on M1, no bf16 concern. Mind the unified worker pool (commit b8c333a): check how the gpu lane discovers operators per node in server/runner.py / server/remote.py before assuming both nodes can serve the op.
2. Benchmark protocol — same cutout image through all three, one at a time (gpu lane serializes anyway):
   • trellis_mac (M3, defaults: 1024 pipeline, 2048 texture)
   • hunyuan3d_mlx on M3 AND on M1 (fp16, defaults)
   • fal_hunyuan3d_v21 (single-image mode — multi-view is broken on fal, verified 2026-07) Record per run: wall time per stage, peak memory, GLB size, vert/face count, and a viewer screenshot. Subjective quality notes: geometry coherence, texture seams, PBR plausibility.
3. Record in BENCHMARKS.md following the existing table format, and update the local-verdict line if Hunyuan changes the recommendation (e.g. "trellis_mac for geometry, hunyuan3d_mlx for textures"). Add the operator row to README.md's table and AGENTS.md's mesh-gen row.

5. Phase D (stretch) — Studio-class fork improvements

Work on the fork's modelbeast branch; each item is independently shippable and upstreamable as a PR to dgrauet:

1. 4096² textures. First just run --texture-size 4096 on the M3 Ultra untiled — the command-buffer watchdog may not fire on Studio GPUs. If it fires: patch uv_feature_map() (mesh_render_mlx.py:728) to route through the existing _rasterize_tiled() using the same ≥3072/≥4096 tile thresholds as extract_textiles() (line 647). Grep for any other _rasterize(..., self.texture_size) call. Validate: tiled output must equal untiled at 2048 (the codebase already did this for extract_textiles; reuse that method). Compare 2048 vs 4096 bakes on the same mesh — if 4096 wins visibly, flip the operator default on the M3.
2. Bigger bake mesh. Raise the remesh decimation target from ~40k to 100-200k faces (flag already exposed via --remesh-faces); check bake time scaling and whether the rasterizer holds.
3. Stage 1 detail: octree_resolution=384 A/B vs 256 on a detail-heavy cutout.
4. More views: max_num_view=8-12 A/B — the candidate camera ring (30° azimuth steps at ±20° elevation, textureGenPipeline_mlx.py:89-100) already supports it. Watch for texture-blend regressions, not just wall time.
5. Log every result in BENCHMARKS.md; if any patch is a clear win, open the upstream PR (small, focused, with the validation evidence — the author clearly values PT-parity proofs).

6. Guardrails

• Never test against the production DB with running generations — scratch DB only (repo rule).
• gpu lane = ONE job at a time; don't parallelize benchmarks on a node.
• First-run weight downloads are long; don't cancel "stuck" jobs that are downloading.
• The reference image must match the mesh content (upstream caveat) — for e2e runs this is automatic (same image drives both stages).
• If Stage 2 quality disappoints out of the box, benchmark Stage 1 + --no-texture anyway: Hunyuan shape + TRELLIS texture routing is a valid future pipeline.
• Ask the owner before: forking under an org account, pushing PRs upstream, or changing existing operator defaults.